Oodle Data Compression and IO

Oodle Development History

This change log includes the Oodle development history.


Release 2.4.3 - February 14, 2017

  • fix : Mermaid & Selkie encoders could cause an access violation reading past the end of the raw buffer if the length was just over 64k mod 128k
  • fix : Some compressors failed to decode files that were (128k+1)mod 256k bytes long due to bug in fuzz safety checks
  • fix : Selkie encoder was incorrectly choosing offset compression in some cases, causing slower decodes

Release 2.4.2 - January 25, 2017

  • new : Nintendo NX (Switch) support (SDK 0.17.13 and 1.0.0)
  • new : example_lz_outputchunking : Example demonstrating cutting OodleLZ compressed output into fixed size chunks

  • enhancement : Kraken, Mermaid & Selkie now support "in-place" decoding ; OodleLZ_Compressor_CanDecodeInPlace now returns true for all OodleLZ decoders

  • fix : a bug could occur in Oodle Ext 32-bit builds on systems with user memory in the top 2 GB
  • fix : rename OodleLog_Printf macro to OodleXLog_Printf to make OodleX lib naming convention
  • fix : LZB16 decoder crashed on 32-bit ARM Android. LZB16 was also used as the fallback for tiny buffers in Kraken/Mermaid/Selkie/Hydra.

Release 2.4.1 - October 3, 2016

  • new : OodleX_ReleaseThreadTLS function to release OodleX TLS resources in the rare usage that you churn through a huge number of thread creates and destroys

  • change : Clean up the behavior of OodleLZ_GetCompressedStepForRawStep when it is not given the entire compressed data; now returns -1 for error and 0 for not enough data.

  • fix : Fix race in the OodleThinSemaphore used in the example code via OodleX_Semaphore_Post; this was only used by example_lz_threadphased , so should not affect any production code
  • fix : Out of memory default handler would just infinite busy-loop on fgetc on non-Windows platforms
  • fix : Android distribution didn't contain AArch64 libs

Release 2.4.0 - August 22, 2016

  • new : Hydra ; automatically selects Kraken/Mermaid/Selkie
  • new : example_lz lz_test_11 demonstrates sliding window with Kraken using memcpy
  • new : Mermaid+ ; slightly higher compression, slightly lower speed ; control Mermaid vs Mermaid+ using spaceSpeedTradeoffBytes

  • fix : Selkie & Mermaid encoders could overrun the compressed buffer when data expanded
  • fix : fix Mac dylib having a different name for its self-reference
  • fix : fix Linux link error with gold on the Oodle static lib
  • fix : fix iOS version min lowered to 7.0 (was 8.1)
  • fix : fix race in some low level thread primitives caused by the threadprofiler. Showed up as ThreadPhased decodes hanging.

  • enhancement : LZNA level 8 (Optimal 3) tries several min match lengths

Release 2.3.0 - July 14, 2016

  • new : Mermaid ! - Mermaid is a new super-fast-to-decode compressor with good compression ratios (comparable to ZLib)
  • new : Selkie ! - Selkie is the fastest-decoding compressor, with low compression ratios (comparable to LZ4)
  • enhancement : Kraken is even faster to decode (10-20%) on all platforms
  • enhancement : Kraken significantly faster to decode on ARM
  • change : WARNING Kraken data made by 2.3.0 by default cannot be loaded by Oodle 2.2.0 ; set m_OodleLZ_BackwardsCompatible_MajorVersion if you need that. Oodle can always load data made by previous versions, but the reverse is not necessarily true.
  • change : OodleLZ Decompress functions args changed. New OodleLZ_FuzzSafe argument. Removed the allowed_compressor_mask.
  • new : new compressor property query OodleLZ_Compressor_CanDecodeFuzzSafe
  • new : OodleConfigValues for core-only compressor config
  • new : OodleConfigValues::m_OodleLZ_BackwardsCompatible_MajorVersion - set to 2 to make Kraken data that can be loaded by Oodle version 2.2.0
  • change : OodleLZ_CompressOptions dictionarySize option added to limit offsets
  • change : OodleLZ_CompressOptions offsetShift option for old LZH encoder removed (now always 0)
  • new : lz_test_10 in example_lz demonstrates decoding quanta from a finite io buffer
  • change : LZB16 decoder is now fuzz safe. Very slightly slower to decode.
  • fix : LZB16 Optimal1 level could crash in encode of files > 2GB
  • fix : OodleLZ_GetInPlaceDecodeBufferSize was too big on tiny buffers (it's now never bigger than complen + rawlen)

Release 2.2.0 - May 11, 2016

  • new : threaded Kraken decoder! See About OodleLZ ThreadPhased Decode
  • new : example_lz_threadphased contains a client-side implementation of a ThreadPhased decoder
  • new : Kraken Optimal3 (Level 7) mode with higher compression ratio
  • enhancement : Kraken optimized for ARM processors
  • enhancement : Kraken optimal parse levels improved, about 1% more compression
  • change : XBox1 now built with VS 2012, but Core lib is compatible with VS 2015 (Ext will work in VC 2012 but not in VC 2015)
  • change : Windows SDK redist64 dir removed; 64-bit DLL's are just in "redist"
  • change : examples now include oodle2.h with a relative path so they compile out of the box
  • change : Oodle2 Core lib default plugins no longer use the CRT on Microsoft platforms. See Core plugins
  • change : minor version bumped for API incompatibilities
  • change : OodleLZ_Decode_ThreadPhase argument added to OodleLZ_Decompress and OodleLZ_DecodeSome for 2-threaded Kraken decode. To retain previous behavior, add OodleLZ_Decode_Unthreaded to your call (or use default argument).
  • change : allowed_compressor_mask argument added to OodleLZ_Decompress and OodleLZ_DecodeSome for fuzz safety. To retain previous behavior, add OODLELZ_ALLOWED_COMPRESSOR_ALL to your call (or use default argument).
  • removed : Oodle Huffman public APIs removed

Release 2.1.5 - April 26, 2016

  • new : Kraken !! An amazing new compressor with high compression and fast decode speed!
  • fix : LZB16 and LZBLW decoders could over-read compressed data if the last quantum was memset
  • fix : example_packet : better randomization of subsets and limits in TestOodleNetwork_SelectDictionaryAndTrain
  • change : example_packet : remove some rarely used options
  • removed : OodleLZ_GetZipLikeCompressionSettings is gone
  • deprecation : LZHLW and LZNIB and LZBLW compressors are now deprecated, use Kraken instead
  • PS4 library now built with SDK 3.0
  • Xbox One library now built with March 2016 QFE1 , VS 2015

Release 2.1.4 - March 20, 2016

  • new : example_lz lz_test_9 demonstrating block composability rules
  • enhancement : LZNA Normal parse compression improved
  • enhancement : LZNA wasn't using SSE on Linux or Mac ; it now does, which makes those platforms much faster
  • fix : OodleLZ_CompressWithContext with NULL CompressOptions crashed
  • fix : LZBLW Normal level encoder would crash when called with dictionary preload (or large buffers)
  • fix : OodleNetwork1UDP fix fuzz safety for split large packets
  • fix : OodleNetwork1UDP fix fuzz safety prevent reading past end of dictionary
  • fix : OodleNetwork1UDP removed all used of release-asserts, now returns failure in those cases
  • fix : OodleNetwork1UDP properly document OODLENETWORK1_DECOMP_BUF_OVERREAD_LEN
  • change : Oodle Logs at verbosity level 2 are now compiled out of release builds
  • change : example_lz_chart compressor set changed
  • change : OodlePlugins_SetAssertion can now be called with a NULL function pointer argument as an easy way to disable all assert handling
  • change : example_lz test_8 shows how to use the WithContext calls for independent buffers
  • change : OodleNetwork1UDP_State_Uncompact return bool to indicate success or failure on bad data

Release 2.1.3 - Feb 1, 2016

  • enhancement : OodleNetwork1UDP_Decode faster
  • fix : OodleNetwork1UDP_Decode fuzz safety
  • fix : OodleNetwork1UDP - fix possible bug in large split packet handling; now properly ensures that complen <= rawlen, and complen == rawlen always means a pass-through uncompressed packet
  • deprecation : OodleLZ_Compressor_CanSPUDecompress function removed

Release 2.1.2 - Jan 16, 2016

  • new : BitKnit ! BitKnit is a new LZ compressor with high compression and good speed, between LZNA and LZHLW
  • enhancement : Most LZ encoders are faster
  • enhancement : LZNA Fast & Normal levels get more compression

Release 2.1.1 - Nov 25, 2015

  • new : example_lz_chart make a neat text chart of the Oodle compressors & levels to test performance
  • new : example_network_client simple Oodle Network client example
  • fix : don't store OODLE_HEADER_VERSION in the example_packet data header; use OODLE2_VERSION_MAJOR instead
  • enhancement : most decoders faster by around 10% !
  • enhancement : LZNIB Normal & Optimal1 parsers much improved, more compression and faster decodes!
  • enhancement : LZNIB decodes faster
  • enhancement : LZNA about 10% faster to decode
  • enhancement : LZHLW decoder about 10% faster in 64-bit
  • enhancement : LZB encoder must faster in 32-bit
  • enhancement : LZNA on iOS now uses NEON, almost 2X faster to decode!
  • deprecation : LZH and LZA compressors are now deprecated; use LZHLW and LZNA instead
  • PS4 library now built with SDK 2.5
  • Windows library now built with MSVC 2013
  • Mac & iOS libraries now built with XCode 7.1

Release 2.1.0 - Oct 10, 2015

  • new : Oodle2 first Beta release

Release 1.45.1 - May 13, 2015

  • fix : fix bug in the LZNA level 7 (Optimal3) encoder

Release 1.45.0 - May 11, 2015

  • new : New LZNA compressor! More compression than LZA and fast to decode on modern CPUs.

Release 1.44.2 - March 4, 2015

  • enhancement : LZA new optimal parse; faster at Optimal1 and more compression at Optimal3
  • enhancement : LZNib Normal encoder faster, better optimal parse
  • enhancement : LZB Fast & VeryFast encoder faster

Release 1.44.1 - December 9, 2014

  • fix : example_packet state header was not the same in 32-bit and 64-bit builds
  • enhancement : example_packet LZ compression level option added

Release 1.44.0 - November 16, 2014

  • new : Mac sdk now includes 64 bit libraries
  • new : Android and IOS sdk's available, for Oodle Network only
  • new : Unreal Engine integration of Oodle Network is available
  • change : example_packet has better trained model IO

Release 1.43.0 - September 8, 2014

  • fix : LZA packet compressor introduced in 1.42 was getting much less compression than it should due a bug; fixed!

Release 1.42.0 - September 3, 2014

  • change : reduce memory use of LZNib and LZBLW compressors at Optimal level
  • new : LZA compressor now has sliding window encoding and decoding
  • new : OodleLZ_CompressWithContext incremental API now supports LZA
  • change : example_packet LZA option added
  • change : Oodle is using the new "radtypes.h" shared header
  • enhancement : OodleNetwork1 UDP new implementation - faster, a little more compression.
  • fix : Optimal parse compressors could hang on very large degenerate files

Release 1.41.0 - July 10, 2014

  • new : OodleNetwork dictionary selection by string matching; see OodleNetwork1_SelectDictionaryFromPackets and example_packet : Example demonstrating network packet compression ; usually improves compression
  • enhancement : OodleNetwork1 TCP memory use reduced to 84104 bytes per channel
  • enhancement : OodleNetwork1 UDP is faster to encode ; saved states are incompatible with older versions
  • enhancement : LZNIB VeryFast and Normal much faster to encode; see FAQ : What are the speeds and ratios of the OodleLZ compressors and levels? for speeds
  • enhancement : LZB16-Fast and VeryFast much faster to encode; see FAQ : What are the speeds and ratios of the OodleLZ compressors and levels? for speeds
  • change : rename OodleNetwork1 TCP variant to OodleNetwork1TCP for clarity
  • change : better default hash table sizes for LZB16
  • fix : Fix bug in which packets that expanded in OodleNetwork1 TCP could cause a write at compbuf[-1]

Release 1.40.0 - June 27, 2014

  • change : OodleStaticLZP renamed to OodleNetwork1
  • enhancement : LZHLW compressor significantly faster to encode at all levels, especially Normal
  • new : New compressor OodleLZ_Compressor_LZA for very high compression!
  • fix : example_packet had a hard-coded limit of 64k for packet sizes; that was removed
  • fix : OodleLZDecoder_Reset was in the header but not implemented
  • new : OodleNetwork1 compressed buffer size is now limited by OodleNetwork1_CompressedBufferSizeNeeded

Release 1.30.0 - April 14, 2014

  • new : Minimal library initialization; see OodleX_Init_GetDefaults_Minimal , OodleX_Init_NoThreads , OodleX_Shutdown_NoThreads
  • change : OodleX_Init no longer accepts NULL options
  • change : Windows : OodleX_Init no longer does timeBeginPeriod(1) ; it's now left up to the app to decide if they want to do that
  • change : api simplified; many unused APIs removed

Release 1.22.0 - March 31, 2014

  • new : redo docs for Oodle Network packet compression

Release 1.21.2 - March 11, 2014

  • fix : OOP tool failed to extract large packages due to exceeding threading limits
  • fix : SPU LZ match offsets fixed again
  • fix : ThreadProfiler code stripped completely in final build

Release 1.21.0 - March 1, 2014

  • new : OodleNetwork1UDP_StateCompacted added to persist the trained state more compactly
  • enhancement : OodleLZH and LZHLW decompressors further optimized; 5-10% faster
  • enhancement : OodleLZH and LZHLW no longer does an allocation to prevent overrun
  • enhancement : OodleNetwork1 for UDP networking significantly faster (about 300%)
  • fix : SPU LZ match offsets could be too large by 1, causing corruption in rare cases

Release 1.20.0 - November 25, 2013

  • new : Oodle now shipping "final" build without debug facilities
  • new : OodleLZ_GetZipLikeCompressionSettings to make it easier to select compressor settings
  • fix : fix possible mismatch between structs in the public header and structs in the lib
  • fix : matchTableSizeLog2 wasn't being used correctly; made LZB and LZNIB compress less than they should

Release 1.19.0 - November 14, 2013

  • new : OodleNetwork1 now provides a UDP (stateless) variant for unordered packet compression (beta)
  • new : OodleHuffman_MultiHuffEncode for UDP unordered packet compression (beta)
  • note : version 119 numbering adjusted to match its intention better. 1.major.minor
  • fix : LZNib could fail when using "in-place" decompression
  • fix : LZH and LZHLW could speculatively read past the end of compbuf in rare cases, potentially causing a (benign) access violation
  • fix : LZ parallel compression with long-range-matcher could fail when parallel chunks didn't align to LRM chunks

Release 1.1.8 - November 4, 2013

  • new : add OodleXLZ_Decompress_MakeSeekTable_Wide_Async , a simple parallel version of OodleLZ_Decompress
  • new : add matchTableSizeLog2 to OodleLZ_CompressOptions ; controls encoder memory use and speed
  • new : add FAQ : What are the speeds and ratios of the OodleLZ compressors and levels? to make choosing a compressor and level easier
  • enhancement : LZB16 encoder faster
  • fix : LZ decoders could speculatively read past the end of compressed buffers; this could cause an access violation if the memory after the compressed buffer was not readable. Fixed.
  • fix : Fix issues with the WithContext family of functions when used with circular window encoding.
  • fix : Fix issues with combining heterogenous OodleLZ data (with different seek chunk settings) causing the decoder to make invalid parallel decodes
  • fix : Fix issues with LZB when used with packed-raw-overlap decode mode
  • enhancement : PS4 : OodleMalloc goes to "default" memory instead of heap memory for large allocations

Release 1.1.7 - October 23, 2013

  • new : unified OodleLZ_CompressContext functions for incremental encoding, with and without sliding window
  • new : LZB16 can encode & decode in a circular window
  • fix : fix scheduling bug in OOZ IO->Decompression loader
  • fix : Linux : fix LZB16 generation page faults due to movdqa gcc bug
  • fix : don't lock Oodle threads to cores except on platforms where it is necessary

Release 1.1.6 - October 7, 2013

  • new : new LZB compressor - LZ-Bytewise (and LZB-LargeWindow), the fastest of all, for when even LZNib isn't fast enough
  • fix : OodleLZ_FillSeekTable could incorrectly mark heterogenous streams as being entirely independent seek chunks
  • fix : Linux & Mac : Fix worker thread stack sizes too small, could crash OOP tool and other uses of the worker threads.
  • fix : In-place decompression of incompressible data could fail if source overlapped dest.
  • fix : Fix OodleLZ_Compress_Async not working (and related APIs, such as OodleLZ_Compress_WriteOOZ_Async) when OodleAsyncSelect_None was passed for asyncSelect (now runs synchronously on the calling thread).

Release 1.1.5 - August 8, 2013

  • enhancement : LZNib now has a sliding window encoder for streaming compression
  • new : new OodleHuffman functions expose huffman-only coding
  • new : OodleNetwork1 compressor, static dictionary ideal for MMO network compression (Beta)

Release 1.1.4 - May 6, 2013

  • enhancement : large IO ops are now broken on the IO thread instead of at call time
  • enhancement : large IO aps are now cancellable in progress and return correct status
  • fix : OOP header could fail to read when it had a seek table
  • enhancement : rev the OOZ header; add the size of the header

Release 1.1.3 - May 1, 2013

  • fix : OOP header reader was truncating 64-bit file sizes to 32-bit
  • new : lzadvanced_output_chunking in example_lz_advanced
  • enhancement: OodleLZ_GetCompressedStepForRawStep allowed to seek quanta

Release 1.1.2 - April 28, 2013

  • fix : Linux dir enumeration was including "."
  • fix : Linux Open for WriteCreate was overwriting existing files too aggressively
  • fix : Linux Open failure had debug perror left in
  • change : better default log dirs and log names
  • change : log file will open in "." if default log dir is not accessible
  • change : Linux & Mac do log->prev cycle like Windows
  • change : disable stack trace on .map file platforms for now
  • enhancement: OodlePackage_FillWithFiles only decompresses chunks it needs (not all)
  • enhancement: OodlePackage_SortFiles now uses extention file groups

Release 1.1.1 - April 24, 2013

  • fix : Oodle_GetExtensionKey was not doing tolower as it should
  • fix : OodlePackage_FillWithFiles could crash if files failed to load
  • fix : fix memory overrun crash in DirListing
  • fix : fix null deref in arg parse in OOP tool
  • change: rename IOQ_DeleteFile_ to IOQ_Delete_ and make it work on dirs as well
  • new: add OodleDirListing_DeleteDirContents

Release 1.1.0 - April 11, 2013

  • new: LZNIB super fast to decode LZ variant added
  • new: LosslessFilters for some common data transformations that improve compression
  • change: Low level IO now allows aligned and unaligned IO modes
  • enhancement: Compression seek chunk size now variable, set by client
  • enhancement: All compressors can run encode and decode in parallel
  • enhancement: Long Range Matcher for LZ on big buffers
  • enhancement: (PS3) SPU waits now without thread switches
  • enhancement: (PS3) SPU after_decompress replaceable function

Release 1.0.0 - August 24, 2012

  • Initial Release.