Oodle Data Compression and IO

Oodle Development History

This change log includes the Oodle development history.


Release 2.5.4 - August 8, 2017

  • new : Windows UWP build added
  • fix : omit-frame-pointer was set on too many builds; it is now only used in x86 (32-bit) builds
  • change : Oodle PS4 now built with SDK 4.500

Release 2.5.3 - June 14, 2017

  • fix : remove alloca from rrHuffman_BuildCodeLens_Limitted ; causing crash on small stack threads on Linux

Release 2.5.2 - May 25, 2017

  • fix : Kraken decoder had a bug that caused it to incorrectly return failure on valid compressed chunks (due to the header corruption check not counting the scratch space needed with correct accounting of the thread phasing header)
  • fix : fix handle leak in OodleXLZ_Compress_WriteOOZFile_AsyncAndWait when file open failed

Release 2.5.1 - May 13, 2017

  • enhancement : Mermaid & Selkie encoding improved (faster and better ratio) on buffers <= 64k bytes long
  • change : Oodle Mac OSX build now includes static libs and dynamic libs.
  • change : Xbox One build is made with March 2017 XDK , still VC 2012 (but VC 2015 compatible)
  • deprecation : LZB16 and LZBLW Optimal level encoders are gone. LZB with CompressionLevel >= Optimal1 now drops down to Normal. Use Selkie instead.
  • obsolescence : libdivsufsort is no longer used

Release 2.5.0 - April 20, 2017

  • new : OodleHuffman_EncodeArrayU8 and OodleHuffman_DecodeArrayU8 reintroduced as public APIs. (not compatible with old public Huffman API)
  • enhancement : Kraken, Mermaid & Selkie - the non-optimal encode levels (SuperFast - Normal) get more compression and encode faster
  • enhancement : Kraken, Mermaid & Selkie - faster to decode, especially on ARM (Android, iOS, Switch)
  • enhancement : Hydra encoding now works in levels below Optimal
  • fix : Kraken, Mermaid & Selkie - SuperFast encode level had bugs causing it to get much worse compression than necessary
  • fix : LZB16 encoder at the Optimal1 level could crash due to uninitialized memory
  • change : NX now uses SDK 3.0

Release 2.4.3 - February 14, 2017

  • fix : Mermaid & Selkie encoders could cause an access violation reading past the end of the raw buffer if the length was just over 64k mod 128k
  • fix : Some compressors failed to decode files that were (128k+1)mod 256k bytes long due to bug in fuzz safety checks
  • fix : Selkie encoder was incorrectly choosing offset compression in some cases, causing slower decodes

Release 2.4.2 - January 25, 2017

  • new : Nintendo NX (Switch) support (SDK 0.17.13 and 1.0.0)
  • new : example_lz_outputchunking : Example demonstrating cutting OodleLZ compressed output into fixed size chunks

  • enhancement : Kraken, Mermaid & Selkie now support "in-place" decoding ; OodleLZ_Compressor_CanDecodeInPlace now returns true for all OodleLZ decoders

  • fix : a bug could occur in Oodle Ext 32-bit builds on systems with user memory in the top 2 GB
  • fix : rename OodleLog_Printf macro to OodleXLog_Printf to match OodleX lib naming convention
  • fix : LZB16 decoder crashed on 32-bit ARM Android. LZB16 was also used as the fallback for tiny buffers in Kraken/Mermaid/Selkie/Hydra.

Release 2.4.1 - October 3, 2016

  • new : OodleX_ReleaseThreadTLS function to release OodleX TLS resources in the rare usage that you churn through a huge number of thread creates and destroys

  • change : Clean up the behavior of OodleLZ_GetCompressedStepForRawStep when it is not given the entire compressed data; now returns -1 for error and 0 for not enough data.

  • fix : Fix race in the OodleThinSemaphore used in the example code via OodleX_Semaphore_Post; this was only used by example_lz_threadphased , so should not affect any production code
  • fix : Out of memory default handler would just infinite busy-loop on fgetc on non-Windows platforms
  • fix : Android distribution didn't contain AArch64 libs

Release 2.4.0 - August 22, 2016

  • new : Hydra ; automatically selects Kraken/Mermaid/Selkie
  • new : example_lz lz_test_11 demonstrates sliding window with Kraken using memcpy
  • new : Mermaid+ ; slightly higher compression, slightly lower speed ; control Mermaid vs Mermaid+ using spaceSpeedTradeoffBytes

  • fix : Selkie & Mermaid encoders could overrun the compressed buffer when data expanded
  • fix : fix Mac dylib having a different name for its self-reference
  • fix : fix Linux link error with gold on the Oodle static lib
  • fix : fix iOS version min lowered to 7.0 (was 8.1)
  • fix : fix race in some low level thread primitives caused by the threadprofiler. Showed up as ThreadPhased decodes hanging.

  • enhancement : LZNA level 8 (Optimal 3) tries several min match lengths

Release 2.3.0 - July 14, 2016

  • new : Mermaid ! - Mermaid is a new super-fast-to-decode compressor with good compression ratios (comparable to ZLib)
  • new : Selkie ! - Selkie is the fastest-decoding compressor, with low compression ratios (comparable to LZ4)
  • enhancement : Kraken is even faster to decode (10-20%) on all platforms
  • enhancement : Kraken significantly faster to decode on ARM
  • change : WARNING Kraken data made by 2.3.0 by default cannot be loaded by Oodle 2.2.0 ; set m_OodleLZ_BackwardsCompatible_MajorVersion if you need that. Oodle can always load data made by previous versions, but the reverse is not necessarily true.
  • change : OodleLZ Decompress functions args changed. New OodleLZ_FuzzSafe argument. Removed the allowed_compressor_mask.
  • new : new compressor property query OodleLZ_Compressor_CanDecodeFuzzSafe
  • new : OodleConfigValues for core-only compressor config
  • new : OodleConfigValues::m_OodleLZ_BackwardsCompatible_MajorVersion - set to 2 to make Kraken data that can be loaded by Oodle version 2.2.0
  • change : OodleLZ_CompressOptions dictionarySize option added to limit offsets
  • change : OodleLZ_CompressOptions offsetShift option for old LZH encoder removed (now always 0)
  • new : lz_test_10 in example_lz demonstrates decoding quanta from a finite io buffer
  • change : LZB16 decoder is now fuzz safe. Very slightly slower to decode.
  • fix : LZB16 Optimal1 level could crash in encode of files > 2GB
  • fix : OodleLZ_GetInPlaceDecodeBufferSize was too big on tiny buffers (it's now never bigger than complen + rawlen)

Release 2.2.0 - May 11, 2016

  • new : threaded Kraken decoder! See About OodleLZ ThreadPhased Decode
  • new : example_lz_threadphased contains a client-side implementation of a ThreadPhased decoder
  • new : Kraken Optimal3 (Level 7) mode with higher compression ratio
  • enhancement : Kraken optimized for ARM processors
  • enhancement : Kraken optimal parse levels improved, about 1% more compression
  • change : XBox1 now built with VS 2012, but Core lib is compatible with VS 2015 (Ext will work in VC 2012 but not in VC 2015)
  • change : Windows SDK redist64 dir removed; 64-bit DLL's are just in "redist"
  • change : examples now include oodle2.h with a relative path so they compile out of the box
  • change : Oodle2 Core lib default plugins no longer use the CRT on Microsoft platforms. See Core plugins
  • change : minor version bumped for API incompatibilities
  • change : OodleLZ_Decode_ThreadPhase argument added to OodleLZ_Decompress and OodleLZ_DecodeSome for 2-threaded Kraken decode. To retain previous behavior, add OodleLZ_Decode_Unthreaded to your call (or use default argument).
  • change : allowed_compressor_mask argument added to OodleLZ_Decompress and OodleLZ_DecodeSome for fuzz safety. To retain previous behavior, add OODLELZ_ALLOWED_COMPRESSOR_ALL to your call (or use default argument).
  • removed : Oodle Huffman public APIs removed

Release 2.1.5 - April 26, 2016

  • new : Kraken !! An amazing new compressor with high compression and fast decode speed!
  • fix : LZB16 and LZBLW decoders could over-read compressed data if the last quantum was memset
  • fix : example_packet : better randomization of subsets and limits in TestOodleNetwork_SelectDictionaryAndTrain
  • change : example_packet : remove some rarely used options
  • removed : OodleLZ_GetZipLikeCompressionSettings is gone
  • deprecation : LZHLW and LZNIB and LZBLW compressors are now deprecated, use Kraken instead
  • PS4 library now built with SDK 3.0
  • Xbox One library now built with March 2016 QFE1 , VS 2015

Release 2.1.4 - March 20, 2016

  • new : example_lz lz_test_9 demonstrating block composability rules
  • enhancement : LZNA Normal parse compression improved
  • enhancement : LZNA wasn't using SSE on Linux or Mac ; it now does, which makes those platforms much faster
  • fix : OodleLZ_CompressWithContext with NULL CompressOptions crashed
  • fix : LZBLW Normal level encoder would crash when called with dictionary preload (or large buffers)
  • fix : OodleNetwork1UDP fix fuzz safety for split large packets
  • fix : OodleNetwork1UDP fix fuzz safety prevent reading past end of dictionary
  • fix : OodleNetwork1UDP removed all used of release-asserts, now returns failure in those cases
  • fix : OodleNetwork1UDP properly document OODLENETWORK1_DECOMP_BUF_OVERREAD_LEN
  • change : Oodle Logs at verbosity level 2 are now compiled out of release builds
  • change : example_lz_chart compressor set changed
  • change : OodlePlugins_SetAssertion can now be called with a NULL function pointer argument as an easy way to disable all assert handling
  • change : example_lz test_8 shows how to use the WithContext calls for independent buffers
  • change : OodleNetwork1UDP_State_Uncompact return bool to indicate success or failure on bad data

Release 2.1.3 - Feb 1, 2016

  • enhancement : OodleNetwork1UDP_Decode faster
  • fix : OodleNetwork1UDP_Decode fuzz safety
  • fix : OodleNetwork1UDP - fix possible bug in large split packet handling; now properly ensures that complen <= rawlen, and complen == rawlen always means a pass-through uncompressed packet
  • deprecation : OodleLZ_Compressor_CanSPUDecompress function removed

Release 2.1.2 - Jan 16, 2016

  • new : BitKnit ! BitKnit is a new LZ compressor with high compression and good speed, between LZNA and LZHLW
  • enhancement : Most LZ encoders are faster
  • enhancement : LZNA Fast & Normal levels get more compression

Release 2.1.1 - Nov 25, 2015

  • new : example_lz_chart make a neat text chart of the Oodle compressors & levels to test performance
  • new : example_network_client simple Oodle Network client example
  • fix : don't store OODLE_HEADER_VERSION in the example_packet data header; use OODLE2_VERSION_MAJOR instead
  • enhancement : most decoders faster by around 10% !
  • enhancement : LZNIB Normal & Optimal1 parsers much improved, more compression and faster decodes!
  • enhancement : LZNIB decodes faster
  • enhancement : LZNA about 10% faster to decode
  • enhancement : LZHLW decoder about 10% faster in 64-bit
  • enhancement : LZB encoder must faster in 32-bit
  • enhancement : LZNA on iOS now uses NEON, almost 2X faster to decode!
  • deprecation : LZH and LZA compressors are now deprecated; use LZHLW and LZNA instead
  • PS4 library now built with SDK 2.5
  • Windows library now built with MSVC 2013
  • Mac & iOS libraries now built with XCode 7.1

Release 2.1.0 - Oct 10, 2015

  • new : Oodle2 first Beta release

Release 1.45.1 - May 13, 2015

  • fix : fix bug in the LZNA level 7 (Optimal3) encoder

Release 1.45.0 - May 11, 2015

  • new : New LZNA compressor! More compression than LZA and fast to decode on modern CPUs.

Release 1.44.2 - March 4, 2015

  • enhancement : LZA new optimal parse; faster at Optimal1 and more compression at Optimal3
  • enhancement : LZNib Normal encoder faster, better optimal parse
  • enhancement : LZB Fast & VeryFast encoder faster

Release 1.44.1 - December 9, 2014

  • fix : example_packet state header was not the same in 32-bit and 64-bit builds
  • enhancement : example_packet LZ compression level option added

Release 1.44.0 - November 16, 2014

  • new : Mac sdk now includes 64 bit libraries
  • new : Android and IOS sdk's available, for Oodle Network only
  • new : Unreal Engine integration of Oodle Network is available
  • change : example_packet has better trained model IO

Release 1.43.0 - September 8, 2014

  • fix : LZA packet compressor introduced in 1.42 was getting much less compression than it should due a bug; fixed!

Release 1.42.0 - September 3, 2014

  • change : reduce memory use of LZNib and LZBLW compressors at Optimal level
  • new : LZA compressor now has sliding window encoding and decoding
  • new : OodleLZ_CompressWithContext incremental API now supports LZA
  • change : example_packet LZA option added
  • change : Oodle is using the new "radtypes.h" shared header
  • enhancement : OodleNetwork1 UDP new implementation - faster, a little more compression.
  • fix : Optimal parse compressors could hang on very large degenerate files

Release 1.41.0 - July 10, 2014

  • new : OodleNetwork dictionary selection by string matching; see OodleNetwork1_SelectDictionaryFromPackets and example_packet : Example demonstrating network packet compression ; usually improves compression
  • enhancement : OodleNetwork1 TCP memory use reduced to 84104 bytes per channel
  • enhancement : OodleNetwork1 UDP is faster to encode ; saved states are incompatible with older versions
  • enhancement : LZNIB VeryFast and Normal much faster to encode; see FAQ : What are the speeds and ratios of the OodleLZ compressors and levels? for speeds
  • enhancement : LZB16-Fast and VeryFast much faster to encode; see FAQ : What are the speeds and ratios of the OodleLZ compressors and levels? for speeds
  • change : rename OodleNetwork1 TCP variant to OodleNetwork1TCP for clarity
  • change : better default hash table sizes for LZB16
  • fix : Fix bug in which packets that expanded in OodleNetwork1 TCP could cause a write at compbuf[-1]

Release 1.40.0 - June 27, 2014

  • change : OodleStaticLZP renamed to OodleNetwork1
  • enhancement : LZHLW compressor significantly faster to encode at all levels, especially Normal
  • new : New compressor OodleLZ_Compressor_LZA for very high compression!
  • fix : example_packet had a hard-coded limit of 64k for packet sizes; that was removed
  • fix : OodleLZDecoder_Reset was in the header but not implemented
  • new : OodleNetwork1 compressed buffer size is now limited by OodleNetwork1_CompressedBufferSizeNeeded

Release 1.30.0 - April 14, 2014

  • new : Minimal library initialization; see OodleX_Init_GetDefaults_Minimal , OodleX_Init_NoThreads , OodleX_Shutdown_NoThreads
  • change : OodleX_Init no longer accepts NULL options
  • change : Windows : OodleX_Init no longer does timeBeginPeriod(1) ; it's now left up to the app to decide if they want to do that
  • change : api simplified; many unused APIs removed

Release 1.22.0 - March 31, 2014

  • new : redo docs for Oodle Network packet compression

Release 1.21.2 - March 11, 2014

  • fix : OOP tool failed to extract large packages due to exceeding threading limits
  • fix : SPU LZ match offsets fixed again
  • fix : ThreadProfiler code stripped completely in final build

Release 1.21.0 - March 1, 2014

  • new : OodleNetwork1UDP_StateCompacted added to persist the trained state more compactly
  • enhancement : OodleLZH and LZHLW decompressors further optimized; 5-10% faster
  • enhancement : OodleLZH and LZHLW no longer does an allocation to prevent overrun
  • enhancement : OodleNetwork1 for UDP networking significantly faster (about 300%)
  • fix : SPU LZ match offsets could be too large by 1, causing corruption in rare cases

Release 1.20.0 - November 25, 2013

  • new : Oodle now shipping "final" build without debug facilities
  • new : OodleLZ_GetZipLikeCompressionSettings to make it easier to select compressor settings
  • fix : fix possible mismatch between structs in the public header and structs in the lib
  • fix : matchTableSizeLog2 wasn't being used correctly; made LZB and LZNIB compress less than they should

Release 1.19.0 - November 14, 2013

  • new : OodleNetwork1 now provides a UDP (stateless) variant for unordered packet compression (beta)
  • new : OodleHuffman_MultiHuffEncode for UDP unordered packet compression (beta)
  • note : version 119 numbering adjusted to match its intention better. 1.major.minor
  • fix : LZNib could fail when using "in-place" decompression
  • fix : LZH and LZHLW could speculatively read past the end of compbuf in rare cases, potentially causing a (benign) access violation
  • fix : LZ parallel compression with long-range-matcher could fail when parallel chunks didn't align to LRM chunks

Release 1.1.8 - November 4, 2013

  • new : add OodleXLZ_Decompress_MakeSeekTable_Wide_Async , a simple parallel version of OodleLZ_Decompress
  • new : add matchTableSizeLog2 to OodleLZ_CompressOptions ; controls encoder memory use and speed
  • new : add FAQ : What are the speeds and ratios of the OodleLZ compressors and levels? to make choosing a compressor and level easier
  • enhancement : LZB16 encoder faster
  • fix : LZ decoders could speculatively read past the end of compressed buffers; this could cause an access violation if the memory after the compressed buffer was not readable. Fixed.
  • fix : Fix issues with the WithContext family of functions when used with circular window encoding.
  • fix : Fix issues with combining heterogenous OodleLZ data (with different seek chunk settings) causing the decoder to make invalid parallel decodes
  • fix : Fix issues with LZB when used with packed-raw-overlap decode mode
  • enhancement : PS4 : OodleMalloc goes to "default" memory instead of heap memory for large allocations

Release 1.1.7 - October 23, 2013

  • new : unified OodleLZ_CompressContext functions for incremental encoding, with and without sliding window
  • new : LZB16 can encode & decode in a circular window
  • fix : fix scheduling bug in OOZ IO->Decompression loader
  • fix : Linux : fix LZB16 generation page faults due to movdqa gcc bug
  • fix : don't lock Oodle threads to cores except on platforms where it is necessary

Release 1.1.6 - October 7, 2013

  • new : new LZB compressor - LZ-Bytewise (and LZB-LargeWindow), the fastest of all, for when even LZNib isn't fast enough
  • fix : OodleLZ_FillSeekTable could incorrectly mark heterogenous streams as being entirely independent seek chunks
  • fix : Linux & Mac : Fix worker thread stack sizes too small, could crash OOP tool and other uses of the worker threads.
  • fix : In-place decompression of incompressible data could fail if source overlapped dest.
  • fix : Fix OodleLZ_Compress_Async not working (and related APIs, such as OodleLZ_Compress_WriteOOZ_Async) when OodleAsyncSelect_None was passed for asyncSelect (now runs synchronously on the calling thread).

Release 1.1.5 - August 8, 2013

  • enhancement : LZNib now has a sliding window encoder for streaming compression
  • new : new OodleHuffman functions expose huffman-only coding
  • new : OodleNetwork1 compressor, static dictionary ideal for MMO network compression (Beta)

Release 1.1.4 - May 6, 2013

  • enhancement : large IO ops are now broken on the IO thread instead of at call time
  • enhancement : large IO aps are now cancellable in progress and return correct status
  • fix : OOP header could fail to read when it had a seek table
  • enhancement : rev the OOZ header; add the size of the header

Release 1.1.3 - May 1, 2013

  • fix : OOP header reader was truncating 64-bit file sizes to 32-bit
  • new : lzadvanced_output_chunking in example_lz_advanced
  • enhancement: OodleLZ_GetCompressedStepForRawStep allowed to seek quanta

Release 1.1.2 - April 28, 2013

  • fix : Linux dir enumeration was including "."
  • fix : Linux Open for WriteCreate was overwriting existing files too aggressively
  • fix : Linux Open failure had debug perror left in
  • change : better default log dirs and log names
  • change : log file will open in "." if default log dir is not accessible
  • change : Linux & Mac do log->prev cycle like Windows
  • change : disable stack trace on .map file platforms for now
  • enhancement: OodlePackage_FillWithFiles only decompresses chunks it needs (not all)
  • enhancement: OodlePackage_SortFiles now uses extention file groups

Release 1.1.1 - April 24, 2013

  • fix : Oodle_GetExtensionKey was not doing tolower as it should
  • fix : OodlePackage_FillWithFiles could crash if files failed to load
  • fix : fix memory overrun crash in DirListing
  • fix : fix null deref in arg parse in OOP tool
  • change: rename IOQ_DeleteFile_ to IOQ_Delete_ and make it work on dirs as well
  • new: add OodleDirListing_DeleteDirContents

Release 1.1.0 - April 11, 2013

  • new: LZNIB super fast to decode LZ variant added
  • new: LosslessFilters for some common data transformations that improve compression
  • change: Low level IO now allows aligned and unaligned IO modes
  • enhancement: Compression seek chunk size now variable, set by client
  • enhancement: All compressors can run encode and decode in parallel
  • enhancement: Long Range Matcher for LZ on big buffers
  • enhancement: (PS3) SPU waits now without thread switches
  • enhancement: (PS3) SPU after_decompress replaceable function

Release 1.0.0 - August 24, 2012

  • Initial Release.