Skip to end of metadata
Go to start of metadata

ZFS has a powerful storage cache architecture built mainly around the ARC or Adaptive Replacement Cache. However, starting around 2006, rapid progress in flash memory technology prompted an expansion of ZFS' cache hierarchy to include support for an additional cache tier that would fit between traditional DRAM caches and disk spindles. This has finally arrived in 2008 in the form of the ZFS Level 2 Adaptive Replacement Cache (L2ARC), which adds the ability to have "cache" devices as part of a storage pool that will effectively expand the space available for data caching to this new class of storage devices. For more background information on the L2ARC, please see the L2ARC article written by Brendan Gregg.

Developed as issue #3137 is a new feature that introduces the ability to compress L2ARC contents. With the inclusion of the very high-performance LZ4 compression algorithm in ZFS, it is now possible to transparently compress and decompress large volumes of data with much lower latency and overhead than was previously possible with LZJB. As such, the L2ARC is the next possible target for transparent compression.

Why Compress the L2ARC?

The L2ARC is an important feature for read-heavy workloads that are sensitive to access latency, or that need to access a large dataset in a fairly unpredictable fashion. While relatively cheaper than DRAM, the L2ARC is still an appreciably precious resource. Thus, having the ability to extend available L2ARC space by compressing data being written to the devices is important.

Performance Numbers

The actual impact that compression has on L2ARC performance depends largely on the dataset composition and available CPU performance:

  • Large-block workloads, such as file and storage servers, will generally tend to see a higher compression ratio, and thus benefit, than small-block workloads (such as databases).
  • In most storage environments, performance is primarily limited by available IO throughput and latency. Compressing the L2ARC partially offloads the IO subsystem (by allowing it to do less physical IO for the same data volume processed) by shifting the workload over to the CPUs.
  • Overall throughput of the machine can drastically improve on workloads that do not fit into L2ARC entirely and must bear some L2 cache misses that then degrade into main pool accesses.

The following benchmark numbers were gathered on a relatively modest home system:

  • HP MicroServer
  • CPU: 1x AMD Athlon II N36L Neo (dual-core 64-bit at 1.3 GHz)
  • RAM: 8GB DDR3
  • Test pool:
    • 4x WD Red 2TB SATA drives in a raidz
    • 1x OCZ Vertex 3 60GB SSD as L2ARC
    • zfs_prefetch_disable=1

The benchmarking workload consisted of a fixed dataset (uncompressed on-disk) being accessed by 50 parallel reader threads. Datasets are constructed in increments of 200 copies of Silesia Corpus (each increment thus being 40GB in size) - given previous LZ4 benchmarking results on ZFS, we know that this dataset compresses down by approximately 51% on ZFS (thus, effectively doubling L2ARC capacity). Each thread, executes an infinite loop of:

  1. randomly select a file in the working set and open it
  2. seek to a random 128k-aligned offset within the file
  3. read eight 128k data blocks
  4. close file and start over


Aggregate Disk Throughput

This result shows the combined throughput of all 50 reader threads.

Aggregate Throughput
Dataset SizeL2ARC Setting
 no_l2arcoff (no_compress)all (lz4_compress)
40 GB69,482 kB/s255,651 kB/s343,181 kB/s
80 GB55,390 kB/s161,845 kB/s304,654 kB/s
120 GB47,124 kB/s82,518 kB/s263,653 kB/s
160 GB41,651 kB/s55,534 kB/s121,294 kB/s
200 GB37,734 kB/s47,531 kB/s76,015 kB/s
240 GB35,210 kB/s42,013 kB/s64,355 kB/s

Aggregate Disk IOs

This result shows the combined number of block IOs of all 50 reader threads. We consider a single block IO to be a single 128k read, i.e. a single file access consisted of 8 consecutive block IOs. Given that block size didn't vary between reads, these results track the above throughput results fairly accurately.

Aggregate IOs
Dataset SizeL2ARC Setting
 no_l2arcoff (no_compress)all (lz4_compress)
40 GB545 io/s2005 io/s2691 io/s
80 GB434 io/s913 io/s2389 io/s
120 GB369 io/s647 io/s2068 io/s
160 GB326 io/s435 io/s951 io/s
200 GB295 io/s372 io/s596 io/s
240 GB276 io/s329 io/s504 io/s

How To Start Using L2ARC Compression

Whether to compress a buffer is automatically determined from the buffer's originating dataset's compress property setting. If it is set to anything other than off, the buffer will be compressed when entering the L2ARC. In other words, compression in L2ARC mirrors compression behavior on the main pool (though the algorithm for the L2ARC is fixed to be LZ4). If multiple datasets reference the same on-pool data blocks, L2ARC compression is applied if at least one of the datasets is compressed (i.e. compression is preferred).

As of the time of writing support for compressed L2ARC exists in OmniOS r151008 bloody and OpenIndiana oi_151a8. For support in proprietary Illumos distros contact your distro vendor.