Overview

Zarr Backend and Utilities

hdmf_zarr implements a Zarr backend for HDMF. Some of the key classes relevant for end-users are:

  • ZarrIO implements an alternative storage backend to store data using HDMF via the Zarr library.

  • NWBZarrIO uses ZarrIO to define a Zarr backend store for integration with PyNWB to simplify the use of hdmf_zarr with NWB (similar to NWBHDF5IO in PyNWB)

  • utils implements utility classes for the ZarrIO backend. For end-users the ZarrDataIO class is relevant for defining advanced I/O options for datasets.

Supported features

  • Write/Read of basic data types, strings and compound data types

  • Chunking

  • Compression and I/O filters

  • Links

  • Object references

  • Writing/loading namespaces/specifications

  • Iterative data write using AbstractDataChunkIterator

  • Parallel write with GenericDataChunkIterator (since v0.4)

  • Lazy load of datasets

  • Lazy load of datasets containing object references (since v0.4)

Known Limitations

  • The Zarr backend is currently experimental and may still change.

  • Zarr v2 read-only: Zarr v2 folders can only be read in read-only mode with the ZarrV2IO and corresponding NWBZarrV2IO classes. Writing of new Zarr v2 files is not supported, i.e., new files can only be written in Zarr v3 via ZarrIO and NWBZarrIO. Zarr v2 folders can be converted to v3 using the convert_to_v3() function. Most Zarr v2 NWB files hold at least one pickle-encoded dataset, because hdmf-zarr 0.13 and earlier encode object datasets (an electrodes table’s object-reference columns, for instance) with the pickle codec by default. Decoding pickle executes arbitrary code, so reading and converting both refuse it until you pass allow_pickle=True for a source you trust, e.g. NWBZarrV2IO.convert_to_v3(source_path, dest_path, allow_pickle=True). (since v0.14)

  • Attributes are stored as JSON documents in Zarr (using the LocalStore). As such, all attributes must be JSON serializable. The ZarrIO backend attempts to cast types to JSON serializable types as much as possible.

  • Currently the ZarrIO backend supports Zarr’s LocalStore for local storage and FsspecStore for remote read-only access. Other Zarr stores could be added but will require proper treatment of links and references for those backends as links are not supported in Zarr (see zarr-python issues #389).

  • Compound dtypes with strings: Compound data types are stored in zarr v3’s struct data type, with string and reference fields as fixed-length Unicode strings (fixed_length_utf32). Both are registered Zarr extension data types, so a zarr implementation that does not support them may not read these datasets.

  • Exporting of HDF5 files with external links is not yet fully implemented/tested (see hdmf-zarr issue #49).

  • Special characters (e.g., :, <, >, ", /, \, |, ?, or *) may not be supported by all file systems (e.g., on Windows) and as such should not be used as part of the names of Datasets or Groups as Zarr needs to create folders on the filesystem for these objects.