hdmf_zarr package
Submodules
- hdmf_zarr.backend module
ROOT_NAMESPEC_LOC_ATTRDEFAULT_SPEC_LOC_DIRREFERENCE_DTYPE_ATTR_VALUECOMPOUND_DTYPE_MIN_STRING_LENGTHSUPPORTED_ZARR_STORESZarrIOZarrIO.can_read()ZarrIO.generate_dataset_html()ZarrIO.pathZarrIO.abspathZarrIO.modeZarrIO.open()ZarrIO.close()ZarrIO.is_remote()ZarrIO.load_namespaces()ZarrIO.load_namespaces_io()ZarrIO.write()ZarrIO.export()ZarrIO.get_written()ZarrIO.get_builder_exists_on_disk()ZarrIO.get_builder_disk_path()ZarrIO.write_builder()ZarrIO.write_group()ZarrIO.write_attributes()ZarrIO.get_zarr_parent_path()ZarrIO.resolve_ref()ZarrIO.write_link()ZarrIO.write_dataset()ZarrIO.get_type()ZarrIO.read_builder()ZarrIO.get_container()ZarrIO.get_builder()
- hdmf_zarr.backend_zarrv2 module
- hdmf_zarr.nwb module
- hdmf_zarr.nwb_zarrv2 module
- hdmf_zarr.utils module
- hdmf_zarr.zarr_utils module
Module contents
- class hdmf_zarr.ZarrIO(path, mode, manager=None, storage_options=None, force_overwrite=False)
Bases:
HDMFIO- Parameters:
path (
strorPathorLocalStoreorFsspecStoreorStore) – the path to the Zarr file or a supported Zarr storemode (
str) – the mode to open the Zarr file with, one of (“w”, “r”, “r+”, “a”, “r-“). the mode r- is used to force open without consolidated metadata in read only mode.manager (
BuildManager) – the BuildManager to use for I/Ostorage_options (
dict) – Zarr storage options to read remote foldersforce_overwrite (
bool) – force overwriting existing object when in ‘w’ mode. The existing file or directory will be deleted when before opening (even if the object is not Zarr, e.g,. an HDF5 file)
- static can_read(path)
Determines whether a given path is readable by this HDMFIO class
- static generate_dataset_html(dataset)
Generates an HTML representation for a dataset for the ZarrIO class.
This method extracts metadata from a Zarr array using its info object and formats it as an HTML table for display in Jupyter notebooks and other HTML-based interfaces.
- Parameters:
dataset (zarr.Array) – The Zarr array for which to generate an HTML representation
- Returns:
HTML representation of the dataset
- Return type:
- property path
The path to the Zarr file as set by the user
- property abspath
The absolute path to the Zarr file
- property mode
The mode specified by the user when creating the ZarrIO instance.
NOTE: The Zarr library may not honor the mode. E.g., LocalStore in Zarr uses append mode and does not allow setting a file to read-only mode.
- open()
Open the Zarr file
- close()
Close the Zarr file
- is_remote()
Return True if the file is remote, False otherwise
- classmethod load_namespaces(namespace_catalog, path=None, file=None, storage_options=None, namespaces=None)
Load cached namespaces from a file.
- Parameters:
namespace_catalog (
NamespaceCatalogorTypeMap) – the NamespaceCatalog or TypeMap to load namespaces intopath (
strorPathorLocalStoreorFsspecStoreorStore) – the path to the Zarr file or a supported Zarr storefile (
Group) – An already opened Zarr groupstorage_options (
dict) – Zarr storage options to read remote foldersnamespaces (
list) – the namespaces to load
- Returns:
dict mapping the names of the loaded namespaces to a dict mapping included namespace names and the included data types
- Return type:
- load_namespaces_io(namespace_catalog, namespaces=None)
Load cached namespaces from this ZarrIO object itself.
- Parameters:
namespace_catalog (
NamespaceCatalogorTypeMap) – the NamespaceCatalog or TypeMap to load namespaces intonamespaces (
list) – the namespaces to load
- Returns:
dict mapping the names of the loaded namespaces to a dict mapping included namespace names and the included data types
- Return type:
- write(container, cache_spec=True, link_data=True, exhaust_dci=True, number_of_jobs=1, max_threads_per_process=None, multiprocessing_context=None, consolidate_metadata=True)
Overwrite the write method to add support for caching the specification and parallelization.
- Parameters:
container (
Container) – the Container object to writecache_spec (
bool) – cache specification to filelink_data (
bool) – If not specified otherwise link (True) or copy (False) Datasetsexhaust_dci (
bool) – exhaust DataChunkIterators one at a time. If False, add them to the internal queue self.__dci_queue and exhaust them concurrently at the endnumber_of_jobs (
int) – Number of worker processes to use in parallel during write (only works with GenericDataChunkIterator-wrapped datasets). For sharded arrays, parallel tasks own separate shards. Shard and iterator buffer shapes therefore affect the number of independent tasks and available parallelism. See the ZarrDataIOshardsparameter for buffer alignment and memory guidance.max_threads_per_process (
int) – Limits the number of threads used by each process. The default is None (no limits).multiprocessing_context (
str) – Context for multiprocessing. It can be None (default), ‘fork’ or ‘spawn’. Note that ‘fork’ is only available on UNIX systems (not Windows).consolidate_metadata (
bool) – Consolidate metadata into the root group’s zarr.json to accelerate read.
- export(src_io, container=None, write_args={}, clear_cache=False, cache_spec=True, number_of_jobs=1, max_threads_per_process=None, multiprocessing_context=None, consolidate_metadata=True)
Export data read from a file from any backend to Zarr. See
hdmf.backends.io.HDMFIO.export()for more details.- Parameters:
src_io (
HDMFIO) – the HDMFIO object for reading the data to exportcontainer (
Container) – the Container object to export. If None, then the entire contents of the HDMFIO object will be exportedwrite_args (
dict) – arguments to pass towrite_builder()clear_cache (
bool) – whether to clear the build manager cachecache_spec (
bool) – whether to cache the specification to filenumber_of_jobs (
int) – Number of worker processes to use in parallel during write (only works with GenericDataChunkIterator-wrapped datasets). For sharded arrays, parallel tasks own separate shards. Shard and iterator buffer shapes therefore affect the number of independent tasks and available parallelism. See the ZarrDataIOshardsparameter for buffer alignment and memory guidance.max_threads_per_process (
int) – Limits the number of threads used by each process. The default is None (no limits).multiprocessing_context (
str) – Context for multiprocessing. It can be None (default), ‘fork’ or ‘spawn’. Note that ‘fork’ is only available on UNIX systems (not Windows).consolidate_metadata (
bool) – Consolidate metadata into the root group’s zarr.json to accelerate read.
- get_written(builder, check_on_disk=False)
Return True if this builder has been written to (or read from) disk by this IO object, False otherwise.
- Parameters:
builder (Builder) – Builder object to get the written flag for
check_on_disk (bool) – Check that the builder has been physically written to disk not just flagged as written by this I/O backend
- Returns:
True if the builder is found in self._written_builders using the builder ID, False otherwise. If check_on_disk is enabled then the function cals get_builder_exists_on_disk in addition to verify that the builder has indeed been written to disk.
- get_builder_exists_on_disk(builder)
Convenience function to check whether a given builder exists on disk in this Zarr file.
- Parameters:
builder (
Builder) – The builder of interest
- get_builder_disk_path(builder, filepath=None)
- write_builder(builder, link_data=True, exhaust_dci=True, export_source=None, consolidate_metadata=True)
Write a builder to disk.
- Parameters:
builder (
GroupBuilder) – the GroupBuilder object representing the NWBFilelink_data (
bool) – If not specified otherwise link (True) or copy (False) Zarr Datasetsexhaust_dci (
bool) – Exhaust DataChunkIterators one at a time. If False, add them to the internal queue self.__dci_queue and exhaust them concurrently at the endexport_source (
str) – The source of the builders when exportingconsolidate_metadata (
bool) – Consolidate metadata into the root group’s zarr.json to accelerate read.
- write_group(parent, builder, link_data=True, exhaust_dci=True, export_source=None)
Write a GroupBuider to file
- Parameters:
parent (
Group) – the parent Zarr objectbuilder (
GroupBuilder) – the GroupBuilder to writelink_data (
bool) – If not specified otherwise link (True) or copy (False) Zarr Datasetsexhaust_dci (
bool) – exhaust DataChunkIterators one at a time. If False, add them to the internal queue self.__dci_queue and exhaust them concurrently at the endexport_source (
str) – The source of the builders when exporting
- Returns:
the Group that was created
- Return type:
Group
- write_attributes(obj, attributes)
Set (i.e., write) the attributes on a given Zarr Group or Array.
- static get_zarr_parent_path(zarr_object)
Get the absolute Unix path to the parent of a zarr_object from the root of the Zarr file :param zarr_object: Object for which we are looking up the path :type zarr_object: Zarr Group or Array :return: String with the path
- resolve_ref(zarr_ref)
Get the full path to the object linked to by the zarr reference
The function only constructs the links to the target object, but it does not check if the object exists
- Parameters:
zarr_ref – A reference to resolve. Can be a dict with source and path keys, a ZarrReference object, a JSON string, or a scalar
np.ndarrayofStringDType.- Returns:
name of the target object
the target zarr object within the target file
- write_link(parent, builder, export_source=None)
- Parameters:
parent (
Group) – the parent Zarr objectbuilder (
LinkBuilder) – the LinkBuilder to writeexport_source (
str) – The source of the builders when exporting
- write_dataset(parent, builder, link_data=True, exhaust_dci=True, force_data=None, export_source=None)
- Parameters:
parent (
Group) – the parent Zarr objectbuilder (
DatasetBuilder) – the DatasetBuilder to writelink_data (
bool) – If not specified otherwise link (True) or copy (False) Zarr Datasetsexhaust_dci (
bool) – exhaust DataChunkIterators one at a time. If False, add them to the internal queue self.__dci_queue and exhaust them concurrently at the endforce_data (None) – Used internally to force the data being used when we have to load the data
export_source (
str) – The source of the builders when exporting
- Returns:
the Zarr array that was created
- Return type:
- classmethod get_type(data)
- read_builder()
- Returns:
a GroupBuilder representing the NWB Dataset
- Return type:
GroupBuilder
- get_container(zarr_obj)
Get the container for the corresponding Zarr Group or Dataset
- Raises:
ValueError – When no builder has been constructed yet for the given h5py object
- Parameters:
zarr_obj (
ArrayorGroup) – the Zarr object to the corresponding Container/Data object for
- get_builder(zarr_obj)
Get the builder for the corresponding Group or Dataset
- Raises:
ValueError – When no builder has been constructed
- Parameters:
zarr_obj (
ArrayorGroup) – the Zarr object to the corresponding Builder object for
- class hdmf_zarr.ZarrDataIO(data, chunks=None, fillvalue=None, compressors=None, compressor=None, filters=None, serializer=None, shards=None, link_data=False)
Bases:
DataIOWrap data arrays for write via ZarrIO to customize I/O behavior, such as compression and chunking for data arrays.
- Parameters:
data (
ndarrayorlistortupleorArrayorIterable) – the data to be written. NOTE: If an zarr.Array is used, all other settings but link_data will be ignored as the dataset will either be linked to or copied as is in ZarrIO.chunks (
listortuple) – Chunk shape. Whenshardsis also specified,chunksdefines the inner chunk shape within each shard (i.e. the fine-grained I/O unit).fillvalue (None) – Value to be returned when reading uninitialized parts of the dataset
compressors (
BytesBytesCodecorlistortupleorbool) – Zarr compressor codecs (BytesBytesCodec) to be used. Can be a single codec or list of codecs. Set to True to use Zarr default. Set to False to disable compression. Use zarr.codecs (e.g., zarr.codecs.BloscCodec()) or zarr.codecs.numcodecs wrappers.compressor (None) – DEPRECATED. Zarr compressor filter to be used. Instead of using this argument, pass a zarr v3 BytesBytesCodec (e.g., zarr.codecs.BloscCodec()) to
compressors.filters (
listortuple) – One or more Zarr-supported codecs (ArrayArrayCodec) used to transform the array before it is serialized to bytes. Use zarr.codecs or zarr.codecs.numcodecs wrappers.serializer (
ArrayBytesCodec) – Zarr codec (ArrayBytesCodec) used to serialize the array to bytes, e.g., zarr.codecs.BytesCodec(). Zarr chooses a default for the dtype when this is not given.shards (
listortupleorstr) – Shard shape in array elements, or ‘auto’ to let Zarr choose the shape. Each shard is a single object in the store and contains multiple inner chunks defined bychunks. Sharding reduces the number of store objects and can improve performance for large arrays.chunksdefines the inner chunk shape. For efficient parallel iterator writes, prefer abuffer_shapeequal to the shard shape or an integer multiple along each axis, so each buffer contains complete shards. Buffers spanning a full array axis may include a partial edge shard. Aligned buffers are written as whole-buffer tasks. Other buffer shapes are supported safely by assigning each shard to one task and sequentially reading and writing the portions of iterator buffers inside it. Partial shard updates read the entire existing encoded shard and assemble its replacement. When shards span multiple buffers, this repeats for successive buffer pieces and can use substantially more memory than the source buffer.buffer_shapelimits source read sizes, not total write memory, including Zarr’s encoded shard buffers and codec workspace. With ‘auto’, scheduling uses the shard shape chosen by Zarr; buffer-shard alignment is not required.link_data (
bool) – If data is an zarr.Array should it be linked to or copied. NOTE: This parameter is only allowed if data is an zarr.Array
- property link_data: bool
Only applies to zarr.Array type data
- Type:
Bool indicating should it be linked to or copied. NOTE
- static from_h5py_dataset(h5dataset, **kwargs)
Factory method to create a ZarrDataIO instance from a h5py.Dataset. The ZarrDataIO object wraps the h5py.Dataset and the io filter settings are inferred from filters used in h5py such that the options in Zarr match (if possible) the options used in HDF5.
- Parameters:
dataset (h5py.Dataset) – h5py.Dataset object that should be wrapped
kwargs – Other keyword arguments to pass to ZarrDataIO.__init__
fillvalue,chunks,compressors, andfiltersoverride the corresponding values inferred fromh5dataset. When omitted, fill value and chunks are copied from the HDF5 dataset, while Zarr compressors and filters are inferred from its HDF5 filter pipeline.
- Returns:
ZarrDataIO object wrapping the dataset
- static hdf5_to_zarr_filters(h5dataset) list
From the given h5py.Dataset infer the corresponding zarr v3 codecs.
Returns a list of zarr v3 codec instances (BytesBytesCodec or ArrayArrayCodec). Uses zarr.codecs.numcodecs wrappers for codecs not natively available in zarr v3.
- static is_h5py_dataset(obj)
Check if the object is an instance of h5py.Dataset without requiring import of h5py
- class hdmf_zarr.NWBZarrIO(path, mode, manager=None, storage_options=None, force_overwrite=False, load_namespaces=True)
Bases:
ZarrIOIO backend for PyNWB for writing NWB files
This class is similar to the
NWBHDF5IOclass in PyNWB. The main purpose of this class is to perform default setup for BuildManager, loading or namespaces etc., in the context of the NWB format.- Parameters:
path (
strorPathorLocalStoreorFsspecStoreorStore) – the path to the Zarr file or a supported Zarr storemode (
str) – the mode to open the Zarr file with, one of (“w”, “r”, “r+”, “a”, “r-“). the mode r- is used to force open without consolidated metadata in read only mode.manager (
BuildManager) – the BuildManager to use for I/Ostorage_options (
dict) – Zarr storage options to read remote foldersforce_overwrite (
bool) – force overwriting existing object when in ‘w’ mode. The existing file or directory will be deleted when before opening (even if the object is not Zarr, e.g,. an HDF5 file)load_namespaces (
bool) – whether or not to load cached namespaces from given path - not applicable in write mode
- export(src_io, nwbfile=None, write_args={})
- static read_nwb(path)
Helper factory method for reading an NWB file and return the NWBFile object.
- Parameters:
path (
strorPathorLocalStoreorFsspecStoreorStore) – the path to the Zarr file or a supported Zarr store
- class hdmf_zarr.ZarrV2IO(path, mode, manager=None, storage_options=None, force_overwrite=False, allow_pickle=False)
Bases:
ZarrIORead-only
ZarrIOextension that handles zarr-v2 NWB files.Compared to
ZarrIOthis class:Falls back to
use_consolidated=Falseif zarr v3 cannot parse the consolidated metadata block (e.g. fill_value incompatibilities).Iterates group children via the store and decodes object-dtype arrays directly when zarr v3 refuses them.
Resolves reference
sourcepaths against the parent directory (old hdmf-zarr convention) in addition to the store itself.Uses
ZarrV2SpecReaderfor cached namespaces.
Note
When zarr v3 cannot parse a v2 array (e.g. object-dtype arrays stored via
pickle/json2/vlen-utf8filters, or fill_values incompatible with the encoded dtype), this backend falls back to decoding the raw chunks directly (see_read_v2_dataset()/_decode_v2_dataset()). Unlike the normal zarr-backed read path, this fallback eagerly loads the entire dataset into memory at read time rather than reading it lazily on access. For the small, object-dtype arrays this path typically handles (specs, references, compound/vlen columns) this is not a concern, but it could matter for large datasets that zarr v3 fails to parse. Lazy decoding could be added in a follow-up if it becomes a performance issue (see https://github.com/hdmf-dev/hdmf-zarr/issues/356).- Parameters:
path (
strorPathorLocalStoreorFsspecStoreorStore) – the path to the Zarr file or a supported Zarr storemode (
str) – the mode to open the Zarr file with, one of (“w”, “r”, “r+”, “a”, “r-“). the mode r- is used to force open without consolidated metadata in read only mode.manager (
BuildManager) – the BuildManager to use for I/Ostorage_options (
dict) – Zarr storage options to read remote foldersforce_overwrite (
bool) – force overwriting existing object when in ‘w’ mode. The existing file or directory will be deleted when before opening (even if the object is not Zarr, e.g,. an HDF5 file)allow_pickle (
bool) – whether to decode unsafe pickle codecs from a trusted v2 file
- property allow_pickle
Whether unsafe pickle decoding is enabled for this trusted v2 file.
- property skipped_entries
Entries this IO could not read, as a list of
(path, reason)pairs.An entry lands here when neither zarr v3 nor the raw v2 chunk decoder could read it. It is absent from the builders this IO produced, so a caller that needs a faithful copy of the file must treat a non-empty list as a failure.
- classmethod load_namespaces(namespace_catalog, path=None, file=None, storage_options=None, namespaces=None)
Load cached namespaces, refusing any cached spec that declares the pickle codec.
- Parameters:
namespace_catalog (
NamespaceCatalogorTypeMap) – the NamespaceCatalog or TypeMap to load namespaces intopath (
strorPathorLocalStoreorFsspecStoreorStore) – the path to the Zarr file or a supported Zarr storefile (
Group) – An already opened Zarr groupstorage_options (
dict) – Zarr storage options to read remote foldersnamespaces (
list) – the namespaces to load
- static can_read(path)
Determines whether a given path is readable by this HDMFIO class
- resolve_ref(zarr_ref)
Resolve zarr_ref as
ZarrIO.resolve_ref()does, enforcing the pickle trust policy on the target.Links and references open their targets here, which can lie outside the groups
_iter_children()walks (for example underspecifications), so the target array is checked before it is returned to be decoded.
- write(container, cache_spec=True, link_data=True, exhaust_dci=True, number_of_jobs=1, max_threads_per_process=None, multiprocessing_context=None, consolidate_metadata=True)
Overwrite the write method to add support for caching the specification and parallelization.
- Parameters:
container (
Container) – the Container object to writecache_spec (
bool) – cache specification to filelink_data (
bool) – If not specified otherwise link (True) or copy (False) Datasetsexhaust_dci (
bool) – exhaust DataChunkIterators one at a time. If False, add them to the internal queue self.__dci_queue and exhaust them concurrently at the endnumber_of_jobs (
int) – Number of worker processes to use in parallel during write (only works with GenericDataChunkIterator-wrapped datasets). For sharded arrays, parallel tasks own separate shards. Shard and iterator buffer shapes therefore affect the number of independent tasks and available parallelism. See the ZarrDataIOshardsparameter for buffer alignment and memory guidance.max_threads_per_process (
int) – Limits the number of threads used by each process. The default is None (no limits).multiprocessing_context (
str) – Context for multiprocessing. It can be None (default), ‘fork’ or ‘spawn’. Note that ‘fork’ is only available on UNIX systems (not Windows).consolidate_metadata (
bool) – Consolidate metadata into the root group’s zarr.json to accelerate read.
- export(src_io, container=None, write_args={}, clear_cache=False, cache_spec=True, number_of_jobs=1, max_threads_per_process=None, multiprocessing_context=None, consolidate_metadata=True)
Export data read from a file from any backend to Zarr. See
hdmf.backends.io.HDMFIO.export()for more details.- Parameters:
src_io (
HDMFIO) – the HDMFIO object for reading the data to exportcontainer (
Container) – the Container object to export. If None, then the entire contents of the HDMFIO object will be exportedwrite_args (
dict) – arguments to pass towrite_builder()clear_cache (
bool) – whether to clear the build manager cachecache_spec (
bool) – whether to cache the specification to filenumber_of_jobs (
int) – Number of worker processes to use in parallel during write (only works with GenericDataChunkIterator-wrapped datasets). For sharded arrays, parallel tasks own separate shards. Shard and iterator buffer shapes therefore affect the number of independent tasks and available parallelism. See the ZarrDataIOshardsparameter for buffer alignment and memory guidance.max_threads_per_process (
int) – Limits the number of threads used by each process. The default is None (no limits).multiprocessing_context (
str) – Context for multiprocessing. It can be None (default), ‘fork’ or ‘spawn’. Note that ‘fork’ is only available on UNIX systems (not Windows).consolidate_metadata (
bool) – Consolidate metadata into the root group’s zarr.json to accelerate read.
- class hdmf_zarr.NWBZarrV2IO(path, mode, manager=None, storage_options=None, force_overwrite=False, allow_pickle=False, load_namespaces=True)
Bases:
ZarrV2IORead-only NWB IO for zarr-v2 files, opened via
ZarrV2IO.Use this class when reading NWB files written by older hdmf-zarr versions. Pickle-encoded datasets require
allow_pickle=Trueand must only be read from trusted files.- Parameters:
path (
strorPathorLocalStoreorFsspecStoreorStore) – the path to the Zarr file or a supported Zarr storemode (
str) – the mode to open the Zarr file with, one of (“w”, “r”, “r+”, “a”, “r-“). the mode r- is used to force open without consolidated metadata in read only mode.manager (
BuildManager) – the BuildManager to use for I/Ostorage_options (
dict) – Zarr storage options to read remote foldersforce_overwrite (
bool) – force overwriting existing object when in ‘w’ mode. The existing file or directory will be deleted when before opening (even if the object is not Zarr, e.g,. an HDF5 file)allow_pickle (
bool) – whether to decode unsafe pickle codecs from a trusted v2 fileload_namespaces (
bool) – whether or not to load cached namespaces from given path
- export_to_v3(path, nwbfile=None, write_args=None, storage_options=None)
Export this zarr-v2 NWB file to a new zarr-v3 NWB file.
Reads the contents of this read-only zarr-v2 file and writes them out as a zarr-v3 file via
NWBZarrIO. Because a v2 → v3 conversion rewrites the storage layout, data chunks are always copied (link_datais forced toFalse) rather than linked back to the source file.Raises
IncompleteConversionErrorwhen any entry of the source file could not be read, listing the entries. Reading a file for inspection reports such an entry as a warning and continues, but a conversion writes a file that is meant to stand in for the source, so a missing entry is an error here.- Parameters:
path (
strorPathorLocalStoreorFsspecStoreorStore) – the destination path (or Zarr store) for the exported zarr v3 NWB filenwbfile (
NWBFile) – the NWBFile object to export. If None, the file is read from this IO and exported in full.write_args (
dict) – arguments to pass towrite_builder()storage_options (
dict) – Zarr storage options for the destination store
- static convert_to_v3(source_path, dest_path, write_args=None, storage_options=None, allow_pickle=False)
Convert a zarr-v2 NWB file to a new zarr-v3 NWB file in one call.
Convenience one-shot wrapper that opens source_path with
NWBZarrV2IOand exports it to dest_path as a zarr-v3 file. For more control (e.g. inspecting or modifying theNWBFilebefore writing), open the file yourself and useexport_to_v3()instead.s3://source_path URLs are read anonymously (storage_options={"anon": True}), which works for public buckets. To read a source that needs credentials or non-default options (a private S3 bucket,gs://, etc.), open it withNWBZarrV2IO(source_path, mode="r", storage_options=...)and callexport_to_v3(). Thestorage_optionsargument of this method configures the destination store, not the source.Example:
NWBZarrV2IO.convert_to_v3("old_v2.nwb.zarr", "new_v3.nwb.zarr")
hdmf-zarr 0.13 and earlier encode object datasets with the pickle codec by default, so most Zarr v2 NWB files hold at least one pickle-encoded dataset (an electrodes table’s object-reference columns, for instance). Decoding pickle executes arbitrary code, so it is refused with
UnsafePickleCodecErrorunless the source is trusted:NWBZarrV2IO.convert_to_v3("old_v2.nwb.zarr", "new_v3.nwb.zarr", allow_pickle=True)
Raises
IncompleteConversionErrorwhen any entry of the source file could not be read; seeexport_to_v3().- Parameters:
source_path (
strorPath) – the path to the source zarr v2 NWB file to convertdest_path (
strorPathorLocalStoreorFsspecStoreorStore) – the destination path (or Zarr store) for the exported zarr v3 NWB filewrite_args (
dict) – arguments to pass towrite_builder()storage_options (
dict) – Zarr storage options for the destination storeallow_pickle (
bool) – whether the trusted source file may decode unsafe pickle codecs
- static read_nwb(path, allow_pickle=False)
Open a zarr-v2 NWB file and return its
NWBFile.s3://path URLs are read anonymously (storage_options={"anon": True}), which works for public buckets. To read a file that needs credentials or non-default options (a private S3 bucket,gs://, etc.), constructNWBZarrV2IOdirectly with the appropriatestorage_optionsand callread().
- hdmf_zarr.is_zarr_v2_file(path, storage_options=None)
Return
Trueif path looks like a zarr v2 hierarchy.Checks the zarr format reported by the opened hierarchy (zarr v2 sets
zarr_format=2in.zgroup; zarr v3 useszarr.json). Works for local paths, opened Zarr stores, and protocol URLs (including chained fsspec URLs such assimplecache::s3://).
- exception hdmf_zarr.IncompleteConversionError
Bases:
RuntimeErrorRaised when a v2 to v3 conversion would omit entries it could not read.