Integrating New Zarr Data Stores¶

ZarrIO by default uses the Zarr DirectoryStore via the zarr.convenience.open() method. ZarrIO further supports all stores listed in SUPPORTED_ZARR_STORES. Users can specify a particular store using the path parameter when creating a new ZarrIO instance. This document discusses key steps towards integrating other data stores available for Zarr with ZarrIO.

Updating ZarrIO¶

Import and add the new storage class to the SUPPORTED_ZARR_STORES. This will in turn allow instances of your new storage class to be passed as a path parameter to __init__() and load_namespaces() and pass docval() validation for these functions.
- If your store has a .path property then the source property will be set accordingly in __init__ in ZarrIO, otherwise __init__ may need to be updated to set a correct source (used, e.g., to define links).
Update open() and close() as necessary.
Depending on the type of data store, it may also be necessary to update the handling of links and references in ZarrIO. In principle, reading and writing of links should not need to change, however, in particular the __resolve_ref() and get_builder_exists_on_disk() method may need to be updated to ensure references are opened correctly on read for files stored with your new store. The __get_ref() function may also need to be updated, in particular in case the links to your store also modify the storage schema for links (e.g., if you need to store additional metadata in order to resolve links to your store).

Updating NWBZarrIO¶

In most cases we should not need to update NWBZarrIO as it inherits directly from ZarrIO. However, in particular if the interface for __init__ has changed for ZarrIO, then we may also need to modify NWBZarrIO accordingly.

Updating Unit Tests¶

Much of the core test harness of hdmf_zarr is modularized to simplify running existing tests with new storage backends. In this way, we can quickly create a collection of common tests for new backends, and new test cases added to the test suite can be run with all backends. The relevant test class are located in the /tests/unit directory of the hdmf_zarr repository.

test_zarrio.py¶

base_tests_zarrio.py provides a collection of base classes that define common test cases to test basic functionality of ZarrIO. Using these base classes, the test_zarrio.py module then implements concrete tests for various backends. To create tests for a new data store, we need to add the following main classes (while <MyStore> in the code below would need to be replaced with the class name of the new data store):

Create tests for new data store: Add the following main classes (while <MyStore> in the code below would need to be replaces with the class name of the new data store):

#########################################
#  <MyStore> tests
#########################################
class TestZarrWriter<MyStore>(BaseTestZarrWriter):
    """Test writing of builder with Zarr using a custom <MyStore>"""
    def setUp(self):
        super().setUp()
        self.store = <MyStore>()
        self.store_path = self.store.path


class TestZarrWriteUnit<MyStore>(BaseTestZarrWriteUnit):
    """Unit test for individual write functions using a custom <MyStore>"""
    def setUp(self):
        super().setUp()
        self.store = <MyStore>()
        self.store_path = self.store.path


class TestExportZarrToZarr<MyStore>(BaseTestExportZarrToZarr):
    """Test exporting Zarr to Zarr using <MyStore>."""
    def setUp(self):
        super().setUp()
        self.stores = [<MyStore>() for i in range(len(self.store_path))]
        self.store_paths = [s.path for s in self.stores]

Update ``base_tests_zarrio.reopen_store`` If our new data store cannot be reused after it has been closed via close(), then update the method to either reopen or create a new equivalent data store that can be used for read. The function is used in tests that write data, then close the ZarrIO, and create a new ZarrIO to read and validate the data.
Run and update tests Depending on your data store, some test cases in BaseTestZarrWriter, BaseTestZarrWriteUnit or BaseTestExportZarrToZarr may need to be updated to correctly work with our data store. Simply run the test suite to see if any cases are failing to see whether the setUp in your test classes or any specific test cases may need to be updated.

test_io_convert.py¶

test_io_convert.py uses a collection of mixin classes to define custom test classes to test export from one IO backend to another. As such, the test cases here typically first write to one target and then export to another target and then compare that the data between the two files is consistent.

Update ``MixinTestHDF5ToZarr``, ``MixinTestZarrToZarr``, and ``MixinTestZarrToZarr`` mixin classes to add the new backend to the WRITE_PATHS (if Zarr is the initial write target) and/or EXPORT_PATHS (if Zarr is the export target) variables to define our store as a write or export store for ZarrIO, respectively. Once we have added our new store as write/export targets to these mixins, all test cases defined in the module will be run with our new backend. Specifically, we here commonly need to add an instance of our new data store to:
- MixinTestHDF5ToZarr.EXPORT_PATHS
- MixinTestZarrToHDF5.WRITE_PATHS
- MixinTestZarrToZarr.WRITE_PATHS and MixinTestZarrToZarr.EXPORT_PATHS
Update tests and ZarrIO as necessary Run the test suite and fix any identified issues.