.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "tutorials/plot_zarr_dataset_io.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_tutorials_plot_zarr_dataset_io.py: Zarr Dataset I/O ================ To customize data write settings on a per-dataset basis, HDMF supports wrapping of data arrays using :py:class:`~hdmf.data_utils.DataIO`. To support defining settings specific to Zarr ``hdmf-zarr`` provides the corresponding :py:class:`~hdmf_zarr.utils.ZarrDataIO` class. Create an example DynamicTable Container ---------------------------------------- As a simple example, we first create a ``DynamicTable`` container to store some arbitrary data columns. .. GENERATED FROM PYTHON SOURCE LINES 16-36 .. code-block:: Python # Import DynamicTable and get the ROOT_NAME from hdmf.common.table import DynamicTable, VectorData from hdmf_zarr.backend import ROOT_NAME from hdmf_zarr import ZarrDataIO import numpy as np # Setup a DynamicTable for managing data about users data = np.arange(50).reshape(10, 5) column = VectorData( name='test_data_default_settings', description='Some 2D test data', data=data) test_table = DynamicTable( name=ROOT_NAME, description='a table containing data/metadata about users, one user per row', columns=(column, ), colnames=(column.name, ) ) .. GENERATED FROM PYTHON SOURCE LINES 38-43 Defining Data I/O settings -------------------------- To define custom settings for write (e.g., for chunking and compression) we simply wrap our data array using :py:class:`~hdmf_zarr.utils.ZarrDataIO`. .. GENERATED FROM PYTHON SOURCE LINES 43-53 .. code-block:: Python from numcodecs import Blosc data_with_data_io = ZarrDataIO( data=data * 3, chunks=(10, 10), fillvalue=0, compressor=Blosc(cname='zstd', clevel=1, shuffle=Blosc.SHUFFLE) ) .. GENERATED FROM PYTHON SOURCE LINES 54-55 Adding the data to our table .. GENERATED FROM PYTHON SOURCE LINES 55-61 .. code-block:: Python test_table.add_column( name='test_data_zstd_compression', description='Some 2D test data', data=data_with_data_io) .. GENERATED FROM PYTHON SOURCE LINES 62-63 Next we add a column where we explicitly disable compression .. GENERATED FROM PYTHON SOURCE LINES 63-71 .. code-block:: Python data_without_compression = ZarrDataIO( data=data*5, compressor=False) test_table.add_column( name='test_data_nocompression', description='Some 2D test data', data=data_without_compression) .. GENERATED FROM PYTHON SOURCE LINES 72-80 .. note:: To control linking to other datasets see the ``link_data`` parameter of :py:class:`~hdmf_zarr.utils.ZarrDataIO` .. note:: In the case of :py:class:`~hdmf.container.Data` (or here :py:class:`~hdmf.common.table.VectorData`) we can also set the ``DataIO`` object to use via the :py:meth:`~hdmf.container.Data.set_dataio` function. .. GENERATED FROM PYTHON SOURCE LINES 83-87 Writing and Reading ------------------- Reading and writing data with filters works as usual. See the :ref:`zarrio_tutorial` tutorial for details. .. GENERATED FROM PYTHON SOURCE LINES 87-95 .. code-block:: Python from hdmf.common import get_manager from hdmf_zarr.backend import ZarrIO zarr_dir = "example_data.zarr" with ZarrIO(path=zarr_dir, manager=get_manager(), mode='w') as zarr_io: zarr_io.write(test_table) .. GENERATED FROM PYTHON SOURCE LINES 96-97 reading the table from Zarr .. GENERATED FROM PYTHON SOURCE LINES 97-102 .. code-block:: Python zarr_io = ZarrIO(path=zarr_dir, manager=get_manager(), mode='r') intable = zarr_io.read() intable.to_dataframe() .. raw:: html
test_data_default_settings test_data_zstd_compression test_data_nocompression
id
0 [0, 1, 2, 3, 4] [0, 3, 6, 9, 12] [0, 5, 10, 15, 20]
1 [5, 6, 7, 8, 9] [15, 18, 21, 24, 27] [25, 30, 35, 40, 45]
2 [10, 11, 12, 13, 14] [30, 33, 36, 39, 42] [50, 55, 60, 65, 70]
3 [15, 16, 17, 18, 19] [45, 48, 51, 54, 57] [75, 80, 85, 90, 95]
4 [20, 21, 22, 23, 24] [60, 63, 66, 69, 72] [100, 105, 110, 115, 120]
5 [25, 26, 27, 28, 29] [75, 78, 81, 84, 87] [125, 130, 135, 140, 145]
6 [30, 31, 32, 33, 34] [90, 93, 96, 99, 102] [150, 155, 160, 165, 170]
7 [35, 36, 37, 38, 39] [105, 108, 111, 114, 117] [175, 180, 185, 190, 195]
8 [40, 41, 42, 43, 44] [120, 123, 126, 129, 132] [200, 205, 210, 215, 220]
9 [45, 46, 47, 48, 49] [135, 138, 141, 144, 147] [225, 230, 235, 240, 245]


.. GENERATED FROM PYTHON SOURCE LINES 103-105 Check dataset settings used. .. GENERATED FROM PYTHON SOURCE LINES 105-111 .. code-block:: Python for c in intable.columns: print("Name=%s, Chunks=% s, Compressor=%s" % (c.name, str(c.data.chunks), str(c.data.compressor))) .. rst-class:: sphx-glr-script-out .. code-block:: none Name=test_data_default_settings, Chunks=(10, 5), Compressor=Blosc(cname='lz4', clevel=5, shuffle=SHUFFLE, blocksize=0) Name=test_data_zstd_compression, Chunks=(10, 10), Compressor=Blosc(cname='zstd', clevel=1, shuffle=SHUFFLE, blocksize=0) Name=test_data_nocompression, Chunks=(10, 5), Compressor=None .. GENERATED FROM PYTHON SOURCE LINES 113-114 .. code-block:: Python zarr_io.close() .. _sphx_glr_download_tutorials_plot_zarr_dataset_io.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_zarr_dataset_io.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_zarr_dataset_io.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_zarr_dataset_io.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_