.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "tutorials/plot_zarr_io.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_tutorials_plot_zarr_io.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_tutorials_plot_zarr_io.py:


.. _zarrio_tutorial:

ZarrIO Overview
===============

The :py:class:`~hdmf_zarr.backend.ZarrIO` backend behaves in general much like the
standard :py:class:`~hdmf.backends.hdf5.HDF5IO` available with HDMF and is an
adaptation of that backend to use Zarr instead of HDF5

Create an example DynamicTable Container
----------------------------------------

As a simple example, we here create a basic :py:class:`~hdmf.common.table.DynamicTable` for
describing basic user data.

.. note::

   When writing a :py:class:`~hdmf.common.table.DynamicTable` (or any Container that is
   normally not intended to be the root of a file) we need to use :py:attr:`hdmf_zarr.backend.ROOT_NAME`
   as the name for the Container to ensure that link paths are created correctly by
   :py:class:`~hdmf_zarr.backend.ZarrIO`. This is due to the fact that the top-level Container
   used during I/O is written as the root of the file. As such, the name of the root Container
   of a file does not appear in the path to locate it.

.. GENERATED FROM PYTHON SOURCE LINES 26-67

.. code-block:: Python


    # Import DynamicTable and get the ROOT_NAME
    from hdmf.common.table import DynamicTable
    from hdmf_zarr.backend import ROOT_NAME

    # Setup a DynamicTable for managing data about users
    users_table = DynamicTable(
        name=ROOT_NAME,
        description='a table containing data/metadata about users, one user per row',
    )
    users_table.add_column(
        name='first_name',
        description='the first name of the user',
    )
    users_table.add_column(
        name='last_name',
        description='the last name of the user',
    )

    users_table.add_column(
        name='phone_number',
        description='the phone number of the user',
        index=True,
    )

    # Add some simple example data to our table
    users_table.add_row(
        first_name='Grace',
        last_name='Hopper',
        phone_number=['123-456-7890']
    )
    users_table.add_row(
        first_name='Alan',
        last_name='Turing',
        phone_number=['555-666-7777', '888-111-2222']
    )

    # Show the table for validation
    users_table.to_dataframe()







.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>first_name</th>
          <th>last_name</th>
          <th>phone_number</th>
        </tr>
        <tr>
          <th>id</th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>Grace</td>
          <td>Hopper</td>
          <td>[123-456-7890]</td>
        </tr>
        <tr>
          <th>1</th>
          <td>Alan</td>
          <td>Turing</td>
          <td>[555-666-7777, 888-111-2222]</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 69-71

Writing the table to Zarr
---------------------------------

.. GENERATED FROM PYTHON SOURCE LINES 71-79

.. code-block:: Python


    from hdmf.common import get_manager
    from hdmf_zarr.backend import ZarrIO

    zarr_dir = "example.zarr"
    with ZarrIO(path=zarr_dir,  manager=get_manager(), mode='w') as zarr_io:
        zarr_io.write(users_table)








.. GENERATED FROM PYTHON SOURCE LINES 80-82

Reading the table from Zarr
----------------------------------

.. GENERATED FROM PYTHON SOURCE LINES 82-86

.. code-block:: Python

    zarr_io = ZarrIO(path=zarr_dir,  manager=get_manager(), mode='r')
    intable = zarr_io.read()
    intable.to_dataframe()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>first_name</th>
          <th>last_name</th>
          <th>phone_number</th>
        </tr>
        <tr>
          <th>id</th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>Grace</td>
          <td>Hopper</td>
          <td>[123-456-7890]</td>
        </tr>
        <tr>
          <th>1</th>
          <td>Alan</td>
          <td>Turing</td>
          <td>[555-666-7777, 888-111-2222]</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 88-91

.. code-block:: Python

    zarr_io.close()









.. GENERATED FROM PYTHON SOURCE LINES 92-101

Converting to/from HDF5 using ``export``
----------------------------------------

Exporting the Zarr file to HDF5
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

To convert our Zarr file to HDF5 we can now simply read the file with our
:py:class:`~hdmf_zarr.backend.ZarrIO` backend and the export the file
using HDMF's :py:class:`~hdmf.backends.hdf5.HDF5IO` backend

.. GENERATED FROM PYTHON SOURCE LINES 101-108

.. code-block:: Python


    from hdmf.backends.hdf5 import HDF5IO

    with ZarrIO(path=zarr_dir,  manager=get_manager(), mode='r') as zarr_read_io:
        with HDF5IO(path="example.h5", manager=get_manager(), mode='w') as hdf5_export_io:
            hdf5_export_io.export(src_io=zarr_read_io, write_args=dict(link_data=False))  # use export!








.. GENERATED FROM PYTHON SOURCE LINES 109-117

.. note::

    When converting between backends we need to set ``link_data=False`` as linking
    between different storage backends (here from HDF5 to Zarr and vice versa) is
    not supported.

Check that the HDF5 file is correct


.. GENERATED FROM PYTHON SOURCE LINES 117-122

.. code-block:: Python

    with HDF5IO(path="example.h5", manager=get_manager(), mode='r') as hdf5_read_io:
        intable_from_hdf5 = hdf5_read_io.read()
        intable_hdf5_df = intable_from_hdf5.to_dataframe()
    intable_hdf5_df  # display the table in the gallery output






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>first_name</th>
          <th>last_name</th>
          <th>phone_number</th>
        </tr>
        <tr>
          <th>id</th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>Grace</td>
          <td>Hopper</td>
          <td>[123-456-7890]</td>
        </tr>
        <tr>
          <th>1</th>
          <td>Alan</td>
          <td>Turing</td>
          <td>[555-666-7777, 888-111-2222]</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 123-129

Exporting the HDF5 file to Zarr
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

In the same way as above, we can now also convert our HDF5 file back to Zarr
simply by reading the HDF5 file using HDMF's :py:class:`~hdmf.backends.hdf5.HDF5IO` backend
and the exporting the file using the :py:class:`~hdmf_zarr.backend.ZarrIO` backend.

.. GENERATED FROM PYTHON SOURCE LINES 129-134

.. code-block:: Python


    with HDF5IO(path="example.h5", manager=get_manager(), mode='r') as hdf5_read_io:
        with ZarrIO(path="example_exp.zarr",  manager=get_manager(), mode='w') as zarr_export_io:
            zarr_export_io.export(src_io=hdf5_read_io, write_args=dict(link_data=False))  # use export!








.. GENERATED FROM PYTHON SOURCE LINES 135-137

Check that the Zarr file is correct


.. GENERATED FROM PYTHON SOURCE LINES 137-143

.. code-block:: Python

    with ZarrIO(path="example_exp.zarr", manager=get_manager(), mode='r') as zarr_read_io:
        intable_from_zarr = zarr_read_io.read()
        intable_zarr_df = intable_from_zarr.to_dataframe()
    intable_zarr_df  # display the table in the gallery output







.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>first_name</th>
          <th>last_name</th>
          <th>phone_number</th>
        </tr>
        <tr>
          <th>id</th>
          <th></th>
          <th></th>
          <th></th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>Grace</td>
          <td>Hopper</td>
          <td>[123-456-7890]</td>
        </tr>
        <tr>
          <th>1</th>
          <td>Alan</td>
          <td>Turing</td>
          <td>[555-666-7777, 888-111-2222]</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 144-164

Using custom Zarr storage backends
-----------------------------------

:py:class:`~hdmf_zarr.backend.ZarrIO` supports a subset of data stores available
for Zarr, e.g., :py:class`~zarr.storage.DirectoryStore`, :py:class`~zarr.storage.TempStore`,
and :py:class`~zarr.storage.NestedDirectoryStore`. The supported stores are defined
in :py:attr:`~hdmf_zarr.backend.SUPPORTED_ZARR_STORES`. A main limitation to supporting
all possible Zarr stores in :py:class:`~hdmf_zarr.backend.ZarrIO` is due to the fact that
Zarr does not support links and references.

.. note:

    See :ref:`sec-integrating-zarr-data-store` for details on how to integrate
    new stores with :py:class:`~hdmf_zarr.backend.ZarrIO`.

To use a store other than the default, we simply need to instantiate the store
and set pass it to :py:class:`~hdmf_zarr.backend.ZarrIO` via the ``path`` parameter.
Here we use a :py:class`~zarr.storage.NestedDirectoryStore` to write a simple
:py:class:`hdmf.common.CSRMatrix` container to disk.


.. GENERATED FROM PYTHON SOURCE LINES 164-186

.. code-block:: Python


    from zarr.storage import NestedDirectoryStore
    from hdmf.common import CSRMatrix

    zarr_nsd_dir = "example_nested_store.zarr"
    store = NestedDirectoryStore(zarr_dir)
    csr_container = CSRMatrix(
        name=ROOT_NAME,
        data=[1, 2, 3, 4, 5, 6],
        indices=[0, 2, 2, 0, 1, 2],
        indptr=[0, 2, 3, 6],
        shape=(3, 3))

    # Write the csr_container to Zarr using a NestedDirectoryStore
    with ZarrIO(path=zarr_nsd_dir,  manager=get_manager(), mode='w') as zarr_io:
        zarr_io.write(csr_container)

    # Read the CSR matrix to confirm the data was written correctly
    with ZarrIO(path=zarr_nsd_dir, manager=get_manager(), mode='r') as zarr_io:
        csr_read = zarr_io.read()
        print(" data=%s\n indices=%s\n indptr=%s\n shape=%s" %
              (str(csr_read.data), str(csr_read.indices), str(csr_read.indptr), str(csr_read.shape)))




.. rst-class:: sphx-glr-script-out

 .. code-block:: none

    /home/docs/checkouts/readthedocs.org/user_builds/hdmf-zarr/checkouts/0.13.0/docs/gallery/plot_zarr_io.py:169: FutureWarning: The NestedDirectoryStore is deprecated and will be removed in a Zarr-Python version 3, see https://github.com/zarr-developers/zarr-python/issues/1274 for more information.
      store = NestedDirectoryStore(zarr_dir)
     data=[1 2 3 4 5 6]
     indices=[0 2 2 0 1 2]
     indptr=[0 2 3 6]
     shape=[3 3]





.. _sphx_glr_download_tutorials_plot_zarr_io.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_zarr_io.ipynb <plot_zarr_io.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_zarr_io.py <plot_zarr_io.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_zarr_io.zip <plot_zarr_io.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_