.. DO NOT EDIT. .. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY. .. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE: .. "tutorials/plot_convert_nwb_hdf5.py" .. LINE NUMBERS ARE GIVEN BELOW. .. only:: html .. note:: :class: sphx-glr-download-link-note :ref:`Go to the end ` to download the full example code. .. rst-class:: sphx-glr-example-title .. _sphx_glr_tutorials_plot_convert_nwb_hdf5.py: Converting NWB HDF5 files to/from Zarr ====================================== This tutorial illustrates how to convert data between HDF5 and Zarr using a Neurodata Without Borders (NWB) file from the DANDI data archive as an example. In this tutorial we will convert our example file from HDF5 to Zarr and then back again to HDF5. The NWB standard is defined using :hdmf-docs:`HDMF <>` and uses the :py:class:`~ hdmf.backends.hdf5.h5tools.HDF5IO` HDF5 backend from HDMF for storage. .. GENERATED FROM PYTHON SOURCE LINES 13-36 Setup ----- Here we use a small NWB file from the DANDI neurophysiology data archive from `Dandiset 001333 `_ as an example. To download the file directly from DANDI we can use: .. code-block:: python :linenos: import os from dandi.dandiapi import DandiAPIClient dandiset_id = "001333" filepath = "sub-healthy-simulated-beta/sub-healthy-simulated-beta_ses-162_ecephys.nwb" # 220 KiB file with DandiAPIClient() as client: asset = client.get_dandiset(dandiset_id, 'draft').get_asset_by_path(filepath) s3_path = asset.get_content_url(follow_redirects=1, strip_query=True) filename = os.path.basename(asset.path) asset.download(filename) We here use a local copy of a small file from this Dandiset as an example: .. GENERATED FROM PYTHON SOURCE LINES 36-60 .. code-block:: Python import os import shutil from pynwb import NWBHDF5IO from hdmf_zarr.nwb import NWBZarrIO # Input file to convert basedir = "resources" filename = os.path.join(basedir, "sub-healthy-simulated-beta_ses-162_ecephys.nwb") # Zarr file to generate for converting from HDF5 to Zarr zarr_filename = "test_zarr_" + os.path.basename(filename) + ".zarr" # HDF5 file to generate for converting from Zarr to HDF5 hdf_filename = "test_hdf5_" + os.path.basename(filename) # Delete our converted HDF5 and Zarr file from previous runs of this notebook for fname in [zarr_filename, hdf_filename]: if os.path.exists(fname): print("Removing %s" % fname) if os.path.isfile(fname): # Remove a single file (here the HDF5 file) os.remove(fname) else: # Remove whole directory and subtree (here the Zarr file) shutil.rmtree(fname) .. GENERATED FROM PYTHON SOURCE LINES 62-68 Convert the NWB file from HDF5 to Zarr -------------------------------------- To convert files between storage backends, we use HDMF's :hdmf-docs:`export ` functionality. As this is an NWB file, we here use the :py:class:`pynwb.NWBHDF5IO` backend for reading the file from from HDF5 and use the :py:class:`~hdmf_zarr.nwb.NWBZarrIO` backend to export the file to Zarr. .. GENERATED FROM PYTHON SOURCE LINES 68-73 .. code-block:: Python with NWBHDF5IO(filename, 'r') as read_io: # Create HDF5 IO object for read with NWBZarrIO(zarr_filename, 'w') as export_io: # Create Zarr IO object for write export_io.export(src_io=read_io, write_args=dict(link_data=False)) # Export from HDF5 to Zarr .. GENERATED FROM PYTHON SOURCE LINES 74-81 .. note:: When converting between backends we need to set ``link_data=False`` as linking from Zarr to HDF5 (and vice-versa) is not supported. Read the Zarr file back in -------------------------- .. GENERATED FROM PYTHON SOURCE LINES 81-85 .. code-block:: Python zarr_io = NWBZarrIO(zarr_filename, 'r') nwb_zarr = zarr_io.read() .. GENERATED FROM PYTHON SOURCE LINES 86-87 The basic behavior of the :py:class:`~pynwb.file.NWBFile` object is the same. .. GENERATED FROM PYTHON SOURCE LINES 87-91 .. code-block:: Python # Print the NWBFile to illustrate that print(nwb_zarr) .. rst-class:: sphx-glr-script-out .. code-block:: none root pynwb.file.NWBFile at 0x132636080918464 Fields: devices: { NEURON_Simulator } electrode_groups: { shank0 , shank1 , shank2 , shank3 } electrodes: electrodes experiment_description: The PESD dataset is generated from a cortico-basal-ganglia network for a Parkinsonian computational model. The computational model of the cortico-basal-ganglia is originally presented by Fleming et al. in the article: 'Simulation of Closed-Loop Deep Brain Stimulation Control Schemes for Suppression of Pathological Beta Oscillations in Parkinson's Disease'. experimenter: ['Ananna Biswas'] file_create_date: [datetime.datetime(2025, 3, 27, 16, 53, 28, 55430, tzinfo=tzoffset(None, -14400))] identifier: 7a68ea11-865a-481a-a5fd-d91fe6def653 institution: Michigan Technological University keywords: lab: BrainX Lab processing: { ecephys } related_publications: ['https://arxiv.org/abs/2407.17756' 'DOI: 10.3389/fnins.2020.00166'] session_description: Parkinson's Electrophysiological Signal Dataset (PESD) Generated from Simulation session_start_time: 2025-03-27 16:53:27.990500-04:00 subject: subject pynwb.file.Subject at 0x132636080916336 Fields: age: P0D age__reference: birth description: This is a simulated dataset generated from a computational model. sex: U species: Homo sapiens subject_id: healthy-simulated-beta timestamps_reference_time: 2025-03-27 16:53:27.990500-04:00 .. GENERATED FROM PYTHON SOURCE LINES 92-94 The main difference is that datasets are now represented by Zarr arrays compared to h5py Datasets when reading from HDF5. .. GENERATED FROM PYTHON SOURCE LINES 94-97 .. code-block:: Python print(type(nwb_zarr.electrodes['label'].data)) .. rst-class:: sphx-glr-script-out .. code-block:: none .. GENERATED FROM PYTHON SOURCE LINES 98-100 For illustration purposes, we here show the NWB :pynwb-docs:`Electrodes ` table. .. GENERATED FROM PYTHON SOURCE LINES 100-104 .. code-block:: Python print(nwb_zarr.electrodes.to_dataframe()) zarr_io.close() .. rst-class:: sphx-glr-script-out .. code-block:: none location ... label id ... 0 Simulated Cortico-basal-ganglia network of brain ... shank0_elec0 1 Simulated Cortico-basal-ganglia network of brain ... shank0_elec1 2 Simulated Cortico-basal-ganglia network of brain ... shank0_elec2 3 Simulated Cortico-basal-ganglia network of brain ... shank1_elec0 4 Simulated Cortico-basal-ganglia network of brain ... shank1_elec1 5 Simulated Cortico-basal-ganglia network of brain ... shank1_elec2 6 Simulated Cortico-basal-ganglia network of brain ... shank2_elec0 7 Simulated Cortico-basal-ganglia network of brain ... shank2_elec1 8 Simulated Cortico-basal-ganglia network of brain ... shank2_elec2 9 Simulated Cortico-basal-ganglia network of brain ... shank3_elec0 10 Simulated Cortico-basal-ganglia network of brain ... shank3_elec1 11 Simulated Cortico-basal-ganglia network of brain ... shank3_elec2 [12 rows x 4 columns] .. GENERATED FROM PYTHON SOURCE LINES 105-109 Convert the Zarr file back to HDF5 ---------------------------------- Using the same approach as above, we can now convert our Zarr file back to HDF5. .. GENERATED FROM PYTHON SOURCE LINES 109-114 .. code-block:: Python with NWBZarrIO(zarr_filename, 'r') as read_io: # Create Zarr IO object for read with NWBHDF5IO(hdf_filename, 'w') as export_io: # Create HDF5 IO object for write export_io.export(src_io=read_io, write_args=dict(link_data=False)) # Export from Zarr to HDF5 .. GENERATED FROM PYTHON SOURCE LINES 115-120 Read the new HDF5 file back --------------------------- Now our file has been converted from HDF5 to Zarr and back again to HDF5. Here we check that we can still read that file. .. GENERATED FROM PYTHON SOURCE LINES 120-124 .. code-block:: Python with NWBHDF5IO(hdf_filename, 'r') as hdf5_io: nwb_hdf5 = hdf5_io.read() print(nwb_hdf5) .. rst-class:: sphx-glr-script-out .. code-block:: none root pynwb.file.NWBFile at 0x132636079836048 Fields: devices: { NEURON_Simulator } electrode_groups: { shank0 , shank1 , shank2 , shank3 } electrodes: electrodes experiment_description: The PESD dataset is generated from a cortico-basal-ganglia network for a Parkinsonian computational model. The computational model of the cortico-basal-ganglia is originally presented by Fleming et al. in the article: 'Simulation of Closed-Loop Deep Brain Stimulation Control Schemes for Suppression of Pathological Beta Oscillations in Parkinson's Disease'. experimenter: ['Ananna Biswas'] file_create_date: [datetime.datetime(2025, 3, 27, 16, 53, 28, 55430, tzinfo=tzoffset(None, -14400))] identifier: 7a68ea11-865a-481a-a5fd-d91fe6def653 institution: Michigan Technological University keywords: lab: BrainX Lab processing: { ecephys } related_publications: ['https://arxiv.org/abs/2407.17756' 'DOI: 10.3389/fnins.2020.00166'] session_description: Parkinson's Electrophysiological Signal Dataset (PESD) Generated from Simulation session_start_time: 2025-03-27 16:53:27.990500-04:00 subject: subject pynwb.file.Subject at 0x132636079838064 Fields: age: P0D age__reference: birth description: This is a simulated dataset generated from a computational model. sex: U species: Homo sapiens subject_id: healthy-simulated-beta timestamps_reference_time: 2025-03-27 16:53:27.990500-04:00 .. _sphx_glr_download_tutorials_plot_convert_nwb_hdf5.py: .. only:: html .. container:: sphx-glr-footer sphx-glr-footer-example .. container:: sphx-glr-download sphx-glr-download-jupyter :download:`Download Jupyter notebook: plot_convert_nwb_hdf5.ipynb ` .. container:: sphx-glr-download sphx-glr-download-python :download:`Download Python source code: plot_convert_nwb_hdf5.py ` .. container:: sphx-glr-download sphx-glr-download-zip :download:`Download zipped: plot_convert_nwb_hdf5.zip ` .. only:: html .. rst-class:: sphx-glr-signature `Gallery generated by Sphinx-Gallery `_