.. DO NOT EDIT.
.. THIS FILE WAS AUTOMATICALLY GENERATED BY SPHINX-GALLERY.
.. TO MAKE CHANGES, EDIT THE SOURCE PYTHON FILE:
.. "tutorials/general/plot_external_resources.py"
.. LINE NUMBERS ARE GIVEN BELOW.

.. only:: html

    .. note::
        :class: sphx-glr-download-link-note

        :ref:`Go to the end <sphx_glr_download_tutorials_general_plot_external_resources.py>`
        to download the full example code.

.. rst-class:: sphx-glr-example-title

.. _sphx_glr_tutorials_general_plot_external_resources.py:


.. _external_resources:

Linking to External Resources (HERD)
====================================

The :py:class:`~pynwb.resources.HERD` (HDMF External Resources Data Structure) class lets you map
terms used in your data to entities defined in external, web-accessible resources such as
ontologies. For example, you may store a species name ``"Mus musculus"`` on a
:py:class:`~pynwb.file.Subject` and want to link it to the corresponding NCBI Taxonomy term so that
the value is standardized and easy to query.

From a user's perspective, a HERD can be treated as a single table that associates a ``key`` (a term
used on an ``object``, i.e. a dataset or attribute in the file) with an ``entity`` (a term in an
external resource, identified by a compact URI and a full URI). Internally, HERD stores this in six
interlinked tables (``keys``, ``files``, ``entities``, ``entity_keys``, ``objects``, and
``object_keys``) and provides convenience methods so you rarely need to interact with those tables
directly.

This tutorial shows how to create a HERD, annotate objects in an NWB file, store the HERD in the
file, and inspect the annotations after reading the file back. For the full HERD API (including
``add_ref_termset`` for validating terms against a :py:class:`~hdmf.term_set.TermSet`, ``get_key``,
and compound-data references), see the
`HDMF HERD tutorial <https://hdmf.readthedocs.io/en/stable/tutorials/plot_external_resources.html>`_.

.. GENERATED FROM PYTHON SOURCE LINES 26-37

.. code-block:: Python


    import os
    from datetime import datetime
    from uuid import uuid4

    from dateutil.tz import tzlocal

    from pynwb import NWBHDF5IO, NWBFile
    from pynwb.file import Subject
    from pynwb.resources import HERD








.. GENERATED FROM PYTHON SOURCE LINES 39-43

Create an NWB file
------------------
Start with an :py:class:`~pynwb.file.NWBFile` that has a :py:class:`~pynwb.file.Subject`. The
subject's species is the value we will annotate with an external resource.

.. GENERATED FROM PYTHON SOURCE LINES 43-51

.. code-block:: Python


    nwbfile = NWBFile(
        session_description="a demonstration of external resources",
        identifier=str(uuid4()),
        session_start_time=datetime(2018, 4, 25, 2, 30, 3, tzinfo=tzlocal()),
        subject=Subject(subject_id="001", species="Mus musculus"),
    )








.. GENERATED FROM PYTHON SOURCE LINES 52-56

Create a HERD and attach it to the file
---------------------------------------
Create a :py:class:`~pynwb.resources.HERD` and assign it to the ``external_resources`` field of the
:py:class:`~pynwb.file.NWBFile`.

.. GENERATED FROM PYTHON SOURCE LINES 56-59

.. code-block:: Python


    nwbfile.external_resources = HERD()








.. GENERATED FROM PYTHON SOURCE LINES 60-66

Add references with ``add_ref``
-------------------------------
Use :py:meth:`~hdmf.common.resources.HERD.add_ref` to add a row that links a key on an object to an
external entity. Here we link the subject's species to the NCBI Taxonomy entry for *Mus musculus*.
Because the subject is already part of the file, the ``file`` argument is resolved automatically
from the parent hierarchy and can be omitted.

.. GENERATED FROM PYTHON SOURCE LINES 66-74

.. code-block:: Python


    nwbfile.external_resources.add_ref(
        container=nwbfile.subject,
        key=nwbfile.subject.species,
        entity_id="NCBITAXON:10090",
        entity_uri="https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/wwwtax.cgi?mode=Info&id=10090",
    )








.. GENERATED FROM PYTHON SOURCE LINES 75-82

References can also point to an attribute of an object, such as a column of a table. Here we record
the brain region of a set of electrodes in the electrodes table and link the region to the
corresponding structure in the
`Allen Mouse Brain Common Coordinate Framework (CCFv3) <https://atlas.brain-map.org/>`_. When the
target is a column, pass the table as the ``container`` and the column name as the ``attribute``;
HERD resolves the reference to the column object itself, which is the closest object that has a
data type.

.. GENERATED FROM PYTHON SOURCE LINES 82-101

.. code-block:: Python


    device = nwbfile.create_device(name="probe")
    electrode_group = nwbfile.create_electrode_group(
        name="shank0",
        description="a shank of the recording probe",
        location="VISp",
        device=device,
    )
    for _ in range(4):
        nwbfile.add_electrode(location="VISp", group=electrode_group)

    nwbfile.external_resources.add_ref(
        container=nwbfile.electrodes,
        attribute="location",
        key="VISp",
        entity_id="385",
        entity_uri="https://api.brain-map.org/api/v2/data/Structure/385.json",
    )








.. GENERATED FROM PYTHON SOURCE LINES 102-106

Inspect the HERD
----------------
:py:meth:`~hdmf.common.resources.HERD.to_dataframe` flattens the interlinked tables into a single
:py:class:`~pandas.DataFrame`, with one row per (object, key, entity) association.

.. GENERATED FROM PYTHON SOURCE LINES 106-109

.. code-block:: Python


    nwbfile.external_resources.to_dataframe()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>file_object_id</th>
          <th>objects_idx</th>
          <th>object_id</th>
          <th>files_idx</th>
          <th>object_type</th>
          <th>relative_path</th>
          <th>field</th>
          <th>keys_idx</th>
          <th>key</th>
          <th>entities_idx</th>
          <th>entity_id</th>
          <th>entity_uri</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>812478a8-2742-4500-af99-bd4424509489</td>
          <td>0</td>
          <td>113f704e-527a-4b8b-9aef-edef149d86a4</td>
          <td>0</td>
          <td>Subject</td>
          <td></td>
          <td></td>
          <td>0</td>
          <td>Mus musculus</td>
          <td>0</td>
          <td>NCBITAXON:10090</td>
          <td>https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/...</td>
        </tr>
        <tr>
          <th>1</th>
          <td>812478a8-2742-4500-af99-bd4424509489</td>
          <td>1</td>
          <td>1eb5c72b-bd0d-4821-942a-e58d72185091</td>
          <td>0</td>
          <td>VectorData</td>
          <td></td>
          <td></td>
          <td>1</td>
          <td>VISp</td>
          <td>1</td>
          <td>385</td>
          <td>https://api.brain-map.org/api/v2/data/Structur...</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 110-112

You can also view the individual tables. Each is a
:py:class:`~hdmf.common.table.DynamicTable` and has its own ``to_dataframe`` method.

.. GENERATED FROM PYTHON SOURCE LINES 112-115

.. code-block:: Python


    nwbfile.external_resources.keys.to_dataframe()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>key</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>Mus musculus</td>
        </tr>
        <tr>
          <th>1</th>
          <td>VISp</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 116-119

.. code-block:: Python


    nwbfile.external_resources.entities.to_dataframe()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>entity_id</th>
          <th>entity_uri</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>NCBITAXON:10090</td>
          <td>https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/...</td>
        </tr>
        <tr>
          <th>1</th>
          <td>385</td>
          <td>https://api.brain-map.org/api/v2/data/Structur...</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 120-122

:py:meth:`~hdmf.common.resources.HERD.get_object_type` returns all annotations for objects of a
given type, for example every annotated :py:class:`~pynwb.file.Subject`.

.. GENERATED FROM PYTHON SOURCE LINES 122-125

.. code-block:: Python


    nwbfile.external_resources.get_object_type(object_type="Subject")






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>file_object_id</th>
          <th>objects_idx</th>
          <th>object_id</th>
          <th>files_idx</th>
          <th>object_type</th>
          <th>relative_path</th>
          <th>field</th>
          <th>keys_idx</th>
          <th>key</th>
          <th>entities_idx</th>
          <th>entity_id</th>
          <th>entity_uri</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>812478a8-2742-4500-af99-bd4424509489</td>
          <td>0</td>
          <td>113f704e-527a-4b8b-9aef-edef149d86a4</td>
          <td>0</td>
          <td>Subject</td>
          <td></td>
          <td></td>
          <td>0</td>
          <td>Mus musculus</td>
          <td>0</td>
          <td>NCBITAXON:10090</td>
          <td>https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/...</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 126-130

Write and read the NWB file
---------------------------
Writing the file stores the HERD inside it. Reading the file back makes the HERD available again
through the ``external_resources`` field.

.. GENERATED FROM PYTHON SOURCE LINES 130-139

.. code-block:: Python


    filename = "external_resources_tutorial.nwb"
    with NWBHDF5IO(filename, mode="w") as io:
        io.write(nwbfile)

    read_io = NWBHDF5IO(filename, mode="r")
    read_nwbfile = read_io.read()
    read_herd = read_nwbfile.external_resources








.. GENERATED FROM PYTHON SOURCE LINES 140-146

Access the loaded data
-----------------------
In a Jupyter notebook, the default display of a read HERD shows collapsible sections that can
appear empty. To see the annotations, use the same accessors as above:
:py:meth:`~hdmf.common.resources.HERD.to_dataframe` for the flattened view, or the individual
tables for a focused view.

.. GENERATED FROM PYTHON SOURCE LINES 146-149

.. code-block:: Python


    read_herd.to_dataframe()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>file_object_id</th>
          <th>objects_idx</th>
          <th>object_id</th>
          <th>files_idx</th>
          <th>object_type</th>
          <th>relative_path</th>
          <th>field</th>
          <th>keys_idx</th>
          <th>key</th>
          <th>entities_idx</th>
          <th>entity_id</th>
          <th>entity_uri</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>812478a8-2742-4500-af99-bd4424509489</td>
          <td>0</td>
          <td>113f704e-527a-4b8b-9aef-edef149d86a4</td>
          <td>0</td>
          <td>Subject</td>
          <td></td>
          <td></td>
          <td>0</td>
          <td>Mus musculus</td>
          <td>0</td>
          <td>NCBITAXON:10090</td>
          <td>https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/...</td>
        </tr>
        <tr>
          <th>1</th>
          <td>812478a8-2742-4500-af99-bd4424509489</td>
          <td>1</td>
          <td>1eb5c72b-bd0d-4821-942a-e58d72185091</td>
          <td>0</td>
          <td>VectorData</td>
          <td></td>
          <td></td>
          <td>1</td>
          <td>VISp</td>
          <td>1</td>
          <td>385</td>
          <td>https://api.brain-map.org/api/v2/data/Structur...</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 150-151

View the individual tables:

.. GENERATED FROM PYTHON SOURCE LINES 151-154

.. code-block:: Python


    read_herd.keys.to_dataframe()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>key</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>Mus musculus</td>
        </tr>
        <tr>
          <th>1</th>
          <td>VISp</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 155-158

.. code-block:: Python


    read_herd.entities.to_dataframe()






.. raw:: html

    <div class="output_subarea output_html rendered_html output_result">
    <div>
    <style scoped>
        .dataframe tbody tr th:only-of-type {
            vertical-align: middle;
        }

        .dataframe tbody tr th {
            vertical-align: top;
        }

        .dataframe thead th {
            text-align: right;
        }
    </style>
    <table border="1" class="dataframe">
      <thead>
        <tr style="text-align: right;">
          <th></th>
          <th>entity_id</th>
          <th>entity_uri</th>
        </tr>
      </thead>
      <tbody>
        <tr>
          <th>0</th>
          <td>NCBITAXON:10090</td>
          <td>https://www.ncbi.nlm.nih.gov/Taxonomy/Browser/...</td>
        </tr>
        <tr>
          <th>1</th>
          <td>385</td>
          <td>https://api.brain-map.org/api/v2/data/Structur...</td>
        </tr>
      </tbody>
    </table>
    </div>
    </div>
    <br />
    <br />

.. GENERATED FROM PYTHON SOURCE LINES 159-164

:py:meth:`~hdmf.common.resources.HERD.get_object_entities` returns the entities annotated on a
single object as a :py:class:`~pandas.DataFrame`. On a HERD read back from a file this accessor
currently requires the fix for
`hdmf #1496 <https://github.com/hdmf-dev/hdmf/issues/1496>`_, which will be resolved soon, so it is
shown here commented out:

.. GENERATED FROM PYTHON SOURCE LINES 164-167

.. code-block:: Python


    # read_herd.get_object_entities(container=read_nwbfile.subject)








.. GENERATED FROM PYTHON SOURCE LINES 168-169

Close the file once you are done reading from it.

.. GENERATED FROM PYTHON SOURCE LINES 169-172

.. code-block:: Python


    read_io.close()








.. GENERATED FROM PYTHON SOURCE LINES 173-181

Alternative: store a HERD outside an NWB file
---------------------------------------------
A HERD can also be saved independently of an NWB file as a zip archive of the underlying tables
using :py:meth:`~hdmf.common.resources.HERD.to_zip`, and read back with
:py:meth:`~hdmf.common.resources.HERD.from_zip`. This is useful when external resources span
multiple files; see :ref:`external_resources_streaming` for an example that annotates many NWB
files with a single HERD. For the full HERD API, see the
`HDMF HERD tutorial <https://hdmf.readthedocs.io/en/stable/tutorials/plot_external_resources.html>`_.

.. GENERATED FROM PYTHON SOURCE LINES 181-183

.. code-block:: Python


    os.remove(filename)








.. _sphx_glr_download_tutorials_general_plot_external_resources.py:

.. only:: html

  .. container:: sphx-glr-footer sphx-glr-footer-example

    .. container:: sphx-glr-download sphx-glr-download-jupyter

      :download:`Download Jupyter notebook: plot_external_resources.ipynb <plot_external_resources.ipynb>`

    .. container:: sphx-glr-download sphx-glr-download-python

      :download:`Download Python source code: plot_external_resources.py <plot_external_resources.py>`

    .. container:: sphx-glr-download sphx-glr-download-zip

      :download:`Download zipped: plot_external_resources.zip <plot_external_resources.zip>`


.. only:: html

 .. rst-class:: sphx-glr-signature

    `Gallery generated by Sphinx-Gallery <https://sphinx-gallery.github.io>`_