NAME

       datalad aggregate-metadata - aggregate metadata of one or more datasets for later query.

SYNOPSIS

       datalad   aggregate-metadata   [-h]   [-d   DATASET]   [-r]   [-R  LEVELS]  [--update-mode  {all|target}]
              [--incremental] [--force-extraction] [--nosave] [--version] [PATH ...]

DESCRIPTION

       Metadata aggregation refers to a procedure that extracts metadata present in a dataset  into  a  portable
       representation  that  is stored a single standardized format. Moreover, metadata aggregation can also ex‐
       tract metadata in this format from one dataset and store it in another (super)dataset. Based on such col‐
       lections of aggregated metadata it is possible to discover particular  datasets  and  specific  parts  of
       their content, without having to obtain the target datasets first (see the DataLad 'search' command).

       To enable aggregation of metadata that are contained in files of a dataset, one has to enable one or more
       metadata extractor for a dataset. DataLad supports a number of common metadata standards, such as the Ex‐
       changeable  Image  File Format (EXIF), Adobe's Extensible Metadata Platform (XMP), and various audio file
       metadata systems like ID3. DataLad extension packages can provide metadata data extractors for additional
       metadata sources. For example, the neuroimaging extension provides extractors for  scientific  (meta)data
       standards  like  BIDS,  DICOM, and NIfTI1.  Some metadata extractors depend on particular 3rd-party soft‐
       ware. The list of metadata extractors available to a particular DataLad installation is reported  by  the
       'wtf' command ('datalad wtf').

       Enabling  a  metadata  extractor  for a dataset is done by adding its name to the dataset's configuration
       file (.datalad/config), e.g.::

       [datalad "metadata"]
         nativetype = exif
         nativetype = xmp

       If an enabled metadata extractor is not available in a particular DataLad installation, metadata  extrac‐
       tion will not succeed in order to avoid inconsistent aggregation results.

       Enabling  multiple  extractors is supported. In this case, metadata are extracted by each extractor indi‐
       vidually, and stored alongside each other.  Metadata aggregation will also extract DataLad's own metadata
       (extractors

       Metadata aggregation can be performed recursively, in order to aggregate all  metadata  across  all  sub‐
       datasets,  for  example, to be able to search across any content in any dataset of a collection. Aggrega‐
       tion can also be performed for subdatasets that are not available locally. In this  case,  pre-aggregated
       metadata from the closest available superdataset will be considered instead.

       Depending on the versatility of the present metadata and the number of dataset or files, aggregated meta‐
       data  can  grow prohibitively large. A number of configuration switches are provided to mitigate such is‐
       sues.

       datalad.metadata.aggregate-content-<extractor-name> If set to false, content  metadata  aggregation  will
       not  be performed for the named metadata extractor (a potential underscore '_' in the extractor name must
       be replaced by a dash '-'). This can substantially reduce the runtime for metadata extraction,  and  also
       reduce  the size of the generated metadata aggregate. Note, however, that some extractors may not produce
       any metadata when this is disabled, because their metadata might come from individual file headers  only.
       'datalad.metadata.store-aggregate-content' might be a more appropriate setting in such cases.

       datalad.metadata.aggregate-ignore-fields Any metadata key matching any regular expression in this config‐
       uration  setting is removed prior to generating the dataset-level metadata summary (keys and their unique
       values across all dataset content), and from the dataset metadata itself. This switch can also be used to
       filter out sensitive information prior aggregation.

       datalad.metadata.generate-unique-<extractor-name> If set to false, DataLad will not auto-generate a  sum‐
       mary  of unique content metadata values for a particular extractor as part of the dataset-global metadata
       (a potential underscore '_' in the extractor name must be replaced by a dash '-').  This can be useful if
       such a summary is bloated due to minor uninformative (e.g.  numerical) differences, or when a  particular
       extractor already provides a carefully designed content metadata summary.

       datalad.metadata.maxfieldsize Any metadata value that exceeds the size threshold given by this configura‐
       tion setting (in bytes/characters) is removed.

       datalad.metadata.store-aggregate-content  If set, extracted content metadata are still used to generate a
       dataset-level summary of present metadata (all keys and their unique values across all files in a dataset
       are determined and stored as part of the dataset-level metadata  aggregate,  see  datalad.metadata.gener‐
       ate-unique-<extractor-name>),  but  metadata on individual files are not stored.  This switch can be used
       to avoid prohibitively large metadata files. Discovery of datasets containing content matching particular
       metadata properties will still be possible, but such datasets would have to be obtained first in order to
       discover which particular files in them match these properties.

OPTIONS

       PATH   path to datasets that shall be aggregated. When a given path is pointing into a dataset, the meta‐
              data of the containing dataset will be aggregated. If no paths given, current dataset metadata  is
              aggregated. Constraints: value must be a string

       -h, --help, --help-np
              show  this  help message. --help-np forcefully disables the use of a pager for displaying the help
              message

       -d DATASET, --dataset DATASET
              topmost dataset metadata will be aggregated into. All dataset between this dataset and  any  given
              path  will  receive updated aggregated metadata from all given paths. Constraints: Value must be a
              Dataset or a valid identifier of a Dataset (e.g. a path)

       -r, --recursive
              if set, recurse into potential subdataset.

       -R LEVELS, --recursion-limit LEVELS
              limit recursion into subdataset to the given number of levels. Constraints: value must be convert‐
              ible to type 'int'

       --update-mode {all|target}
              which datasets to update with newly aggregated metadata: all datasets from any leaf dataset to the
              top-level target dataset including all intermediate datasets (all), or just the  top-level  target
              dataset (target). Constraints: value must be one of ('all', 'target') [Default: 'target']

       --incremental
              If  set,  all information on metadata records of subdatasets that have not been (re-)aggregated in
              this run will be kept unchanged. This is useful when (re-)aggregation only a subset of  a  dataset
              hierarchy, for example, because not all subdatasets are locally available.

       --force-extraction
              If  set,  all  enabled extractors will be engaged regardless of whether change detection indicates
              that metadata has already been extracted for a given dataset state.

       --nosave
              by default all modifications to a dataset are immediately saved. Giving this option  will  disable
              this behavior.

       --version
              show the module and its version which provides the command

AUTHORS

        datalad is developed by The DataLad Team and Contributors <team@datalad.org>.

datalad aggregate-metadata 0.15.5                  2022-02-10                      datalad aggregate-metadata(1)