Working with overlays

Overlays provide a means to store and access per item metadata.

Display table with all per item metadata

It is possible to display all the per item metadata as a CSV table using the command dtool overlays show.

$ dtool overlays show

The dataset above has three overlays named: pair_id, is_read1, and useful_name. The columns named identifiers and relpaths are reported for bookkeeping purposes.

Accessing an overlay value of a specific dataset item

It is possible to get access to the value stored in an overlay for a specific item using the command dtool item overlay.

$ dtool item overlay  \
    is_read1  \  \

Creating overlays

Overlay creation happens in two steps.

  1. Create a template overlay CSV file using the format above
  2. Use the template to write all overlays in the template to the dataset

Creating overlay templates

A starting template can be created using the dtool overlays show command. For a dataset with no overlays this will result in a table with the columns identifiers and relpaths. The table will have one row for each item in the dataset. One can then add columns for the overlays one would wish to create.

However, in many cases one would want to use metadata in the items’ relapths to generate a starting CSV template. This can be achieved using the commands:

  • dtool overlays template parse
  • dtool overlays template glob
  • dtool overlays template pairs

Consider for example the dataset below:

$ dtool ls
8bda245a8cd526673aab775f90206c8b67d196af  ERR022075_2.fastq.gz
9760280dc6313d3bb598fa03c5931a7f037d7ffc  ERR022075_1.fastq.gz

The command below could be used to generate a template for the overlays “useful_name” and “read”:

$ dtool overlays template parse  \  \

Results in the CSV output below:


To ignore a variable element when parsing one can use unnamed curly braces. The command below for example only generates the overlay “useful_name”:

$ dtool overlays template parse  \  \

Sometimes one simply wants to create a boolean overlay based on weather or not a particular file matches a glob pattern. The command below can be used to create a CSV template for an overlay named is_read1:

$ dtool overlays template glob  \  \
    is_read1  \

Sometimes it is useful to be able to find pairs of items. For example when dealing with genomic sequencing data that has forward and reverse reads.

One can create a “pair_id” overlay CSV template for this dataset using the command below:

$  dtool overlays template pairs .fastq.gz

In the above the suffix “.fastq.gz” is used to extract the prefix ERR022075_ that is used to find matching pairs.

Writing an overlay template to a dataset

Once one has a overlay template CSV file one can write this to a dataset:

$ dtool overlays write <DS_URI> overlays.csv

Further reading

For more information see the at