Query metadata

To query the logged metadata, use the Neptune Fetcher API. You can:

• List experiments of a project.
• List attributes of a project's experiments.
• Fetch experiment metadata as a data frame.
• Define filters, to fetch experiments and attributes matching certain criteria.

You can also use corresponding functions for runs rather than experiments.

note

This page shows how to use the next major version of the Fetcher API (alpha). We may introduce non-backward-compatible changes in upcoming releases.

• For the current version 0.x of the API, see Fetch runs or experiments.
• To search and filter experiments or runs in the Neptune app, use the runs table.

Setting up Neptune Fetcher

Install neptune-fetcher:

pip install -U neptune-fetcher

Set your Neptune API token and project name as environment variables:

export NEPTUNE_API_TOKEN="h0dHBzOi8aHR0cHM.4kl0jvYh3Kb8...ifQ=="

export NEPTUNE_PROJECT="workspace-name/project-name"

For help, see Get started.

tip

To change the token or project, you can also set the context directly in the code. This way, you can work with multiple projects at the same time.

To start using the API, import the alpha module:

import neptune_fetcher.alpha as npt

Listing experiments

Filter experiments by name that match a regex:

npt.list_experiments(r"exp_.*")

['exp_xjjrq', 'exp_ymgun', 'exp_nmjbt', 'exp_ixgwm', 'exp_rdmuw']

Listing unique attributes

The following example looks for attributes matching the regular expression metrics.*|config/.*, among experiments whose name match exp.*:

npt.list_attributes(
    experiments=r"exp.*",
    attributes=r"metrics.*|config/.*",
)

['config/batch_size',
 'config/epochs',
 'config/last_event_time',
 'config/learning_rate',
 'config/optimizer',
 'config/use_bias',
 'metrics/accuracy',
 'metrics/loss',
 'metrics/val_accuracy',
 'metrics/val_loss']

Fetching metadata as table

To fetch experiment metadata from your project, use the fetch_experiments_table() function.

To limit the scope, use the filtering parameters:

• experiments: Only fetch attributes from experiments that pass the filter.
• attributes: Only include attributes that pass the filter as columns.

For both arguments, you can specify a simple string to match experiment or attribute names against. To construct more complex filters, use the Filter classes.

npt.fetch_experiments_table(
    experiments=r"exp.*",
    attributes=r".*metric.*/val_.+",
)

           metrics/val_accuracy metrics/val_loss
                           last             last
experiment
exp_ergwq              0.278149         0.336344
exp_qgguv              0.160260         0.790268
exp_cnuwh              0.702490         0.366390
exp_kokxd              0.301545         0.917683
exp_gyvpp              0.999489         0.069839

note

Fetching metrics this way returns an aggregated value for each attribute. The default aggregation is the last logged value.

To fetch actual metric values at each step, see Fetching metric values.

The following example fetches metadata from experiments that meet two criteria:

• The validation/loss attribute value is less than 0.1
• The experiment name matches the regex exp.*

Additionally, only attributes matching validation are included as columns.

loss_filter = Filter.lt("validation/loss", 0.1)
name_filter = r"exp.*"

combined_filter = loss_filter & name_filter

npt.fetch_experiments_table(
    experiments=combined_filter,
    attributes=r"validation",
)

            validation/accuracy  validation/loss
                           last             last
experiment
exp_ergwq              0.278149         0.336344
exp_qgguv              0.160260         0.790268
exp_cnuwh              0.702490         0.366390
exp_kokxd              0.301545         0.917683
exp_gyvpp              0.999489         0.069839

Fetching metric values

To fetch individual values from one or more FloatSeries attributes, use the fetch_metrics() function:

npt.fetch_metrics(
    experiments=r"exp.*",
    attributes=r"metrics/.*",
)

  experiment  step  metrics/accuracy  metrics/loss  metrics/val_accuracy  metrics/val_loss
0  exp_dczjz     0          0.819754      0.588655              0.432187          0.823375
1  exp_dczjz     1          0.347907      0.297161              0.649685          0.971732
2  exp_dczjz     2          0.858863      0.988583              0.760142          0.154741
3  exp_dczjz     3          0.217097          None              0.719508          0.504652
4  exp_dczjz     4          0.089981      0.463146              0.180321          0.503800
5  exp_hgctc     0          0.318828      0.386347              0.012390          0.171790
6  exp_hgctc     1          0.035026      0.087053              0.392041          0.768675
7  exp_hgctc     2          0.004711      0.061848                  None          0.847072
8  exp_hgctc     3          0.359770      0.130022              0.479945          0.323537
9  exp_hgctc     4          0.007815      0.746344              0.102646          0.055511

Fetch metric previews

To fetch point previews, set the include_point_previews argument to True:

npt.fetch_metrics(
    experiments=r"exp.*",
    attributes=r"metrics/.*",
    include_point_previews=True,
)

When previews are included, the returned data frame includes additional sub-columns with preview information: is_preview and preview_completion.

Constructing filters

Use the Filter class to specify criteria when fetching experiments or runs.

Examples of filters:

• Name or attribute must match regular expression.
• Attribute value must pass a condition, like "greater than 0.9".

You can negate a filter or join multiple filters with logical operators.

To use a filter in a query, pass it as the argument to a fetching or listing method:

npt.fetch_experiments_table(Filter.lt("validation/loss", 0.1))

Methods

Operators and methods available for attribute values:

• eq(): Equals
• ne(): Doesn't equal
• gt(): Greater than
• ge(): Greater than or equal to
• lt(): Less than
• le(): Less than or equal to
• matches_all(): Matches regex or all in list of regexes
• matches_none(): Doesn't match regex or any of list of regexes
• contains_all(): Tagset contains all tags, or string contains substrings
• contains_none(): Tagset doesn't contain any of the tags, or string doesn't contain the substrings
• exists(): Attribute exists

Examples

Import the needed classes:

import neptune_fetcher.alpha as npt
from npt.filters import Filter

Filter on name

The simplest filter is a string representing a regular expression. To pass the filter, the experiment name must match the regex:

name_filter = r"kittiwake$"

npt.fetch_experiments_table(name_filter)

Filter on string value

To match an arbitrary string field against a regex:

optimizer_filter = Filter.matches_all("config/optimizer", r"Ada.+")

Require tags

Don't allow the tags "test" or "val":

no_test_or_val = Filter.contains_none("sys/tags", ["test", "val"])

Filter on attribute value

Set a condition for the last logged value of a metric:

loss_filter = Filter.lt("validation/loss", 0.1)

For more control over the selected metric, use the Attribute helper class.

Negate a filter

Call negate() or prepend with ~:

not_loss_filter = ~loss_filter
# equivalent to
not_loss_filter = Filter.negate(loss_filter)

Combining filters

• To join with AND: Use & or pass the filters to the all() method.
• To join with OR: Use | or pass the filters to the any() method.

name_and_loss_filter = name_filter & loss_filter
# equivalent to
name_and_loss_filter = Filter.all(name_filter, loss_filter)

Filtering attributes

When fetching metadata with the fetch_experiments_table() function, in the returned table, each column represents an attribute.

To select specific metrics or other attributes based on various criteria, use the AttributeFilter class.

Examples

Specify an attribute by its exact name:

from npt.filters import AttributeFilter

AttributeFilter(name_eq="config/optimizer")

Select metrics that don't match regexes ^test or loss$ and pick the "average" and "variance" aggregations:

AttributeFilter(
    type_in=["float_series"],
    name_matches_none=[r"^test", r"loss$"],
    aggregations=["average", "variance"],
)

In this case, the returned table includes "average" and "variance" columns for each metric:

attribute              train/accuracy             validation/accuracy
aggregation                   average  variance               average  variance
experiment
exp-1738662528               0.000000  0.000273                   0.0  0.000269
exp-1738325381               0.000000  0.594614                   0.0  0.595119
...

Related

• Attribute types

Combine multiple filters with the pipe character:

filter_1 = AttributeFilter(...)
filter_2 = AttributeFilter(...)
filter_3 = AttributeFilter(...)
alternatives = filter_1 | filter_2 | filter_3

To use an attribute filter, pass it to the attributes argument of the fetch_experiments_table() function:

npt.fetch_experiments_table(
    attributes=AttributeFilter(...),
)

Attribute helper class

The Attribute helper class is used for specifying a single attribute and picking a metric aggregation function.

When fetching experiments or runs, use this class to filter and sort the returned entries.

Examples

Select a metric and pick variance as the aggregation:

import neptune_fetcher.alpha as npt
from npt.filters import Attribute, Filter


val_loss_variance = Attribute(
    name="val/loss",
    aggregation="variance",
)

Construct a filter around the attribute, then pass it to a fetching or listing method:

tiny_val_loss_variance = Filter.lt(val_loss_variance, 0.01)
npt.fetch_experiments_table(experiments=tiny_val_loss_variance)

Working with multiple projects

To work with multiple projects simultaneously, you can use contexts. This way, you can set the scope for individual fetching calls or globally for your session.

• To set a new project or API token globally, use set_project() or set_api_token():

  npt.set_project("some-workspace/another-project")

• To create a context object that you can pass to the context argument of a fetching method:

  another_project_context = npt.get_context().with_project("some-workspace/another-project")
  npt.list_experiments(r"exp_.*", context=another_project_context)

Example flow:

1. NEPTUNE_PROJECT environment variable is read on module initialization:

   import neptune_fetcher.alpha as npt

2. Work on the default project inferred from the environment variables:

   npt.list_experiments(r"exp_.*")

   ['exp_dhxxz', 'exp_saazg', 'exp_sxlrq', 'exp_vlqfe', 'exp_fpogj']

3. Work on another project without changing the global context:

   another_project_ctx = npt.get_context().with_project("some-workspace/another-project")
   npt.list_experiments(r"exp_.*", context=another_project_ctx)

   ['exp_oecez', 'exp_ytbda', 'exp_khfym']

4. Change the project globally:

   npt.set_project("some-workspace/another-project")

5. Do some more work on another project:

   npt.list_experiments(r"exp_.*")

   ['exp_oecez', 'exp_ytbda', 'exp_khfym']

Working with runs

Experiments consist of runs. By default, the Fetcher API returns metadata only from experiment head runs, not all individual runs.

To work with runs:

1. Import the corresponding method from the runs module:

   from neptune_fetcher.alpha.runs import (
       fetch_metrics,
       fetch_runs_table,
       list_attributes,
       list_runs,
   )

2. Instead of the experiments parameter, use runs.

Note the difference in identifying runs versus experiments:

• Experiments are characterized by name (sys/name). This is the string passed to the name argument at creation.
• Runs are characterized by ID (sys/custom_run_id). This is the string passed to the run_id argument at creation.

Example:

import neptune_fetcher.alpha as npt

npt.runs.fetch_metrics(
    runs=r"^speedy-seagull.*_02",  # run ID
    attributes=r"losses/.*",
)