Skip to content

Data Collection and Metrics#


Metrics are collected by the Unryo Collector(s) and sent to the Unryo Time-Serie database for storage.

To store metrics, Unryo uses a schemaless database. Designed for real-time analytics, high-speed data ingestion and fast queries, it stores data points into Measurements, Tags and Fields. Because Unryo receives metrics from many external data sources, it enriches them with metadata tags, so that they are aligned with a common data model. This data unification allows the upper engines - analytic, correlation, reporting - to efficiently process the information. The metric tagging is extensible and customizable, and customers can enrich metrics by adding their own dimensions to get better context.

All the metric-related configuration work is available from the console UI. The Unryo Vault stores all the configurations, while the configuration UI edits them and dictates how to propagate them securely to the target components.


Measurements, Tags and Fields#

Time Series data points are written to the time-series database by collectors, that use a text-based format and a structure composed of measurement, tags and fields.

<measurement>,<tag_key>=<tag_value>[,<tag_key>=<tag_value>] <field_key>=<field_value>[,<field_key>=<field_value>] [<timestamp>]

The example below shows a data point for an NGINX server:

  nginx,                         <-- measurement
  resource=,             <-- tag
  resource_type=Nginx Server,    <-- tag
  technology=Nginx,              <-- tag
  port=80,                       <-- tag
  <space>                        <-- separator
  accepts=605,                   <-- field
  active=2,                      <-- field
  requests=12132,                <-- field
  <space>                        <-- separator
  145669099470178433             <-- timestamp


  • Measurement is a data structure that stores data over time. Typically, it stores fields that are meant to coexist together. You could think of a measurement as an 'SQL table'.
  • Tags represent 'columns' where you could store data. The biggest difference between tags and fields is that tags are indexed, and fields are not. So, queries on tags are highly performant. Tags can be seen as metadata defining data in the measurement. They are hints giving additional information about data, but not data itself. For example, the region 'column' would be a tag.
  • Fields, on the other hand, are literally data. For example, the temperature 'column' would be a field. The field value can be a string, float, integer, or boolean.
  • Timestamp is a date and time defined in RFC3339 format.

Unryo Data Model#

Unryo applies a common set of tags to data points, in order to provide a unified data model to simplify grouping, searching and filtering when querying data.

Those common tags are:

Tag Description
resource name of the resource
resource_type type of the resource. Metadata associated with this tag is listed here.
technology technology (or domain) of the resource. Metadata associated with this tag is listed here.
resource_parent parent resource
resource_component resource components (i.e. a component can be a network interface, a processor, ...)
resource_component_type resource component type

In addition to those common tags, all points have an additional set of tags that are specific to their domain. For example, a Google VM resource has specific tags such as the project_id, the zone, etc.

Data Enrichment#

With Unryo, the tagging is extensible and customizable. The Unryo tagger allows you to:

  • certify your custom data, by adding the appropriates tags resource, resource_type, technology or any other tags
  • add your own tags to enrich your data, for example adding a customer tag, a geolocation tag, based on an external data source.

The tagger comes also with some predefined configurations covering the most popular tagging use-cases:

Unryo Vault & Configuration UI#

In Unryo, all your configurations are stored encrypted in a vault, which serves as the communication channel back to your collectors or to other modules (analytic engine, topology engine, ...).

Learn more about the Unryo Vault and Configuration UI here.

Collect Metrics#

About Metrics#

In Unryo, the Collector is responsible to gather metrics. Because it is template-based and extensible, a metric can come from various sources:

  • from a device or application directly, via periodic polling (SNMP, API endpoint, ...) or ingestion (TCP stream, Webhook, ...).
  • from a synthetic test, such as ICMP, TCP or Speedtest queries.
  • from an external system, such as a monitoring tool or a custom data source.
  • from log data, when a log-to-metric aggregation is configured.
  • calculated/derived from other metrics, such as a KPI, a business service status, etc.

To configure collectors and select configurations to apply on, you can use the Configuration UI or the Vault API.

Use the Configuration UI#

The Configuration UI centrally manages all your configurations, and facilitates the operation to add, update, push telemetry configurations to one or multiple collectors, just in few clicks, from a single pane.

The Configuration UI

To use the Configuration UI, follow the procedure here.

All your configurations are stored encrypted in a central vault, which serves as the communication channel back to your collectors.

Activate a MLPacks#

Unryo supports 100+ predefined integrations to collect metrics and metadata on popular technologies and across all the stack: server, network, applications, clouds.


To start collecting data, follow the procedure here.

Use the Unryo tagger#

Within Unryo, administrators can:

You can add your own dimensions, called tags, on metrics. For example, you might want to add a tag named Region or Customer or Business Unit, or any other tag that is particular to your environment. This feature allows you to add an arbitrary number of tags before storage. Customers define the tagging rules (key/value pairs) in a simple CSV file.

  • Tags are indexed, which means queries on tags are fast.
  • You should use tags for storing commonly-queried metadata, when you plan to use them for grouping, searching and filtering.
  • Don't use tags that contain highly variable information like UUIDs, hashes or random strings, otherwise it can lead to a large number of series in the database, known as high series cardinality. High series cardinality is a primary driver of high memory usage for many database workloads.

How the tagging works#

The goal of the tagging processor is to add additional tags or string fields to metrics, based on the value of their existing tags' or fields' values. For example, for some input metrics:

measurement resource_type=Router,resource=R1 cpu=33,memory=42
measurement resource_type=Host,resource=H1 cpu=12,memory=1
measurement resource_type=Router,resource=R2 cpu=65,memory=12

A tagging processor could be configured to add a city and country tag based on the value of the resource_type and resource tags in order to transform the metrics into:

measurement resource_type=Router,resource=R1,city=Montreal,country=Canada cpu=33,memory=42
measurement resource_type=Host,resource=H1,city=Paris,country=France cpu=12,memory=1
measurement resource_type=Router,resource=R2,city=Naples,country=USA cpu=65,memory=12

For convenience and maximum flexibility, the tagging processor is configured, from the Unryo UI, using a CSV along with some metadata describing the columns. In our example above, the CSV file may look like:


Along with some metadata that explains that the first two columns are the resource and resource tag names, whose values we have to match, and the two next ones are the new city and country tags that we want to add.

Add tags#

Go in Configuration Management.


Go in Tagging

Create your metadata file (for example tagger.conf) and the associated CSV file (for example tagger.csv)

The metadata file must be configured using the following:

  ## the path to the CSV file
  file = "/etc/telegraf/telegraf.d/tagger.csv"

  ## the list of keys in the CSV file
    tag = "resource_type"
    tag = "resource"

  ## the list of values in the CSV file
    tag = "city"
    tag = "country"

The provided CSV file must have, in the same order as the configuration file, one column per declared key, then one column per declared value.


That's it! The next data points will be enriched with the new tags.

Copy tags#

You can create a new tag from an existing tag. The new tag will have the same value of the existing tag. A typical use-case for this is to regroup similar tags under a single one for data model unification purpose.


Incoming data:

system,region=us-midwest temperature=82 1465839830100400200

With this tagging configuration:

      tag = "resource"
      type = "copy-tag"

And CSV configuration:


Would give:

system,region=us-midwest,resource=us-midwest temperature=82 1465839830100400200

Certify custom devices#

We recommend enrich your custom metrics with Unryo well-known metadata tags such as technology, resource, resource_type. By doing this, it will simplify grouping, searching and filtering when querying data.

Go in Configuration Management.


Go in Tagging and edit the configuration Data Model UserDefined CSV.

In the configuration, add your new line(s) as the examples shown below:

## columns: <measurement>,<technology>,<resource_type>,<resource>,1,<resource_parent>,<resource_parent_type>,<resource_component>,<resource_component_type>,<metric>
## example: docker_container_cpu,Docker,Docker Container,container_name,1,engine_host,Docker Host,cpu,Processor,usage_percent 


  • column 1 (measurement): the name of your (time-serie db) measurement
  • column 2 (technology): a string that identify the technology (i.e. Azure, Cisco IPSLA, ...). Can be an existing one or a new one.
  • column 3 (resource_type): a string that identify the resource type (i.e. VMware vSphere VM, Windows Server, ...).
  • column 4 (resource): the tag from the measurement that identifies the resource.
  • column 5 (resource_availability): set always 1.
  • column 6 (resource_parent): optional. the tag from the measurement that identifies the resource parent.
  • column 7 (resource_parent_type): optional. the resource type of parent.
  • column 8 (resource_component): optional. If the resource has components (i.e. multiple cpu, filesystems, interfaces, ...), the tag from the measurement that identifies the component.
  • column 9 (resource_component_type): optional. The resource component type (i.e. Processor, Interface, ...)
  • column 10 (metric): optional. A field from the measurement that is a key metric (will be displayed in portal views)
  • Other columns are optional and let you model the topology hierarchy. Read the configuration header for details.

Once done, click Apply. Your new data model will take effect for new collected data.

Add Maintenance Periods#

To set a resource in maintenance:

  • Go to Settings > Configuration Management > Tagging
  • Edit the configuration In Maintenance.

Maintenance Periods can be set manually (from the Configuration UI) or through the Unryo Vault API.

Add SNMP Vendor and Model#

To set a resource in maintenance:

  • Go to Settings > Configuration Management > Tagging
  • Edit the configuration SNMP Oid2Type.

Unify naming#

To unify naming from various data sources:

Example below unify SNMP naming with SNMP trap source.

  • Go to Settings > Configuration Management > Tagging
  • Edit the configuration SNMP Traps IP-to-Name Resolving.

Naming resolution can be done manually (from the Configuration UI) or through the Unryo Vault API.


Data Retention & Clean Up#

Unryo uses a time-serie database to continuously store metric data points.

Thus, to manage the storage used up by the database, Unryo provides features to:

Configure Data Retention#

The time-serie database has a retention policy which automatically manages the lifespan of the data points in each measurement. It deletes data that are older than the retention policy. Unryo offers the ability to modify this retention policy.

Go to Settings > Platform Administration > Data Retention


Use the number field to specify the retention policy (in days):

  • By default, this lifespan is infinite, meaning data points are kept forever.
  • Unryo requires that any retention policy must be greater than 14 days.
  • To set an infinite retention policy (i.e points are never deleted), the number of days should be set to 0.

Delete data for inactive/outdated resources#

This feature allows administrators to configure deletes based on the inactivity of a resource. More specifically, administrators can specify a consecutive number of days of inactivity (Threshold) which Resources must meet to automatically have all data points dropped and alerts cleared. Unryo checks the local server’s timestamp with the Resource’s last active timestamp and drops all series if the duration is greater than the threshold.

Go to Settings > Platform Administration > Data Retention

Administrators can specify a default threshold which applies to all resources. To modify the default threshold, navigate to “All” Resource Types section and change the value in the corresponding number field: If the administrator does not want a default configuration, they can set the Threshold to 0.


Administrators can also use the Override section to specify thresholds for Resources under Collectors, Technologies, and Resource Types of their choice. To add an override, navigate to the Overrides table and click the green plus button. Use the dropdowns to specify the Collector, Technology, and Resource Type of the Resources you wish to have your threshold applied to. This override will take precedence over any default threshold.

image image

These checks and deletes are scheduled to occur once daily. Large deletes in InfluxDB can often increase server load, thus Unryo has provided the option to specify at which hour they should occur. Navigate to the dropdown and specify at which hour the delete should occur daily:


Search and Delete Resources#

Unryo offers the ability for administrators to manually drop all data points and clear all alerts from any resource.


To search resources:

  1. Go to Settings > Platform Administration > Search and Delete
  2. Use the dropdowns to narrow down and search for resources by specifying the Collector, Technology, Resource Type, and Status.
  3. Click List Resources.


The table displays all results which match the query, specifying its Host, Technology, Resource Type, Resource name, Status, and Last Active timestamp (if status is Offline).

  • A resource is considered Offline if it is currently Unreachable or if it’s last Reachable timestamp was over 7 days ago.
  • Only the first 1000 results of the query.
  • You can view the Resource’s dashboard by clicking on the icon under Actions.

To delete resources:


  • To delete a resource: navigate to a specific row and click the icon under Actions, where they will be asked to confirm the delete.
  • To delete multiple resources at a time: select multiple rows and click the DELETE SELECTED button which appears in the top right.
  • To delete all listed resources at once: set the Rows per page to All, click on the column header checkbox to select everything and then click the DELETE SELECTED button which appears in the top right.