Data Collection and Metrics

Introduction

Metrics are collected by the Unryo Collector(s) and sent to the Unryo Time-Serie database for storage.

To store metrics, Unryo uses a schemaless database. Designed for real-time analytics, high-speed data ingestion and fast queries, it stores data points into Measurements, Tags and Fields. Because Unryo receives metrics from many external data sources, it enriches them with metadata tags, so that they are aligned with a common data model. This data unification allows the upper engines - analytic, correlation, reporting - to efficiently process the information. The metric tagging is extensible and customizable, and customers can enrich metrics by adding their own dimensions to get better context.

All the metric-related configuration work is available from the console UI. The Unryo Vault stores all the configurations, while the configuration UI edits them and dictates how to propagate them securely to the target components.

• Add a collect using the Configuration UI
• Enable a specific MLPack
• Add tags
• Configure Data Retention
• Clean-up inactive metric series
• Search and delete data

Concepts

Measurements, Tags and Fields

Time Series data points are written to the time-series database by collectors, that use a text-based format and a structure composed of measurement, tags and fields.

<measurement>,<tag_key>=<tag_value>[,<tag_key>=<tag_value>] <field_key>=<field_value>[,<field_key>=<field_value>] [<timestamp>]

The example below shows a data point for an NGINX server:

  nginx,                         <-- measurement
  resource=10.1.1.1,             <-- tag
  resource_type=Nginx Server,    <-- tag
  technology=Nginx,              <-- tag
  port=80,                       <-- tag
  <space>                        <-- separator
  accepts=605,                   <-- field
  active=2,                      <-- field
  requests=12132,                <-- field
  <space>                        <-- separator
  145669099470178433             <-- timestamp

Where:

• Measurement is a data structure that stores data over time. Typically, it stores fields that are meant to coexist together. You could think of a measurement as an 'SQL table'.
• Tags represent 'columns' where you could store data. The biggest difference between tags and fields is that tags are indexed, and fields are not. So, queries on tags are highly performant. Tags can be seen as metadata defining data in the measurement. They are hints giving additional information about data, but not data itself. For example, the region 'column' would be a tag.
• Fields, on the other hand, are literally data. For example, the temperature 'column' would be a field. The field value can be a string, float, integer, or boolean.
• Timestamp is a date and time defined in RFC3339 format.

Unryo Data Model

Unryo applies a common set of tags to data points, in order to provide a unified data model to simplify grouping, searching and filtering when querying data.

Those common tags are:

Tag	Description
resource	name of the resource
resource_type	type of the resource. Metadata associated with this tag is listed here.
technology	technology (or domain) of the resource. Metadata associated with this tag is listed here.
resource_parent	parent resource
resource_component	resource components (i.e. a component can be a network interface, a processor, ...)
resource_component_type	resource component type

In addition to those common tags, all points have an additional set of tags that are specific to their domain. For example, a Google VM resource has specific tags such as the project_id, the zone, etc.

Data Enrichment

With Unryo, the tagging is extensible and customizable. The Unryo tagger allows you to:

• certify your custom data, by adding the appropriates tags resource, resource_type, technology or any other tags
• add your own tags to enrich your data, for example adding a customer tag, a geolocation tag, based on an external data source.

The tagger comes also with some predefined configurations covering the most popular tagging use-cases:

• Add Maintenance Periods
• Add SNMP Vendor and Model
• Unify naming

Unryo Vault & Configuration UI

In Unryo, all your configurations are stored encrypted in a vault, which serves as the communication channel back to your collectors or to other modules (analytic engine, topology engine, ...).

Learn more about the Unryo Vault and Configuration UI here.

Collect Metrics

About Metrics

In Unryo, the Collector is responsible to gather metrics. Because it is template-based and extensible, a metric can come from various sources:

• from a device or application directly, via periodic polling (SNMP, API endpoint, ...) or ingestion (TCP stream, Webhook, ...).
• from a synthetic test, such as ICMP, TCP or Speedtest queries.
• from an external system, such as a monitoring tool or a custom data source.
• from log data, when a log-to-metric aggregation is configured.
• calculated/derived from other metrics, such as a KPI, a business service status, etc.

To configure collectors and select configurations to apply on, you can use the Configuration UI or the Vault API.

Use the Configuration UI

The Configuration UI centrally manages all your configurations, and facilitates the operation to add, update, push telemetry configurations to one or multiple collectors, just in few clicks, from a single pane.

To use the Configuration UI, follow the procedure here.

All your configurations are stored encrypted in a central vault, which serves as the communication channel back to your collectors.

Activate a MLPacks

Unryo supports 100+ predefined integrations to collect metrics and metadata on popular technologies and across all the stack: server, network, applications, clouds.

To start collecting data, follow the procedure here.

Use the Unryo tagger

Within Unryo, administrators can:

• Add tags for example a customer name or service ID
• Extend the Unryo data model to add your custom devices or apps

You can add your own dimensions, called tags, on metrics. For example, you might want to add a tag named Region or Customer or Business Unit, or any other tag that is particular to your environment. This feature allows you to add an arbitrary number of tags before storage. Customers define the tagging rules (key/value pairs) in a simple CSV file.

• Tags are indexed, which means queries on tags are fast.
• You should use tags for storing commonly-queried metadata, when you plan to use them for grouping, searching and filtering.
• Don't use tags that contain highly variable information like UUIDs, hashes or random strings, otherwise it can lead to a large number of series in the database, known as high series cardinality. High series cardinality is a primary driver of high memory usage for many database workloads.

How the tagging works

The goal of the tagging processor is to add additional tags or string fields to metrics, based on the value of their existing tags' or fields' values. For example, for some input metrics:

measurement resource_type=Router,resource=R1 cpu=33,memory=42
measurement resource_type=Host,resource=H1 cpu=12,memory=1
measurement resource_type=Router,resource=R2 cpu=65,memory=12

A tagging processor could be configured to add a city and country tag based on the value of the resource_type and resource tags in order to transform the metrics into:

measurement resource_type=Router,resource=R1,city=Montreal,country=Canada cpu=33,memory=42
measurement resource_type=Host,resource=H1,city=Paris,country=France cpu=12,memory=1
measurement resource_type=Router,resource=R2,city=Naples,country=USA cpu=65,memory=12

For convenience and maximum flexibility, the tagging processor is configured, from the Unryo UI, using a CSV along with some metadata describing the columns. In our example above, the CSV file may look like:

Router,R1,Montreal,Canada
Host,H1,Paris,France
Router,R2,Naples,USA

Along with some metadata that explains that the first two columns are the resource and resource tag names, whose values we have to match, and the two next ones are the new city and country tags that we want to add.

Add tags

Go in Configuration Management.

Go in Tagging

Create your metadata file (for example tagger.conf) and the associated CSV file (for example tagger.csv)

The metadata file must be configured using the following:

[[processors.tagger]]
  ## the path to the CSV file
  file = "/etc/telegraf/telegraf.d/tagger.csv"

  ## the list of keys in the CSV file
  [[processors.tagger.key]]
    tag = "resource_type"
  [[processors.tagger.key]]
    tag = "resource"

  ## the list of values in the CSV file
  [[processors.tagger.value]]
    tag = "city"
  [[processors.tagger.value]]
    tag = "country"

The provided CSV file must have, in the same order as the configuration file, one column per declared key, then one column per declared value.

Router,R1,Montreal,Canada
Host,H1,Paris,France
Router,R2,Naples,USA

That's it! The next data points will be enriched with the new tags.

Copy tags

You can create a new tag from an existing tag. The new tag will have the same value of the existing tag. A typical use-case for this is to regroup similar tags under a single one for data model unification purpose.

Example:

Incoming data:

system,region=us-midwest temperature=82 1465839830100400200

With this tagging configuration:

[[processors.tagger]]
   ...
   [[processors.tagger.value]]
      tag = "resource"
      type = "copy-tag"

And CSV configuration:

...,region

Would give:

system,region=us-midwest,resource=us-midwest temperature=82 1465839830100400200

Certify custom devices

We recommend enrich your custom metrics with Unryo well-known metadata tags such as technology, resource, resource_type. By doing this, it will simplify grouping, searching and filtering when querying data.

Go in Configuration Management.

Go in Tagging and edit the configuration Data Model UserDefined CSV.

In the configuration, add your new line(s) as the examples shown below:

## columns: <measurement>,<technology>,<resource_type>,<resource>,1,<resource_parent>,<resource_parent_type>,<resource_component>,<resource_component_type>,<metric>
## example: docker_container_cpu,Docker,Docker Container,container_name,1,engine_host,Docker Host,cpu,Processor,usage_percent 

Where:

• column 1 (measurement): the name of your (time-serie db) measurement
• column 2 (technology): a string that identify the technology (i.e. Azure, Cisco IPSLA, ...). Can be an existing one or a new one.
• column 3 (resource_type): a string that identify the resource type (i.e. VMware vSphere VM, Windows Server, ...).
• column 4 (resource): the tag from the measurement that identifies the resource.
• column 5 (resource_availability): set always 1.
• column 6 (resource_parent): optional. the tag from the measurement that identifies the resource parent.
• column 7 (resource_parent_type): optional. the resource type of parent.
• column 8 (resource_component): optional. If the resource has components (i.e. multiple cpu, filesystems, interfaces, ...), the tag from the measurement that identifies the component.
• column 9 (resource_component_type): optional. The resource component type (i.e. Processor, Interface, ...)
• column 10 (metric): optional. A field from the measurement that is a key metric (will be displayed in portal views)
• Other columns are optional and let you model the topology hierarchy. Read the configuration header for details.

Once done, click Apply. Your new data model will take effect for new collected data.

Add Maintenance Periods

To set a resource in maintenance:

• Go to Settings > Configuration Management > Tagging
• Edit the configuration In Maintenance.

Maintenance Periods can be set manually (from the Configuration UI) or through the Unryo Vault API.

Add SNMP Vendor and Model

To set a resource in maintenance:

• Go to Settings > Configuration Management > Tagging
• Edit the configuration SNMP Oid2Type.

Unify naming

To unify naming from various data sources:

Example below unify SNMP naming with SNMP trap source.

• Go to Settings > Configuration Management > Tagging
• Edit the configuration SNMP Traps IP-to-Name Resolving.

Naming resolution can be done manually (from the Configuration UI) or through the Unryo Vault API.

Data Retention & Clean Up

Unryo uses a time-serie database to continuously store metric data points.

Thus, to manage the storage used up by the database, Unryo provides features to:

• configure the data retention
• delete data for inactive/outdated resources
• search and manually delete data for a group of resources

Configure Data Retention

The time-serie database has a retention policy which automatically manages the lifespan of the data points in each measurement. It deletes data that are older than the retention policy. Unryo offers the ability to modify this retention policy.

Go to Settings > Platform Administration > Data Retention

Use the number field to specify the retention policy (in days):

• By default, this lifespan is infinite, meaning data points are kept forever.
• Unryo requires that any retention policy must be greater than 14 days.
• To set an infinite retention policy (i.e points are never deleted), the number of days should be set to 0.

Delete data for inactive/outdated resources

This feature allows administrators to configure deletes based on the inactivity of a resource. More specifically, administrators can specify a consecutive number of days of inactivity (Threshold) which Resources must meet to automatically have all data points dropped and alerts cleared. Unryo checks the local server’s timestamp with the Resource’s last active timestamp and drops all series if the duration is greater than the threshold.

Go to Settings > Platform Administration > Data Retention

Administrators can specify a default threshold which applies to all resources. To modify the default threshold, navigate to “All” Resource Types section and change the value in the corresponding number field: If the administrator does not want a default configuration, they can set the Threshold to 0.

Administrators can also use the Override section to specify thresholds for Resources under Collectors, Technologies, and Resource Types of their choice. To add an override, navigate to the Overrides table and click the green plus button. Use the dropdowns to specify the Collector, Technology, and Resource Type of the Resources you wish to have your threshold applied to. This override will take precedence over any default threshold.

These checks and deletes are scheduled to occur once daily. Large deletes in InfluxDB can often increase server load, thus Unryo has provided the option to specify at which hour they should occur. Navigate to the dropdown and specify at which hour the delete should occur daily:

Search and Delete Resources

Unryo offers the ability for administrators to manually drop all data points and clear all alerts from any resource.

To search resources:

1. Go to Settings > Platform Administration > Search and Delete
2. Use the dropdowns to narrow down and search for resources by specifying the Collector, Technology, Resource Type, and Status.
3. Click List Resources.

The table displays all results which match the query, specifying its Host, Technology, Resource Type, Resource name, Status, and Last Active timestamp (if status is Offline).

• A resource is considered Offline if it is currently Unreachable or if it’s last Reachable timestamp was over 7 days ago.
• Only the first 1000 results of the query.
• You can view the Resource’s dashboard by clicking on the icon under Actions.

To delete resources:

• To delete a resource: navigate to a specific row and click the icon under Actions, where they will be asked to confirm the delete.
• To delete multiple resources at a time: select multiple rows and click the DELETE SELECTED button which appears in the top right.
• To delete all listed resources at once: set the Rows per page to All, click on the column header checkbox to select everything and then click the DELETE SELECTED button which appears in the top right.