Ubuntu Manpage: pmseries - display information about performance metric timeseries

NAME

       pmseries - display information about performance metric timeseries

SYNOPSIS

       pmseries  [-adFiIlLmMnqsStvV?]   [-c  config]  [-g pattern] [-h host] [-p port] [-w window] [-Z timezone]
       [query | labels ... | series ... | source ... ]

DESCRIPTION

pmseries displays various types of information about performance metrics available through the scalable
timeseries facilities of the Performance Co-Pilot (PCP) and a distributed key-value data store such as
https://valkey/io/..

By default pmseries communicates with a local key-value server however the -h and -p options can be used
to specify an alternate key-value server. If this instance is a node within a cluster, all other
instances in the cluster will be discovered and used automatically.

pmseries runs in several different modes - either querying timeseries identifiers, metadata or values
(already stored in the key-value store), or manually loading timeseries. The latter mode is generally
only used for loading previously collected (inactive) archives, since pmproxy(1) automatically performs
this function for "live" (actively growing) local pmlogger(1) instances, when running in its default time
series mode. See the TIMESERIES LOADING section below and the -L option for further details.

Without command line options specifying otherwise, pmseries will issue a timeseries query to find
matching timeseries and values. All timeseries are identified using a unique SHA-1 hash which is always
displayed in a 40-hexdigit human readable form. These hashes are formed using the metadata associated
with every metric.

Importantly, this includes all metric metadata (labels, names, descriptors). Metric labels in particular
are (as far as possible) unique for every machine - on Linux for example the labels associated with every
metric include the unique /etc/machine-id, the hostname, domainname, and other automatically generated
machine labels, as well as any administrator-defined labels from /etc/pcp/labels. These labels can be
reported with pminfo(1) and the pmcd.labels metric.

See pmLookupLabels(3), pmLookupInDom(3), pmLookupName(3) and pmLookupDesc(3) for detailed information
about metric labels and other metric metadata used in each timeseries identifier hash calculation.

The timeseries identifiers provide a higher level (and machine independent) identifier than the
traditional PCP performance metric identifiers (pmID), instance domain identifiers (pmInDom) and metric
names. See PCPIntro(1) for more details about these traditional identifiers. However, pmseries uses
timeseries identifiers in much the same way that pminfo(1) uses the lower level indom, metric identifiers
and metric names.

The default mode of pmseries operation (i.e. with no command line options) depends on the arguments it is
presented. If all non-option arguments appear to be timeseries identifiers (in 40 hex digit form)
pmseries will report metadata for these timeseries - refer to the -a option for details. Otherwise, the
parameters will be treated as a timeseries query.

TIMESERIES QUERIES

       Query expressions are formed using the pmseries query language described below, but can be as simple as a
       metric name.

       The following is an example of querying timeseries from all  hosts  that  match  a  metric  name  pattern
       (globbed):

         $ pmseries kernel.all.cpu*
         1d7b0bb3f6aec0f49c54f5210885464a53629b60
         379db729afd63fb9eff436625bd6c55a7adc5cfd
         3dd3b45bb05f96636043e5d58b52b441ce542285
         [...]
         ed2bf325ff6dc7589ec966698e5404b67252306a
         dcb2a032a308b5717bf605ba8f8737e9c6e1ed19

       To identify timeseries expression operands, the query language uses the general syntax:

         [metric.name] '{metadata qualifiers}' '[time specification]'

       The  metric.name  component  restricts  the timeseries query to any matching PCP metric name (the list of
       metric names for a PCP archive or live host is reported by pminfo(1) with no arguments beyond  --host  or
       --archive).   The pmseries syntax extends on that of pminfo and allows for glob(7) based pattern matching
       within the metric name.  The above describes operands available as the leaves  of  pmseries  expressions,
       which  may include functions, arithmetic operators and other features.  See the EXPRESSIONS section below
       for further details.

METADATA QUALIFIERS AND METADATA OPERATORS

       Metadata qualifiers are enclosed by ``curly'' braces ({}), and further  restrict  the  query  results  to
       timeseries  operands  with various metadata properties.  These qualifiers are based on metric or instance
       names, and metric label values, and take the general form metadata.name OPERATOR value, such as:

         instance.name == "cpu0"
         metric.name != "kernel.all.pswitch"

       When using label names, the metadata qualifier is optional and can be dropped, such as:

         label.hostname == "www.acme.com"
         hostname == "www.acme.com"

       For metric and instance names only the string operators apply, but for metric label values all  operators
       are available.  The set of available operators is:

   Boolean operators
       All string (label, metrics and instances) and numeric (label) values can be tested for equality ("==") or
       inequality ("!=").

   String operators
       Strings can be subject to pattern matching in the form of glob matching ("~~"), regular expression match‐
       ing  ("=~"),  and regular expression non-matching ("!~").  The ":" operator is equivalent to "~~" - i.e.,
       glob matching.

   Relational operators (numeric label values only)
       Numeric label values can be subject to the less than ("<"),  greater  than  (">"),  less  than  or  equal
       ("<="), greater than or equal (">="), equal ("==") and not equal ("!=") operators.

   Logical operators
       Multiple  metadata  qualifiers can be combined with the logical operators for AND ("&&") and OR ("||") as
       in many programming languages.  The comma (",") character is equivalent to logical AND ("&&").

TIME SPECIFICATION

       The final (optional) component of a query allows the user to specify a specific time window of  interest.
       Any time specification will result in values being returned for all matching timeseries only for the time
       window specified.

       The specification is ``square'' bracket ([]) enclosed, and consists of one or more comma-separated compo‐
       nents.   Each  component  specifies some aspect related to time, taking the general form: keyword: value,
       such as:

         samples:10

   Sample count
       The number of samples to return, specified via either the samples or  (equivalent)  count  keyword.   The
       value  provided must be a positive integer.  If no end time is explicitly set (see ``Time window'' later)
       then the most recent samples will be returned.

   Sample interval
       An interval between successive samples can be requested using the interval or (equivalent) delta keyword.
       The value provided should be either a numeric or string value that will be parsed by  pmParseInterval(3),
       such as 5 (seconds) or 2min (minutes).

   Time window
       Start and end times, and alignments, affecting the returned values.  The keywords match the parameters to
       the pmParseTimeWindow(3) function which will be used to parse them, and are: start or (equivalent) begin,
       finish or (equivalent) end, align and offset.

   Time zones
       The  resulting  timestamps can be returned having been evaluated for a specific timezone, using the time‐
       zone or hostzone keywords.  The value associated with timezone will be interpreted  by  pmNewZone(3).   A
       true  or false value should be associated with hostzone, and when set to true this has the same effect as
       described by pmNewContextZone(3).

EXPRESSIONS

As described above, operands are the leaves of a query expression tree.

[metric.name] '{metadata qualifiers}' '[time specification]'
Note in most of the query expression examples below, the metadata qualifiers have been omitted for brevi‐
ty. In all cases, multiple time series may qualify, particularly for the hostname label.

In the simple case, a query expression consists of a single operand and may just be a metric name. In
the more general case, a query expression is either an operand or the argument to a function, or two
operands in a binary arithmetic or logical expression. Most functions take a single argument (an expres‐
sion), though some require additional arguments, e.g. rescale.

operand | expr operator expr | func(expr[, arg])

This grammar shows expressions may be nested, e.g. using the addition (+) operator as an example,

func1(func2(expr))
func1(expr) + func2(expr)
expr + func(expr)
func(expr) + expr
expr + expr

Rules governing compatibility of operands in an expression generally depend on the function and/or opera‐
tors and are described below individually. An important rule is that if any time windows are specified,
then all operands must cover the same number of samples, though the time windows may differ individually.
If no time windows or sample counts are given, then pmseries will return a series identifier (SID) in‐
stead of a series of timestamps and values. This SID may be used in subsequent /series/values?series=SID
RESTAPI calls, along with a specific time window.

Arithmetic Operators
pmseries support addition, subtraction, division and multiplication on each value in the time series of a
binary pair of operands. No unary or ternary operators are supported (yet). In all cases, the instance
domain and the number of samples of time series operands must be the same. The metadata (units and di‐
mensions) must also be compatible. Depending on the function, the result will usually have the same in‐
stance domain and (unless noted otherwise), the same units as the operands. The metadata dimensions
(space, time, count) of the result may differ (see below).

Expression operands may have different qualifiers, e.g. you can perform binary arithmetic on metrics
qualified by different labels (such as hostname), or metric names. For example, to add the two most re‐
cent samples of the process context switch (pswitch) counter metric for hosts node88 and node89, and then
perform rate conversion:

$ pmseries 'rate(kernel.all.pswitch{hostname:node88}[count:2] +
kernel.all.pswitch{hostname:node89}[count:2])'
1cf1a85d5978640ef94c68264d3ae8866cc11f7c
[Tue Nov 10 14:39:48.771868000 2020] 71.257509 8e0a59304eb99237b89593a3e839b5bb8b9a9924

Note the resulting time series of values has one less sample than the expression operand passed to the
rate function.

Other rules for arithmetic expressions:

1. if both operands have the semantics of a counter, then only addition and subtraction are allowed

2. if the left operand is a counter and the right operand is not, then only multiplication or division
are allowed

3. if the left operand is not a counter and the right operand is a counter, then only multiplication is
allowed.

4. addition and subtraction - the dimensions of the result are the same as the dimensions of the
operands.

5. multiplication - the dimensions of the result are the sum of the dimensions of the operands.

6. division - the dimensions of the result are the difference of the dimensions of the operands.

Functions
Expression functions operate on vectors of time series values, and may be nested with other functions or
expressions as described above. When an operand has multiple instances, there will generally be one re‐
sult for each series of instances. For example, the result for

$ pmseries 'min_sample(kernel.all.load[count:100])'

will be the smallest value of the 100 most recent samples, treating each of the three load average in‐
stances as a separate time series. As an example, for the two most recent samples for each of the three
instances of the load average metric:

$ pmseries 'kernel.all.load[count:2]'
726a325c4c1ba4339ecffcdebd240f441ea77848
[Tue Nov 10 11:52:30.833379000 2020] 1.100000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
[Tue Nov 10 11:52:30.833379000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
[Tue Nov 10 11:52:30.833379000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133
[Tue Nov 10 11:52:20.827134000 2020] 1.120000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
[Tue Nov 10 11:52:20.827134000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
[Tue Nov 10 11:52:20.827134000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133

Using the min_sample function :

$ pmseries 'min_sample(kernel.all.load[count:2])'
11b965bc5f9598034ed9139fb3a78c6c0b7065ba
[Tue Nov 10 11:52:30.833379000 2020] 1.100000e+00 a7c96e5e2e0431a12279756d11590fa9fed8f306
[Tue Nov 10 11:52:30.833379000 2020] 9.900000e-01 ee9b506935fd0976a893dc27242926f49326b9a1
[Tue Nov 10 11:52:30.833379000 2020] 1.070000e+00 d5e1c360d13064c461169091997e1e8be7488133

For singular metrics (with no instance domain), a single value will result, e.g. for the five most recent
samples of the context switching metric:

$ pmseries 'kernel.all.pswitch[count:5]'
d7832c4fba33bcc980b1a1b614e0508043288480
[Tue Nov 10 12:44:59.380666000 2020] 460774294
[Tue Nov 10 12:44:49.382070000 2020] 460747232
[Tue Nov 10 12:44:39.378545000 2020] 460722370
[Tue Nov 10 12:44:29.379029000 2020] 460697388
[Tue Nov 10 12:44:19.379096000 2020] 460657412

$ pmseries 'min_sample(kernel.all.pswitch[count:5])'
1b6e92fb5bc012372f54452734dd03f0f131fa06
[Tue Nov 10 12:44:19.379096000 2020] 460657412 d7832c4fba33bcc980b1a1b614e0508043288480

Some pmseries functions provide operations across both time and instances domain. For example, max_in‐
st(expr) finds the maximum value across instances while max_sample(expr) finds the maximum value across
time.

Future versions of pmseries may provide functions that perform aggregation, interpolation, filtering or
transforms in other ways.

Function Reference
max_inst(expr) the maximum value in the time series for each instance of expr. For backwards compatibil‐
ity, the synonym max is equivalent to max_inst.

max_sample(expr) the maximum value in the time series for each sample of expr across time.

min_inst(expr) the minimum value in the time series for each instance of expr. For backwards compatibil‐
ity, the synonym min is equivalent to min_inst.

min_sample(expr) the minimum value in the time series for each sample of expr across time.

sum_inst(expr) sum of the values in the time series for each instance of expr. For backwards compatibil‐
ity, the synonym sum is equivalent to sum_inst.

sum_sample(expr) sum of the values in the time series for each sample of expr across time.

avg_inst(expr) average of the values in the time series for each instance of expr. For backwards compat‐
ibility, the synonym avg is equivalent to avg_inst.

avg_sample(expr) average of the values in the time series for each sample of expr across time.

rate(expr) the rate with respect to time of each sample. The given expr must have counter semantics and
the result will have instant semantics (the time dimension reduced by one). In addition, the result will
have one less sample than the operand - this is because the first sample cannot be rate converted (two
samples are required).

rescale(expr,scale) rescale the values in the time series for each instance of expr to scale (units).
Note that expr should have instant or discrete semantics (not counter - rate conversion should be done
first if needed). The time, space and count dimensions between expr and scale must be compatible. Exam‐
ple: rate convert the read throughput counter for each disk instance and then rescale to mbytes per sec‐
ond. Note the native units of disk.dev.read_bytes is a counter of kbytes read from each device instance
since boot.

$ pmseries 'rescale(rate(disk.dev.read_bytes[count:4]), "mbytes/s")'

abs(expr) the absolute value of each value in the time series for each instance of expr. This has no ef‐
fect if the type of expr is unsigned.

floor(expr) rounded down to the nearest integer value of the time series for each instance of expr.

round(expr) rounded up or down to the nearest integer for each value in the time series for each instance
of expr.

log(expr) logarithm of the values in the time series for each instance of expr

sqrt(expr) square root of the values in the time series for each instance of expr

stdev_inst(expr) standard deviation of the values in the time series for each instance of expr.

stdev_sample(expr) standard deviation of the values in the time series for each sample of expr across
time.

topk_inst(expr,k) the top k values in the time series for each instance of expr.

topk_sample(expr,k) the top k values in the time series for each sample of expr across time.

nth_percentile_inst(expr,percentile_value) the nth percentile of the values in the time series for each
instance of expr. Note that percentile_value has value in the range 0 to 100.

nth_percentile_sample(expr,percentile_value) the nth percentile of the values in the time series for each
sample of expr across time. Note that percentile_value has value in the range 0 to 100.

Compatibility
All operands in an expression must have the same number of samples, but not necessarily the same time
window. e.g. you could subtract some metric time series from today from that of yesterday by giving dif‐
ferent time windows and different metrics or qualifiers, ensuring the same number of samples are given as
the operands.

Operands in an expression must either all have a time window, or none. If no operands have a time win‐
dow, then instead of a series of time stamps and values, the result will be a time series identifier
(SID) that may be passed to the /series/values?series=SID REST API function, along with a time window.
For further details, see PMWEBAPI(3).

If the semantics of both operands in an arithmetic expression are not counter (i.e. PM_SEM_INSTANT or
PM_SEM_DISCRETE) then the result will have semantics PM_SEM_INSTANT unless both operands are PM_SEM_DIS‐
CRETE in which case the result is also PM_SEM_DISCRETE.

TIMESERIES METADATA

Using command line options, pmseries can be requested to provide metadata (metric names, instance names,
labels, descriptors) associated with either individual timeseries or a group of timeseries, for example:

$ pmseries -a dcb2a032a308b5717bf605ba8f8737e9c6e1ed19

dcb2a032a308b5717bf605ba8f8737e9c6e1ed19
PMID: 60.0.21
Data Type: 64-bit unsigned int InDom: PM_INDOM_NULL 0xffffffff
Semantics: counter Units: millisec
Source: f5ca7481da8c038325d15612bb1c6473ce1ef16f
Metric: kernel.all.cpu.nice
labels {"agent":"linux","domainname":"localdomain",\
"groupid":1000,"hostname":"shard",\
"latitude":-25.28496,"longitude":152.87886,\
"machineid":"295b16e3b6074cc8bdbda8bf96f6930a",\
"userid":1000}

The complete set of pmseries metadata reporting options are:

-a, --all
Convenience option to report all metadata for the given timeseries, equivalent to -deilms.

-d, --desc
Metric descriptions detailing the PMID, data type, data semantics, units, scale and associated in‐
stance domain. This option has a direct pminfo(1) equivalent.

-F, --fast
Query or load series metadata only, not values.

-g pattern, --glob=pattern
Provide a glob(7) pattern to restrict the report provided by the -i, -l, -m, and -S.

-i, --instances
Metric descriptions detailing the PMID, data type, data semantics, units, scale and associated in‐
stance domain.

-I, --fullindom
Print the InDom in verbose mode. This option has a direct pminfo(1) equivalent.

-l, --labels
Print label sets associated with metrics and instances. Labels are optional metric metadata de‐
scribed in detail in pmLookupLabels(3). This option has a direct pminfo(1) equivalent.

-m, --metrics
Print metric names.

-M, --fullpmid
Print the PMID in verbose mode. This option has a direct pminfo(1) equivalent.

-n, --names
Print comma-separated label names only (not values) for the labels associated with metrics and in‐
stances.

-s, --series
Print timeseries identifiers associated with metrics, instances and sources. These unique identi‐
fiers are calculated from intrinsic (non-optional) labels and other metric metadata associated with
each PMAPI context (sources), metrics and instances. Archive, local context or pmcd(1) connections
for the same host all produce the same source identifier. This option has a direct pminfo(1) equiv‐
alent. See also pmLookupLabels(3) and the -l/--labels option.

TIMESERIES SOURCES

       A  source is a unique identifier (represented externally as a 40-byte hexadecimal SHA-1 hash) that repre‐
       sents both the live host and/or archives from which each timeseries originated.  The context for a source
       identifier (obtained with -s) can be reported with:

       -S, --sources
            Print names for timeseries sources.  These names are either hostnames  or  fully  qualified  archive
            paths.

       It is important to note that live and archive sources can and will generate the same SHA-1 source identi‐
       fier  hash,  provided  that  the  context  labels remain the same for that host (labels are stored in PCP
       archives and can also be fetched live from pmcd(1)).

TIMESERIES LOADING

       Timeseries metadata and data are loaded either automatically by a local pmproxy(1), or manually  using  a
       specially crafted pmseries query and the -L/--load option:

         $ pmseries --load "{source.path: \"$PCP_LOG_DIR/pmlogger/acme\"}"
         pmseries: [Info] processed 2275 archive records from [...]

       This  query  must  specify a source archive path, but can also restrict the import to specific timeseries
       (using metric names, labels, etc) and to a specific time window using the time specification component of
       the query language.

       As a convenience, if the argument to load is a valid file path as determined by access(2), then a  short-
       hand form can be used:

         $ pmseries --load $PCP_LOG_DIR/pmlogger/acme.0

       NOTE:  Timeseries loading is append-only (timestamp-wise) and if more than stream.maxlen entries (defined
       in $PCP_SYSCONF_DIR/pmseries/pmseries.conf) are loaded  for  a  given  metric,  the  oldest  entries  are
       dropped.

OPTIONS

The available command line options, in addition to timeseries metadata and sources options described
above, are:

-c config, --config=config
Specify the config file to use.

-h host, --host=host
Connect to the key-value server at host, rather than the one the localhost.

-L, --load
Load timeseries metadata and data into the key-value cluster.

-p port, --port=port
Connect to the key-value server at port, rather than the default 6379.

-q, --query
Perform a timeseries query. This is the default action.

-t, --times
Report time stamps numerically (in milliseconds) instead of the default human readable form.

-v, --values
Report all of the known values for given label name(s), or report values for the given series iden‐
tifiers.

-w, --window
Provide a time specification that will be applied to values being returned when returning values via
use of series identifiers (i.e. when not using a query string). The time specification uses the
same square-bracket enclosed form described earlier in the ``TIME SPECIFICATION'' section.

-V, --version
Display version number and exit.

-Z timezone, --timezone=timezone
Use timezone for the date and time. Timezone is in the format of the environment variable TZ as de‐
scribed in environ(7).

-?, --help
Display usage message and exit.

EXAMPLES

       The following sample query shows several fundamental aspects of the pmseries query language:

         $ pmseries 'kernel.all.load{hostname:"toium"}[count:2]'

         eb713a9cf472f775aa59ae90c43cd7f960f7870f
             [Thu Nov 14 05:57:06.082861000 2019] 1.0e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
             [Thu Nov 14 05:57:06.082861000 2019] 6.8e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
             [Thu Nov 14 05:57:06.082861000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480
             [Thu Nov 14 05:57:16.091546000 2019] 1.6e-01 b84040ffccd54f839b65140cf139bab51cbbcf62
             [Thu Nov 14 05:57:16.091546000 2019] 6.7e-01 a60b5b3bf25e71071c41934fa4d7d251f765f30c
             [Thu Nov 14 05:57:16.091546000 2019] 6.4e-01 e1974a062375e6e62370ffadf5b0650dad739480

       This query returns the two most recent values for all instances of the kernel.all.load metric with a  la‐
       bel.hostname  matching  the regular expression "toium".  This is a set-valued metric (i.e., a metric with
       an ``instance domain'' which in this case consists of three instances: 1, 5 and 15 minute averages).  The
       first column returned is a timestamp, then a floating point value, and  finally  an  instance  identifier
       timeseries  hash  (two  values returned for three instances, so six rows are returned).  The metadata for
       these timeseries can then be further examined:

         $ pmseries -a eb713a9cf472f775aa59ae90c43cd7f960f7870f

         eb713a9cf472f775aa59ae90c43cd7f960f7870f
             PMID: 60.2.0
             Data Type: float  InDom: 60.2 0xf000002
             Semantics: instant  Units: none
             Source: 0e89c1192db79326900d82131c31399524f0b3ee
             Metric: kernel.all.load
             inst [1 or "1 minute"] series b84040ffccd54f839b65140cf139bab51cbbcf62
             inst [5 or "5 minute"] series a60b5b3bf25e71071c41934fa4d7d251f765f30c
             inst [15 or "15 minute"] series e1974a062375e6e62370ffadf5b0650dad739480
             inst [1 or "1 minute"] labels {"agent":"linux","hostname":"toium"}
             inst [5 or "5 minute"] labels {"agent":"linux","hostname":"toium"}
             inst [15 or "15 minute"] labels {"agent":"linux","hostname":"toium"}

PCP ENVIRONMENT

       Environment variables with the prefix PCP_ are used to parameterize the file and directory names used  by
       PCP.   On  each  installation, the file /etc/pcp.conf contains the local values for these variables.  The
       $PCP_CONF variable may be used to specify an alternative configuration file, as described in pcp.conf(5).

       For environment variables affecting PCP tools, see pmGetOptions(3).