Ubuntu Manpage: varnishncsa - Display Varnish logs in Apache / NCSA combined log format

NAME

       varnishncsa - Display Varnish logs in Apache / NCSA combined log format

SYNOPSIS

       varnishncsa  [-a]  [-b] [-c] [-C] [-d] [-D] [-E] [-F <format>] [-f <formatfile>] [-g <request|vxid>] [-h]
       [-j] [-k <num>] [-L <limit>] [-n <workdir>] [-P <file>] [-Q <file>]  [-q  <query>]  [-r  <filename>]  [-R
       <limit[/duration]>] [-t <seconds|off>] [-V] [-w <filename>]

DESCRIPTION

The varnishncsa utility reads varnishd(1) shared memory logs and presents them in the Apache / NCSA
"combined" log format.

Each log line produced is based on a single Request type transaction gathered from the shared memory log.
The Request transaction is then scanned for the relevant parts in order to output one log line. To filter
the log lines produced, use the query language to select the applicable transactions. Non-request
transactions are ignored.

The following options are available:

-a When writing output to a file, append to it rather than overwrite it. This option has no effect
without the -w option.

-b Log backend requests. If -c is not specified, then only backend requests will trigger log lines.

-c Log client requests. This is the default. If -b is specified, then -c is needed to also log client
requests

-C Do all regular expression and string matching caseless.

-d Process log records at the head of the log and exit.

-D Daemonize.

-E Show ESI requests, implies client mode.

-F <format>
Set the output log format string.

-f <formatfile>
Read output format from a file. Will read a single line from the specified file, and use that line
as the format.

-g <request|vxid>
The grouping of the log records. The default is to group by vxid.

-h Print program usage and exit

-j Make format-specifier replacements JSON-compatible. When escaping characters, use JSON-style
\uXXXX escape sequences instead of C-style \xXX sequences. Empty strings will be replaced with ""
instead of "-", and empty integers will be replaced with null. Use -F or -f in combination with -j
to write JSON logs.

-k <num>
Process this number of matching log transactions before exiting.

-L <limit>
Sets the upper limit of incomplete transactions kept before the oldest transaction is force
completed. A warning record is synthesized when this happens. This setting keeps an upper bound on
the memory usage of running queries. Defaults to 1000 transactions.

-n <workdir>
Specify the varnish working directory of the instance to attach to. See varnishd(1) -n option
documentation for additional information and defaults.

-P <file>
Write the process' PID to the specified file.

-Q <file>
Specifies the file containing the VSL query to use. When multiple -Q or -q options are specified,
all queries are considered as if the 'or' operator was used to combine them.

-q <query>
Specifies the VSL query to use. When multiple -q or -Q options are specified, all queries are
considered as if the 'or' operator was used to combine them.

-r <filename>
Read log in binary file format from this file. The file can be created with varnishlog -w
filename. If the filename is -, logs are read from the standard input. and cannot work as a
daemon.

-R <limit[/duration]>
Restrict the output to the specified limit. Transactions exceeding the limit will be suppressed.
The limit is specified as the maximum number of transactions (with respect to the chosen grouping
method) and an optional time period. If no duration is specified, a default of s is used. The
duration field can be formatted as in VCL (e.g. -R 10/2m) or as a simple time period without the
prefix (e.g. -R 5/m). When in -g raw grouping mode, this setting can not be used alongside -i, -I,
-x or -X, and we advise using -q instead.

-t <seconds|off>
Timeout before returning error on initial VSM connection. If set the VSM connection is retried
every 0.5 seconds for this many seconds. If zero the connection is attempted only once and will
fail immediately if unsuccessful. If set to "off", the connection will not fail, allowing the
utility to start and wait indefinitely for the Varnish instance to appear. Defaults to 5 seconds.

-V Print version information and exit.

-w <filename>
Redirect output to file. The file will be overwritten unless the -a option was specified. If the
application receives a SIGHUP in daemon mode the file will be reopened allowing the old one to be
rotated away. This option is required when running in daemon mode. If the filename is -,
varnishncsa writes to the standard output and cannot work as a daemon.

--optstring
Print the optstring parameter to getopt(3) to help writing wrapper scripts.

MODES

       The default mode of varnishncsa is "client mode".  In this mode, the log will be similar to  what  a  web
       server would produce in the absence of varnish.  Client mode can be explicitly selected by using -c.

       If  the  -b  switch  is  specified,  varnishncsa  will operate in "backend mode".  In this mode, requests
       generated by varnish to the backends will be logged.   Unless  -c  is  also  specified,  client  requests
       received by varnish will be ignored.

       When  running  varnishncsa  in both backend and client mode, it is strongly advised to include the format
       specifier %{Varnish:side}x to distinguish between backend and client requests.

       Client requests that results in a pipe (ie. return(pipe) in vcl), will not generate  logging  in  backend
       mode. This is because varnish is not generating requests, but blindly passes on bytes in both directions.
       However,  a  varnishncsa  instance  running  in  normal  mode  can  see  this case by using the formatter
       %{Varnish:handling}x, which will be 'pipe'.

       In backend mode, some of the fields in the format string get different meanings.  Most notably, the  byte
       counting formatters (%b, %I, %O) considers varnish to be the client.

       It  is  possible  to keep two varnishncsa instances running, one in backend mode, and one in client mode,
       logging to different files.

FORMAT

Specify the log format to use. If no format is specified the default log format is used:

%h %l %u %t "%r" %s %b "%{Referer}i" "%{User-agent}i"

Escape sequences \n and \t are supported.

Supported formatters are:

%b In client mode, size of response in bytes, excluding HTTP headers. In backend mode, the number of
bytes received from the backend, excluding HTTP headers. In CLF format, i.e. a '-' rather than a
0 when no bytes are sent.

%D In client mode, time taken to serve the request, in microseconds. In backend mode, time from the
request was sent to the entire body had been received. This is equivalent to %{us}T.

%H The request protocol. Defaults to HTTP/1.0 if not known.

%h Remote host. Defaults to '-' if not known. In backend mode this is the IP of the backend server.

%I In client mode, total bytes received from client. In backend mode, total bytes sent to the
backend.

%{X}i The contents of request header X. If the header appears multiple times in a single transaction,
the last occurrence is used in backend mode and the first one in client mode.

%l Remote logname. Always '-'.

%m Request method. Defaults to '-' if not known.

%{X}o The contents of response header X. If the header appears multiple times in a single transaction,
the last occurrence is used in client mode and the first one in backend mode.

%O In client mode, total bytes sent to client. In backend mode, total bytes received from the
backend.

%q The query string. Defaults to an empty string if not present.

%r The first line of the request. Synthesized from other fields, so it may not be the request
verbatim. See the NOTES section.

%s Status sent to the client. In backend mode, status received from the backend.

%t In client mode, time when the request was received, in HTTP date/time format. In backend mode,
time when the request was sent.

%{X}t In client mode, time when the request was received, in the format specified by X. In backend
mode, time when the request was sent. The time specification format is the same as for
strftime(3) with these extensions:

• %{sec}: number of seconds since the Epoch

• %{msec}: number of milliseconds since the Epoch

• %{usec}: number of milliseconds since the Epoch

• %{msec_frac}: millisecond fraction

• %{usec_frac}: microsecond fraction

The extensions cannot be combined with each other or strftime(3) in the same specification. Use
multiple %{X}t specifications instead.

%T In client mode, time taken to serve the request, in seconds. In backend mode, time from the
request was sent to the entire body had been received. This is equivalent to %{s}T.

%{X}T In client mode, time taken to serve the request, in the format specified by X. In backend mode,
time from the request was sent to the entire body had been received. The time specification format
can be one of the following: s (same as %T), ms or us (same as %D).

%U The request URL without the query string. Defaults to '-' if not known.

%u Remote user from auth.

%{X}x Extended variables. Supported variables are:

Varnish:default_format
The log format used when neither -f nor -F options are specified. Useful for
appending/prepending with other formatters.

Varnish:time_firstbyte
Time from when the request processing starts until the first byte is sent to the client, in
seconds. For backend mode: Time from the request was sent to the backend to the entire
header had been received.

Varnish:hitmiss
In client mode, one of the 'hit' or 'miss' strings, depending on whether the request was a
cache hit or miss. Pipe, pass and synth are considered misses. In backend mode, this field
is blank.

Varnish:handling
In client mode, one of the 'hit', 'hitmiss', 'hitpass', 'miss', 'pass', 'pipe' or 'synth'
strings indicating how the request was handled. In backend mode, this field is blank.

Varnish:side
Backend or client side. One of two values, 'b' or 'c', depending on where the request was
made. In pure backend or client mode, this field will be constant.

Varnish:vxid
The VXID of the varnish transaction.

VCL_Log:key
The value set by std.log("key:value") in VCL.

VSL:tag:record-prefix[field]
The value of the VSL entry for the given tag-record prefix-field combination. Tag is
mandatory, the other components are optional.

The record prefix will limit the matches to those records that have this prefix as the
first part of the record content followed by a colon.

The field will, if present, treat the log record as a white space separated list of fields,
and only the nth part of the record will be matched against. Fields start counting at 1 and
run up to 255.

Defaults to '-' when the tag is not seen, the record prefix does not match or the field is
out of bounds. If a tag appears multiple times in a single transaction, the first
occurrence is used.

SIGNALS

       • SIGHUP

         Rotate  the  log file (see -w option) in daemon mode, abort the loop and die gracefully when running in
         the foreground.

       • SIGUSR1

         Flush any outstanding transactions.

NOTES

       The %r formatter is equivalent to %m http://%{Host}i%U%q %H. This differs from the Apache HTTP Server  %r
       behavior, equivalent to %m %U%q %H.

       Note  that  request  fields  are  collected on a first match basis in client mode and last match basis in
       backend mode. Similarly, response fields are collected on a first match basis in backend  mode  and  last
       match basis in client mode.

       In other words, request headers are logged as they were received from the client and as they were sent to
       the  backend, while response headers are logged as they were sent to the client and as they were received
       from the backend.

       Furthermore, these rules also apply to items that appear multiple times in a transaction. For example, if
       a header appears multiple times in a client request, the first occurrence is logged in client mode, while
       in backend mode the last occurrence is logged.

EXAMPLE

       Log the second field of the Begin record, corresponding to the VXID of the parent transaction:

          varnishncsa -F "%{VSL:Begin[2]}x"

       Log the entire Timestamp record associated with the processing length:

          varnishncsa -F "%{VSL:Timestamp:Process}x"

       Log in JSON, using the -j flag to ensure that the output is valid JSON for all inputs:

          varnishncsa -j -F '{"size": %b, "time": "%t", "ua": "%{User-Agent}i"}'

HISTORY

       The varnishncsa utility was developed by Poul-Henning Kamp  in  cooperation  with  Verdens  Gang  AS  and
       Varnish  Software AS. This manual page was initially written by Dag-Erling Smørgrav < <des@des.no> >, and
       later updated by Martin Blix Grydeland and Pål Hermunn Johansen.

COPYRIGHT

       This document is licensed under the same licence as Varnish itself. See LICENCE for details.

       • Copyright (c) 2006 Verdens Gang AS

       • Copyright (c) 2006-2016 Varnish Software AS

                                                                                                  VARNISHNCSA(1)