Provided by: guerillabackup_0.5.0-1.1_all 
NAME
gb-storage-tool - Manage guerillabackup backup data storages
SYNOPSIS
gb-storage-tool --Config file --DryRun
DESCRIPTION
This is the manual page for the gb-storage-tool command. The tool is used to perform operations on backup
file storage locations as used by gb-backup-generator or gb-transfer-service to store backup data.
Currently the tool supports checking storage file naming to identify incomplete backups due to aborts
during backup generation or transfer e.g. by reboots or crashes. To ignore files for a reason, e.g.
notes, add entries to the status file, e.g.
For all files defining valid backup data elements, configurable policies are applied. See POLICIES
section below for supported policies.
OPTIONS
--Config file
This optional parameter specifies an alternative configuration file instead of
/etc/guerillabackup/storage-tool-config.json.
--DryRun
This optional parameter will make gb-storage-tool perform policy checks only but will not modify the
storage, e.g. by deleting files flagged for deletion by a retention policy.
POLICIES
gb-storage-tool can apply multiple policies to each backup data source but it is only possible to have
one policy of a given type (see policy types below). Which policies to apply is defined by the
gb-storage-tool configuration file "Policies" parameter. A regular expression is used to select which
sources policies should be applied to with the first matching expression taking precedence. For each
regular expression a list of polices with parameters is defined. See
/data/etc/storage-tool-config.json.template for examples.
To ease policy selection in large setups, policy inheritance can be used. A included configuration (see
"Include" configuration parameter) may also define policies, which can extend or override the policies
from the parent configuration(s) but also policies defined just earlier in the same configuration. The
overriding policy definition has to have a higher priority, otherwise it will be ignored. To disable
policy inheritance a subconfiguration may set the "Inherit" configuration parameter to false (default is
true). This will also prevent any policies defined earlier in the very same configuration to be ignored.
Thus to disable inheritance for all sources in a configuration, the first entry in the policy list should
match all sources (.*) and disable inheritance.
Each policy defined in the gb-storage-tool configuration file may also keep policy status information in
the status file. The status data is usually updated as the policy is applied unless there is a
significant policy violation. That will require the user either to fix the root cause of the violation
(e.g. backup data was found to be missing) or the user may update the status to ignore the violation. The
later cannot be done interactively via gb-storage-tool yet, one has to adjust the storage status
configuration manually. Therefore the user has to create or update the status configuration with the the
backup element name (the filename relative to the data directory without any suffix) as key and the
status information for the policy as described below (and sometimes given as hint on the console too).
gb-storage-tool supports following policies:
Interval:
Verify that all sources generate backups at expected rates and all backups were transferred
successfully. Thus this policy eases spotting of system failures in the backup system. An example
policy configuration is:
...
"Policies": [
{
"Sources": "^(.*/)?root$",
"Inherit": false,
"List": [
{
"Name": "Interval",
"Priority": 100,
"FullMin": "6d20H",
"FullMax": "6d28H",
"IncMin": "20H",
"IncMax": "28H"
}, {
...
This configuration specifies that to all backups from source with name "root" (the default backup
created by the gb-backup-generator) an Interval policy shall be applied. The policy will expect full
backups every 7 days +- 4 hours and incremental backups each day +- 4 hours.
When policy validation fails for a given source, the policy configuration may be adjusted but also
the violation may be ignored by updating the check status. Thus the validation error will not be
reported any more in the next run. The status data in that case may look like:
...
"20200102000000-root-full": {
"Interval": {
"Ignore": "both"
}
},
...
This status indicates, that the both interval checks for the interval from the previous full and
incremental backup to the backup named above should be disabled. Do disable only one type of checks,
the "full" or "inc" type keyword is used instead of "both".
While above is fine to ignore singular policy violations, also the policy itself may be adjusted.
This is useful when e.g. the backup generation intervals where changed at the source. The status data
in that case could look like:
...
"20200102000000-root-full": {
"Interval": {
"Config": {
"FullMax": "29d28H",
"FullMin": "29d20H",
"IncMax": "6d28H",
"IncMin": "6d20H"
}
}
},
...
LevelRetention:
This defines a retention policy defined by retention levels, e.g. on first level keep each backup for
30 days, next level keep 12 weekly backups, on the next level keep 12 monthly backups, then 12 every
three month and from that on only yearly ones.
...
"Policies": [
{
"Sources": "^(.*/)?root$",
"List": [
{
"Name": "LevelRetention",
"Levels": [
# Keep weekly backups for 30 days, including incremental backups.
{
"KeepCount": 30,
"Interval": "day",
"TimeRef": "latest",
"KeepInc": true
},
# Keep weekly backups for 3 month, approx. 13 backups.
{
"KeepCount": 13,
"Interval": "day",
"AlignModulus": 7
},
...
{
"KeepCount": 12,
"Interval": "month",
"AlignModulus": 3,
"AlignValue": 1
},
...
This configuration defines, that on the finest level, backups for 30 days should be kept counting
from the most recent on ("TimeRef": "latest"), including incremental backups ("KeepInc": true). Thus
for machines not producing backups any more, the most recent ones are kept unconditionally.
On the next level, 13 weekly backups are kept, while may overlap with backups already kept due to the
first level configuration from above. But here only full backups will be kept, that were generated
after every 7th day due to "AlignModulus", preferring the one generated on day 0.
At another level, only one backup is kept every three month, preferring the one from the month
numbered 1, 4, 7, 10 due to "AlignModulus" and "AlignValue". Hence the first backup in January,
April, ... should be kept.
Size:
This policy checks that backup data sizes are as expected as size changes may indicate problems, e.g.
a size increase due to archives, database dumps, local file backups ... forgotten by the
administrator (thus wasting backup space but sometimes also causing security issues due to lack of as
strict access permissions on those files compared to their source), size increase due to rampant
processes filling up database tables or log files in retry loops (also monitoring should catch that),
core dumps accumulating, ...
A "Size" policy can be defined for both full and incremental backups. For each backup type, the
accepted size range can be defined by absolute or relative values. Without providing an expected
size, the size of the first backup of that type seen is used. Therefore for servers without
accumulating data, following policy could be defined:
...
"Policies": [
{
"Sources": "^(.*/)?root$",
"List": [
{
"Name": "Size",
"Priority": 0,
"FullSizeMinRel": 0.9,
"FullSizeMaxRel": 1.1,
"IncSizeMin": 100000,
"IncSizeMaxRel": 10.0
}, {
...
This configuration will check sizes of "root" backups using the first full and incremental size as
reference. Full backups may vary in size between 90% and 110% while incremental backups have to be at
least 100kb large but may vary 10-fold in size. All supported policy parameters are:
FullSizeExpect [size (int)]
Specify the expected full backup size. When missing the size of first full backup seen is used as
default.
FullSizeMax [size (int)]
Specify the absolute maximum backup size. You cannot use "FullSizeMaxRel" at the same time.
FullSizeMin [size (int)]
Specify the absolute minimum backup size. You cannot use "FullSizeMinRel" at the same time.
IncSizeExpect, IncSizeMax, IncSizeMin [size (int)]
Same as "Full..." parameters just for incremental backups. See above.
FullSizeMaxRel [ratio (float)]
Specify the maximum backup size in relation to the expected size (see "FullSizeExpect"). You
cannot use "FullSizeMax" at the same time.
FullSizeMinRel [ratio (float)]
Specify the minimum backup size in relation to the expected size (see "FullSizeExpect"). You
cannot use "FullSizeMin" at the same time.
IncSizeExpectRel [ratio (float)]
Specify the expected incremental backup size in relation to the expected full backup size (see
"FullSizeExpect"). You cannot use "IncSizeExpect" at the same time.
IncSizeMaxRel, IncSizeMinRel [ratio (float)]
Same as "Full..." parameters just for incremental backups. See above.
When policy validation fails for a given source, the policy configuration may be adjusted but also
the violation may be ignored by updating the check status. Thus the validation error will not be
reported any more in the next run. The status data in that case may look like:
...
"20200102000000-root-full": {
"Size": {
"Ignore": true
}
},
...
While above is fine to ignore singular policy violations, also the policy itself may be adjusted.
This is useful when e.g. the size of backups changed due to installing of new software or services.
The updated policy configuration can then be attached to the first element it should apply to:
...
"20200102000000-root-full": {
"Size": {
"Config": {
"FullSizeExpect": 234567890,
"FullSizeMinRel": 0.9,
"FullSizeMaxRel": 1.1,
"IncSizeMin": 100000,
"IncSizeMaxRel": 10.0
}
}
},
...
FILES
/etc/guerillabackup/storage-tool-config.json
The default configuration file for gb-storage-tool tool. Use storage-tool-config.json.template to
create it. The template also contains the documentation for each available parameter. The most
relevant parameters for gb-storage-tool are DataDir, Include and Status.
/var/lib/guerillabackup/state/storage-tool-status.json
This is the recommended location for the toplevel gb-storage-tool status file. The file has to
contain valid JSON data but also comment lines starting with #. See POLICIES section above for
description of policy specific status data.
BUGS
For guerillabackup setups installed from packages, e.g. .deb or .rpm files usually installed via package
management software, e.g. apt-get, aptitude, rpm, yast, please report bugs to the package maintainer.
For setups from unpackaged software trunk, please report at
https://github.com/halfdog/guerillabackup/issues.
SEE ALSO
gb-transfer-service(1)
AUTHOR
halfdog <me@halfdog.net>
Wrote this manual page.
COPYRIGHT
Copyright © 2022-2023 halfdog
This manual page was written for guerillabackup system on Linux systems, e.g. Debian.
Permission is granted to copy, distribute and/or modify this document under the terms of the Lesser GNU
General Public License, Version 3.
On Debian systems, the complete text of the Lesser GNU General Public License can be found in
/usr/share/common-licenses/LGPL-3.
guerillabackup 09/15/2024 GB-STORAGE-TOOL(1)