NAME

       ganesha-rados-grace - manipulate the shared grace management database

SYNOPSIS

       ganesha-rados-grace [ --cephconf /path/to/ceph.conf ] [--ns namespace] [ --oid obj_id ] [ --pool pool_id ] [ --userid cephuser ] dump|add|start|join|lift|remove|enforce|noenforce|member [ nodeid ... ]

DESCRIPTION

       This tool allows the administrator to directly manipulate the database used by the rados_cluster recovery
       backend.  Cluster  nodes  use  that  database  to  indicate  their current state in order to coordinate a
       cluster-wide grace period.

       The first argument should be a  command  to  execute  against  the  database.   Any  remaining  arguments
       represent the nodeids of nodes in the cluster that should be acted upon.

       Most  commands will just fail if the grace database is not present. The exception to this rule is the add
       command which will create the pool, database and namespace if they do not already exist.

       Note that this program does not consult ganesha.conf.  If  you  use  non-default  values  for  ceph_conf,
       userid,  grace_oid, namespace or pool in your RADOS_KV config block, then they will need to passed in via
       command-line options.

OPTIONS

       --cephconf

       Specify the ceph.conf configuration that should be used (default is to use the normal search path to find
       one)

       --ns

       Set the RADOS namespace to use within the pool (default is NULL)

       --oid

       Set the object id of the grace database RADOS object (default is "grace")

       --pool

       Set the RADOS poolid in which the grace database object resides (default is "nfs-ganesha")

       --userid

       Set the cephx user ID to use when contacting the cluster (default is NULL)

COMMANDS

       dump

       Dump the current status of the grace period database to stdout. This will show the current  and  recovery
       epoch serial numbers, as well as a list of hosts currently in the cluster and what flags they have set in
       their individual records.

       add

       Add  the  specified  hosts  to the cluster. This must be done before the given hosts can take part in the
       cluster. Attempts to modify the database by cluster hosts that have not yet  been  added  will  generally
       fail.  New  hosts  are  added with the enforcing flag set, as they are unable to hand out new state until
       their own grace period has been lifted.

       start

       Start a new grace period. This will begin a new grace period in the cluster if one is not already  active
       and  set  the  record for the listed cluster hosts as both needing a grace period and enforcing the grace
       period. If a grace period is already active, then this is equivalent to join.

       join

       Attempt to join an existing grace period. This works like start, but only if there is already an existing
       grace period in force.

       lift

       Attempt to lift the current grace period. This will clear the need grace flags for the listed  hosts.  If
       there  are no more hosts in the cluster that require a grace period, then it will be fully lifted and the
       cluster will transition to normal operations.

       remove

       Remove one or more existing hosts from the cluster. This will remove the  listed  hosts  from  the  grace
       database, possibly lifting the current grace period if there are no more hosts that need one.

       enforce

       Set  the  flag for the given hosts that indicates that they are currently enforcing the grace period; not
       allowing the acquisition of new state by clients.

       noenforce

       Clear the enforcing flag for the given hosts, meaning that  those  hosts  are  now  allowing  clients  to
       acquire new state.

       member

       Test  whether  the  given  hosts are members of the cluster. Returns an error if any of the hosts are not
       present in the grace db omap.

FLAGS

       When the dump command is issued, ganesha-rados-grace will display a list of all of the nodes in the grace
       database, and any flags they have set.  The flags are as follows:

       E (Enforcing)

       The node is currently enforcing the grace period by rejecting requests from clients to acquire new state.

       N (Need Grace)

       The node currently requires a grace period. Generally, this means that the node has clients that need  to
       perform recovery.

NODEID ASSIGNMENT

       Each  running  ganesha daemon requires a nodeid string that is unique within the cluster. This can be any
       value as ganesha treats it as an opaque string. By default, the ganesha daemon will use the  hostname  of
       the node where it is running.

       This  may  not  be  suitable  when  running under certain HA clustering infrastructure, so it's generally
       recommended to manually assign nodeid values to the hosts in the RADOS_KV config block of ganesha.conf.

GANESHA CONFIGURATION

       The ganesha daemon will need to be configured with the RecoveryBackend set to rados_cluster. If you use a
       non-default pool, namespace or oid, nodeid then those values will need  to  be  set  accordingly  in  the
       RADOS_KV config block as well.

STARTING A NEW CLUSTER

       First,  add  the  given  cluster nodes to the grace database. Assuming that the nodes in our cluster will
       have nodeids ganesha-1 through ganesha-3:

       ganesha-rados-grace add ganesha-1 ganesha-2 ganesha-3

       Once this is done, you can start the daemons on each host and they will coordinate to start and lift  the
       grace periods as-needed.

ADDING NODES TO A RUNNING CLUSTER

       After this point, new nodes can then be added to the cluster as needed using the add command:

       ganesha-rados-grace add ganesha-4

       After  the node has been added, ganesha.nfsd can then be started. It will then request a new grace period
       as-needed.

REMOVING A NODE FROM THE CLUSTER

       To remove a node from the cluster, first unmount any clients that have that node mounted (possibly moving
       them to other servers). Then execute the remove command with the nodeids to be removed from the  cluster.
       For example:

       ganesha-rados-grace remove ganesha-4

       This  will remove the ganesha-4's record from the database, and possibly lift the current grace period if
       one is active and it was the last one to need it.

                                                  Apr 18, 2025                            GANESHA-RADOS-GRACE(8)