Ubuntu Manpage: buffindexed.conf - Configuration for the buffindexed overview method

NAME

       buffindexed.conf - Configuration for the buffindexed overview method

DESCRIPTION

buffindexed.conf, found in pathetc, specifies the buffers that the buffindexed overview method should
use. It is required if the server uses buffindexed (as configured by the ovmethod parameter in
inn.conf).

Buffindexed uses pre-built buffer files to store overview data and indexes to that data. The buffers are
divided into 8 KB internally, and a given block is used either for overview data or for index data. A
block is always allocated to a single newsgroup and is never shared among newsgroups. It also means that
overview data is limited to 8 KB per article, which may lead to the lack of integration of a few articles
with headers of unusual length into the overview database.

In addition to the buffers, buffindexed also stores information in a file named group.index in pathdb.
(This file should not be mistaken for the one named group.index in pathoverview which is used by the
tradindexed overview method.) It contains information about each newsgroup: the pointer to the index
block for the newsgroup, the high mark, the low mark, the flag of the group, the number of articles, and
so forth. This file is created automatically when all buffers are initialized and should not be manually
edited.

Buffindexed buffers are of fixed size, so buffindexed will never use more space than what is available in
those buffers. If all buffers are full, innd will throttle when it attempts to store overview
information for any additional articles until space is freed (with expireover, for instance) or another
buffer is added. This is unlike the CNFS storage method.

You can see the current usage of the buffers with the -o option to inndf.

The overview data of received articles are flushed to disk every ovflushcount articles, as set in
inn.conf.

In the buffindexed.conf file, blank lines and lines beginning with a number sign ("#") are ignored. All
other lines must be of the format:

The order of lines is not significant.

<index> is the index of this overview buffer and must be unique. Other than that constraint, it can be
any number between 0 and 65535.

<filename> is the path to the buffer. The length of the path should not be longer than 63 characters.

<size> is the length of the buffer in kilobytes (1 KB = 1024 bytes). If <filename> does not specify a
special device, the file size of the buffer must be <size> * 1024 bytes. If it does specify a special
device, that device must have at least <size> space available. For more information on setting up the
buffers, see "CREATING BUFFERS".

An example of buffindexed.conf file can be:

0:<pathoverview in inn.conf>/OV1:1536000
1:<pathoverview in inn.conf>/OV2:1536000

When you first start innd with everything configured properly, you should see messages like this in
pathlog/news.notice:

Aug 27 00:00:00 kevlar innd: buffindexed: no magic cookie found
for ovbuff 0, initializing

You MUST recreate overview completely using makehistory if you remove or replace buffers. However, new
buffers can be added without any special care (other than restarting innd after modifying
buffindexed.conf). If you need to rebuild overview, you should zero all of the buffers first.

We recommend not to reserve too much spare space in existing buffers, so that to minimize the duration of
the expireover process, and to just add new buffers when space left is low (see the result of "inndf
-no"). Plan on needing at least 0.65 KB for every article in your spool (not counting crossposts). So,
if you have 5 million articles, you'll need at least 3.25 GB of disk space for buffindexed.

CREATING BUFFERS

       There are two methods to create a new buffindexed buffer:

       1.  Create  a  large  file  on top of a regular file system.  The easiest way to do this is probably with
           dd(1), using a command like:

               dd if=/dev/zero of=/path/to/cycbuff bs=1024 count=<size>

           where <size> is the size from the relevant line in buffindexed.conf.

           This is the simplest method, but has the disadvantage that very large files on regular  file  systems
           can  be  fairly  slow to access, particularly at the end of the file, and INN incurs unnecessary file
           system overhead when accessing the buffer.

       2.  Use block devices directly.  If your operating system allows you to  call  mmap()  on  block  devices
           (Solaris and recent versions of Linux do, FreeBSD at last report does not), this method can avoid all
           of the native file system overhead.  Note, however, that Solaris has problems with byte range locking
           on block devices, and therefore this method should not be used on Solaris.

           Partition  the  disk.  If you're using Solaris, set up your partitions to avoid the first cylinder of
           the disk (or otherwise the buffindexed header will overwrite the disk partition table and render  the
           buffers inaccessible).  Then, create device files for each block device you're going to use.

           It's  not  recommended  to  use  the  block  device files in /dev, since the news system doesn't have
           permission to write to them and changing the permissions  of  the  system  device  files  may  affect
           something  else.   Instead,  use  mknod(1)  to  create  a new set of block devices (in somewhere like
           pathspool/overview that's only writable by the news user).  To do this, run "ls -Ll" on  the  devices
           in  /dev  that  correspond  to  the  block  devices that you want to use.  The major and minor device
           numbers are in the fifth and sixth columns (right before the date),  respectively.   Then  run  mknod
           like:

               mknod <filename> b <major> <minor>

           where <filename> is the path to the device to create (matching the <filename> part of the buffindexed
           configuration  line)  and  <major>  and  <minor> are the major and minor device numbers as discovered
           above.

           Here's a short script to do this when given the path to the system device file as an argument:

               #!/bin/sh
               base=`echo "$1" | sed 's%.*/%%'`
               major=`ls -Ll "$1" | awk '{print $5}' | tr -d ,`
               minor=`ls -Ll "$1" | awk '{print $6}`
               mkdir -p <pathoverview in inn.conf>
               mknod <pathoverview>/"$base" b "$major" "$minor"
               chown news:news <pathoverview>/"$base"
               chmod 644 <pathoverview>/"$base"

           Make sure that the created files are owned by the news user and news group, as specified at configure
           time (the default being "news" for both).  Also make sure that the permissions on the  devices  allow
           the news user to read and write.

HISTORY

       Written  by  Katsuhiro  Kondou  <kondou@nec.co.jp>  for  InterNetNews.   Converted to POD by Russ Allbery
       <eagle@eyrie.org>.

NAME

DESCRIPTION

CREATING BUFFERS

HISTORY

SEE ALSO