The Big Brother Systems and Network Monitor ------------------------------------------------------------------------ Installation and Configuration Manual Version 1.4h - Jun 07th 2000 (BETA) ------------------------------------------------------------------------ 1.0 Quick and Dirty Install for Big Brother 1.1 Installing Big Brother 1.2 Upgrading BB 2.0 Configuring the BB hosts file 2.1 The etc/bb-hosts file 2.2 Basic format of etc/bb-hosts 2.3 etc/bb-hosts special directives 3.0 Customizing your BB installation 3.1 The runbb.sh script 3.2 Configuring the etc/bbdef.sh file 3.3 Localizing programs paths 3.4 Creating BB clients 3.5 Configuring the Web Stuff 3.6 History 3.7 Adding your own tests 3.8 Starting BB at system boot 4.0 Configuring the Notification feature 4.1 A Little History 4.2 Setting up notification 4.3 Creating notification rules 4.4 Acknowledging notification messages 4.5 Manual notification (Web based) 4.6 SMS notification 4.7 Using kermit 4.8 Adding a custom notification 4.9 Notification/acknowledgement logs 4.10 Disabling notifications temporarely 5.0 Customizing your Big Brother display 5.1 HTMLized status logs 5.2 headers / footers 5.3 Creating skins 5.4 Inserting notes about your hosts 5.5 Extending the contents of the main HTML pages 5.6 Extending the contents of the history HTML pages 5.7 Working with historical logs 5.8 Hiding the bb.html/bb2.html page header 6.0 Miscellaneous Big Brother configuration options 6.1 Specifying filesystem specific values in the disk test 6.2 Enabling security 6.3 Using bbnet to test TCP services 6.4 Sending custom one line status with bb 6.5 Changing the 'purple' interval 6.6 Specifying processes to look for 6.7 Sending arbritary data to a BBDISPLAY(s) 6.8 Disabling the messages file(s) zero-length test 6.9 Setting a Time To Live to a status message 6.10 Disabling the connectivity test 6.11 Reducing connectivity errors 6.12 Setting up BBPAGERs and BBNETs host in failover mode 6.13 Speeding up the network tests 7.0 Other documentation sources ------------------------------------------------------------------------ This Installation and Configuration Guide is Copyright 1997-2000 by The MacLawran Group Inc. This document may be reproduced, so long as it is kept in its entirety and in its original format. ------------------------------------------------------------------------ ------------------------------------------------------------------------ Section 1: Quick and Dirty Install for Big Brother 1.1 Installing Big Brother WARNING: For security reasons, it is best to install and run BB as its own user and not as root. *** READ THE README.SECURITY FILE BEFORE PROCEEDING *** From the directory you extracted the BB archive: 1. cd ./install, and run ./bbconfig where OS-NAME is bsdi sco3 sco freebsd solaris hpux9 hpux linux sunos netbsd osf ultrix irix unixware redhat aix dynix debian dgux Note that is optional, BB will try to figure it out. If you are running Linux you may have to provide the proper distribution name. If you're not running one of the above, read the install/README ./bbconfig will ask you questions about your setup be ready with: BBHOME: directory where BB resides (usually the choice given by BB is correct) If you intend to use FQDN(Fully qualified domain names) Which host(s) is(are) the BBDISPLAY(s) Which host(s) is(are) the BBPAGER(s) If the current host is a BBDISPLAY/BBPAGER Default e-mail recipient for notification URL you intend to view BB with URL of the BB CGI scripts 2. cd ../src type "make" then type "make install" If you have trouble compiling, refer to the README file located at src/README for those who are upgrading then go to "upgrading BB" section 3. cd ../etc edit bb-hosts, put your hosts names in there. Refer to install/README. This is the core of Big Brother. You must read the docs here. edit bbdef.sh, set alarm levels and things. If you want to use fully qualified domain name hosts then make sure you set FQDN=TRUE in etc/bbdef.sh. If this is the first host you install and intend to use it as the display/notification server as well as the server that tests the network services then your bb-hosts should contain this line: xxx.xxx.xxx.xxx this.host.name # BBDISPLAY BBPAGER BBNET 4. ./bbchkcfg.sh Checks the bbdef.sh/bbinc.sh/bbsys.sh source scripts for invalid entries ./bbchkhosts.sh Checks the bb-hosts file for errors 5. cd ../.. ln -s bbdir bb where bbdir is the new version directory (e.g. ln -s bb13a bb) This is useful as you don't have to change the directory in your startup script. (see section 11) cd bb chown -R bbuser . where bbuser is the user you defined in the install process. This makes sure that the bbuser can write/read into the BB directory structure as you will probably install it while in the root account. 6. ln -s /full_path_to_bb/www /WWW/bb (where /WWW is the Document Root dir). Make sure the permissions are correct. Configure your web server for this directory if need be. Also make sure that your web server follows symbolic links 7. cd bbdir where bbdir is the new version directory (e.g. bb13a) or cd bb if you've followed section 5 ./runbb.sh start examine the BBOUT file for any errors NB The HTML summary pages (bb.html/bb2.html) should be available 2 minutes after the startup. Don't panic the'll be there if you are patient. In your browser, you should be able to see the results at http://yourwebhost/bb/ or http://yourwebhost/bb/bb.html (assuming you used /bb as BB URL root) 8. Debug, and look at all the docs. 9. Check the online documentation! It lives under: http://bb4.com/bb/bb-help.html 10. This will have enabled you to have a display/pager server set up. At this point no clients are running. When you have all of your hosts defined in etc/bb-hosts then use install/bbclient to create a tarball for BB clients of the same OS/HW type. If you have different OS/HW platforms then reinstall BB on each one (then use the bbclient to create a tarball for each identical OS/HW client) and don't forget to copy your master bb-hosts file to it. Run through the install procedure to make sure that the clients are also installed properly(bbchkcfg.sh/bbchkhosts.cfg) 11. If you wish to start BB automatically at startup, we suggest the following command in your startup script: su - <bbuser> -c "cd <BBHOME>;./runbb.sh start" <bbuser> is the user you chose at install <BBHOME> is the directory in which you installed BB or where you've created the bb directory link as per section 5. If you're new to BB, we suggest that you subscribe to the BB mailing list. To do so, follow these instructions: Send an e-mail to MajorDomo@bb4.com In the body of the e-mail message (not the Subject line), place the statement: subscribe bb Nick Silberstein has made an archive of the Big Brother mailing list available. It can be found at the URL: http://www.tpdinc.com/~bb/ When you have finished installing BB on all of your monitored hosts, you might want to check out the FTP archive that contains many contributed scripts. Check it out at ftp://ftp.deadcat.net/pub/BB 1.2 Upgrading BB If you are upgrading from a previous version then do these steps after the preliminary install of BB: cd where is the directory of the new install cd www rm -r logs acks hist histlogs notes data # Replace OLDBBHOME by the full path of BBHOME defined in your previous install mv OLDBBHOME/www/logs . mv OLDBBHOME/www/acks . mv OLDBBHOME/www/hist . mv OLDBBHOME/www/histlogs . mv OLDBBHOME/www/notes . mv OLDBBHOME/www/data . cd .. cd ext If you have external scripts cp OLDBBHOME/ext/* . cp OLDBBHOME/ext/hist/* hist/ cp OLDBBHOME/ext/mkbb/* mkbb/ cd .. cd disabled If you currently have any disabled entries cp OLDBBHOME/disabled/* . cd .. vi runbb.sh Change the BBEXT variable to contain your external scripts cd etc cp OLDBBHOME/etc/bb-hosts . Depending of your current setup cp OLDBBHOME/etc/bbwarnrules.cfg . copy and edit appropriate config files cp OLDBBHOME/etc/bbwarnsetup.cfg . cp OLDBBHOME/etc/bb-dftab . cp OLDBBHOME/etc/bb-proctab . cp OLDBBHOME/etc/security . vi bbdef.sh Set environment variables for your setup vi bbinc.sh It is always useful to run a diff getween your old version of bbdef.sh/bbinc.sh to view your own changes cd ../.. if you already have a link called bb used to access your BB install rm bb ln -s bb Setup bb as a link to the new bbdir directory such that it's easier to refer rc scripts to bb/ without having to change them every time you upgrade. ------------------------------------------------------------------------ Section 2: Configuring the Big Brother hosts file 2.1 The etc/bb-hosts file The etc/bb-hosts file controls where Big Brother looks for things and the actions that are taken. The format is identical to the standard /etc/hosts file, except with additional directives for Big Brother. 2.2 Basic format of etc/bb-hosts Lines are of the format: IP-ADDR HOSTNAME # DIRECTIVES IP-ADDR: XXX.XXX.XXX.XXX HOSTNAME: host.domain.com if FQDN="TRUE" host if FQDN="" FQDN (display with Fully Qualified Domain Name) is a variable that you set in the etc/bbdef.sh file. Directives the Big Brother knows about are: BBDISPLAY This host displays the HTML results You need a Web server on this host BBPAGER This host acts as the notification server You can have multiple BBDISPLAYs & BBPAGERs. Just put the BBDISPLAY/BBPAGER directive on the config line for each host that acts as BBDISPLAY/BBPAGER. BBNET Indicates that this host checks the ip network services Many hosts can act as a BBNET. If you do have more than one BBNET, make sure that the bb-hosts file only contains the hosts that you will test against. You must be very careful because if a host is tested by many BBNETs then the status display for the services on that host will only show that last test status. You may get errors for that host even though it shows green, it's just that another BBNET send a status message afterwards and that message returned OK. On the other hand you may want to have tests from different location :) http://www_path The host is tested for http connections using the www_path https://www_path The host is tested for https connections using the www_path (requires lynx: http://lynx.browser.org/ or curl: http://curl.haxx.nu/ ) Note that you can specify multiple URLs by joining the URLs with '|': http://www_path|http://www_path1 or by specifying them individually: http://www_path http://www_path1 ftp Test for ftp service smtp Test for smtp server pop3 Test host for pop3 server telnet Check telnet service ssh Test ssh server nntp Test nntp news server Check /etc/services for proper service name especially pop3 (sometimes referred as pop-3). These directives names MUST appear in /etc/services Any text based service can be checked by putting the /etc/services name as the directive and adding in the list of services in bb-network.sh The '!' can be prefixed to a service such it will be checked that the service is NOT running (!ftp). dns Checks for name resolution server noping Don't do ping test for this host noconn Don't do ping test for this host and don't generate a clear dot dialup If host is down then display clear button otherwise display a green status You can add your own directives that you access through an external test. Known directives from user contributed tests: oracle/fping/trap/sybase/... These contributions can be found on the FTP archive. This is a sample bb-hosts file: # # BIG BROTHER HOSTS FILE # 192.168.110.102 bobo # BBDISPLAY BBPAGER ftp smtp pop3 192.168.110.95 admin # http://admin/ 192.168.110.108 mynet # BBNET nntp! bobo is the display and notification server. mynet is the host that runs ip network services checks it will check the ftp/smtp/pop3 services on bobo, then the http service on admin and finally it makes sure that nntp is NOT running on mynet You can also group hosts within a seperate HTML table. There are three group tags defined: group, group-compress and group-only. Results can also be saved to a specific page using the 'page' directive. It is best to have the master bb-hosts file identical on all BB server and client hosts but it is not required (unless you have multiple BBNETs, see previous comment) Changes to the etc/bb-hosts file are automatic. You do not have to restart BB to initialize the changes. 2.3 etc/bb-hosts special directives 2.3.1 Grouping on the display The Web display may be broken into tables to create a more aesthetic and sensible display. It's also much faster to load small tables, than one giant table. So consider grouping your etc/bb-host entries logically and separating them using the "group" , "group-compress" or "group-only" directives. group Intranet Servers The group directive defines a block of hosts to be grouped in the same HTML table. All hosts lines following the group directive, until a new group/group-compress is defined, belong to that group. The text that follows the directive is the title given to the HTML table. Note that you can embed HTML code in the title: Italic / H3, but use with caution. group-compress California Servers The group-compress is identical to the "group" directive except it will only display services (columns) containing data for that group. group-only conn|cpu|disk Restricted Services The group-only will create a table with only the colums defined in the directive. The columns are '|' delimited. 2.3.2 DHCP hosts (no fixed IP address) 0.0.0.0 dhcphost.domain # The 0.0.0.0 indicates that this host is a DHCP host and that the BBNET host will not try to run the connection test on that host. 2.3.3 Modem Banks dialup modem-bank 204.19.116.20 4 The dialup directive (not to be confused with the other dialup tag which displays a clear button if the host is down) is used to specify connectivity for a bank of modems. The 2nd parameter is the name to be displayed on the display page. The 3rd argument is the starting IP address of the modem bank. The last argument is the number of modems on that bank. 2.3.4 Summary Lines summary quebec.mtl 255.255.255.255 http://cafe.domain.com/bb/ This indicates to the BBDISPLAY machine that summary information about the state of this display is to be forwarded to the IP address noted on the line. The summary can be sent to more than one parent machine. You can send the generated bb.html/bb2.html or subpages created with the 'page' directive to any other BBDISPLAY host(s). 2.3.5 Display results in an HTML subpage page nyrouters New York Routers BB always creates two HTML pages: bb.html/bb2.html. When the page directive is encountered, BB will then create a new HTML subpage until another page directive is found and will save the output from that point on to that new subpage. It will keep a pointer of the new sub page in bb.html. The first argument is the directive, the second one is the name of the page (i.e. nyrouters.html) and the remaining arguments is the caption that will appear in bb.html when it creates the link to that subpage. Always place your page directives after all hosts that you want to appear in the bb.html page. When you use a page directive then all subsequent ouput will be in a subpage. N.B. summaries and dialup directives will always appear on bb.html/b2.html and not in any HTML subpages ------------------------------------------------------------------------ Section 3: Customizing your BB installation 3.1 The runbb.sh script The runbb.sh script is the master BB script. It sets up the BB environment with the values from the configuration files. You MUST set the BBHOME variable to the directory path where you have installed BB. Without it, BB will not run properly. If you create external monitoring agents then you can copy them to the $BBHOME/ext directory and define them in the BBEXT variable. cd $BBHOME (to your BB install directory) To start BB: ./runbb.sh or ./runbb.sh start To stop BB: ./runbb.sh stop To restart BB: ./runbb.sh restart When BB starts/restarts it checks that there is no processes already running. If this is the case, BB will complain but will continue to start as it cannot determine if the process was really started by BB. BB checks for any processes that contain the BBHOME in its definition. If you ever have to add a script that runs as a deamon use the following template to add it in runbb.sh: echo "Starting snmptrapd" echo "Starting snmptrapd" >> BBOUT MIBFILE=$BBHOME/cmu/etc/mib.txt export MIBFILE $BBHOME/cmu/bin/snmptrapd -B | $BBHOME/bin/bb $BBDISP - & set `$PS | $GREP snmptrapd | $GREP -v "$GREP"` > /dev/null BBPID="$BBPID $1" This is from an SNMP trapd extension available at the FTP archive site (ftp://ftp.deadcat.net/pub/BB). You'll notice that the PID is added to $BBPID, This way, you'll be able to stop the extension at the same time as BB when executing "runbb.sh stop". 3.2 Configuring the etc/bbdef.sh file etc/bbdef.sh is where you configure how Big Brother reacts to a variety of situations that may arise. You can establish levels for warnings(yellow), panics(red) values. Please note that any modifications to this file do not come into effect until the next restart of BB. It is best to stop/start BB immediately after this file has been modified. Use FQDN (fully qualified domain naming) FQDN="" - Host names rendered on the BB status pages will only include the host name, not including the domain name. FQDN="TRUE" - Host names rendered on the BB status pages will include the host name and the domain name. Which program to use to check the HTTP connectivity: LYNX="$BBHOME/bin/bbnet" - This indicates where bbnet (a C program that checks the IP services) is located. LYNX="/usr/local/bin/lynx -dump -head" - Use lynx if you need to check password protected pages. Please use the proper directory path as per your host installation. NONETPAGE defines which IP services not to notify on if an error is encountered. Include the services as they are named in the etc/bb-hosts file. This is only valid on a host defined as a BBNET host. BBTMP indicates the location of the BB temporary directory. You may want to locate it outside of the BB structure. The DFWARN and DFPANIC variables contain the % levels at which the disk test will fail. By default DFWARN is 90% and DFPANIC is 95%. These values can be overriden by values in the etc/bb-dftab file described in section 5.2. CPUWARN and CPUPANIC are default levels based on the load average. CPUWARN is set at 150 and CPUPANIC at 300. These values are the load average (from uptime) multiplied by 100. You can define the processes for which you want to monitor if they are up and running. You do this by setting the PROCS and PAGEPROC variables. The processes included in PROCS will lead to a yellow condition if one or more the processes is down. Processes in PAGEPROC will generate a red condition if a process is down. MSGS and PAGEMSGS are used to monitor error messages in the system logs. MSGS contain keywords to look for in the system logs. If a keyword is found then this will create a yellow condition and if the keyword is in the PAGEMSGS variable then it will be upgraded to a red condition. You can also discard certain messages by defined portions of that message in the IGNMSGS variable. You can define multiple messages by delimiting each message by a ';'. BBWEB="http://bb4.com/bb/" This variable defines the path to the root page of BB. Make sures it always ends with "/". This value is used in a tag and it doesn't like it with the "/" BBWEBHTMLLOGS="${BBWEB}html/" This variable defines the path to the HTMLized status logs. This path is concatenated with the name of the HTMLized status file and is appended to the e-mail notification message for a quickpath to the error information. The PAGELEVELS variable indicates to the 'bb' process at which level(s) it should send a notification message. The 'bb' program is used to send status messages and if the status level of the message is found in the PAGELEVELS variable then a notification message is also sent to the BBPAGER host. This is the default value. It can be overridden by the 'pagelevels:' token in the etc/bbwarnsetup.cfg file. The PAGELEVELSMAIL contains the color level(s) on which only e-mail is to be sent. The color level must also be found in PAGELEVELS. The WEBHISTORY defines if you want to use the web-based history display script. This script gives you a 24hr HTML chart and the last 50 status changes. The SAVESTATUSLOG variable defines if the historical log of the host.service is kept or not. The logs are saved in www/histlogs. The BBLOGSTATUS variable determines if each log received is rendered into a static HTML page, dynamically by CGI or not at all (it appears as plain text). If it is set to DYNAMIC then a CGI will create the HTMLized status log only when requested. That's when you click on a colored dot on the BB display pages. The dynamic CGI is called bb-hostsvc.sh. If is set to DYNAMIC then a static HTML status log will be generated the www/html directory. If it is set to TEXT then no HTMLized status logs are available. The status will appear as plain text. NOPINGCLEAR lets you decide if you want clear dots for host that have the noping directive. If you set to TRUE then clear dots will be displayed for hosts with the noping as well as hosts with IP address 0.0.0.0 (DHCP hosts or hosts accessed only with its full name). You can use the noconn directive in conjunction with NOPINGCLEAR to select which hosts you don't want a clear dot. The IPTEST_2_CLEAR_ON_FAILED_CONN variable, if set to TRUE, defines that if a network test fails and that the connectivity (ping) test also failed then display the test results as clear. You can turn off this setting individually by test by using the ~ modifier. in etc/bbdef.sh: IPTEST_2_CLEAR_ON_FAILED_CONN=TRUE in etc/bb-hosts 123.123.123.123.host1 # ftp If the connectivity test to 123.123.123.123 failed and the ftp test also failed (very likely) then display the test with a clear dot. If you had set the directive to ~ftp then it would have been displayed as red. 3.2 Localizing programs paths Sometimes the OS specific programs paths found in the installation might be wrong for a particular host(s) OS. Change the paths in the etc/bbsys.local file. The variables contained in this file all have default values from the bbsys.OS file. The original bbsys.local is copied from the etc/bbsys.OS file during installation (bbconfig); so keep track of your changes as they will automatically be overwritten if you re-run bbconfig. 3.4 Creating BB clients Creating BB clients is very simple: cd $BBHOME cd install ./bbclient <client-hostname> <client-hostname> must be found in the etc/bb-hosts file. This will create a tarball for the client that has the same OS/HW platform type as the host on which the bblient command is executed. The tarball is found in the directory above $BBHOME. Don't forget to configure the PROCS and PAGEPROC variables in etc/bbdef.sh on the client after you've untarred the archive such that the correct processes are monitored. If you have different OS/HW platforms then install BB, as a client, on each OS/HW platform (then use the bbclient to create a tarball for each identical OS/HW client) and don't forget to copy your master bb-hosts file to it. Run through the install procedure to make sure that the clients are also installed properly (bbchkcfg.sh/bbchkhosts.cfg). 3.5 Configuring the Web Stuff All Web-based things live in the $BBHOME/www directory. In order for the bb stuff to work correctly, this directory must be linked into your Web site somewhere. I suggest something like the following: ln -s /home/sean/bb/www /WWW/bb (where /WWW is the Document Root dir of your web server). You should then be able to access BB with the URL http://.../bb/bb.html where ... is your webserver address. You also have to configure the script which generates the web-based history page if you set the WEBHISTORY variable to TRUE in etc/bbdef.sh. This page is accessible from the "History" button when you click on a colored dot in one of the display pages (bb.html/bb2.html). You must copy the web/bb-hist.sh script to your /cgi-bin directory and set the BBHOME. Make sure the script has the proper permissions to execute: system and web server permissions. If you want only a text based history display then set the WEBHISTORY variable to "FALSE" in etc/bbdef.sh. Set the BBHIST_IGNOREBLUE variable to TRUE in etc/bbdef.sh if you don't want the blue state to be used in the calculations of daily % use. By default the CGIBINURL (in etc/bbdef.sh) points to /cgi-bin but if your web server cofiguration has a different CGI directory then change it here or your history feature won't work. If you intend to enable manual notification/acknowledgement ("PAGE/ACK"button in bb.html/bb2.html) then copy the web/bb-ack.sh script to you /cgi-bin directory and set the BBHOME and BBWEB variables in the script. Make sure the script has the proper permissions to execute: system and web server permissions. Acknowledgments generate a log entry in acks/acklog file. 3.6 History Each time a status log is received it is checked against its previous status to see if it has changed. If it has, then the duration of the previous status is recorded and a new line is appended to the history file with the timestamp of the new status. There's a history file for each host.service (status log) received. The history files are located in the www/hist directory. Here's a sample: Sun Jan 24 18:30:45 1999 green 917220645 232068 Wed Jan 27 10:58:33 1999 red 917452713 1560 Wed Jan 27 11:24:33 1999 green 917454273 Notice that last line doesn't have a duration because it is the current status. The duration will be appended only when the status changes. If you do not wish to have historical data then rename www/hist to something else or remove it. 3.7 Adding your own tests You can easily add your own tests. Start with the template available at ext/template, add your code. Look at bb-local.sh, bb-network.sh for an example on how to send data to BB. Then in runbb.sh, specify the name of your script in the BBEXT variable. Restart BB and your test should be running. But before you use it within BB, I suggest you test it for errors by using the method: cd /home/bb (or wherever your BB is located) BBHOME=/home/bb export BBHOME . ./etc/bbdef.sh cd ext ./yourexternaltest Look for errors, fix them, rerun your test until you're satisfied, then update runbb.sh. Note that all temporary files should be created in $BBTMP and make sure you remove them after use. Also, remember that you don't have to deal with sending notification messages, the 'bb' process send a 'page' type message to the BBPAGER host when the status color is found in the PAGELEVELS variable defined in bbdef.sh. When it's ready don't forget to update the svcerrlist token in the bbwarnsetup.cfg file on your BBPAGER host. You must assign a numeric code to your column name. You can also set the lifetime of the status sent by your script by following the instructions described in section 6.8: "Setting a Time To Live to a status message". 3.8 Starting BB at system boot Depending on U*X operation system version (Linux/BSD/Solaris/...) your startup procedures will vary from OS to OS. In short, you have to create a startup script that has a start/stop capabilities. Under a lot of OSes, you'll want to create your script in the init.d directory and create an S89bb link in rc3.d that links to init.d/S89bb (you may also want to create a K11bb link in rc3.d that'll be used when the system shuts down). Use an existing startup script has an example and substitute with these commands: To start BB, use this command: su - -c "cd ;./runbb.sh start" or su - -c "cd ;./runbb.sh restart" To stop BB, use this command: su - -c "cd ;./runbb.sh stop" is the user that BB will execute has make sure that bbuser has all permissions under BBHOME is the location of your BB install e.g. su - bb -c "cd /home/bb;./runbb.sh start" ------------------------------------------------------------------------ Section 4: Configuring the Notification feature (BBWARN) 4.1 A little History In the Beginning :), BB could only e-mail and notify by pager. The configuration required that all BB hosts (servers and clients) have the PAGER variable set with the recipients of the notifications. There were no possibilities of customizing the notification feature. This led to the: The Pager Protection Act of 1997 ================================ Use of BB has been linked to the untimely death of at least one pager in Texas. In order to protect pagers from their users, a notification scheme has been created to allow to specify a delay between the notifications. But this wasn't enough. BB netizens got restless and finally a solution was delivered by Robert-Andre Croteau, and described in an article in SysAdmin magazine. It was called BBWARN and it was good. It allowed an admin to specify rules which enable him/her to get notified based on the source of the error (host) and the service involved and also based on the day/time. Different rules could be set up and different recipients could be specified depending on the characteristics of the problem (host/service/day/time). Numerous changes have been made and various configuration options are available to the BB admin(s). Notification methods You can currently get notified by email, numeric pager and SMS communication device. Users have reported custom notification by alpha-numeric pagers and other PCS devices (some are available on the FTP archive or a search of the mailing list archives will lead you to them). To use the email feature, your BBPAGER host (in bb-hosts) must have an email program. This is not a problem on UNIX: the mail (mailx) program is available. Sending a notification message to a numeric pager requires kermit and a modem. The path of kermit should be defined in the KERMIT variable of bbsys.sh/bbsys.local. 4.2 Setting up notification All of the BB notification configuration is specified in the etc/bbwarnsetup.cfg file. Here are the various options that can be set. Note that instructions are included in the configuration file as comments. All tokens in the etc/bbwarnsetup.cfg file must be defined on the BBPAGER host. ***** The 'pagelevels:' token must be defined on all BB hosts (servers & clients) because the 'bb' executable sends the notification and that is done locally on each client host. ***** bbwarn: Set to "TRUE" is you want to enable the notification feature svcerrlist: This contains a list of service/code pairs. The service name is the column found in the HTML display and the code is the value displayed on a numeric message. If you add a custom check then the column name must have a corresponding code. ignforall: Is a regular expression that is used to temporarely disable notification a host.service combinations e.g. .*.cpu|.*.msgs|host11.* Don't notify for any cpu/msgs errors and any errors concerning host11. N.B. This feature can be duplicated but the ! rule (below). pagehelpcode: Numeric code to use when a user sends a manual notification. See section 4.5. ttyline: List of modems devices. You can specify more than one. prefix: Prefix to use when dialing out suffix: Suffix to use when dialing out (like a hangup) pagelevels: At which color level to send a notification. This token MUST be set on every host (clients & servers). You must restart BB if you modify it. pagerecovered: Set to TRUE if you want to be notified when a problem has been fixed. This feature is only available when 'pagetype:' is set to EVENT. pagetype: Defines how the pager delay is handled. There are 4 choices (RCPT|EVENT|HOST|GROUP). RCPT: the recipient is not notified until pager delay expires. EVENT: the recipient is not notified for a particular host.service combination until the pager delay expires. HOST: the recipient is not notified for a particular host until the pager delay expires. GROUP: the recipient is not notified for a particular host.service within the same etc/bb-hosts group combination until the pager delay expires. pagemaster: Recpient(s) to receive an e-mail notification if a page notification could not be sent. pageaddhtmlpath: Set to TRUE if you want the HTML path of the status log to be appended to an e-mail notification. If this is set to TRUE, make sure the BBWEBHTMLLOGS variable is set correctly in etc/bbdef.sh. cfgdelim: entry delimiter in the etc/bbwarnrules.cfga briefrcpt: define the recipients that should receive a brief notifications message. hg-xxxxxxx: You can have multiple hg-xxxxxxx tokens that define a group of hosts/devices. So you can e.g. create hostgroup 'hg-routers' as an alias or shorthand of a list your routers. These tokens can then be specified in the bbwarnrules.cfg file in the host-fields columns (1st/2nd column) instead of having to type all of them individually in all rules lines. pg-yyyyyyy: You can have multiple pg-yyyyyyy tokens that define a group of recipients. These tokens are expanded in the rules lines inthe recipients definition. On the client make sure that PAGELEVELS is set (etc/bbdef.sh) and on the server, if appropriate, that PAGELEVELSMAIL(etc/bbdef.sh) is set. Refer to section 3.2 for more details. 4.3 Creating notification rules You define the notification rules in the etc/bbwarnrules.cfg file. Rules are written in the following format: hosts;exhosts;services;exservices;day;time;recipients hosts: match on these hosts (* is a wildcard for all hosts) exhosts: exclude these hosts services: match on these services (* is wildcard for all hosts) exservices: exclude these services day: 0-6 (sunday-saturday) time: 0000-2359 recipients: email address, numeric pager, sms number, qpage recipients are defined with qp- sendpage recipients are defined with sp- There's also a special format of the rule line: !hosts;exhosts;services;exservices;day;time;recipients If a rule line starts with ! the event thats matches the rule line will disable notification to any recipient defined on that rule line. If the recipients field is '*' then no notification will occur for that event. Here's an example !*;;*;;*;*;* This will in effect disable all notifications and render useless any other rule that you have defined :) !*;;*;;*;*;robert@localhost 9999999 This will remove robert@localhost and 9999999 from the list of recipients for the current event if they were defined on another rule that matched the same event. The value set by 'cfgdelim' (in bbwarnsetup.cfg) is your field delimiter. In this case the value is ';' Even though egrep regular expressions are allowed, do not use the .* construct, just use '*'. It will be replaced with .* in the regexp. It's just that * is more readable than .* Also the default 'pagedelay' value (see bbwarnsetup.cfg) which indicates how long before the next notification occurs can be overridden on the rule line for a specific recipient by appending the time value to the recipient: recipient:XX where XX is the value in minutes. For examples and complete information please refer to the etc/bbwarnrules.cfg and the etc/bbwarnsetup.cfg files. Escalation To escalate a notification, you use the following format for the recipient: recipient:^XX[-YY] XX is the initial wait before sending the notification YY is the delay for each subsequent call. If it is not specified then the 'pagedelay' value is used. An escalation can only be acknowledged by the recipient. Initial delay An initial delay can be specified when configuring a recipient. The format is: recipient:~XX[-YY] XX is the initial delay before sending the notification YY is the delay for each subsequent call. If it is not specified then the 'pagedelay' value is used. An initial delay can be reset if a recipient acknowledges a notification for all recipients of that notification (see below). There's a simple checking program that validates the contents of your bbwnarnrules.cfg file. It is located in $BBHOME/etc. cd $BBHOME/etc ./bbchkwarnrules.cfg Note that it only checks if your config file is not missing a column. 4.4 Acknowledging a notification When an admin is notified, the admin is always sent an acknowlegement tag with the message. This tag number is seven digits: the first five digits are a unique number and the last two digits are the recipient's ID for that notification (the recipient's ID can change, it is valid only for the current notification). In an e-mail notification, the ack number is in the subject and the body of the message: in the subject it is found at the beginning as in "!BB - XXXXXXX!", and in the body of the message it is at the beginning of the body delimited by [XXXXXXX]. In a numeric page, the acknowledgement is always after the BB numeric code. It is the last 7 digits of the numeric message. NOTE, if your pager doesn't support more than 21 digits, then you're out of luck, you'll be missing some digits. You could always try to shorten the ack number but that is left as an exercise for the reader ;-). If you get the first 5 digits then you can always use recipient 99 to acknowledge for all recipients but that may not fly at your site especially if you have multiple recipients. 4.5 Manual notification (Web based "PAGE/ACK" button) You can enable the manual notification feature and let the user reach you by pager/e-mail if they can't get a hold of you. If this is enabled, then you'll receive a numeric message starting with 911 (pagecode in etc/bbwarnsetup.cfg) followed by the phone number to call. Copy the web/bb-ack.sh script (if it was not copied during the installation) to your cgi-bin directory and make sure all permissions are correct. You can setup who gets called when, by adding rules to the etc/bbwarnrules.cfg file. The pseudo host to use is notify-admin*. There's an example in the config file. 4.6 SMS notification This a user custom notification scheme for SMS devices. What follows is the README file for his custom hack. Last updated: 1997-01-11 by jaclu@ibk.se This section describes the usage of a kermit script that sends SMS messages. It is used as a mail -> SMS gateway, and also for the "Big Brother" network monitor(obviously). The kermit script for SMS can be found in etc/sms.scr My operator uses NOKIA server software, hopefully you can use this also on Motorola servers, without too much hassle. Terms SMS Short Message Service SMSC Short Message Service Center CIMD Computer Interface to Message Distribution Syntax kermit sms.scr device file number [number [number [...]]] device is modem device like /dev/cua0 The file should contain the message, only first line of file will be sent, and it MUST be terminated with LF, otherwise the kermit script won't be able to extract the message. SMS messages can only be 160 chars, and since nobody would want a lengthly message on a 8 character display, I haven't bothered to handle message splitting. If message is too long, it is truncated. If one or more SMS phonenumbers can be specified, the message will be sent to each one of them. Configuration I have done this the easy way, that is I start CIMD with faked checksums, if anybody manages to generate correct checksums, please mail me. Log in to SMSC You have to set two variables in sms.scr: \%b phonenumber to you operators SMSC \%c login to access CIMD For Europolitan (Swedish operator) you can use: \%b 46708222902 \%c cimd3 # (use faked checksums) This starts the CIMD server, and from now we must ACK all responses. Don't forget to include your modem prefix to \%b!! I don't use the "prefix" token from BB, since I also use this script as a mail->SMS gateway. If you are going to use this, you must anyhow configure this script, so I hope you don't mind. Start a CIMD session The first step in a CIMD session is to identify yourself You have to set two variables in sms.scr: \%d CIMD account \%e CIMD password Unfourtunately, I cant give you theese values ;) Now you have configured your script and can start to use it! Send to all recipients BB calls the script one time for each recipient, that's a bit of a waste, but it would just be to much work to get around. If you call it manually or from sendmail, you can use multiple recipients. More info The (english) spec of NOKIAS CIMD implementation can be found on: http://www.europolitan.se/europolitan/fick/tjanst/cimdspec.htm If you happen to know swedish, you can read more on Europolitans implementation on: http://www.europolitan.se/europolitan/fick/tjanst/faqcimd.htm Feedback Please let me know if you use this with a non NOKIA server, or if you fix real checksums! Jacob Lundqvist (SMS:+46-708-555 456) Thanks Jacob !!! 4.7 Using kermit By default, BB expects kermit 5 to be installed on your BBPAGER host. On certain installations it is V6 and up that is installed. If that's your case then you must (might) modify the bin/bb-page1.sh script: change numeric.scr to numeric-k6.scr in the script. But try with numeric.scr at first and if it doesn't work then replace it with numeric-k6.scr. Don't forget to set KERMIT in bbsys.sh. 4.8 Adding/modifying for a custom notification procedure You can change the look of a notificaion message by modifying the bin/bb-page1.sh script. Go at the last case statement at the bottom of that script and make the appropriate changes: ${BBALPHAMSG} is the text body ${randvalsubjtag} is the security associated with the notification $BBHOSTSVC is the hostname.service tag $BBNUMERIC is a numeric value made up of 22+ digits XXXYYYYYYYYYYYYZZZZZZZ XXX - corresponding code of service defined in bbwarnsetup.cfg YYYYYYYYYYYY - IP address ZZZZZZZ - security code - first 5 digits: event ID code last 2 digits: user ID code $BBSVCNUM - equivalent of XXX in BBNUMERIC $BBCOLORLEVEL - Color level causing the notification $BBHOSTSVCDOTS - Name of host in the xxx.yyy.zzz format $BBHOSTSVCCOMMAS - Name of host in the xxx,yyy,zzz format $BBSVCNAME - Name of service causing the notification If you want to create your own custom notification method then you must add a case statement with a unique prefix associated with it. Look at Sendpage (sp-) and Qpage (qp-) for examples on how to do this. 4.9 Notification/acknowledgement logs You'll find logs of who got notified for what reasons, logs about recoveries, who got a recovery message and logs about who acknowledged what notification. They are all located in the www/acks directory: acklog notifications.log recoveries.log recoverymsgs.log 4.10 Disabling notifications temporarely You can temporarely disable notifications without having to modify the etc/bbwarnrules.cfg file. All of you to do is send a 'disable' message to the BBDISPLAY(s) using the following format: MAKE SURE THAT THE BB ENVIRONMENT HAS BEEN SET PRIOR TO USE 'bb' directly ./bb $BBDISP "disable 'host regular expression' 'duration' [reason]" You can match multiple hosts/services by specifying a regular expression instead of a real host name. The duration is in minutes. You can also add an optional reason that will be displayed in the status. AGAIN, MAKE SURE THAT THE BB ENVIRONMENT HAS BEEN SET PRIOR TO USE 'bb' directly e.g. cd $BBHOME/bin ./bb $BBDISP "disable www.bb4.com.disk 240" Disable notifications on the disk event of www.bb4.com for 240 minutes. cd $BBHOME/bin ./bb $BBDISP "disable www.bb4.com* 240" Disable notifications for all events for host www.bb4.com for 240 minutes cd $BBHOME/bin ./bb $BBDISP "disable www.bb4.com* 240 Taking www.bb4.com offline for a new disk" Disable notifications for all events for host www.bb4.com for 240 minutes and specify the reason NOTE: duration can be expressed in seconds/minutes/hours/days 30s: 30 seconds 15m: 15 minutes 1h: 1 hour 1d: 1 day by default, minutes are used if a qualifier is not used To reenable a disabled host(s), send the "enable" message ./bb $BBDISP "enable 'host regular expression'" As an example: cd $BBHOME/bin ./bb $BBDISP "enable www.bb4.com*" Enable notifications on all events for host www.bb4.com Note that when you use the enable message, the colored dot will stay blue until a new status is received by the BBDISPLAY(s) This feature is not automatically available. It must be explicitly compiled in. YOu must add the -DENABLE_DISABLE directive to the CFLAGS variable of the makefile: CFLAGS=-DENABLE_DISABLE -O Please be warned that if you enable this feature, a hacker could disable notifications while cracking into your systems. ------------------------------------------------------------------------ Section 5: Customizing your Big Brother display 5.1 Customizing the HTMLized status logs If you don't like the blinking gifs, you can change them to non-blinking by copying over the files found in www/gifs. The non-blinking gifs start with the nb- prefix. You can change Sean's mean face in the bb.html/bb2.html and HTMLized status logs by replacing the bb.gif file. This only applies to pre-1.3 versions. You can change the header/footer for the HTMLized status pages. The files are located in the /web directory. Note that your customized version can contain special tags which can be replaced by values: &BBDATE: current date/time &BBBACKGROUND: with the current background color (red/green/yellow/clear/purple) &BBRELDATE: Release date of the current version &BBREL: Current version number &BBCOLOR: Color of this status file &BBHOST: Name of host of current status file &BBSVC: Service name of current status file. &BBIPNAME: IP address of host, if 0.0.0.0 then use hostname &BBIP: IP address of host, always return IP address unless not found You can add tags by modifying the bbd.c file. Every time a status log is received by bbd, an HTML version of the status log is created by prefixing the status with the header file and suffixing with the footer file. You can put tags also in the status logs: &red &green &yellow &clear &purple These tags are replaced with a corresponding image source tag for the HTMLized version. Just like the bb-local.sh script does for the disk and procs tests. So, if you create your own custom check, enter these tags if you want to see a colored dot in the HTMLized status. NOTE: If you do not wish to have HTMLized status files, then rename the www/html directory and the main BB display pages will use the plain text status logs which are located in the www/logs directory. 5.2 Customizing headers / footers bb.html/bb2.html and various HTML page generation can be customized by modifying their respectable headers/footers. These are located in the $BBHOME/web directory. NOTE: The link to bb4.com MUST be kept in the footer page as per your agreement of the license agreement. 5.3 Creating skins You can create your own look and feel by customizing various HTML aspects of Big Brother. You can use your own gifs by creating a new directory under $BBHOME/www. As an example, a directory 'psy' has been created with more psychedelic colored dots It lives in $BBHOME/www/psy. To use those images then set the BBSKIN variable in etc/bbinc.sh to "psy". To further modify the appearance of your display, modify the header/footer files found in $BBHOME/web. You also have some control over the bb.html/bb2.html and the status logs HTML pages by modifying the following variables in etc/bbinc.sh: MKBBLOCAL MKBBREMOTE MKBBTITLE MKBBROWFONT MKBBCOLFONT MKBBMANAGE DAYS HOURS MINS STATUNCHNMSG RECVFROMMSG DOTHEIGHT DOTWIDTH BBSKIN 5.4 Inserting notes about your hosts You can setup HTML links on the host name in the bb.html/bb2.html pages to point to an information page for that particular host. If a file exists in the www/notes directory that matches the system name as displayed on the bb pages, then they are linked into both the bb.html page and the bb2.html summary page. These files can end in .htm, .html, .shtml, .php3 or nothing :) You can create these files manually or you can use the "notes" messages type: "notes " Here's an example: cd $BBHOME/bin ./bb $BBDSIP "notes www.bb4.com.html It's a hit!" This feature is not automatically available. It must be explicitly compiled in. You must add the -DNOTESMSG directive to the CFLAGS variable of the makefile: CFLAGS=-DNOTESMSG -O Please be warned that if you enable this feature, a hacker could fill the disk with dummy notes file. Also note that the file size is limited to the incoming buffer of BB and by the size of the environment buffer. Also note that if you source the BB environment prior to sending the "notes" message and you have multiple BBDISPLAYs then the "notes" message will be sent to all BBDISPLAYs. If you want to send it to only one BBDISPLAY, then use that IP address as the command line argument and set BBDISPLAYS to "" just before sending your message. 5.5 Extending the contents of the main HTML pages You can tack on some extra HTML code at the bottom of the bb.html /bb2.html pages before the insertion of the footer. You save the scripts that generate the HTML code in the $BBHOME/ext/mkbb directory. And in etc/bbdef.sh, you specify the name of those scripts in the BBMKBBEXT variable for output on the bb.html page and in the BBMKBB2EXT variable for output in bb2.html. There's an example of a script in $BBHOME/ext/mkbb called eventlog.sh that displays the last global events in a table of the bb2.html page. Make sure that the scripts are only writable by the owner to prevent someone to execute malicious code. 5.6 Extending the contents of the history HTML pages You can tack on some extra HTML code at the bottom of a history display page before the insertion of the footer. You save the scripts that generate the HTML code in the $BBHOME/ext/hist directory. And in etc/bbdef.sh, you specify the name of those scripts in the BBHISTEXT variable. Make sure that the scripts are only writable by the owner to prevent someone to execute malicious code. 5.7 Working with historical logs There are many historical logs available with BB. These logs contain all events that occured based on the host.service combination, on the host itself or for all hosts. These logs are found in $BBHOME/www/hist. You can use these logs to view that last events that occured. By default, if you view a status log, you can click the history button to view the last 50 events for that host.service combination. You can also use the ext/mkbb/eventlog.sh to view the last 20 events that happened for all hosts (you can view more by changing the NUMEVENTS variable in eventlog.sh to the size you want). You can disable the logging of events by renaming the www/hist directory. You can set BBHOSTHISTLOG=FALSE or BBALLHISTLOG=FALSE to prevent the logging of host or all hosts events. You set those values in etc/bbdef.sh There's a lot of interesting information available in those logs. Check the FTP site for scripts that take advantage of that information. There's also a directory that keeps a copy of the status logs when they change state such that they can be viewed later on. The status logs are saved in the www/histlogs directory. A directory is created with the name of the hosts and then each service has its own directory where the status logs are saved based on the time of the occurance. 5.8 Hiding the bb.html/bb2.html page header You can position the bb.html/bb2.html pages at the beginning of the first block by accessing the page as follows: .../bb.html#begindata You can bookmark this position to hide the header. ------------------------------------------------------------------------ Section 6: Miscellaneous Big Brother configuration options 6.1 Specifying filesystem specific values in the disk test You can define warning and panic levels for specific filesystems. They are defined in the etc/bb-dftab file. This file contains definitions for all hosts all in a single file. You just need to redistribute across your hosts. The format is as follows: [hostname]:/filesystem:warning level:panic level example: /:85:95 www.maclawran.ca:/oradata:98:99 / uses 85% for warning level and 95% for panic, this is valid for all hosts (as no host name was define in the definition) Only on www.maclawran.ca /oradata uses 98% for warning and 99% for panic All other filesystems use the default values defined in the etc/bbdef.sh file. Use the etc/bb-dftab.DIST file as a starting point. 6.2 Enabling security When BB is installed, it accepts connections from any hosts. You can specify from which hosts only to accept status logs using the etc/security file. You can define individual hosts or networks in the etc/security file. Using this format: 192.168.1.0 192.168.2.125 10.0.0.0 Accept from network 192.168.1.0, for 192.168.2.125 host and the 10.0.0.0 network. 6.3 Using bbnet to test TCP services You can test any text based TCP services using bbnet. To do so add the test in bb-network.sh. If you need just a basic test, then just add the service to the list of other services. If you need to send a special command to the service daemon, then add a case statement entry like the imap test. Set change the textmsg variable to the text to be sent to the daemom. 6.4 Sending custom one line status with bb The bb executable used to send statuses/notifications to the BBDISPLAY/BBPAGER can also be used by a custom check using standard input: custom_check | $BB $BBDISP - The custom_check program just needs to send a single line status to stdout (use the exact format of status files found in www/logs). This is very useful if the custom check program is a daemon. 6.5 Changing the 'purple' interval By default, BB will send a 'purple' status to indicate that a status has not been received in the last 30 minutes (default). You can change this value to a more suitable value by modifying PURPLEDELAY in etc/bbdef.sh 6.6 Specifying processes to look for You can define processes to check for existence. They are defined in the etc/bb-proctab file. This file contains definitions for all hosts all in a single file. You just need to redistribute this file across your hosts. The format is as follows: [hostname]:process list for yellow : process list for red example: localhost:smtp:httpd www.maclawran.ca:!oracle "sleep 30":smtp !process - check that host does not exists "sleep 30" - use '"' when you want to check on a process that spawns multiple words Use the etc/bb-proctab.DIST file as a starting point. If bb-proctab exists, then it overrides the PROCS/PAGEPROCS values in bbdef.sh 6.7 Sending data to the BBDISPLAY to be handled by an external script You can send data to your BBDISPLAY host to be processed by an external script, a cron job or a command line program/script. The data is appended to the specified file name in the $BBHOME/www/data directory. To use this feature you have to send a "data" message to the BBDISPLAY in this format: "data " Here's an example: cd $BBHOME ./bb $BBDISP "data www.bb4.com.hits 952281437 10m 5000" The "952281437 10m 5000" would be appended to the www/data/www.bb4.com.hits file. This feature is not automatically available. It must be explicitly compiled in. You must add the -DDATAMSG directive to the CFLAGS variable of the makefile: CFLAGS=-DDATAMSG -O Please be warned that if you enable this feature, a hacker could fill the disk with dummy data files. Also note that if you source the BB environment prior to sending the "data" message and you have multiple BBDISPLAYs then the "data" message will be sent to all BBDISPLAYs. If you want to send it to only one BBDISPLAY, then use that IP address as the command line argument and set BBDISPLAYS to "" just before sending your message. 6.8 Disabling the messages file(s) zero-length test By default Big Brother verifies that none of the messages file specified in the MSGFILE variable is of zero-length. This is a simple test to make sure that a hacker hasn't linked your messages file(s) to /dev/null. Unfortunately, on some systems, when the logs are rotated they are left empty by the procedure that rotates the logs. If this is your case then you have 2 choices: Immediately have the log rotation, add this command: echo "`date`" >> or use a syslogd feature to add a line in the messages file(s) or simply disable the zero-length check by setting the CHKMSGLEN variable to FALSE in bbdef.sh. Note, IMHO, the zero-length should always be there. Simply echo the data to the messages file(s) or use syslogd features to add an entry in the message file(s). Security is too important to ignore. Your best bet is to install a log scanner to analyze your messages file(s). Which one ? It is our of BB's scope to tell which one(s) but ask around for opinions of people who use log scanners. 6.9 Setting a Time To Live to a status message When you create external scripts, you can now tell how long the status is valid for. Just specify the length in minutes of the status. This is to prevent purple status on certain service that are not run in the regular interval of 5 minutes. To use this feature just follow this example $BB $BBDISP "status+1560 www,bb4,com.backup green `date` Backup Successful" A purple indicator will be triggered only if no status is received within 1560 minutes (26hours). 6.10 Disabling the connectivity test In some cases you may want to disable the connectivity test altogether because you don't need it or because you want to use the fping.sh external script available at ftp://ftp.deadcat.net . To disable it, just set the CONNTEST variable in etc/bbdef.sh, on the BBNET host, to FALSE. Also take in consideration that by disabling this test you will lose the ability to generate clear conditions for the other network tests because you will not have access to then connectivity status for that host. 6.11 Reducing connectivity errors If you are prone to a lot of red status for your connectivty tests even though the host responds when you ping by hand. To possibly reduce these false alarms, you may want to add/change the PINGPAR1 or PINGPAR2 parameters in etc/bbsys.local on the BBNET host. Adjust according to the BBNET's ping parameters. PINGPAR1 are options that goes before the hostname/ IP address in the ping command while PINGPAR2 are options that goes after the hostname/IP address. Adjust the number of packets sent for the ping test. Start with sending 2 packets and analyze the results. If it did not make a difference, try with 3 packets. Note that when you specify more than one packet to be sent, you inherently add a 1 second sleep on each new packet. This will delay, in seconds, your whole network test by (n-1) * hosts. If you test a large number of hosts, you will see a noticable difference in how often your hosts gets tested. 6.12 Setting up BBPAGERs and BBNETs host in failover mode You can declare failover BBPAGER and BBNET hosts and have them be in standby mode by checking for a failure on the master BBPAGER or BBNET host and when that happens they start behaving like the master BBPAGER or BBNET host until the master host is back online. To setup failover BBPAGER/BBNET hosts, define the primaries BBPAGER/BBNET in the bb-hosts file with the BBPAGER or BBNET directive and enable the failover external script on the failover hosts by putting the failover script name into the BBEXT variable of the runbb.sh script for each host that should be considered a failover host. Do not put failover in BBEXT on the primary BBPAGER or BBNET host ! For complete details please refer to the failover script in $BBHOME/ext. Each failover host will generate a "fo" status log. When a host will be in standby mode, it will generate a clear status and when failover mode is enabled it will send a yellow status. 6.13 Speeding up the network tests If you have a faily large number of servers to monitor there's a very good possibility that the network tests takes a while to complete. You can speed up the tests by parallelizing the network tests from a single bb-network.sh script to running multiple copies of bb-network.sh at once. All you have to do is to increase the value of BBNETTHREADS in etc/bbdef.sh. Indicate how many bb-network.sh you'd like to have working concurrently. You can also speed up the network tests by using the fping program to do the connectivity test. You'll have to download fping and also get the fping.sh external script at the FTP archive site: http://www.deadcat.net. You can disable the regular ping test by setting CONNTEST to FALSE in etc/bbdef.sh. ------------------------------------------------------------------------ Section 7: Other documentation sources $BBHOME/README $BBHOME/LICENSE $BBHOME/README.INSTALL $BBHOME/README.CHANGES $BBHOME/install/README $BBHOME/src/README