Services/Backups

From popdata
Jump to: navigation, search

Bacula Backup

Basics:

  • Version: 5.0.3 (04 August 2010)
  • Runs on fidler (disk backup; old frobisher did tape backups)
  • logs are at /var/log/bacula/ . See also Backup Logs
  • config files in /etc/bacula
  • clients run on each machine; under /etc/bacula/ in "filelist" & "exclude" put real folder names (e.g. mount points; no symlinks)
    • To start bacula client, run "/etc/init.d/bacula-fd restart"
  • Clients are defined on the backup server (in the file /etc/bacula/bacula-sd.conf AND the conf files for each client contained in /etc/bacula/clients/ directory.)
    • If bacula-sd.conf is modified, bacula needs to be restarted on the server. /etc/init.d/bacula-fd restart
  • To change the list of backup clients:
    • Client config file (on the client machine) bacula-fd.conf has the "Director" (backup server) defined in a few places. The FileDaemon section at the bottom has an "FDAddress =" line, which is the IP of the CLIENT.
    • You need to modify the list of clients in the /etc/bacula/clients/ directory.
    • "/etc/init.d/bacula-director restart" on the server is also be necessary when client list changes.
    • Also need to restart bacula client, run "/etc/init.d/bacula-fd restart"
    • Also check the /etc/hosts.allow file on the client. It needs to have a entry with it's own hostname and the server ip address. ex: verendrye-fd: 10.80.90.35
    • For a conplete restart, the Director needs a restart in addition to the File daemon (fd) and Storage daemon (sd). run "/etc/init.d/bacula-director restart" This should reset the list of running jobs if they are backed up.

Bconsole

  • bconsole is command line console, runs only on server
  • prompt "*"
    • Status of jobs: list jobs (thousands)
    • Status of media: list media (thousands)
    • Messages: m (show all messages unseen by this user, possibly thousands)
    • Reload is the command to reload list of backup clients after a modification.
  • messages also in /var/log/bacula
  • bconsole command "restore" prompts for JobID or date; client; provides interactive selection mode (commands "help", "cd", "ls", "dir"; "add" "find PATTERN" is similar to "find / -name PATTERN"); selecting location of restore happens later (default: /nonexistant/path/to/file/archive/dir/bacula-restores/ ), after "done" by responding "mod" to prompt "OK to run? (yes/mod/no):"
    • Prompt for "file selection mode" is "$". After zero or more "add", do "lsmark" to list files marked for restoration; exit selection mode with "done" to proceed to selection of client and target directory for restoration (or "quit" to abort").
      • bacula displays various stats, including "Where: /nonexistant/path/to/file/archive/dir/bacula-restores", and prompts "OK to run? (yes/mod/no):"
      • Enter "mod" to set better destination folder, ex: "e:/alfresco/_restore_Privacy"; also change "Replace: always" to "never".
      • Start restore by entering "yes" to prompt "OK to run? (yes/mod/no):". Prompt returns to "*" for top-level command mode
      • enter "m" to display unseen messages and see progress of queued job.
  • bconsole command "list media jobid=xxxxx" will list files for a job.
      • list media jobid=39675
      • Jobid 39675 used 3 Volume(s): bondar-2151|bondar-2152|bondar-2153
      • +--------+--------+---------------------+------+-------+----------+-------------------+-----------+
      • | JobId | Name | StartTime | Type | Level | JobFiles | JobBytes | JobStatus |
      • +--------+--------+---------------------+------+-------+----------+-------------------+-----------+
      • | 39,675 | bondar | 2017-01-21 01:15:02 | B | F | 351,676 | 1,471,853,358,149 | T |
      • +--------+--------+---------------------+------+-------+----------+-------------------+-----------+

Restore from backup disks

  • 2017 Feb : Harold requested to restore 8mth-old files from defuca backup disk.

Method 1 - restore bconsole index/catalogs

  • (1) insert and mount external backup disks (USB doc)
  • cryptsetup luksFormat /dev/sdd1
  • cryptsetup luksOpen /dev/sdd1 backup1
  • mount /dev/mapper/backup1 /offsite1
  • repeat above with different mount name & disk location if there are more than 1 disks
  • (2) protect the ongoing defuca backups
  • cd /home/bacula/data (there are symlink "defuca" which is linked with /data3/backup/defica)
  • mv defuca defuca-current (rename defuca symlink to defuca-current for backup up)
  • ln -s /offsite1/defuca defuca (create a new symlink called "defuca" which is now linked with mounted disk /offsite1/defuca)
  • (3) stop all of bacula and mysql
  • /etc/init.d/bacula-fd stop
  • /etc/init.d/bacula-sd stop
  • /etc/init.d/bacula-director stop
  • /etc/init.d/mysql stop
  • (4) compare conf files to make sure no major differences
  • diff defuca-dir.conf /etc/bacula/client
  • diff bacula-dir.conf /etc/bacula
  • (5) backup entire mysql directory
  • mv /home/bacula/mysql mysql.bak
  • (6) export mysql database
  • mysql dump -p -u root bacula > bacula > bacula-20170222.sql (dumped to /data3/extra-backup/bacula-20170222.sql)
  • (7) import mysql database
  • mysql -p -u root bacula < bacula.sql
  • start the bscan
  • bscan -S -h localhost -P '%password%' -V 'defuca-0030|defuca-0033' defuca-dev ("defuca-xxxx" is the job number)
  • now the bconsole will include the backup disk's index/catalogs


Method 2 - replace entire bacula/mysql folders to trick bconsole

  • stop all of bacula and mysql
  • /etc/init.d/bacula-fd stop
  • /etc/init.d/bacula-sd stop
  • /etc/init.d/bacula-director stop
  • /etc/init.d/mysql stop
  • copy all of /etc/bacula /var/lib/mysql /home/bacula/var-lib /home/bacula/mysql to /data3/extra-backup
  • mkdir /data3/extra-backup /data3/extra-backup/var-lib-mysql
  • cp -a /etc/bacula /var/lib/mysql /home/bacula/var-lib /data3/extra-backup
  • cp -a /home/bacula/mysql /data3/extra-backup/var-lib-mysql
  • start up bacula
  • /etc/init.d/mysql start
  • /etc/init.d/bacula-fd start
  • /etc/init.d/bacula-sd start
  • /etc/init.d/bacula-director start
  • protect the defuca backups
  • cd /home/bacula/data
  • mv defuca defuca-current
  • ln -s /offsite1/defuca .
  • start the bscan
  • bscan -S -h localhost -P '%password%' -V 'defuca-0030|defuca-0033|defuca-0043|defuca-0048|defuca-0055|defuca-0060|defuca-0061|defuca-0328|defuca-0848' defuca-dev
  • This will take a long time to run since it needs to read about 4TB of data. (100MB/s = 40000 seconds - hmmm.)

Reread inventory in changer

  • unmount both of the tape drives
  • change tapes in changer
  • and run update slots.
# bconsole
Connecting to Director frobisher:9101
1000 OK: frobisher-dir Version: 2.2.8 (26 January 2008)
Enter a period to cancel a command.
*unmount
Automatically selected Storage: HP6030-1
Enter autochanger drive[0]: 
3301 Issuing autochanger "loaded? drive 0" command.
3302 Autochanger "loaded? drive 0", result: nothing loaded.
3901 Device "Drive-1" (/dev/nst0) is already unmounted.
*update slots
Automatically selected Storage: HP6030-1
Enter autochanger drive[0]: 
Connecting to Storage daemon HP6030-1 at frobisher:9103 ...
3306 Issuing autochanger "slots" command.
Device "HP6030-1" has 30 slots.
Connecting to Storage daemon HP6030-1 at frobisher:9103 ...
3306 Issuing autochanger "list" command.
Catalog record for Volume "PH1001" updated to reference slot 1.
Catalog record for Volume "PH1002" updated to reference slot 2.
Catalog record for Volume "PH1003" updated to reference slot 3.
Catalog record for Volume "PH1004" updated to reference slot 4.
Volume "PH1031" not found in catalog. Slot=5 InChanger set to zero.
Catalog record for Volume "PH1006" updated to reference slot 6.
Catalog record for Volume "PH1007" updated to reference slot 7.
Catalog record for Volume "PH1008" updated to reference slot 8.
Catalog record for Volume "PH1009" updated to reference slot 9.
Catalog record for Volume "PH1010" updated to reference slot 10.
Catalog record for Volume "PH1011" updated to reference slot 11.
Catalog record for Volume "PH1012" updated to reference slot 12.
Catalog record for Volume "PH1013" updated to reference slot 13.
Catalog record for Volume "PH1014" updated to reference slot 14.
Catalog record for Volume "PH1019" updated to reference slot 15.
Volume "PH1035" not found in catalog. Slot=17 InChanger set to zero.
Catalog record for Volume "PH1018" updated to reference slot 18.
Volume "PH1034" not found in catalog. Slot=19 InChanger set to zero.
Volume "PH1033" not found in catalog. Slot=20 InChanger set to zero.
Catalog record for Volume "PH1021" updated to reference slot 21.
Catalog record for Volume "PH1022" updated to reference slot 22.
Catalog record for Volume "PH1023" updated to reference slot 23.
Catalog record for Volume "PH1024" updated to reference slot 24.
Catalog record for Volume "PH1025" updated to reference slot 25.
Catalog record for Volume "PH1026" updated to reference slot 26.
Catalog record for Volume "PH1027" updated to reference slot 27.
Volume "PH1032" not found in catalog. Slot=28 InChanger set to zero.
Volume "PH1029" not found in catalog. Slot=29 InChanger set to zero.

Label a tape

*label
Automatically selected Storage: HP6030-1
Enter autochanger drive[0]: 
Enter new Volume name: PH1029
Enter slot (0 or Enter for none): 29
Defined Pools:
     1: Default
     2: FullPool
Select the Pool (1-2): 2
Connecting to Storage daemon HP6030-1 at frobisher:9103 ...
Sending label command for Volume "PH1029" Slot 29 ...
3301 Issuing autochanger "loaded? drive 0" command.
3302 Autochanger "loaded? drive 0", result: nothing loaded.
3304 Issuing autochanger "load slot 29, drive 0" command.
3305 Autochanger "load slot 29, drive 0", status is OK.
3301 Issuing autochanger "loaded? drive 0" command.
3302 Autochanger "loaded? drive 0", result is Slot 16.
3000 OK label. VolBytes=64512 DVD=0 Volume="PH1029" Device="Drive-1" (/dev/nst0)
Catalog record for Volume "PH1029", Slot 29  successfully created.
Requesting to mount HP6030-1 ...
3301 Issuing autochanger "loaded? drive 0" command.
3302 Autochanger "loaded? drive 0", result is Slot 16.
3001 Mounted Volume: PH1029
3001 Device "Drive-1" (/dev/nst0) is already mounted with Volume "PH1029"

Accessing the tape library remotely

http://phlo-tape1

Various database backups

LDAP

On champlain as root crontab:

# m h  dom mon dow   command
15 5 * * * /usr/sbin/slapcat -n 1 -l /home/backups/ldap/phlo-`date +\%Y\%m\%d`.ldif ; gzip /home/backups/ldap/phlo-`date +\%Y\%m\%d`.ldif
20 5 * * * /usr/sbin/slapcat -n 2 -l /home/backups/ldap/snag-`date +\%Y\%m\%d`.ldif ; gzip /home/backups/ldap/snag-`date +\%Y\%m\%d`.ldif
  • slapcat dumps the ldap database in ldif form (slapadd will load it again)
  • the first database (-n 1) is dc=phlo,dc=ubc,dc=ca
  • the second database (-n 2) is dc=snag,dc=ubc,dc=ca
  • backups go into /home/backups/ldap
  • backup is gzip'ed after completion

Mysql on Cartier

14 4 * * 0 mysql-admin -bpmysql-backup -bt/home/backups/mysql -bxmysql -cdefault ; gzip /home/backups/mysql/mysql_`date +\%Y-\%m-\%d`*.sql
  • Profile mysql-backup is setup using mysql-admin which backs up all tables
  • Connection default is setup for root
  • backups go into /home/backups/mysql
  • backup is gzip'ed after completion
  • Currently there are databases for the wikis, snag mailing list, IT blog, and drupal test www

Mysql on cabot

14 4 * * 0 mysql-admin -bpmysql-backup -bt/home/backups/mysql -bxmysql -cdefault ; gzip /home/backups/mysql/mysql_`date +\%Y-\%m-\%d`*.sql
  • Profile mysql-backup is setup using mysql-admin which backs up all tables
  • Connection default is setup for root
  • backups go into /home/backups/mysql
  • backup is gzip'ed after completion
  • Currently there are databases for the todo lists, and syspulse

Postgres on defuca

15 3 * * * mkdir /home/backups/postgres/`date +\%Y\%m\%d`; /usr/bin/pg_dumpall | split -b 500m - /home/backups/postgres/`date +\%Y\%m\%d`/defuca-`date +\%Y\%m\%d` ; gzip /home/backups/postgres/`date +\%Y\%m\%d`/*
  • backups go into /home/backups/postgres
  • a seperate directory is made for each date
  • the backup is slip into 500MB chunks
  • backup is gzip'ed after completion
  • Currently there are databases for subversion, and trac

SecurID on cabot

45 4 * * 1-5 /usr/local/ace/rdbms/bin/sdbkup online /usr/local/ace/data/sdlog /home/backups/ace/sdlog_`date +\%Y-\%m-\%d` ; gzip /home/backups/ace/sdlog_`date +\%Y-\%m-\%d`
50 4 * * 1-5 /usr/local/ace/rdbms/bin/sdbkup online /usr/local/ace/data/sdserv /home/backups/ace/sdserv_`date +\%Y-\%m-\%d` ; gzip /home/backups/ace/sdserv_`date +\%Y-\%m-\%d`
  • backups go into /home/backups/ace
  • backup is gzip'ed after completion
  • can be restored with /usr/local/ace/rdbms/bin/sdrest


Connection reset by peer error during long backup

The fix for this is to increase the ipv4.tcp keepalive to 60 -- issue the following command as root.

  sysctl -w net.ipv4.tcp_keepalive_time=60