Services/Backups/Offsite

From popdata
Jump to: navigation, search

Offsite Backups

Temporary Document Scratch Space

Cleaning up /dev/mapper device-mapping targets.

dmsetup remove [target]

Jim's Notes from the IT Blog

I connected the external USB dock to cheadle.  I decided it might be easiest to make the partition on it encrytped.  
That way we can just separately encrypt the backups of the really important stuff (defuca, george, and franklin) but everything else is actually encrypted offsite too.  
Setup was:

apt-get install cryptsetup

Prepare drives for backup

NEW DRIVES

ssh -X fidler (same for cheadle in red zone)
sudo bash (to be root)
export HOME=/home/davidg
dmesg -T (to see if the drives are there - sdd and sde)
Another check that you are formatting the correct drive: do a "df" to check current devices, and lsblk.

run gparted for new drives only. Then choose the correct disk in the top right. (/dev/sdd for example)
Click "Device" in the menu, then "Create Partition Table".
In the next window under "advanced" choose GPT as the partition table type.
Right click the unallocated space in the main window. Choose "New" partition table. 
Change the file system to unformatted, then click the "Add" button to create a single partition on each disk.
Create the GPT partition for the sde drive in the same way.
Click Edit in the Gparted menu. Then "Apply all operations". Exit Gparted.
Repeat this process on cheadle.

In the Terminal window:
cryptsetup luksFormat /dev/sdd1
cryptsetup luksOpen /dev/sdd1 backup1
mkfs.ext4 -m 0 /dev/mapper/backup1
mount /dev/mapper/backup1 /offsite1
(repeat for /dev/sde1 using "backup2" and "offsite2".)
Repeat this process on cheadle.

REUSING RETURNED DRIVES

Normally we only need the "cryptsetup luksOpen" line from above.
Then mount using the command from above.
Then delete all the data on the drive.
  • Note that luksFormat and luksOpen prompt for a password. See password store in Red Zone.

Run copy script

Move to the folder /home/bacula/scripts/
Run ./newoffsite.py (on cheadle run ./newoffsite.py -h to get help on the command - you need to specify the disk size)
This script will choose the latest full backups for each server and decide which of the two backups drives
to copy it to, then it creates two scripts (YYYY-MM-DD-1.sh and YYYY-MM-DD-2.sh, one for each drive) 
The three scripts can then be run simultaneously (add a "&" character for background processing)

A spot check should be done occasionally before starting the backup to confirm that the script has chosen the correct backup files (from the last full backup of each server):
Cat one of the backup scripts and make note of which files it has chosen to backup for defuca (for example).
Then go into /home/bacula/data/defuca and list (ls -lt) the files to see if those files are the latest full backup files.

Check copy progress during or just after to see if successful. Command "df" to see if offsite1 or offsite2 is 100% full. The files may not have been properly divided among the two drives.

Prepare the drives for pickup

In a terminal connected to fidler, run these commands:

umount /offsite1
umount /offsite2
cryptsetup luksClose backup1
cryptsetup luksClose backup2
eject -s /dev/sdd (on fidler "-s" seems to be necessary).
eject /dev/sde

Run the same set of commands on cheadle to prepare red zone drives.
Turn off power on USB drive bay, wait a moment for drives to spin down, then physically eject the drives.
Write the backup date on the drives and pack them into the correct box for Iron Mountain pickup.
Record info in table below.
* Also need to connect to Iron Mountain SecureSync website and arrange the return of one drive box (oldest) before the Thurs pick up day.

run bconsole on fidler and cheadle, then "list jobs" to see what was the last full backup date.

Offsite backup Drives List

Drive #s: YZ Backup Date yyyy-mm-dd RZ Backup Date yyyy-mm-dd Box #: Status: Pickup/Return Date Notes
YZ-115, YZ-116, RZ-115, RZ-116 -Archived 2012-03-15 2012-03-15 ----- in office - Archived ----- 2011 Archive
YZ-113, YZ-114, RZ-113, RZ-114 -Archived- 2012-12-04 2012-12-05 ----- in office - Archived ----- 2012 Archive
YZ-104, YZ-105, RZ-104, RZ-105 -Archived- 2014-01-04 2013-12-28 ----- in office - Archived ----- 2013 Archive
YZ-100, YZ-101, RZ-100, RZ-101, RZ-S1 -Archived- 2015-01-17 2015-01-24 ----- in office - Archived ----- 2014 Archive
YZ-108, YZ-109, RZ-108, RZ-109, YZ-S1 -Archived- 2016-01-23 2016-01-16 ----- in office - Archived ----- 2015 Archive
YZ-102, YZ-103, RZ-102, RZ-103, YZ-S6 -Archived 2016-12-24 2016-12-31 ----- in office - Archived ----- 2016 Archive
YZS119(8TB), YZ120(3TB), RZ-119(6TB), RZ-120(6TB), RZ-S2(3TB) 2018-01-20 2018-01-13 ----- in office - Archived ----- 2017 Archive
YZ-125(6TB), YZ-126(6TB), RZ-125(8TB), RZ-126(8TB), RZ-S6(6TB) 2018-12-08 2018-12-15 ----- in office - Archived ----- 2018 Archive
YZ-127(6TB), YZ-128(6TB), YZ-112(6TB), RZ-127(8TB), RZ-106(8TB), RZ-107(8TB) 2019-11-02 2019-10-26 95300 at IM 2019-11-14 -----
YZ-123(10TB), YZ-124(8TB), YZ-S3(8TB), RZ-123(10TB), RZ-124(10TB), RZ-S3(3TB) 2019-12-07 2019-11-18 95303 at IM 2019-12-12 -----
YZ-1A(10TB), YZ-1B(10TB), RZ-1A(8TB), RZ-1B(8T), RZ-1C(6TB), RZ-1D(6TB) 2020-01-18 2020-01-25 95301 at IM 2020-02-20
  • YZ needs 14TB, RZ needs 21TB
YZ-117(8T), YZ-118(8T), RZ-117(6TB), RZ-118(6TB), RZ-S5(10TB) 2019-06-01 2019-06-08 95302 in office 2020-03-05 -----
YZ-107(6TB), YZ-110(6TB), YZ-119(6TB), RZ-110(6TB), RZ-133(6TB), RZ-111(8TB), RZ-134(3TB) 2019-09-28 2019-09-21 95304 at IM 2019-10-10
YZ-2A(10T), YZ-2B(6T:full used), YZ-2C(3T: full used), RZ-2A(10T), RZ-2B(10T), RZ-2C(8T -50%used) 2020-02-15 2020-02-22 95305 at IM 2020-03-05
  • YZ needs 20TB, RZ needs 24TB
YZ-135(10T), YZ-136(10T), RZ-135(10T), RZ-136(10T), RZ-137(3T) 2019-08-03 2019-07-27 ----- in office 2020-01-09
YZ-140(10T), YZ-141(10T), RZ-140(10T), RZ-141(10T), RZ-142(3T) 2019-08-31 2019-08-24 ----- in Office 2019-09-12