Systems/Storage

From popdata
Jump to: navigation, search

Disk Benchmarks

Disk Benchmarks

NetAPP Storage Area Network

OS/Firmware Upgrade

OnTap Upgrade

Creating a LUN

  • Create a volume if needed:
    • Volumes -> Add
    • Type: Flexible
    • Name: vol1
    • Language: POSIX
    • Aggregate: ...
    • Size: 1000 GB
    • Space Guarantee: Volume
  • Create Initiators if needed:
    • Luns -> Initiator Groups -> Add
    • Group Name: cartier
    • Type: iSCSI
    • Operating System: Linux
    • Initiators: iqn.1993-08.org.debian:01.a68fc7a2cacf (gotten from /etc/initiatorname.iscsi on client machine)
    • Add
  • Create a lun:
    • Luns -> Add
    • Path: /vol/vol1/lun0
    • LUN Type: Linux
    • Description: my lun
    • Size: 200 GB
    • Space Reserved: check
    • Add
  • Give initiators access to lun:
    • Luns -> Manage LUNs
    • click on Maps Group: LUN ID for the LUN
    • Add Groups to Map
    • Choose: cartier
    • Lun ID: 0 (or the next higher number for that host)
    • Apply


Maintenance

  • Turn off snapshots
vol options vol10 nosnap on
  • remove snapshots
snap list vol14
snap delete -a vol 14
  • show space
df -k
aggr show_space -h
  • thin provision
lun set reservation /vol/vol4/lun0 disable
vol options /vol/vol4 guarantee none
  • dedup
sis start /vol/vol3/lun0
sis start -s /vol/vol3
sis status

Connection on a Linux host

  • Summary:
    • iscsiadm --mode discovery --type sendtargets --portal 10.190.0.210
    • iscsiadm --mode node --portal 10.190.0.210 --login
    • you should see the new slice appear in dmesg
    • partition it: fdisk /dev/sde (or gpt if larger than 2TB)
    • mirror if needed
    • mount
  • apt-get install open-iscsi
  • Search for targets:
root# iscsiadm --mode discovery --type sendtargets --portal 10.190.0.210
10.190.0.20:3260,1000 iqn.1992-08.com.netapp:sn.101191046
10.190.0.21:3260,2000 iqn.1992-08.com.netapp:sn.101191046
  • Login: iscsiadm --mode node --portal 10.190.0.210 --login
  • Using dmesg you should see:
  Vendor: NETAPP    Model: LUN               Rev: 0.2 
  Type:   Direct-Access                      ANSI SCSI revision: 04
SCSI device sdb: 419430400 512-byte hdwr sectors (214748 MB)
sdb: Write Protect is off
sdb: Mode Sense: bd 00 00 08
SCSI device sdb: drive cache: write through
SCSI device sdb: 419430400 512-byte hdwr sectors (214748 MB)
sdb: Write Protect is off
sdb: Mode Sense: bd 00 00 08
SCSI device sdb: drive cache: write through
 sdb: sdb1
sd 7:0:0:2: Attached scsi disk sdb
sd 7:0:0:2: Attached scsi generic sg1 type 0
  • alternatively to make it automatically login

iscsiadm --mode node --portal 10.190.0.210 -o update -n node.conn[0].startup -v automatic

  • Partition the lun if needed: fdisk /dev/sdb
  • Make a filesystem: mke2fs -j /dev/sdb
  • Mount it: mount /dev/sdb1 /mnt

Example with Mirroring and Encryption

iscsiadm --mode discovery --type sendtargets --portal 10.190.0.200
iscsiadm --mode discovery --type sendtargets --portal 10.190.0.210
iscsiadm --mode node --portal 10.190.0.200 --login
iscsiadm --mode node --portal 10.190.0.210 --login
dmesg   # find scsi device names
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdf1 /dev/sdg1
cryptsetup luksFormat /dev/md0
cryptsetup luksOpen /dev/md0 newdata
mkfs.ext4 /dev/mapper/newdata
mount /dev/mapper/newdata /newdata


SAN Layout

SAN 1

  • 14 x 300 GB SCSI
    • 2.57 TB Aggragate
      • vol0 10 GB (system)
      • vol1 880 GB fraser:/home
      • vol2 400 GB franklin:/local(RZ)
      • vol3 1 TB george:/bclhd/projects (RZ)
  • 14 x 1 TB SATA
    • 8TB Aggragate
      • vol4 1 TB sherry:/projects2 (pharma)
      • vol5 300 GB delasalle (RZ VM)
      • vol8 500 GB defuca:/?
      • vol8 500 GB george:/popdata/content(RZ)
      • vol9 1 TB defuca (RZ)
      • vol10 600 GB old SRE vmware (Not Used)
      • vol11 300 GB joliet (RZ VM)
      • vol12 600 GB kelsey (PD admin)
      • vol13 1200 GB kelsey (gilbert homes)
      • vol14 500 GB kelsey (gilbert homes)
      • vol17 1.2 TB helpsvr1

Initiators

Logged in:

  • fraser iqn.1993-08.org.debian:01:5333bb4cf7aa
  • defuca iqn.1993-08.org.debian:01.a68fc7a2cacf
  • sherry iqn.2000-09.ca.ubc.chspr:yellow-sn-1
  • franklin iqn.1993-08.org.debian:01:44f19ac436d
  • george iqn.2000-09.ca.ubc.chspr:red-bclhd-projects
  • kelsey iqn.1998-01.com.vmware:kelsey-3080563f
  • joliet iqn.1998-01.com.vmware:joliet-4087a6d0

Not Logged in:

  • helpsvr1 iqn.1991-05.com.microsoft:helpsvr1.help.local
  • delasalle iqn.1998-01.com.vmware:delasalle-74783679
  • verendrye iqn.1998-01.com.vmware:localhost:1147490437:35

SAN 2

  • 14 x 500 GB SATA
    • 4 TB
      • vol0 22 GB (system)
      • vol1 500 GB sherry:/projects (pharma)
      • vol2 481 GB helpsvr1 (HELP)
      • vol3 800 GB sre vmware (SRE) Not Used
      • vol4 500 GB franklin:/data (RZ)
      • vol5 400 GB helpsvr1 (CPHPR)
      • vol6 712 GB george:/bclhd/datasave (RZ)
      • vol7 316 GB sre vnware (SRE) Not Used
      • vol8 300 GB joliet (RZ VM)
      • vol9 400 GB old verrazzano Not Used
      • vol10 300 GB kelsey (PD admin)

Initiators

Logged In:

  • helpsrv1 iqn.1991-05.com.microsoft:helpsvr4.help.local
  • lily iqn.2000-09.ca.ubc.chspr:yellow-sn-1
  • franklin iqn.1993-08.org.debian:01:44f19ac436d
  • george iqn.2000-09.ca.ubc.chspr:red-bclhd-projects
  • joliet iqn.1998-01.com.vmware:joliet-4087a6d0
  • kelsey iqn.1998-01.com.vmware:kelsey-3080563f

Not Logged In:

  • delasalle iqn.1993-08.org.debian:01.80e2a67d48ff
  • defuca iqn.1993-08.org.debian:01.a68fc7a2cacf
  • helpsvr1 iqn.1991-05.com.microsoft:helpsvr1.help.local

SAN 3

  • 2017-08-10 found pd-san3 = beaulieu = 10.190.0.200 (+ 10.191.0.200 , 10.192.0.200) /var/log/kernel shows disk errors
    STATUS: "targetcli sessions"
    pd-san3 Used on Defuca Franklin Fraser George2 Kelsey Stefansson Verrazzano
    pd-san4 used on Defuca Franklin Fraser George2 Kelsey Mackenzie Stefansson Verrazzano

* 18 * 1.5 TB SATA (21 TB /dev/sdb) Updated July 2012


  • 12 * 1.5TB SATA (13.6 TB /dev/sdb)
    • vg_san3.1 13.64 TiB
  • 10 * 3TB SATA (19.1 TB /dev/sdc(one 3TB drive Hot Spare))
    • vg_san3.2 19.10 TiB


San 3 had an issue with upper left-most drive slot. swapped cable connections with slot underneath -- fixed issue.  Corrected port labels.


iqn.2011-06.ca.bc.popdata:san3.extproj		3.0TB	frobisher	10.190.0.30				yes
iqn.2011-06.ca.bc.popdata:san3.rtlvm		1.2TB	rtl		10.190.0.155				no
iqn.2011-06.ca.bc.popdata:san3.sre2		1.2TB	sre		10.190.0.71/155				no
iqn.2011-06.ca.bc.popdata:san3.zimbra-extra	3.0TB	verrazzano	10.190.0.142				yes

SAN 4

  • 22 * 3TB SATA
    • vg_san4.1 (/dev/sdb) 27TB
    • vg_san4.2 (/dev/sdc) 19TB
														mirror?
iqn.2011-06.ca.bc.popdata:san4.yz-backup	4.0TB	Thompson	10.190.0.55				no
iqn.2011-06.ca.bc.popdata:san4.linkage		4.0TB	defuca		10.190.0.120				yes
iqn.2011-06.ca.bc.popdata:san4.rz-backup	2.0TB	mackenzie	10.190.0.160/170        .170 not used	no
iqn.2011-06.ca.bc.popdata:san4.sitka		0.8TB	sitka		10.190.0.145		     	 	yes
iqn.2011-06.ca.bc.popdata:san4.sre		1.0TB	sre		10.190.0.71/170/180/135  .71/.135 not	yes
iqn.2011-06.ca.bc.popdata:san4.zimbra		1.0TB	old verr.	10.190.0.140		not used  	no
iqn.2011-06.ca.bc.popdata:san4.kelsey		1.4TB	kelsey		10.190.0.110/115        .110 not used	no
iqn.2011-06.ca.bc.popdata:san4.rzvm		1.2TB	delasalle?	10.190.0.117				no
iqn.2011-06.ca.bc.popdata:san4.newrzvm		1.2TB	verendrye	10.190.0.71/170/180/117  .71 not used	no
iqn.2011-06.ca.bc.popdata:san4.qlab		2.0TB	qserver		10.190.0.105				yes
  • 2017-08-10 pd-san4 used on Defuca Franklin Fraser George2 Kelsey Mackenzie Stefansson Verrazzano

New LIO iSCSI Targets

  • first you need a block device. /dev/sda would work, but more likely a lvm logical volume as above
  • lvcreate -v -n jim-test -L 1000G vg_san3.1
  • run /usr/local/bin/targetcli
    • Create the backstore device:
# /usr/local/bin/targetcli
cd /backstores/block
/backstores/block> create sre-kvm-cortereal /dev/vg_san3.2/sre-kvm-cortereal
Created block storage object sre-kvm-cortereal using /dev/vg_san3.2/sre-kvm-cortereal.
/backstores/block> ls
o- block ...................................................................................................... [Storage Objects: 2]
  o- jimtest ............................................................ [/dev/vg_san3.1/jim-test (1000.0GiB) write-thru activated]
  o- sre-kvm-cortereal ....................................... [/dev/vg_san3.2/sre-kvm-cortereal (1000.0GiB) write-thru deactivated]
    • create the iscsi iqn:
/iscsi> create iqn.2014-02.ca.bc.popdata:san3.sre-kvm-cortereal
Created target iqn.2014-02.ca.bc.popdata:san3.sre-kvm-cortereal.
Created TPG 1.
/iscsi> ls
o- iscsi .............................................................................................................. [Targets: 1]
  o- iqn.2014-02.ca.bc.popdata:san3.sre-kvm-cortereal .................................................................... [TPGs: 1]
    o- tpg1 ................................................................................................. [no-gen-acls, no-auth]
      o- acls ............................................................................................................ [ACLs: 0]
      o- luns ............................................................................................................ [LUNs: 0]
      o- portals ...................................................................................................... [Portals: 0]
    • Assign the backstore:
/iscsi> cd iqn.2014-02.ca.bc.popdata:san3.sre-kvm-cortereal
/iscsi/iqn.20...kvm-cortereal> cd tpg1
/iscsi/iqn.20...ortereal/tpg1> cd luns
/iscsi/iqn.20...eal/tpg1/luns> create /backstores/block/sre-kvm-cortereal
Created LUN 0.
    • create a portal, making a daemon listen for incoming connections:
/iscsi/iqn.20...eal/tpg1/luns> cd ../portals
/iscsi/iqn.20.../tpg1/portals> create
Using default IP port 3260
Binding to INADDR_ANY (0.0.0.0)
Created network portal 0.0.0.0:3260.
    • assign the initiator name from the client
/iscsi/iqn.20.../tpg1/portals> cd ../acls
/iscsi/iqn.20...eal/tpg1/acls> create iqn.1993-08.org.debian:01:556899e128f
Created Node ACL for iqn.1993-08.org.debian:01:556899e128f
Created mapped LUN 0.
    • exit and save
/iscsi/iqn.20...eal/tpg1/acls> cd /
/> exit
Global pref auto_save_on_exit=true
Last 10 configs saved in /etc/target/backup.
Configuration saved to /etc/target/saveconfig.json

SAN 4 Setup

I've decide not to use openfiler for SAN4, since our setup is quite simple, and openfiler add the complexity of a different distribution, and a sketchy maintenance schedule from them. All we need is a way to make partitions that is flexible, and then to share those partitions via iSCSI.

Setup is as follows:

  • Physical Disks
    • 2 x 2TB Disks as a mirror which makes the boot disk (larger than 2TB won't work as a boot drive)
    • 10 x 3TB Disk as a RAID 6
    • 11 x 3TB Disk as a RAID 6
    • 1 3TB Hot Spare
  • use gparted to create a GPT partition table, and a single partition for the whole disk
  • Use LVM to partition the huge disks. A single physical volume per volume group, and then logical volumes for the partitions.
    • Physical Volume:
      • pvcreate --metadatacopies 2 /dev/sdb1
      • pvcreate --metadatacopies 2 /dev/sdc1
    • Volume Group:
      • vgcreate vg_san4.1 /dev/sdb1
      • vgcreate vg_san4.2 /dev/sdc1
    • Logical Volume:
      • lvcreate -v -n yz-backup -L 4000G vg_san4.1
      • lvcreate -v -n linkage -L 4000G vg_san4.2
  • Following is obsolete: see LIO aboveShare the partition via iscsi
    • Install tgt package, make an /etc/init.d/tgt file to start /usr/sbin/tgt
    • Add to /etc/tgt/targets.conf
<target iqn.2011-06.ca.bc.popdata:san4.yz-backup>
   backing-store /dev/vg_san4.1/yz-backup

   initiator-address 10.80.90.30     # Allowed IP
</target>
  • run tgt-admin -v -e

To create an extra iscsi lun:

  • lvcreate -v -n yz-backup -L 4000G vg_san4.1
  • edit: /etc/tgt/targets.conf add new target section
  • tgt-admin -v -e

Add an extra share:

  • tgtadm --lld iscsi --op show --mode target # find the target id (tid)
  • tgtadm --lld iscsi --op bind --mode target --tid 6 -I 10.190.0.71

Remove a share

  • tgtadm --lld iscsi --op unbind --mode target --tid 6 -I 10.190.0.71