Services/KVM

From popdata
Jump to: navigation, search
  • Back to [Systems]
  • See also [Install_image#Latest_Master_Image]

KVM General Usage

  • Monitor machines using: virt-manager
  • same thing on the command line with virsh
virsh list --all         # list all VMs that it knows about
virsh vol-list kvm-host  # list all physical volumes in group kvm-host.  kvm-source is the other one
virsh pool-list          # list the pools, make sure they are active and autostarted
virsh vol-dumpxml /home/kvm2/base/w7base.img  # dump the xml for an image (can be fed in to build a new image)
virsh start sre14        # start a machine (also destroy, shutdown, reboot) 
virsh edit red13         # change machine config such as RAM HDD sizes
virsh shutdown sre10
virsh destroy --graceful sre10 # when shudown does not work, force shutdown but still gracefully
  • dump a machine definition: virsh dumpxml sre13
  • create a machine based on an xml file: virsh define /tmp/jimbo.xml
  • start a machine: virsh start sre12
    • No automatic start of VMs on host reboot. TRY: for I in 1 2 3 4 5 6 7 8 9 ; do virsh start sre$I; done
  • shutdown (nicely) a machine: virsh shutdown sre12
  • reboot (nicely) a machine: virsh reboot sre12
  • turn off (not nicely) a machine: virsh destroy sre12
  • completely delete a machine: virsh undefine sre12
  • start a storage pool
virsh pool-list
virsh pool-start kvm-host
  • connect to the serial port: virsh -c qemu:///system console thompson
  • testing Remote Desktop accessibility using netcat (-v: report result; -z: transmit nothing; -w2: timeout 2 seconds)
    • for I in 1 2 3 4 5 6 7 8 9 10 11 12; do nc -vz -w2 fast$I 3389; done

SRE VMs

Creating SRE VMs

  • everything in /home/kvm2
  • clone.py
  • ./clone.py -h gives help
  • there are usually 2 images (w7base-A.img, and w7base-B.img)
  • vminfo will list all machines and which image they use (or by hand: for P in extra home; do for V in /home/kvm2/$P/*.ovl; do echo -n "`basename $V` `dirname $V` : "; virsh vol-dumpxml --pool kvm-${P} $V | fgrep img ; done; done)
  • Usually newest is the one being used
  • Prior to cloning, create on Gilbert DNS entries (.popdata.local) and on Champlain DNS (.popdata.bc.ca), and DHCP entries.
  • Prior to running clone.sh , make sure "ssh Administrator@gilbert" from the SRE server does not ask for a password. (Either have an ssh private key memorized by ssh-agent or setup ssh multiplexing (drake, amundsen and cortereal were setup as per Software-user-maintenance#SSH_Connection_multiplexing; As root (sudo -s), in a separate window, do "ssh Administrator@gilbert" and keep that connection open)
  • examples:
./clone -t sre -l home -b w7base-A 10 11 12       # will create sre10, sre11, sre12 using the home partition, with w7base-A.img as the base
./clone -t srtl -l extra -b w7base-A 1 2 3        # will create srtl1, srtl2, srtl3 using the extra partition, with w7base-A.img as the base
./clone -t sre -l data -b w7base-B 2 3            # will create sre2, srt3 using the home partition, with w7base-B.img as the base
  • Notes:
    • normally all sre machines are on home
    • don't create all machines for a server at the same time (e.g. sre1...sre14) or you will run out of memory. We oversubscribe memory a bit - each machine uses 16GB. So for a machine with 160GB of memory, create/start 8 or 9 machines, and then wait an hour or two, check available memory and start the rest.
    • New machines may sometimes need to manually join domain "popdata.local" : login with local account (ex: srtl9\tech on srtl9), right-click on Computer -> Properties -> workgroup&domain -> Change domain; reboot.
    • New machines may sometimesneed Remote Desktop group configured srtl-users. System properties -> remote -> select users -> add popdata\srtl-users (or as appropriate). Default seems to be rtl-users and sre-users.
    • see hardware database for which servers run which VMs

Removing SRE VMs

  • To remove machines just add the -r flag:
./clone -r -t sre -l home 13 14  # remove and delete sre13, and sre14 from the home partition

Updating SRE VMs

  • 2019-08-27 update sre30 on Hubbard after patching w10image-B.img
  1. Keep a separate window logged in from root to Gilbert: sudo -s ... ssh Administrator@gilbert
  2. vminfo ; cd /home/kvm2/base ; fuser w10base-* ## check which machines have which image
  3. pds..sreinfo ## check which machines are Available
  4. cd /home/kvm2/base ; BASE=A; NUM=29 ; ../clone.w10.py -r -t sre -l home ${NUM} ; ../clone.w10.py -t sre -l home -b w10base-${BASE} ${NUM} ## Combined shutdown and create
    • takes several minutes. Despite the error messages below, sre30 was successfully set up with the new image (except with light blue desktop).
    • Sre27 was totally successful with no error messages and black desktop in about 3 minutes.
==================== Creating sre30 ====================
Vol sre30.ovl created

Formatting '/home/kvm2/home/sre30.ovl', fmt=qcow2 size=236223201280 backing_file=/home/kvm2/base/w10base-A.img encryption=off cluster_size=65536 lazy_refcounts=off refcount_bits=16

Starting install...
Creating domain...                                                                                                                                                          |    0 B  00:00:00     
Domain creation completed.
Waiting for VM to become available . 
==================== Performing both for sre30 ================
WARNING: The changes will take effect after you restart the computer 
win10master.
Waiting for VM to become available 
==================== Updating GPO for sre30 ================
[10.80.50.230] Connecting to remote server 10.80.50.230 failed with the 
following error message : Access is denied. For more information, see the 
about_Remote_Troubleshooting Help topic.
    + CategoryInfo          : OpenError: (10.80.50.230:String) [], PSRemotingT 
   ransportException
    + FullyQualifiedErrorId : AccessDenied,PSSessionStateBroken

Diagnosing VMs

  • List of reboots
    • egrep '^2017.*ing ' /var/log/libvirt/qemu/sre*.log | sed -e 's"^.*/""' -e 's/\.log:/ /' | sort -k2
      sre22 2017-07-17 16:52:35.802+0000: shutting down
      sre22 2017-07-17 16:52:53.874+0000: starting up
    • join down/up pairs Emacs query-replace-regexp  : ^\([^^I]+\)^I\(.*down\)^J\1^I\(.*up\) -> \1^I\2^I\3)

Updating the Base Image

For example on verendrye machine w7base-A can be used either as a base for others, or run on its own.

  • Look at /home/kvm2/base for the base files.
  • Make sure no running VM is using the base by running vminfo
  • you can copy the base. If all running VMs are using w7base-A:
cd /home/kvm2/base
cp w7base-A w7base-B
  • Then start the unused base you wish to update: virsh start w7base-A
    • NOTE: virsh can only work with machines as defined in /etc/libvirt/qemu/*.xml
  • it will get a DHCP address from champlain (usually around 10.80.50.170)
  • remote desktop to 10.80.50.170 and login as .\tech
  • you may need to update firewall: edit then enable group pd-sre (incoming) line ~18 "src=10.80.50.175 dst=any IP permit". Disable the rule and apply when done, and test chrome to ubc.ca .
  • For tests of R, SPSS, SAS that require R:, see below for "net use R: \\gilbert\alfresco\..."
  • Before using the base in other machines: virsh shutdown w7base-B
  • you can now use clone.py to create VMs using this new base

SRE Server Setup

Now for Debian 7 (wheezy)

  • Install the extra packages:
apt-get install libvirt-bin virt-manager qemu-kvm mdadm open-iscsi sysfsutils
  • add line for KSM to /etc/sysfs.conf
kernel/mm/ksm/run = 1
  • add bridge to /etc/network/interfaces
auto lo eth0 br0

iface eth0 inet manual

iface br0 inet static
  address 10.80.50.70
  network 10.80.50.0
  netmask 255.255.255.0
  broadcast 10.80.50.255
  gateway 10.80.50.254
  bridge_ports eth0
  bridge_stp off
  bridge_fd 5
  bridge_maxwait 0
  bridge_maxage 12
  • set up directories
mkdir /home/kvm2
mkdir /home/host
mkdir /extra/host # if needed
mkdir /data/host # if needed
cd /home/kvm2
mkdir base
ln -s /home/host home
ln -s /extra/host extra
ln -s /data/host data
  • add files:
scp cortereal:/home/kvm2/libvirt-files.tar /etc/libvirt
cd /etc/libvirt
tar xvf libvirt-files.tar
mv clone.py /home/kvm2
scp cortereal:/home/kvm2/base/w7base-A.img /home/kvm2/base
/etc/init.d/libvirt-bin restart
  • iscsi stuff. Login and make the login automatic for next time (if needed)
iscsiadm --mode node --portal 10.190.0.200 --login
iscsiadm --mode node --portal 10.190.0.210 --login
iscsiadm --mode node --portal 10.190.0.200 -o update -n node.conn[0].startup -v automatic
iscsiadm --mode node --portal 10.190.0.210 -o update -n node.conn[0].startup -v automatic
  • set up mirror (if needed)
fdisk /dev/sdb  # create partition
fdisk /dev/sdc  # create another partition
mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sdb1 /dev/sdc1
mkdir /data
mkfs.ext4 /dev/md0
  • add it to fstab
/dev/md0                /data   ext4    rw      0       0       2

Notes

  • config stuff for libvirt lives in /etc/libvirt (use virsh to change it, otherwise complete restart needed)
  • bridges:
    • br0: 10.80.50.0/24 (SRE)
    • br1: 10.80.52.0/24 (RTL) (missing on Drake)
    • br2: 10.80.54.0/24 (SRTL) (missing on Cortereal)
    • br3: 10.80.90.0/24 (adm - FAI) ?
    • br4: 10.80.20.0/24 (Gilbert WDS)
  • show bridge info: brctl show

Red Zone

  • Currently all Red Zone machines are under KVM on the hosts
    • Stefansson (10.180.10.130): extract1-5,11-14; red13; larsen
    • Hearne (10.180.10.90)
  • Some of machines are on the internal disk /home/kvm2/host2 and some on the SAN /home/kvm2/host.

The Active Directory server (Beare) is still running under VMware on host 10.180.10.120. The old VMs are also here (harold still has his running while he syncs the two)

Each of the 3 categories of Windows virtual machines connects only to the appropriate file server: "prepare" => Defuca; "extract" => George2; "link" => Ericsson .

RZ VM by user

  • Franklin: grep '^Exec=' /home/*/Desktop/*.desktop
    • Ex: tchoi/Desktop/extract.desktop: Exec=xfreerdp /f /u:tchoi /d:redzone -sen-nla /cert-ignore /v:10.140.50.101
    • extract: 10.140.50.* ; link: 10.130.50.* ; prepare: 10.180.50.*

101 tchoi (extract) 102 msimpson (extract) 103 mgagne (extract) -> pass it on to Michelle 104 jgalo (extract) 105 kalke (extract) 110 lyan (e+l+p) 111 csian (e+l+p) 112 hyip (e+l+p) 113 fxu (e+l+Red13) 114 spare(e+l+p) *temporary give to Michelle 115 bhills (e+l+p) 116 msimpson (p) Others (all on KVM host "hearne" except Kerry) 10.180.10.6 lchen (red6=lchen) 10.180.10.7 Ryan (red7=Ryan's) 10.180.10.8 bweinberg (red8=bweinberg) 10.180.10.9 tark (red9=jboonstra) 10.180.10.209 kkerluke (not kkerluke=10.180.10.154)

RZ VM by server

Server Name IP Path Win ver. File Server VM Name Assigned User Notes
Hearne 10.180.10.6 /home/kvm2/home/red6.ovl Win7 franklin red6 HELP-lchen Deleted
Hearne 10.180.10.7 /home/kvm2/home/red7.ovl Win7 franklin red7 HELP-SPARE Deleted
Hearne 10.180.10.8 /home/kvm2/home/red8.ovl Win7 franklin red8 HELP-bweinberg Deleted
Hearne 10.180.10.9 /home/kvm2/home/red9.ovl Win7 franklin red9 HELP-SPARE Deleted
Hearne 10.130.50.210 /home/kvm2/home/ Win10 ericsson link-10 lyan
Hearne 10.130.50.211 /home/kvm2/home/ Win10 ericsson link-11 csian
Hearne 10.130.50.212 /home/kvm2/home/ Win10 ericsson link-12 hyip
Hearne 10.130.50.213 /home/kvm2/home/ Win10 ericsson link-13 fxu
Hearne 10.130.50.214 /home/kvm2/home/ Win10 ericsson link-14 mjarrett
Hearne 10.130.50.215 /home/kvm2/home/ Win10 ericsson link-15 bhills
Hearne 10.130.50.216 /home/kvm2/home/ Win10 ericsson link-16 msimpson Use this for LinXmart
Hearne 10.130.50.217 /home/kvm2/home/ Win10 ericsson link-17 tark
Hearne 10.180.50.210 /home/kvm2/home/ Win10 defuca prepare-10 lyan
Hearne 10.180.50.211 /home/kvm2/home/ Win10 defuca prepare-11 csian
Hearne 10.180.50.212 /home/kvm2/home/ Win10 defuca prepare-12 hyip
Hearne 10.180.50.213 /home/kvm2/home/ Win10 defuca prepare-13 fxu
Hearne 10.180.50.214 /home/kvm2/home/ Win10 defuca prepare-14 mjarrett
Hearne 10.180.50.215 /home/kvm2/home/ Win10 defuca prepare-15 bhills
Hearne 10.180.50.216 /home/kvm2/home/ Win10 defuca prepare-16 msimpson
Hearne 10.180.50.217 /home/kvm2/home/ Win10 defuca prepare-17 tark
Stefansson /home/kvm2/base/virtio-win-0.1-74.iso larson RZ domain controller RZ domain controller
Stefansson 10.180.10.2 /home/kvm2/home/red2.ovl Win10 franklin red2 HELP-spare NO MACHINE CREATED
Stefansson 10.180.10.3 /home/kvm2/home/red3.ovl Win10 franklin red3 HELP-huishen
Stefansson 10.180.10.4 /home/kvm2/home/red4.ovl Win10 franklin red4 HELP-lchen
Stefansson 10.180.10.5 /home/kvm2/home/red5.ovl Win10 franklin red5 HELP-bweinberg
Stefansson 10.140.50.201 /home/kvm2/home/ Win10 george2 extract-1 tchoi
Stefansson 10.140.50.202 /home/kvm2/home/ Win10 george2 extract-2 mmedearis
Stefansson 10.140.50.203 /home/kvm2/home/ Win10 george2 extract-3 mwong
Stefansson 10.140.50.204 /home/kvm2/home/ Win10 george2 extract-4 jgalo
Stefansson 10.140.50.205 /home/kvm2/home/ Win10 george2 extract-5 kalke NO MACHINE CREATED
Stefansson 10.140.50.210 /home/kvm2/home/ Win10 george2 extract-10 lyan
Stefansson 10.140.50.211 /home/kvm2/home/ Win10 george2 extract-11 csian
Stefansson 10.140.50.212 /home/kvm2/home/ Win10 george2 extract-12 hyip
Stefansson 10.140.50.213 /home/kvm2/home/ Win10 george2 extract-13 fxu
Stefansson 10.140.50.214 /home/kvm2/home/ Win10 george2 extract-14 mjarrett
Stefansson 10.140.50.215 /home/kvm2/home/ Win10 george2 extract-15 bhills
Stefansson 10.140.50.216 /home/kvm2/home/ Win10 george2 extract-16 msimpson
Stefansson 10.140.50.217 /home/kvm2/home/ Win10 george2 extract-17 tark


Below is old win7 information. All had been deleted.

Deleted VM List IP Path VM Name Win ver. File Server Assigned User Notes
Hearne 10.140.50.117 /home/kvm2/extra/extract17.ovl Win7 george2 extract17 tark
Hearne 10.130.50.110 home/kvm2/extra/link10.ovl Win7 ericsson link10 lyan
Hearne 10.130.50.111 home/kvm2/extra/link11.ovl Win7 ericsson link11 csian
Hearne 10.130.50.112 home/kvm2/extra/link12.ovl Win7 ericsson link12 hyip
Hearne 10.130.50.113 home/kvm2/extra/link13.ovl Win7 ericsson link13 fxu
Hearne 10.130.50.114 home/kvm2/extra/link14.ovl Win7 ericsson link14 mjarrett
Hearne 10.130.50.115 home/kvm2/extra/link15.ovl Win7 ericsson link15 bhills
Hearne 10.130.50.116 home/kvm2/extra/link16.ovl Win7 ericsson link16 msimpson
Hearne 10.130.50.117 home/kvm2/extra/link17.ovl Win7 ericsson link17 tark
Hearne 10.180.50.110 /home/kvm2/extra/prepare10.ovl Win7 defuca prepare10 lyan
Hearne 10.180.50.111 /home/kvm2/extra/prepare11.ovl Win7 defuca prepare11 csian
Hearne 10.180.50.112 /home/kvm2/extra/prepare12.ovl Win7 defuca prepare12 hyip
Hearne 10.180.50.114 /home/kvm2/extra/prepare14.ovl Win7 defuca prepare14 mjarrett
Hearne 10.180.50.115 /home/kvm2/extra/prepare15.ovl Win7 defuca prepare15 bhills
Hearne 10.180.50.116 /home/kvm2/extra/prepare16.ovl Win7 defuca prepare16 msimpson
Hearne 10.180.50.117 /home/kvm2/extra/prepare17.ovl Win7 defuca prepare17 tark
Stefansson (/home/jim/vminfo) 10.140.50.101 /home/kvm2/data/extract1.ov Win7 george2 extract1 tchoi
Stefansson 10.140.50.102 /home/kvm2/data/extract2.ov Win7 george2 extract2 mmedearis
Stefansson 10.140.50.103 /home/kvm2/data/extract3.ov Win7 george2 extract3 mwong
Stefansson 10.140.50.104 /home/kvm2/data/extract4.ov Win7 george2 extract4 jgalo
Stefansson 10.140.50.105 /home/kvm2/data/extract5.ov Win7 george2 extract5 kalke
Stefansson 10.140.50.110 //home/kvm2/extra/extract10.ovl Win7 george2 extract10 lyan
Stefansson 10.140.50.111 //home/kvm2/extra/extract11.ovl Win7 george2 extract11 csian
Stefansson 10.140.50.112 //home/kvm2/extra/extract12.ovl Win7 george2 extract12 hyip
Stefansson 10.140.50.113 //home/kvm2/extra/extract13.ovl Win7 george2 extract13 fxu
Stefansson 10.140.50.114 //home/kvm2/extra/extract14.ovl Win7 george2 extract14 mjarrett
 ??? 10.140.50.115 /home/kvm2/extra/extract15.ovl Win7 george2 extract15 bhills
Hearne 10.140.50.116 /home/kvm2/extra/extract16.ovl Win7 george2 extract16 msimpson
Stefansson 10.180.50.113 /home/kvm2/base/virtio-win-0.1.102.iso Win7 defuca red13 (prepare13) fxu
Stefansson /home/kvm2/base/virtio-win-0.1-74.iso red10
Stefansson /home/kvm2/base/virtio-win-0.1-74.iso red11
Stefansson /home/kvm2/base/virtio-win-0.1-74.iso red12
Stefansson /home/kvm2/base/virtio-win-0.1-74.iso red14
Stefansson /home/kvm2/base/virtio-win-0.1-74.iso red15

Virtual machines - other (desktop)

Services/VM-desktop