Install image

From popdata
Jump to: navigation, search

System image plans! See also

Suggested improvements to procedure below

  • PERMISSIONS: C: is writable by "Authenticated Users"
  • Johns Hopkins ACG : after install, copy Start Menu shortcut to "Johns Hopkins ACG 11.0.1-PD"; new target "\\SREfiles\sre\.scripts\acg11-clickthrough.pyw"; delete original shortcut (which won't work, because "clicktrhough" handles license and data file)
  • Firefox: remove duplicate
  • 7-zip
    • Get new version through SourceForge, which has https (but no digital signatures either).
  • python
    • check dependencies before deleting old versions?
  • Stata
    • additional packages:
    • customize local shared library: put in c:\Program Files(x86)\Stata12\profile.do "sysdir set SITE r:\.libs\STATA\ado\site" and "sysdir set PERSONAL u:\ado"
    • consider moving packages from c:\ado\plus\{LETTER} to r:\.libs\STATA\ado\site\{LETTER}
    • make sure that C: and c:\ado is not writeable by users [PLEASE FIX]
  • R and Rstudio
    • Update R before RStudio, because RStudio may pick the newest version it finds at install time.
    • consider updating all shared packages in R:/.libs/RLIBS/3.1 Software-user-maintenance via mirror on Alfresco.
    • to upgrade decimal version (e.g. 3.2 to 3.3), first copy all packages from RLIBS/3.2 to RLIBS/3.3, then change configuration for R_LIBS_SITE to 3.3 in
      • after upgrade of RLIBS, cleanup old versions no longer in use
  • send to Bryony updates for https://my.popdata.bc.ca/html/SRE/software.html
  • SAS
    • update sas9.cfg for MEMSIZE (=> Alfresco/Systems & Security/SRE+RTL+SRTL/Software/sas-9.4-tweak/ )
      "-MEMSIZE 6G" except Fast5,7,8,9,11,12 (8G); Fast14 (0=unlimited)
      "-Xms1024M" and "-Xmx1024M" (Java memory size, default 128m)
  • Desktop : remove desktop icons for Mplus and Google Chrome

Win10 implement process 1/3 (base image file creation)

(May 2017 Windows 10 from scratch)

  • Create a new VM "w10base-" by this command "qemu-img create -f qcow2 w10base-A.img 220G"
  • -> open virt-manager and create a new virtual machine (might need "/etc/init.d/libvirtd restart")
  • -> import existing disk image
  • -> locate above /home/kvm2/base/w10base-A.img
  • -> RAM: 16386MB, CPUs: 2
  • -> name it w10base-A
  • -> check "customize config before install and finish"
  • -> add CD-ROM (Storage > IDE > CDROM device) and connect with win10 installation dvd
  • -> set boot order to boot from CDROM
  • -> start VM and install windows10. reboots a few times. must select NO to personal/data collection settings. shutdown VM when all finished.
  • -> virt-manager > reconnect CDROM with /home/kvm2/base/virtio-win-0.1.126.iso (always download the newest virtIO drivers for Windows)
  • -> virt-manager > add VirtIO disk (Storage > VirtIO). this storage is dummy and gets deleted later so size does not matter here.
  • -> virt-manager > change NIC to virtio
  • -> start VM
  • -> login as local tech and open device manager. search for unrecognized devices.
  • -> update unrecognized devices' driver with virtio-win-0.1.126.iso (VirtIO drivers will be installed)
  • -> shutdown
  • -> virt-manager > remove dummy VirtIO disk (it should have created a qcow2 file /var/lib/libvirt/w10base-A.qcow2. manually delete this file later)
  • -> (also manually delete /var/lib/libvirt/images/xxxx.qcows2 temporary files)
  • -> virt-manager > change IDE bus (/home/kvm2/base/w10base-A.img) to VirtIO
  • -> start VM
  • -> start config windows (add to domain, activate windows, change environmental setting)
  • -> virt-manager -> View -> Snapshot to take a snapshot of only Win10
  • -> install application (take a snapshot every time when big app is installed)
  • -> Some tweaks :
Prevent sysprep failure due to <PackageFullName> was installed for a user, but not provisioned for all users error, disable store apps from being installed or updated automatically. Add this reg key. 
reg add HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\WindowsStore /v AutoDownload /t REG_DWORD /d 00000002 /f
NOTE - remove this key while MS Deployment Toolkit is pushing out my base image.
 Disable the picture background from regedit: 
HKEY_LOCAL_MACHINE\SOFTWARE\Policies\Microsoft\Windows\System\ -> create "DisableLogonBackgroundImage"=dword:00000001
 Remove Authenticated Users group from C:
* File manager -> right mouse click C: to display properties-> Security tab -> Advanced -> Change Permissions
* Select Authenticated Users (one with "this folder" & other with "this folder, subfolders and files") and Remove both
* Errors will popup "An error occurred while applying security information to C:\cygwin64 Access is denied" ... "hyberfil.sys" ... "pagefile.sys" ...etc. but just skip those files and continue. 
* To confirm: open cmd as Administrator -> cd \ (to goto root directory) -> icacls . (there is a period)-> will disply current user groups with access. You should not see Authenticated Users in the list.
  • -> when the master image is ready, check 2 things
  • -> (1) services -> Windows Remote Management (WS-management) -> set this to Automatic
  • -> update VM drivers for Windows by looking up keywords such as "KVM drivers", "Windows VirtIO Drivers"
  • -> Found the latest (as of 2018 Dec) https://fedorapeople.org/groups/virt/virtio-win/direct-downloads/
  • -> download "virtio-win-0.1.141.iso" and move it to the SRE server where the master image is located (verendrye:/home/kvm2/base/)
  • -> open virt-manager -> open w10base-B and add Storage >> IDE CDROM >> select the above downloaded source (might need to restart libvirt)
  • -> Update driver one at the time from Device Manager
  • -> System devices >> VortIO Balloon Driver >> update with new driver in CD-ROM:\Baloon\w10\smd64\
  • -> Action menu >> install legacy device >> Install the hardware that i manually select from a list >> show all devices >> have disk >> Select each of driver
  • -> CD-ROM:\Balloon >> Virtio Memory Balloon driver (UPDATED with this driver)
  • -> CD-ROM:\NetKVM >> Virtio Network driver (UPDATED with this driver)
  • -> CD-ROM:\pvpanic\ >> pvpanic device is a simulated ISA device to generate a guest panic event
  • -> CD-ROM:\qemufwcfg\ >> QEMU Firmware Configuration Device
  • -> CD-ROM:\qemupciserial >> PCI serial device driver
  • -> CD-ROM:\qxl >> QXL graphics driver for Windows 8 and later
  • -> CD-ROM:\qxldod >> QXL graphics driver for Windows 8 and later.
  • -> CD-ROM:\smbus >> Red Hut SCII controller (MUST install)
  • -> CD-ROM:\vioinput
  • -> CD-ROM:\viorng >> Virtio RNG driver
  • -> CD-ROM:\vioscsi >> Virtio SCSI driver
  • -> CD-ROM:\vioserial
  • -> CD-ROM:\viostor >> Virtio Block driver
  • ... done!

Win10 implement process 2/3 (Firewall, DNS, clone.py, DHCP)

  • decide a new pattern of hostnames and IP range
  • 10.130.50.201-229 link-1 - link-16
  • 10.180.50.201-229 prepare-1 - prepare-16
  • 10.140.50.201-229 extract-1 - extract-16
  • MAC for link- 52.54.08.54.53.xx
  • MAC for extract- 52.54.09.54.53.xx
  • MAC for prepare- 52.54.00.54.53.xx
  • not important but update larsen:/cygdrive/c/cygwin64/home/Administrator/prepare-kvm.sh (add new hosts & change local tech passwd)


  • 1: Update REDZONE firewall rules on Vancouver
    • Change the IP range from 10.130.50.100-10.130.50.126 to 10.130.50.100-10.130.50.226 for each vlan/VMs.
  • 2: REDZONE DNS is mackenzie
    • When you want to update hostname/IP you need to modify 2 files "popdata.bc.ca.host" and "bind10.rev"
    • mackenzie: /etc/bind/popdata.bc.ca.host (this file contains the list of hostname and IP)
    • mackenzie: /etc/bind/10.rev (DNS entry file. this file contains the list of hostname and IP in reverse order)
    • (NOTE1) cp to save a backup file first before you vi to above 2 files.
    • (NOTE2) when you edit above files, INCREASE 1 and modify the very top of number in the file. (i.e. 1229586903 -> 1229586904) this is how bind will notice the update.
  • 3: hearne:/home/kvm2/clone.py & stefansson:/home/kvm2/clone.py
    • Add new hostname "link-" "extract-" "prepare-" and also at the "types" section
    • Add "link-" "extract-" "prepare-" new hostnames.
    • Write down the MAC address (you will need them for adding the hostname/IP on DHCP conf file)
  • 4: REDZONE DHCP is defuca, george, ericsson
    • Why 3 servers? prepare, extract and link run on different vlan. defuca = prepare VMs, george = extract VMs, ericsson = link VMs
    • /dhcp/dhcpd.conf -> backup this file
    • /dhcp/dhcpd.conf -> add a set of new hostname/IP. (10.180.50.201 to 10.180.50.217 & prepare-1 to prepare-17)
      • MAC address needs to match with clone.py
      • last 2 digits of MAC address is HEX number of 1 to 16 (HEX 01,02,03,04,05,06,07,08,09,0a,0b,0c,0d,0e,0f,10,11)
    • /etc/ini.d/ise_dhcp_server restart -> restart DHCP service
    • /var/log/dhcp/dhcp.leases -> to monitor errors
    • defuca: ericsson: george: /var/log/syslog -> if new VM can't retrieve IP form DHCP, check this log file too.

Win10 implement process 3/3 (Image file transfer and VM creation)

  • Move w10base-X.img to redzone: franklin
  • scp to stefansson:/home/kvm2/base and hearne:/home/kvm2/base
  • virt-manager
  • create a new virtual machine with using the virt-manager GUI menu
    • change IDE disk to virtO
  • Login to w10base-X from virt-manager GUI and check Win/Office license (activate if necessary)
  • shutdown w10base-X
  • use clone.py to build VM
    • /home/kvm2/clone_w10.py -t link- -l home -b w10base-X X (create link-1 on hearne)
    • if VM has network problem (cannot receive IP/host from DHCP, cannot connect to DNS), check "Win10 implement process 2/3"
  • login to newly created VM and join domain (this should be the only thing you need to manually do after the clone.py creation)
  • done!

w7base-X.img steps to update

(April 19th, 2016, updating existing Windows 7 image)

import site
site.addsitedir('R:\\.libs\\PYLIBS\\3.5')

General Use Image

  • Install SPSS Python Essentials Plugin (64bit)
  • Update Tech account to be a member of the SophosAdministrators group.
  • Zimbra Connector update

Red Zone

  • Remove Sophos Anti-Virus.

Laptop Image

  • New local admin account for instructors:
    • login: courseadmin
    • password: POP!!2006

Upload Image from SRE server to WDS

  1. at bondar verendrye: copy source img file to /home/kvm2/base/w10base-cdrom.img (w10base-cdrom.xml is configured to have CD-ROM added)
    as of 2017-10-31: correction - verendrye has pd-net 10.80.20.x connection (vlan82 via pd-dmzsw port#15)
  2. copy the latest image to w10base-cdrom.img
  3. at bondar verendrye: open virt-manager and manually start w10base-cdrom
  4. connect w10base-cdrom to pd-net (10.80.20.x) *see below for how
    pd-net is 10.80.20.x subnet group which is connected to Bridge 4 on verendrye. This is how to confirm/change.
    virt-manager NIC -> Network source: BR4 (it should be Bridge br4: Host device eh0)
    NIC -> Device model change it to rtl8139
    Boot Options -> Boot device order should be 1) IDE Disk 2) CDROM
    Restart w10base-cdrom to take an effect
  5. virt-manger (not remote desktop): login w10base-cdrom as local tech
  6. While w10base-cdrom is running thru virtual machine (not remote desktop!), disconnect virtual network cable by running below commands from verendrye terminal. *see below box for how
    This is ONLY valid when SRE server is connected to pd-dmzsw switch
    ssh verendrye
    virsh domiflist w10base-cdrom (to grab the vnet info)
    ------------------------------------------------------------
    Interface_____Type_____Source_____Model_____MAC
    ------------------------------------------------------------
    vnet12________bridge_____br4______rtl8139__52:54:00:83:fa:c9
    -------------------------------------------------------------
    virsh domif-setlink w10base-cdrom vnet12 down (to turn off vnet12)
    You will see "no connections available" error message on the network status icon on right bottom corner.
    *virsh domif-setlink w10base-cdrom vnet8 up (to connect back the network if necessary)
  7. Go back to w10base-cdrom virtual machine (it should be logged on as tech with no network) and do below.
  8. Regedit
    HKEY_LOCAL_MACHINE\SYSTEM\Setup\Status\SysprepStatus\, set GeneralizationState to 7
    HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\SoftwareProtectionPlatform\, change SkipRearm to 1 (it will reset windows data)
  9. MSDTC - Microsoft Distributed Transaction Coordinator config
    msdtc -uninstall
    msdtc -install
    *Microsoft Distributed Transaction Coordinator, is a Windows service providing transaction infrastructure for distributed systems.
    Transaction processing is a software technology that makes distributed computing reliable and is providing protection against operation failure.
    By using transactions, you can simplify the error recovery.
  10. Run sysprep
    c:\windows\system32\sysprep\sysprep.exe
    System Clean up Action: Enter System Out-of-Box Experience (OOBE)
    Generalize: mark this checkbox checked
    Shutdown option: shutdown
    OK to shutdown
  11. DON't boot up the virtual machine yet until completing this step: change the bootup option from IDE Disk to CD-ROM ONLY. Also change the disk type from virIO to IDE via virt-manager.
  12. Finally - boot up w10base-cdrom.img (virsh start w10base-cdrom)
  13. Select PXE boot if it prompted to enter key etc
  14. When PXE boot prompts, select w64 win10 capture only
  15. Windows Deployment Service will start
    Choose D: as destination
    Set name and detail as sre-20160817 (today's date)
    Click on Browse button and select D:\sre-20160817
    WDS starts capturing process that takes 30min - 1.5hrs
    Successful capturing process will return this message: The operation is complete. & The install image was created successfully.
  16. Change the boot up device order back to IDE Disk from CD-Rom from virt-manager
  17. Start up w10base-cdrom which is a fresh OS and needs to complete the initial set up wizard -> system will restart -> log off manually
  18. Login to w10base-cdrom as ./tech and and run cmd as administrator
    Copy captured file to WDS server (gilbert)
    Map drive from cmd: net use e: \\gilbert\e$
    e:
    cd jim (to check if you can see all other WIM files)
    copy c:\sre-20160817 e:\jim\ (in my case, it was somehow stored in c:)
  19. Logon to Gilbert and open WDS
  20. Open Windows Deployment Services and open menu trees (left side pane) as Servers -> GILBERT.popdata.local -> Install Images -> LPC
  21. Right mouse click on LPC and choose Add Install Image... -> Browse to select e:\jim\sre-20160817.wim (I had to add .wim file extension manually) -> Next -> Next -> Finish
  22. Done!


WINDOWS 10 you need to remove Windows 10 builtin apps before running the sysprep.exe

  • The Error: <PackageFullName> was installed for a user, but not provisioned for all users. This package will not function properly in the sysprep image.
  • The Cause: Windows 10 has a number of built-in Appx apps, such as the Microsoft Office Hub, Tips, the Feedback Hub, Duolingo, XBox and others.
    • When preparing a base image for an enterprise, you may decide that you want to remove these apps to cut down on the size of your image. However, if these apps are unprovisioned from Windows profiles, but still installed on a user’s account, Sysprep’s validation mechanism can fail.
  1. Powershell as admin -> Set-ExecutionPolicy Unrestricted (set the Execution Policy in Powershell to allow you to run the script)
  2. Generate a list of installed Win 10 app and also the installed user account information (very important to understand who installed which apps)
    Import-Module AppX
    Import-Module Dism
    Get-AppxPackage -AllUser | FL -Property PackageFullName,PackageUserInformation > c:\users\tech\document\appx.txt
  3. You can either create a script to remove unwanted app, or remove them one by one.
    Write PS script, save it as appx,ps1, and run it from elevated powershell command prompt
    PS> c:\users\tech\download\appx.ps1 (for example)
    This is a sample PS script:
    Import-Module AppX
    Import-Module Dism
    #Remove AppX Packages for unnecessary Windows 10 AppX Apps
    Get-AppxPackage *Microsoft.BingNews* | Remove-AppxPackage
    Get-AppxPackage *Microsoft.DesktopAppInstaller* | Remove-AppxPackage
    #Remove AppX Packages for Sponsored Windows 10 AppX Apps
    Get-AppxPackage *AdobeSystemsIncorporated.AdobePhotoshopExpress* | Remove-AppxPackage
    Get-AppxPackage *EclipseManager* | Remove-AppxPackage
    #Remove AppX Provisioning for unnecessary Windows 10 AppX apps
    Get-AppxProvisionedPackage -Online | where Displayname -EQ "Microsoft.WindowsCamera" | Remove-AppxProvisionedPackage -Online
    Get-AppxProvisionedPackage -Online | where Displayname -EQ "Microsoft.Xbox.TCUI" | Remove-AppxProvisionedPackage -Online
  4. Alternatively, remove unwanted Win app manually:
    Get-AppxPackage *Microsoft.Lines* | Remove-AppxPackage
    Get-AppxPackage *Microsoft.Office.Sway* | Remove-AppxPackage
    Get-AppxPackage *Microsoft.OfficeLins* | Remove-AppxPackage
    Get-AppxPackage *Microsoft.Whiteboard* | Remove-AppxPackage
    Get-AppxPackage *Microsoft.Todos* | Remove-AppxPackage

Some tweaks when you update softwares versions (Ryoko's note)

  • 2019-07-10: After installing cumulative patches with new build 1903, several configurations (in this list) got deleted.
  1. Firewall
    one of enabled inbound rules: Windows Remote Management (HTTP-In)'s profile was changed to Public > manually change this back to Private
  2. Local administrator group
    lusrmgr.msc > Group > Administrator > domain admin were deleted > add machine to popdata > lusrmgr.msc: added podpata\administrator, popdata\cyg_server, popdata\domain admins > change machine back to workgroup > reboot
  3. SNMP packet list
    SNMP service packet list was gone > add > 10.80.90.10, 10.40.10.68, 10.80.20.30, 10.80.40.10, 10.80.50.100, 10.80.20.160 (kateri)


turn on remote desktop

for new desktop you must add SNMP serves - security tab > add pd-admin and IP addresses :

  • Popdata AD Group Policy >> background-w10
    • Remove Shutdown button
    • Hide Power button from Start Menu

system environment

  • Remove PYPATH from Windows system environment variables (requested by Denis)
  • Add c:\program files\7-zip to system variables PATH (requested by IDD project)
  • Add R:\.libs\Misc\spark-2.4.0-bin-hadoop2.7\tmp\hadoop\bin to system variable path
  • Add JAVA_HOME = c:\program files\java\jdk-xx.x.x (it's important to point to JDK not JSD! This is for R's rJava package to find where the JDK is)
  • Add SPARK_HOME > R:\.libs\Misc\spark-2.4.0-bin-hadoop2.7\ (this is for PySpark python package to properly run on Windows machine)
  • Add HADOOP_HOME > %SPARK_HOME%\tmp\hadoop (this is for PySpark python package to properly run on Windows machine)
  • Add path for Rtools (R package) to be able to compile. C:\Rtool\bin (this is for make.exe) and c:\Rtool\mingw_64\bin (this is for gcc.exe)
  • Add C:\program files\postgresql\11\bin to system environment variables PATH (for Postgresql client)


install pgAdmin for postgreSQL


install python packege pyqt5 for loginv3.py

  • Note: this script needs a very specific version of python and python modules!
  • 1) Python must to be version 3.5.3 (for Python 3.5)
  • 2) Python module pyqt5 must be version 5.9 (the current one is 5.11 so it's downgrade)
  • open cmd as admin
  • cd c:\program files\python35\Script\
  • pip install pyqt5==5.9 (version 5.9 specific installation)
  • pip install PyYAML (pyyaml can be any version)
  • pip install -U pgcli (postgres client)
    • one of dependent packages "setproctitle" gets error. Manually install this by following command:
    • pip install --upgrade "c:\path\to\file\setproctitle-1.1.10-cp35m-win_amd64.whl"
    • pip install -U pgcli (postgres client)


python R: library path config

  • In SRE python library path is defined by a file called sitecustomize.py under C:\path-to-python\Lib\
    • C:\Python27\Lib\sitecustomize.py
    • C:\Program Files\Python35\Lib\sitecustomize.py
      • This is the contents of sitecusomize.py
      • import site
      • site.addsitedir('R:\\.libs\\PYLIBS\\3.5') *or 2.7

python PySpark package need LZO library

    • Copy and paste \\Gilbert\Alfresco\S&S\Software\lzo2.dll to C:\WINDOWS\System32\ so that it can be found by the PyTables extensions.

C: permission reconfig

  • Open cmd as Administrator
  • Goto c:\
  • run command: icacls . (there is a dot) it will show icacls options
  • run command: icacls . /remove "NT Authority\Authenticated Users"
      • It took 20 seconds, and the output was:
      • > processed file: .
      • > Successfully processed 1 files; Failed processing 0 files
 If above icacls command cannot remove this user group, do following. 
* File manager -> right mouse click C: to display properties-> Security tab -> Advanced -> Change Permissions
* Select Authenticated Users (one with "this folder" & other with "this folder, subfolders and files") and Remove both
* Errors will popup "An error occurred while applying security information to C:\cygwin64 Access is denied" ... "hyberfil.sys" ... "pagefile.sys" ...etc. but just skip those files and continue. 
* To confirm: open cmd as Administrator -> cd \ (to goto root directory) -> icacls . (there is a period)-> will disply current user groups with access. You should not see Authenticated Users in the list.
Need to manually change permission on following folders
... Administrators SRE30+None        0 Aug 10  2018 Program Files/GeoDa Software
        icacls GeoDa Software: Everyone:(OI)(CI)(f)  [full access to all 21 subfolders !?!]
... Administrators SRE30+None        0 Aug 10  2018 Program Files/GeoDa Software/basemap_cache
... SYSTEM         SYSTEM            0 Dec  6  2017 Program Files/UNP/Logs
        icacls UNP/Logs:               Everyone:(OI)(CI)(f)  [full access to all subfolders !?!]
... SYSTEM         SYSTEM            0 Oct  3 17:31 Program Files (x86)/Common Files/Adobe/Adobe PCD/cache
... Administrators SYSTEM            0 Oct 18 14:39 Program Files (x86)/Common Files/Adobe/AdobeGCData
... SYSTEM         SYSTEM            0 Mar  4 11:26 Program Files (x86)/Common Files/Adobe/SLCache

... SYSTEM         SYSTEM            0 Nov 20  2017 Program Files (x86)/Microsoft Visual FoxPro 9/Samples
        & 41 subfolders
... SYSTEM         SYSTEM            0 Feb 28 14:53 ProgramData
        [icacls ProgramData BUILTIN\Users:(CI)(WD,AD,WEA,WA)]
        [FLEXnet is part of ARCgis - owener = local Administrators group]]
... Administrators SRE30+None        0 Feb 28 14:46 ProgramData/Adobe/SLStore
... SYSTEM         SYSTEM            0 Aug 22  2018 ProgramData/FLEXnet
        [icacls  ProgramData/FLEXnet BUILTIN\Users:(I)(CI)(WD,AD,WEA,WA)
... Administrators SRE30+None        0 Jan 10 19:09 ProgramData/Git
... SYSTEM         SYSTEM            0 Sep 29  2017 ProgramData/Microsoft/DataMart/PaidWiFi
        [+ 9 more Microsoft]
... Administrators SRE30+None        0 Jan 10 19:11 ProgramData/MiKTeX
        [+18 more MikTeX]
... SYSTEM         SYSTEM            0 Aug  3  2018 ProgramData/Oracle
... SYSTEM         SYSTEM            0 Feb 25 19:55 ProgramData/Oracle/Java
... Administrators SRE30+None        0 Aug  3  2018 ProgramData/Oracle/Java/.oracle_jre_usage
... SYSTEM         SYSTEM            0 Aug  3  2018 ProgramData/regid.1986-12.com.adobe
... SYSTEM         SYSTEM            0 Sep 24 15:19 ProgramData/Sophos/Sophos Anti-Virus/INFECTED
        [you don't have access]
... SYSTEM         SYSTEM            0 Feb 28 15:40 ProgramData/Sophos/Sophos Anti-Virus/Temp
        [can create folder]
... SYSTEM         SYSTEM            0 May 30  2018 ProgramData/SPSS
... SYSTEM         SYSTEM            0 Aug 15  2018 ProgramData/USOShared
        [ok to delete these system trace log files; OK by me to remove write access]
... SYSTEM         SYSTEM            0 Mar  3 17:13 ProgramData/USOShared/Logs
... SRE30+tech     SRE30+None        0 Aug 10  2018 ProgramData/USOShared/WinBUGS14
... SRE30+tech     SRE30+None        0 Aug 10  2018 ProgramData/USOShared/WinBUGS14/Bugs
        [+77 more USO...Winbugs ]
... Administrators SRE30+None        0 Aug 10  2018 ProgramData/{3E6E09B5-18F5-46F0-A5C1-984CA9BE58C9}
        [uninst.dat - OK to remove write access]


cygwin conifg

  • (1) add following packages:
    • sshd module - search by openssh > select openssh: The OpenSSH server and client programs
    • emacs (editor) - search by emacs > select emacs: the extensible, customizable, self-documenting real-time display editor
    • emacs-X11
    • xinit
    • xorg-server
    • cygrunsrv (admin)
    • gcc-g++ GNU compiler collection
  • (2) configure sshd
    • start cygwin terminal as administrator
    • command: ssh-host-config
    • You’ll see the script generate some default files, and then you’ll be prompted for whether or not you want to enable “Privilege Separation.” It’s on by default in standard installations of OpenSSH on other systems, so go ahead and say “yes” to the prompt.
    • You’ll be prompted to create a new account with special privileges. Select “yes” and the script will continue.
    • Next, you’ll be asked if you want sshd to run as a service. This will allow you to get SSH access regardless of whether or not Cygwin is currently running, which is what we want. Go ahead and hit “yes” to continue.
    • You’ll be prompted to create a new account with special privileges. Select “yes” and the script will continue.
    • Next, you’ll be asked if you want sshd to run as a service. This will allow you to get SSH access regardless of whether or not Cygwin is currently running, which is what we want. Go ahead and hit “yes” to continue.
    • Next, you’ll be asked to enter a value for the daemon. Enter "ntsec"
    • You’ll see the script give you some information on your system and then it will ask you to create a privileged account with the default username “cyg_server”. The default works well, so type “no” when it asks you if you want to use a different account name, although you can change this if you really like.
    • enter a password for this account as well. (I used same windows admin password)
    • Cygwin will show you your password in plain text for verification, so be sure you’re in a secure place. You’ll see some extra info come up and if all’s well, you’ll get a message that says it successfully completed.
    • You can either restart, or enter the following command to start the sshd service: "net start sshd"
  • (3) create ssh folder, permission, copy key files
    • At the cygwin terminal, cd to /home and create .ssh (chmod 755)
    • copy .ssh/authorized_keys (chmod 400) from existing sre machine
    • do same for /home/Administrator
    • chown -R Administrator /home/Administrator/.ssh
    • cygrunsrv -S sshd (this is a cygwin command to start sshd service manually)
    • cygrunsrv -E sshd (this is to shutdown sshd service)
  • (4) change service login account from .\cyg_server to popdata\cyg_server1 (passwd same as win admin root)
    • Open services >> cyg_sshd >> log on tab >> this account popdata\cyg_server1
  • Popdata AD >> Group Policy >> gpedit.msc >> sre/srtl/rtl-new -> Edit
    • make sure to add popdata\cyg_server1 to following policies.
    • Act as part of the operating system
    • Create a token object
    • Deny log on locally
    • Deny log on through Remote Desktop
  • Another note on how to config cygwin
    • ssh-host-config
    • overwrite = yes
    • strict = yes
    • CYGWIN = ntsec
    • user to run = cyg_server
    • password = {...}


WINRM configuration

  • open powershell as admin
  • Get-NetConnectionProfile to check the network property. If it said public, you need to change it to private by:
  • Set-NetConnectionProfile -NetworkCategory Private


  • open powershell as admin
  • Enable-PSRemoting


  • open cmd as administrator
  • winrm get winrm/config (command to check the state of config)
  • if nothing has ever configured, then run winrm quickconfig to go thru the configuration
  • set the winrm service type to delayed auto start -> make this changes? -> Y
  • enable the winrm firewall exception -> make this changes? -> Y
  • winrm e winrm/config/listener (compare output of this command with other sre machines)
  • Other helpful commands:
    • winrm help config
    • winrm get winrm/config
    • winrm get winrm/config/client
    • winrm get winrm/config/service
  • After finish Windows Remote Management service configuration, open service > select win RM > change Automatic Delayed start to Start (for faster clone.py creation)

Local firewall setting

  • inbound rules
  • there should be 2 Windows Remote Management rules ENABLED.
  • windows remote management (HTTP-In) properties > Advanced tab > profile set to Private > Scope tab > Remote IP address: Local subnet and 10.80.20.0/24 (gilbert)
  • windows remote management (HTTP-In) properties > Advanced tab > profile set to Domain > Scope tab > leave both blank (default)
    • Make sure to "enable" both of above WinRM (HTTP-IN) inbound rules


Registry key for Remote UAC and WMI issue

    • Remote UAC and WMI issue - https://rakhesh.com/windows/wmi-access-denied-for-remote-machine-etc/
    • In a workgroup, the account connecting to the remote computer is a local user on that computer. Even if the account is in the Administrators group, UAC filtering means that a script runs as a standard user.
    • Create a registry key: HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System – and create a new name called LocalAccountTokenFilterPolicy (of type DWORD_32) with a value of 1


Local Users and Groups

  • run lusrmgr.msc
  • open Groups > Administrators > make sure you have below list
    • Administrator
    • cyg_server
    • popdata\administrator
    • popdata\cyg_server
    • popdata\domain admins
    • tech


Add SNMP protocol

  • control panel > turn windows features on or off menu > tick SNMP and also WMI SNMP provider
  • open services > open SNMP Services > security tab > add "pd-admin" as READ ONLY > add below IP for accept SNMP packet list
10.80.90.10
10.40.10.68
10.80.20.30
10.80.40.10
10.80.50.100
10.80.20.160 (kateri)


Windows System Restore config

  • System property -> System Protection tab -> Configure button
  • Leave the Restore Protection "on" by NOT selecting "Turn off system protection"
  • Set "disk space usage" down to the lowest (1%) so that there is only room to save one restore
  • Delete button to delete all restore points

R

  • When you upgrade R, you must also config c:/Program files/R/R-3.2.5/etc/Renviron.site
  • ## Local for Popdata SRE
  • R_LIBS=R:/.libs/RLIBS/3.2
  • R_MAX_VSIZE=100Gb
  • Also make sure you see R_LIBS_USER -> U:/RLIBS at System Variables under Environment Variables
  • At the Command Prompt (CMD) run these commands to set the variables:
  • "C:\Program Files\R\R-3.2.5\bin\x64\Rgui.exe" R_LIBS=R:/.libs/RLIBS/3.2
  • "C:\Program Files\R\R-3.3.2\bin\x64\Rgui.exe" R_LIBS=R:/.libs/RLIBS/3.3
  • Confirm above updates with this R command .libPaths() you must see: (1) "R:/.libs/RLIBS/3.2", "U:/RLIBS", (3) "C:/Program files/R/R-3.2.5/library"
  • Open each version of R and run the command "update.packages(checkBuilt=TRUE, ask=FALSE)" (This command has to be run every time minor version changes
  • Install Rtools (R package compiler) to default local path C:\Rtools. Always download the newest and compatible version. Current installed version is Rtools35.exe. Make sure to tick on "add to path" option to add c:\Rtools to system environment PATH.

RStudio

  • Set Rstudio's version of R: open menu Tools and pick Global Options; select R version, then Change, and select R-5.3.6 (the most updated R)


SAS

  • For DSU team (Fan and Brent RZ machines) you need to install SQL and MS Office connector drivers for SAS
  • (1) SQL server native client
  • (2) MySQL connector/c 6.1
    • search and download: mysql-connector-c-6.1.2-winx64.msi
  • (3) Add MySQL connector's library path to PATH
    • "C:\program~\MySQL\MySQL connector\lib"
  • (4) The SAS PC Files Server (It communicates with the SAS LIBNAME engine, PCFILES, running on Windows, Linux, or UNIX.)


  • Installation file: //teamshare.ead.ubc.ca/Team/SLMS/DesktopApps/SAS/SAS 9.4/SAS 9.4 Windows/Win x64 February 2016
  • Run setup.exe will start SAS Deployment Wizard
  • Install SAS software (new installation)
  • SAS Home: c:\program files\SASHome
  • Install SAS Foundation and Related Software
  • leave default and Next
  • 64bit Native Mode
  • leave default "select sas foundation products" and Next
  • Specify sas installation data file... -> Browse... and locate the newest license file
  • un-select all the languages except English
  • leave the rest of wizard as "default"
  • Manage this SAS Software Depot (updates, patches, hot fixes etc)
  • Check for Hot Fixes (for new updates)
  • Open SAS Enterprise Guide and run the updater (download the newest version. Do not use 7.100.1.2856)
  • (below SAS tweak is a future plan and do not configure unless Denis adds temp folder in the create user folder script)
      • Map U: to any of sre server's home directory and create "temp" folder
      • Set the U:\temp folder permission: drwxrwx, owner: user-project-id, group: users
      • Modify SAS short-cut icon on desktop/installed program list with below as "Start in":
      • "C:\Program Files\SASHome2\SASFoundation\9.4\sas.exe" -CONFIG "C:\Program Files\SASHome2\SASFoundation\9.4\nls\en\sasv9.cfg" -work "U:\temp"

SAS "PROC TRAJ * the newest installation package from teamshare comes with traj macro

  • Installation file: http://www.andrew.cmu.edu/user/bjones/download.htm
  • Unzip trajv9-32.zip
  • traj.dll, trajplot.sas, trajplotnew.sas, trajtest.sas
  • DLL files must be individually copied to: C:\program files\sashome2\sasfoundation\9.4\core\sasexe\traj.dll
  • SAS files must be individually copied to: C:\program files\sashome2\sasfoundation\9.4\core\sasmacro\trajplotnew.sas

SAS JAVA MEMSIZE

  • C:\Program Files\SASHome\SASFoundation\9.4\nls\en\sasv9.cfg
  • Change java memory size to 1028MB at JREOPTIONS section: Xms128m -> Xms1024M, Xmx128m -> Xmx1024M
  • Change the MEMSIZE 2G -> 0G (0G will set to max. I tested with 8G which didn't change the SAS speed much).


Pharmacoepiemiology Toolbox SAS and R

  • Pharmacoepidemiology Toolbox. Java script.
  • http://www.drugepi.org/dope-downloads/
  • Denis copied at R: .libs folder.
  • we do not need to install this tool individually. Just give users "path" to this toolbox files.


SPSS

  • License server: spss.lm.ok.ubc.ca
  • Installation file: //teamshare.ead.ubc.ca/Team/SLMS/DesktopApps/?
  • Modify C:\Program files\IBM\SPSS\Statistics\23\readerprod.inf to change the "DaemonHost" to spss.lm.ok.ubc.ca
  • Modify C:\Program files\IBM\SPSS\Statistics\23\spssprod.inf to change the "DaemonHost" to spss.lm.ok.ubc.ca
  • Set Environment Variables -> System variables -> Variable: LSHOST & Value: spss.lm.ok.ubc.ca
  • c:\program files\IBM\SPSS\Statistics\23\lswhere.exe (This program connects to configured SPSS server)

Stata

  • Installation file: \\Gilbert\Alfresco\S&S\Software
  • Stata uses ado files as functional module and the default directory is C:\Windows\ado
  • centralized STATA module location = fraser:/data/sre/.libs/STATA/ado/site (alphabetically stored)
  • To change the ado file default directory from C: to R:, copy "profile.do" from fraser:/data/sre/.libs/STATA/ to C:\Program files\Stata16\
  • Open STATA and run command "sysdir" to confirm the changes
   Successful result should look as this:
   STATA: c:\program files\stata16
    BASE: c:\program files\stata16\ado\base
    SITE: r:\.libs\stata\ado\site\
    PLUS: c:\ado\plus\
   PERSONAL: u:\ado\
   OLDPLACE: c:\ado\


MikTex

  • To use R:/.libs/MIKTEX as the centralized package repository, you have to do following steps (a bit tricky).
  1. Firstly this has to be done on the main image w10base-X.img, so use virt-manager to open the main image
  2. Open MikTex console as Admin
  3. Select "Settings" menu from left pane
  4. Package are installed from: click on Change... button on right
  5. Select Local package repository and Next button
  6. Browse to bring out the popup window and map R: to \\fraser\sre\99-t01 (as test user)
  7. When you done you should see R:/.libs/MIKTEX
  8. Done!


Tree.exe

  1. copy tree.exe from \\gilbert\alfresco\S&S\software\ido_sofware\tree.exe
  2. paste to c:\program files\git\usr\bin\tree.exe


Adobe to stop opening the cloud at the startup

  • Regedit
  • sets Adobe Creative Cloud to "Disabled" in Task Manager > Startup. I have yet to figure out how to script the checking of that "Launch at Login" box, but this solution works for us.
  • Path: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Explorer\StartupApproved\Run32
  • Name: Adobe Creative Cloud
  • Type: REG_BINARY
  • Data: 030000009818FB164428D501

Requested Software

Not that we will necessarily install it.

  • OTRS Simio
  • OTRS rserver from Microsoft
  • OTRS SAS enterprise Miner
  • OTRS JAGS & STAN

Latest Master Image

See also [Services/KVM]

Server Machines Image-file date bytes C:\flagfile MD5(1G) NOTE
Bondar OCWA 7 ocwa *.qcow2 Aug-28-2019
Drake rtl1-2 srtl1-9 w10base-A.img Aug-21-2019 81543233536 SRE-20190821 b19423be70f76b5be604b2037c2c1517 new
Drake rtl1-2 srtl1-9 w10base-B.img May-31-2019 81543233536 7e7296fe1a647983d9d4905f73c253fe
Gaboury bruce1-6 w10base-A.img Jul-30-2019 80942006272 BRUCE-20190718 6f230993525cf84098237af3c938c131 fix R config
Hubbard sre26,27,29,30 w10base-A.img Aug-21-2019 89068929024 SRE-20190821 b19423be70f76b5be604b2037c2c1517 new
Hubbard sre23-31 w10base-B.img Jul-31-2019 89062113280 SRE-20190731 debe2106f5ae94db7219ae17e3a841d2 old
Mance sre38-48 w10base-B.img Jul-31-2019 89068929024 SRE-20190731 674f1da55f415425a33f3f69209d40c4
Noyon sre ?? w10base-A.img Aug-21-2019 89068929024 SRE-20190731 b19423be70f76b5be604b2037c2c1517 new
Noyon sre1-10 w10base-B.img Jul-31-2019 89060343808 SRE-20190731 bf100983f19f68e5d20d14b66507127c
Verendrye sre ?? w10base-A.img Aug-21-2019 89068929024 SRE-20190821 b19423be70f76b5be604b2037c2c1517 new
Verendrye sre11-22 w10base-A.img Jul-31-2019 89068929024 SRE-20190731 fb4783d17f2a4c6f7f90df229c84e903
  • 2019-08-27 updated by DL in preparation for patching.
  • DO NOT modify a base image while vminfo says it's in use.
  • Gaboury - latest BRUCE image w10base-A.img C:\BRUCE-20190718
    • (this version has a wrong R config file. Need to fix it in image, or manually after each rebuild.)
    • DO NOT update during office hours.
    • DO NOT USE sre images from other sre servers! Always use images within gaboury only. This is because images on gaboury have different hard disk size (not 220 but 720GB) which somehow cause CPU overload.


  • drake - RTL w10base-B.img C:\SRE-IMAGE-20190515
  • drake - SRTL w10base-B.img C:\SRE-IMAGE-20190515


  • FAST1 to 14
    • WDS: gilbert\e:\RemoteInstall\Images\LPC\SRE-IMAGE-20190712 (C:\SRE-20190710)
    • 2019-08-02: in next upgrade, add tree.exe to c:\program files\git\usr\bin\, upgrade winstep to 4.4.5 add SAS add-on PC file server
  • FAST15 to 20 & BRUCE11, 12, 13
    • WDS: kateri\c:\win10images\Images\Windows 10\SRE-IMAGE-20190712 (C:\SRE-20190710)
    • 2019-08-02: in next upgrade, add tree.exe to c:\program files\git\usr\bin\, upgrade winstep to 4.4.5, add SAS add-on PC file server


  • Hearne (redzone) - currently most of VM use w10base-A.img but w10base-B.img is the most updated image (SRE-IMAGE-20180924) so do upgrade when you have a chance.
  • Stefansson (redzone) - currently most of VM use w10base-A.img but w10base-B.img is the most updated image (SRE-IMAGE-20180924) so do upgrade when you have a chance.
  • SFU2 - latest SRE image w10base-B.img C:\SRE-IMAGE-20180110
    • including the software for IDO, Mplus Mixture add-on, ActivePearl, R3.5.1, R3.5.2, git, MikTex


  • Command to check md5sum of the image file : head -c 1G w7base-A.img | md5sum
    • Note that due to the need for updating image after every copy for updating software licenses (ex: arcgis), md5 sum different on each server.


latest SRE image w10base-A.img C:\SRE-IMAGE-20190313 including the software for IDO, Miktex, Cygwin g++ compiler, Mplus 8.2 with combo, ActivePearl, R3.5.1, R3.5.2, path c:\prog~\7-zip, Winsteps, jdk11.0.2 with Java_Home path, Rtools with 2 paths, removed writable access by icacls (only half way due to the error), Gephi, UCINET Trial edition (this has to be reinstalled with a paid license 60days from 2019-03-13)

* Latest image with update KVM drivers 2017/06/16: a4fad160329d972568ee378eb3918bc7

  • Hubbard - latest SRE image w7base-B.img C:\SRE-IMAGE-20180713
  • Verendry - latest SRE image w7base-B.img C:\SRE-IMAGE-20180731
  • Noyon - latest SRE image w7base-A.img C:\SRE-IMAGE-20180713
  • drake - SRTL/RTL w7base-B.img C:\SRE-IMAGE-20180514
  • FAST WDS: gilbert\e:\RemoteInstall\Images\LPC\SRE-IMAGE-20180918 (C:\SRE-IMAGE-20180828)
    • Missing software/actions (compared with SRE) M-Plus -combo ver8.2, Edge icon not removed. sasv9.cfg MEMSIZE needs to set 0G, IDO software, ActivePearl, remove R-3.5.0 and install R3.5.1 & R3.5.2 (change path R:), git, gvim, jre, sublime, weka, MikTex (install on-the-fly & R: config), add c:\program files\7-zip to system PATH, add cygwin GNU compiler and rest of listed software in SRE.