From popdata
Jump to: navigation, search

Back to Systems & Security

General Info

Active Directory Domain Controller

  • gilbert (

SRE Virtual Machines

  • on 6 KVM servers: Bondar, Drake, Hubbard, Noyon, Verendrye
  • can be accessed with VMware vSphere client on (login as Administrator)

File Server:

  • and the 6 KVM servers.
  • samba shares:
    • //fraser/${USERNAME} (per-user home directories) have symlinked sub-folders EXPORT_FROM_SRE and IMPORT_TO_SRE
    • //${SERVER}/sre share on 7 servers is accessible from all SRE machines.
  • yellow folders

Systems: servers, storage, VMs ...

See Systems

Software: Install, License, Notes

See Software Authentication: Passwords, LDAP servers, Yubikeys =

  • VPN server ( , Cisco ASA): authentication via passphrase+Yubikey for groups SreYubi PopData and Sysadmin, but via password only (server ldap.popdata) for Groups CHSPR, Course, Help.
  • SRE windows machines: authentication via password through Active Directory server Gilbert, which uses the same hashes as ldap.popdata. Authentication to SMB shares on Fraser also uses LDAP. Note that most SRE users have a separate account for separate projects; these accounts normally have the same password, as set on the portal page login tab.
  • some PopData web servers: password (popdata LDAP server), in particular portal.PopData.
  • PopData administrative web servers: password on ldap.popdata .
  • SSH to various PopData servers: password on ldap.popdata .
  • shares on Gilbert (chspr-shared ...): password on ldap.popdata
  • Changing LDAP password: by user on portal.popdata (if they remember the current password, labelled "Old Password" in form), or by an administrator on pds.popdata.
  • Setting up Yubikeys:

Cloning a machine (by hand)

  • power off template and targe
  • ssh to ESXi server
  • nohup cp .../template/template-flat.vmdk .../target/target-flat.vmdk
  • power on target
  • change name, change IP, reboot
  • join domain, reboot

Configuring Fraser

Configuring system - samba dnotify yellowfolders base directories

  • apt-get install dnotify samba
  • install /etc/incron.d/yellowfolders [list of watched files/directories; was: /etc/init.d/yellowfolders]
  • install /etc/cron.daily/yellowfolders-maint [clean up old files; collect MD5 from new project Data]
  • logfile rotation: /etc/logrotate.conf , /etc/logrotate.d/
  • data partition
mkdir /data
mkdir /home/share
mkdir /home/sre
mkdir /home/saved
cd /data
ln -s /home/share .
ln -s /home/sre .
ln -s /home/saved .
cd /home/share
mkdir incoming outgoing
mkdir /home/sre/users

See also SRE-in-out-software for details of yellowfolder transfer software.

create directories

See also Systems/Account_Management

scripts in /usr/local/bin/ :

  • create_user_dirs : hourly cron job (*:01) runs 5 other create_* scripts.
    • create_homes : Create directories for all sre-user: incoming, outgoing, EXPORT_FROM_SRE, IMPORT_FROM_SRE, users, home (& links to incoming & outgoing .
      • 2014-06-10 DL Stop creating obsolete users/USER .
    • create_project  : Creates directories and links (e.g. to "create_home" directories) for all members in a project (as previously set up in PDS).
    • create_rtl_user_dirs {username} : create directories for student logins.
    • create_rtl_free_user_dirs {username} : create directories for self-registered student logins. No IMPORT links.
    • create_srtl_user_dirs {username} : create directories for student logins on Secure Remote Training Lab. No IMPORT or EXPORT links.
      • Edit script to add new courses.

Folders and permissions for SRTL courses

See Courses#Creating a course on SRTL

Changes for RTL / SRTL

2012-apr-11 planning for RTL and SRTL

  • use same SMB "home" share for outside view of IMPORT/EXPORT
  • change RTL users to use "home" folder
  • use different SMB share for inside view (R:, U:)
  • different Active Directory GPO for RTL, SRTL
    • different scripts:
  • separate back-end scripts and cron jobs (create_rtl_user_dirs create_srtl_user_dirs)
    • create_rtl_user_dirs [done]
      • use /data/sre/users/$i as test, so can disable transfer per user)
      • create subfolder users/$i/TRANSFER/ with symlinks IMPORT_TO_RTL EXPORT_FROM_RTL IMPORT_IT.lnk EXPORT_IT.lnk
      • create subfolder EXPORT_FROM_RTL and IMPORT_TO_RTL
  • create parallel directory trees /data/rtl /data/srtl (mirrors of /data/sre) and /data/savertl
    • users/ groups/ data/ courses/
    • /data/sre/IMPORT_TO_RTL /data/sre/EXPORT_TO_RTL .../.trigger
    • .scripts/ off.bat transfer-in.lnk transfer-in.pyw -> transfer-out.lnk transfer-out.pyw ->
    • .login/ .login/logins.txt .login/users/ .login/hosts/ .login/2010/ .login/2011/
  • Status script on Cartier => rtl/available.
    • Create new webpage rtl/available ; Create DNS aliases RTL1->SRE1...
  • All courses get a shared folder "working" writeable by all. Inherited permissions: writeable by group.
  • Some courses want a folder "SUBMIT" containing 1 folder for each student. Permissions: owned by student, read+write access by instructor group.
    • EX: STUDENTS=`getent group course-phda05 | sed -e 's/.*://' -e 's/,/ /g' -e 's/rarim//'`
    • cd /data/rtl/Courses/PHDA05 && mkdir SUBMIT && chown rarim:course-phda05-instr SUBMIT && chmod g+sw SUBMIT
    • cd SUBMIT && for D in $STUDENTS; do mkdir $D; chown $D $D; chmod g=rwx $D; done

link spaghetti

root@fraser/sbin# df -h /data /extra /home
Filesystem            Size  Used Avail Use% Mounted on
/dev/cciss/c0d0p7      45G  181M   42G   1% /data
/dev/sdb1             980G  395G  575G  41% /extra
/dev/mapper/sre-data  867G  750G   73G  92% /home

# /extra was created to expand the conceptual folder /data from being all on /home

# 2012-07-10,10:00  cd /data/sre && find . \( -maxdepth 2 -type l \) -exec ls -ld \{\} \; 

# The following should be changed to /extra/sre , leaving room for /extra/rtl ...
 11-012 -> /extra/11-012
 users -> /extra/users
 groups -> /extra/groups

# The following is for sre-users who need to look at course materials (unlikely?)
 Courses -> /data/rtl/Courses

# The following are for pretty clickable link, with no black window
 .scripts/transfer-in.pyw ->
 .scripts/transfer-out.pyw ->
 IMPORT_TO_SRE/GET-IT.lnk -> /data/sre/.scripts/transfer-in.lnk
 EXPORT_FROM_SRE/SEND-IT.lnk -> /data/sre/.scripts/transfer-out.lnk

# The following per-project links could be eliminated after changing all [done] 
# , and login.bat Active Directory scripts to use \\fraser\ instead of R:\ 
{PROJECT}/.trigger-in  -> /data/sre/IMPORT_TO_SRE/.trigger
{PROJECT}/.trigger-out -> /data/sre/EXPORT_FROM_SRE/.trigger
{PROJECT}/.login -> /data/sre/.login
{PROJECT}/.scripts -> /data/sre/.scripts

# On the other hand, the user-specific links in TRANSFER directory provide convenience
  find finished at Tue Jul 10 09:52:53

setup samba smb.conf

    obey pam restrictions = no
    domain master = no
    local master = yes
    preferred master = no
    encrypt passwords = true
    security = user
    wins support = no
    wins server =
    dns proxy = no
    netbios name = fraser
    workgroup = POPDATA
    os level = 60
    server string = POPDATA SRE Server
    passdb backend = ldapsam:ldaps://
    ldap machine suffix = ou=Computers
    ldap group suffix = ou=Groups
    ldap suffix = dc=popdata,dc=bc,dc=ca
    ldap user suffix = ou=Users
    ldap admin dn = cn=admin,dc=popdata,dc=bc,dc=ca
    ldap idmap suffix = ou=Users
    os level = 60
    syslog = 0
    max log size = 1000
    log level = 3 passdb:5 auth:10 winbind:2
    log file = /var/log/samba/log.%m
    panic action = /usr/share/samba/panic-action %d

   comment = All Printers
   browseable = no
   path = /var/spool/samba
   printable = yes
   public = no
   writable = no
   create mode = 0700

   comment = Printer Drivers
   path = /var/lib/samba/printers
   browseable = yes
   read only = yes
   guest ok = no

   comment = SRE Data
   path = /data/sre
   browseable = yes
   writable = yes
   read only = no
   guest ok = no
   admin users = jim

   comment = Data incoming to SRE
   path = /data/share/incoming
   browseable = yes
   writable = yes
   read only = no
   guest ok = no

   comment = Data outgoing from SRE
   path = /data/share/outgoing
   browseable = yes
   writable = yes
   read only = no
   guest ok = no

Authorization scheme for SRE/RTL/SRTL and VPN via PDS web form

(prior to rtl change ~ 2012-05) A username can be a member of one or more of following LDAP groups:

user type django boolean ldap group employeeType VPN group
SRE sre_user sre-users (SecurID) sre
RTL student_user rtl-users rtl (was pd-course-policy) rtl
SRTL srtl-users srtl-users (SecurID) srtl
TEST sre_test_user  ?? pd-sretest-policy  ??
Disabled - - pd-no-access -

After RTL change, the employeeType may change to be a multi-valued list identical to VPN group.
Similarly one username may be member of more than one ldap group, but this makes implementation of RTL machines on SRE hardware difficult within Active Directory.

Add a SRE user

Performed by RLU staff: Follow the steps below to set up approved team members with SRE access:

Prior to giving a user SRE access: Ensure the individual has met all of the requirements for project-specific SRE access prior to setting them up:

  • They are approved to have data access under the specific project.
  • They have signed a project-specific Pledge of Confidentiality.
  • They have completed and passed PopData's Privacy Training.
  • They are listed as team members in the project Ethics application.






One of the following should work to securely block import/export

  • Options file. create /data/saved/*PORT_*_SRE/user/.options/inspect-all (script yf-restrict)
  • Trigger links. remove project symlinks .trigger-in -> /data/sre/IMPORT_TO_SRE/.trigger and/or .trigger-out. NOTE: SRE users still have access to create trigger directories if they know how.
  • Access to folders. Remove or change permission of directory {user}/TRANSFER or script or folder symlinks.

IDO/IDD projects use OCWA

See also SRE-in-out-software#Transfer_software

  • "IDO" Project 18-g01 disables import/export for each user - create-project takes care of it when run hourly
    • OBSOLETE Ask IT staff to: Remove all access to TRANSFER folder from group and others, making it invisible and unusable.
    • Done on Mance and Gaboury:
      cd /data/sre && find *-g0* -maxdepth 3 -path '[12][0-9]-g0[0-9]/*/TRANSFER' -ls -exec chmod 500 \{} \; -exec ls -ld \{} \;
  • P='18-g01'; U='rafaelph-18-g01' ; yf-restrict in $U "never" ; yf-restrict out $U "never" ; chmod 000 /data/sre/$P/$U/TRANSFER

POC ODC projects restrict exports

  • Provincial Overdose Cohort projects (all on server Hubbard)
    19-023o DPanagioto- , jlim-
    19-024o harderk-
    18-176 mmedearis- (?data preparation project?)
    99-odc Export control. aslaunwhit- cxavier- mmedearis- (export control)
  • See also drawing file ODC-import-export.odg in //Gilbert/Alfresco/Systems & Security/SRE+RTL+SRTL/Planning/
  • For export: use symlinks to a folder shared between ODC projects and 99-odc reviewers
    • /data/sre/{project}/{user}/TRANSFER/EXPORT_TO_SRE -> /data/sre/ODC_EXPORT_SUBMIT/{user}
    • /data/sre/99-odc/OUTGOING/{user} -> /data/sre/ODC_EXPORT_SUBMIT/{user}
      • ODC_EXPORT_SUBMIT has read-only permission by group odc-exporters (ODC researchers + members of 99-odc)
      • per-users subdirectories are owned by {user} and full access by group 99-odc (ODC reviewers),
        but no access by other users even if members of group odc-exporters
  • Do once for project 99-odc
    • add all members of group 99-odc (inspectors) to group odc-exporters (exporters + inspectors)
    • BASE="/data/sre/ODC_EXPORT_SUBMIT"; mkdir ${BASE} && chown root:odc-exporters ${BASE} && chmod u=rwx,g=rx,o= ${BASE}&& ln -s ${BASE} /data/sre/99-odc/ODC_EXPORT_SUBMIT
  • Do for each user
    • add to group odc-exporters
  • Then create directory, and replace links in TRANSFER folder
    • P="18-176"; U="mmedearis-18-176"; REAL_FOLDER="/data/sre/ODC_EXPORT_SUBMIT/${U}";
    • mkdir ${REAL_FOLDER} && chown ${U}:99-odc ${REAL_FOLDER} && chmod u=rwx,g=rwxs,o= ${REAL_FOLDER}
    • cd /data/sre/${P}/${U}/TRANSFER && rm -i *PORT* && ln -s -T ${REAL_FOLDER} /data/sre/${P}/${U}/TRANSFER/EXPORT_FROM_SRE
    NOTE: "-T" means that LINK_NAME must not be a pre-existing directory

    • As per "Researcher Guide ... export", user emails ;
    • (if permission lacking, Melissa emails sre@popdata.. & IT staff change permissions.
    • Melissa-99-odc inspects, exports then emails the file to researcher.

Delete a SRE group

  • (1) Delete from PDS
    • Click on group under "Authentication and Authorization" menu
    • Find the group "Woods" to be deleted
    • Click on "X DELETE" button _> Yes I'm sure button
You will be prompted with "Are You Sure?" message. You should see two groups "group" and "pop group". In this example it said "Group: Woods", "Pop group: Woods". 
If you see any other group, then it means the group account is cascaded to other group. Be careful to delete cascaded accounts.
    • Now click on pop group under "groups" menu
    • Find the group "Woods" again (you should not see it but if you do, delete it as well)
  • (2) Delete from LDAP
    • Select "Popdata LDAP Server" and login
    • User: cn=admin,dc=popdata,dc=bc,dc=ca & Passwd: unix root
    • Expand "dc=popdata,dc=bc,dc=ca (18)"
    • Expand "ou=groups (42)"
    • Expand "ou=sre-groups (50+)" (or maybe "ou=sre-projects (50+)" if you can't find it)
    • Click on Woods and select "Delete This Entry" menu
    • Delete button
  • All done!

Prepare data for release to SRE

Release data to a group/user on SRE

RLU reviews the data release on George using their RedZone Windows virtual machines. The files appear to them as:
G:\popdata\out\transferred\sre~ {PROJECTNUMBER} ~ {COHORTID} ~ {DSUMEMBER} ~ {TIMESTAMP} ~ {RLUMEMBER} ((ignore spaces, substitute UPPERCASE, ex: G:\popdata\out\transferred\sre~06-018~cohort1~bhills~20120302_104856~tchoi\ ); as part of the "release" software, the file is then transferred by RLU to Fraser in a folder by the above name, under /data/transfer/outgoing-sre/

S&S staff when instructed by RLU move data under the specified project in sub-folder data/sre/ {PROJECTNUMBER} /DATA/ {FINALTRANSFERDATE} /

The file permissions should be changed to "ug=r,o=" (440), owner=datamaster, group={PROJECTNUMBER} , and folder permissions to "ug=rx,o=" (550) owner=root group={PROJECTNUMBER}

Maintenance of documentation for SRE users (vpn, rdc ...)

Moving data in/out of SRE -- how, restrictions. ...)

Contacting SRE users by Email

See SRE-user-lists for how to list and manage various users of SRE using PDS web form, or django command-line, or ldapsearch command-line.

  • is maintained via Sympa on
  • New SRE users automatically added via email to from PDS when creating research project sub-accounts.
  • Deletions are manual (basically on request).

Alternately from a Unix machine do: set shell variables:
-or- LDAP_LOGIN="CN=admin,dc=popdata,dc=bc,dc=ca"

List members of a group (* for all groups)
ldapsearch -b ou=groups,$LDAP_BASE -h $LDAP_HOST -D $LDAP_LOGIN -W -x cn='sre-users' memberUID
enter password as requested
SAMPLE RESULT: memberUid: dlaplante-11-s02 <newline> memberUid: dlaplante [...]

List all logins, with emails:
ldapsearch -b $LDAP_BASE -h $LDAP_HOST -D $LDAP_LOGIN -W -x dn uid displayName mail employeeType > ldap_champlain_popdata_110607_all.txt
enter password as requested

  1. < ldap_champlain_popdata_110607_all.txt >
  2. Open the .tab file in excel

Monthly reports to Data Stewards

See SRE-transfer-reporting

Course access

See Courses Issues

PHDA01 "course" "project" 11-s07 issues


Other SRE user issues

For example: multiple machines; can't reconnect; reconnect vs logout See SRE-User Issues

SRE uptime and usage Monitoring

Script runs on gilbert. It is started via the Task Scheduler (on start menu)

Click on the the "Task Scheduler Library", and you will see "SREStatus" as the name. You can right click and choose "Run" or "End". The normal status is "running" (click on action "refresh" to confirm).

The actual script is on gilbert c:\python2.6\python.exe c:\

The script saves lots of things in the "pds" mysql database running on cartier. This is then read by the portal/pds application (cartier: /home/www/pds)

In addition, is run via AD GPO on each login, and updates PDS database via URL . See Software-sys-maintenance

  • Data usage logged weekly (Wed 05:33) for each project's DATA and other folders

total, data))

    • Visible in PDS group view fields: Total Space, Data Space, Net Space, Average net Space.

SRE SAMBA log tracing

It's laborious to check who opened which files when.

  • See sample _export-xhuang-17-113_2018-10-30.txt in Alfresco/Systems & Security/SRE Reporting/Research Notes/2018-10/
    1. To trace files opened by user UU-PP-NNN, first look at VPN login on Cabot (or for older, copy under "SRE Reporting", and get IP address
      EX: xhuang 2018/10/30 09:41:15 Canada sreyubi
    2. run " -a" on Project's file-server to find if user is logged in and has files open
    3. Ex: find name of SMB logfile for user xhuang in past 120 minutes on date 2018/10/30 using internal VPN IP
      cd /var/log/samba ; find . -maxdepth 1 -mmin -120 -exec /usr/local/bin/ -m 2 -Hn -e '^2018/10/30.*(10\.80\.15\.164|xhuang)' \{} \;
    4. Examine in more details the SMB log files of interest, which may cover many weeks. "" without arguments will show usage message.
      For example "-e filename.xlsx" would only show lines matching that extended regular expression.
      Note that most SMB sessions will have authentication logs in file named from IP, and file access logs in files named after windows client's internal name.
    5. Make a copy of important log files, because they get discarded after they fill up (first they get renamed

SRE and VPN access logs

See about VPN logs at Software-sys-maintenance#scanvpn.pl_on_Cabot_-_track_VPN_logins

SRE multiple file-servers / KVM servers

  • groupinfo 99-t01
    • Group: 99-t01
    • Dscpt: Testing by SRE / RLU
    • Members: dlaplante-99-t01, testuser1-99-t01, testuser2-99-t01
    • Server: verendrye
  • To update table below: for S in fraser noyon verendrye drake; do echo -n "** $S: "; wget -q -O-${S}/ | sed -e '/|$/d' -e 's/|.*//' | sort | fmt -w 999; done
    2014-08-20 (leaving out groups with no members: sed -e '/|$/d')
fraser 04-005 05-005 05-013 06-017 08-015 08-016 09-008 09-017 10-004 10-005BCCA 10-006 10-018 10-c04 11-011 11-012 11-013 11-014 11-c05 11-s07 11-s08 12-006 12-012 12-c01 12-icis 13-002 13-016 13-032 13-053 13-054 13-083 13-s01 14-065 14-077 14-082 14-113 14-bmk 14-s01 14-s06 15-041 15-043 15-052 15-100 15-101 15-104 15-c01 99-t02
noyon: SRE 15-26 08-003 08-005 08-005-admins 10-007 10-008 10-009 11-003 11-c03 12-001 12-005 12-010 12-011 12-017 13-009 13-084 99-t01
verendrye: SRE 27-34, StatTransfer1 09-015 10-005 10-010 11-002 11-015 12-009 12-014 13-014 13-031 13-035 13-037 13-070
drake: SRTL 1-16 13-061
hubbard: SRE 1-14 01-011 01-09M 03-027 04-015 05-029 05-034 05-036 06-012 06-c01 07-007 07-011 08-001 08-008 08-013 08-017 09-013 09-014 09-016 11-009 11-s01 12-019 13-001 13-003 13-010 13-039 13-044 13-082 13-s02 14-020 14-030 14-036 14-058 14-073 14-074 14-079 14-080 14-083 14-090 14-092 14-105 14-112 14-131 15-006 15-021 15-025 15-026 15-042 15-070 15-071 15-076 15-081 15-092 15-097 15-109 15-111 15-119 15-124 15-129 15-132 15-143 15-144 15-153 15-154 15-159 15-171 15-s01 16-047 16-063 16-065

Synchronizing scripts on SRE file-servers

  • 4 SRE file-servers (Fraser Drake Noyon Verendrye Hubbard Bondar) may host R: drive for SRE machines
  • They all need a number of directories, files, scripts
    • /usr/local/sbin/create-yf-dirs will (or should) create
      • under /data/sre/ : EXPORT_FROM_SRE , EXPORT_FROM_SRE/.trigger , IMPORT_TO_SRE , IMPORT_TO_SRE/.trigger , .login , .login/etc (writeable by sre-users)
      • under /data/saved/ : EXPORT_FROM_SRE , IMPORT_TO_SRE
      • under /usr/local/var/ : empty.dir (for , sre (for account tracking) , sre/SRE-reports (for monthly reports), md5/Data/Backup/ (for
    • Cron jobs:
# m h  dom mon dow   command
# hourly create_user_dirs does all creation
1 * * * * /usr/local/bin/create_user_dirs
# at 03:05 on the first of each month, report on SRE transfers
5 3 1 * * /usr/local/bin/
# Wednesdays at 05:33 calculate disk usage for billing
33 5 * * 3 /usr/local/sbin/ > /usr/local/var/quotas-`date +\%Y-\%m-\%d`.out
    • yellowfolders-maint in /etc/cron.daily/* runs around 06:25 daily. yellowfolders-maint on Verendrye & Noyon updates MD5 database of Data files; on fraser also deletes stale user files in shares incoming / outgoing .
    • (/usr/local/bin/) can pull from Fraser the files specified in rsync_usr_local.list :
      • Under /usr/local/bin/  : create_* sre-archive sre-checksizes sre-transfers-report*
      • Under /usr/local/sbin/ : and md5-* update scripts
    • (/usr/local/bin/) can pull from fraser the ".libs" tree (Misc, STATA, SAS ...)
    • Manual transfer : Under /data/sre/.scripts various windows scripts: , off.bat , , transfer-in.lnk -> transfer-in.pyw -> , transfer-out.lnk -> transfer-out.pyw -> ...
    • 'sre-transfers-report.{pl,sh}  ; requires libmagic
    • Currently the following is not done:
      • Under /usr/local/var/md5/Data/ the MD5 checksums of data file contents. We still want to guard against a data file from one project being left on a C: drive and read by a researcher from another project, so we need to have the same md5 files on all servers that run yellowfolders.
        • Proposal: rsync update daily from (drake noyon verendrye) to fraser, then reverse an hour later.

Migrating Projects Between Servers

On original server

  • verifiy that previous nights backup completeled successfully.
  • groupinfo 17-777
  • make sure that no one is logged into the project
  • cd /data/sre
  • du -sh 17-777
  • rsync -va 17-777 rztransfer@mance::datamove/
  • rsync -va *-17-777 rztransfer@mance::datamove/out/
  • rsync -va *-17-777 rztransfer@mance::datamove/in/
  • groupinfo 17-777
  • make sure that no one logged in while you were transferring

On destination server

  • cd /data/sre/move
  • mv 17-777 ..
  • cd out
  • rsync -va *-17-777 /data/sre/EXPORT_FROM_SRE/
  • rm -rf *-17-777
  • cd ../in
  • rsync -va *-17-777 /data/sre/IMPORT_TO_SRE/
  • cd /data/sre
  • du -sh 17-777
  • make sure the size is the same as the original server
  • go to PDS and edit group and change the project mapping server to the new one
  • groupinfo 17-777
  • verify that it is now pointing to the new one.

If you are sure that everything is good, and ideally have waited at least one day, on the old server

  • cd /data/sre
  • groupinfo 17-777
  • double check the server is not this server
  • rm -rf 17-777