Services/transfer

From popdata
Jump to: navigation, search
Back to >> Systems >> Services
See also >> RLU >> RLU Process and Procedures Manual >> Secure_Upload/Download for instructions to researchers.
... and Ryoko/notes-all
  • See RedZone#Yellow_Folders for description of yellowfolder software and folders for moving files between Franklin / Cartier2 / Fraser.

SRE Transfers to/from RedZone

SRE transfers from RedZone

  • On George (or some machine connected to George fileshare)
    • Someone in RLU does "release"
    • This triggers a "yellowfolder" process that does rsync from George to Franklin.

In the yellowzone on one of the SRE servers type: groupinfo 01-001 to determine the destination server.

Connect to Franklin in the red zone.

    • sudo bash
    • cd /data/transfer/outgoing-sre
    • ls (to confirm that the expected folder exists here, from the support request)
    • It's probably a good idea to cd into the release folder to run "sre-checksizes" to make sure it's all there (if it's very big)

Move it to the data folder and send to the destination sre server (e.g. hubbard in the following)

  • cd ..
  • mv sre~12-013~cohort1~csian~20130423_150717~sarabsky ./hubbard
  • ./sendit hubbard
  • tail -f /var/log/yellowfolders.log (wait and verify successful completion).

Connect to appropriate sre server (e.g. hubbard

    • ssh hubbard.popdata.bc.ca (in a terminal window)
    • sudo bash

Check the data

    • cd /data/transfer/outgoing-sre/hubbard
    • ls (to confirm that the expected folder exists here, successfully transferred from Franklin)
    • cd sre~12-013~cohort1~csian~20130423_150717~sarabsky/ (cd into that folder)
    • sre-checksizes
      • 2016-01-20 Denis fixed sre-checksizes (sort order: export LC_ALL=C)
      • Note on speed: 60GB takes ~40m. Failure of sre-checksizes is usually due to unfinished transfer.

Move the data

    • Make a directory named with today's date, as a subfolder of the project's /DATA directory
    • If there are two cohorts in a request, add a -cohort1 to the directory name.
      • mkdir /data/sre/12-013/DATA/2013-04-23 (Use today's date)
    • mv * .filesizes-checksum /data/sre/12-013/DATA/2013-04-23/

Set Permissions

    • Grant read permissions to the researcher group (12-013) using Jim's script.
      • cd /data/sre/12-013/DATA (cd to the DATA directory)
      • setperms 12-013 2013-04-23 (add a -cohort1 to the directory if there was more than one cohort in a request)
      • ls -l (to check that permissions are now correct)

Delete original folder

    • cd /data/transfer/outgoing-sre/hubbard
    • ls
    • rmdir sre* (this will delete only empty directories)
  • If the above fails because directories are not empty, it's probably after copying to another SRE file-server. It is even more important to delete the redundant data, to avoid it getting accidentally copied again to another server. So double-check that the transfer was successful, then be very careful to delete the correct directory.
    • ls -ld sre*
    • rm -r sre~{PROJECT}~...{DATE}_{TIME}~{RLUPERSON}
  • Reply to Support Ticket "Data moved and permissions set:
  • hubbard:/data/sre/12-013/DATA/2013-04-23 "

SRE transfers within YellowZone

  • Rsync configuration may allow transfers between SRE file servers. "rsyncd.conf" defines "modules" that are referenced with "::" and specify accessible directories and conditions.
    • EX: rsync -a --progress --password-file=/etc/rsync.pw LOCALDIRECTORY rztransfer@REMOTE_SERVER::datamove/SUBDIRECTORY
      will create LOCALDIRECTORY under /data/sre/move/SUBDIRECTORY and copy all its files.
    • Notes: avoid trailing slash for LOCALDIRECTORY (else only contents get transferred).
      SUBDIRECTORY is not required.
      rsync with module is better than scp, because scp cannot be run as root, so ownership is not preserved; also destination is confined to specific folder, reducing the risk of accidents.
    • Additional options: -n (not-really); -i (itemize) - different format than -v; --delete (delete from destination anything not found in source); -u (Update: skip files that are newer on the receiver); ...
  • TEST: to check that transfer is likely to work, try: just listing the remote directory:
    rsync -a --password-file=/etc/rsync.pw rztransfer@REMOTE_SERVER::datamove
  • The two scripts rsync_sre_.libs.sh and rsync_usr_local.sh are designed to be run from each SRE server to pull changes from Fraser.
    • Both have "-n" option to just list the changes that could be copied.
    • rsync_sre_.libs.sh copies everything under /data/sre/.libs . It only copies files listed in /usr/local/bin/rsync_sre_.libs.list . It deletes from local server files missing on Fraser.
    • rsync_usr_local.sh copies everything under /usr/local . It only copies files listed in /usr/local/bin/rsync_usr_local.list . It deletes nothing.

MoH SAE and MTP

  • Minitry of Health Secure Analysis Enviroment is a set of Citrix virtual windows machines accessible via https://dts.gov.bc.ca/vpn/index.html . See also docs at //Gilbert/Alfresco/Systems & Security/MOH-SAE . SAE.Guide.UserGuide.EnhanceFTS.DOCX has on page 7 a nice diagram.
  • See also Software-citrix-SAE , or Brent's notes Health_Ideas_-_Direct_Access for DSU point of vue
  • Most scripts on SAE VM require that drive letter P: be mapped to \\semillon\Groups\PopDataDirectAccess .
  • For transfers to/from PopData, SAE uses "MTP" (Managed Transfer Process), a GlobalScape server acting as an intermediate hop from SAE transfer to/from MTP , or from PopData transfer from/to MTP via sftp or scp saemtp.healthideas.gov.bc.ca (winscp for windows or filezilla for X11).
    Note that DSU users cannot access the MTP server, only IT staff with "self-delegate" accounts.
  • starting in early 2017, the MoH IDIR username (8 letters: initiial & lastname) and password (8-14 chars lc+UC+digit+symbol) works for all these 3 connections. Note that the password expires every few months and needs to be reset at https://dts.gov.bc.ca/vpn/index.html (it just asks for a new password). Accounts get locked after 90 days idle.

Popdata RedZone to SAE data transfer

  1. Christina/Mike copies GPG encrypted data to franklin:/data/transfer/outgoing-data/
  2. ./create_dl.pl (run create.pl script) ->...sftp?: leave it blank -> file or directory: cohort (select the folder to be transferred) ...create?: y
    create_dl.pl will check if the content of specified directory is GPG file or not
    create_dl.pl will then transfer *.GPG file to cartier2:/data/transfer/outgoing-sftp/data/
    create_dl.pl will NOT encrypt folder as it's already been encrypted
  3. Download *.GPG from cartier2 to local PC
    scp rnorden@cartier2.popdata.bc.ca:/data/transfer/outgoing-sftp/data/*.GPG /home/rnorden/Download/
  4. At my local PC open FileZilla
    Host: sftp://saemtp.healthideas.gov.bc.ca
    Username & password
    Port: 22
    Copy *.GPG file to Remote site/ SAE_INBOUND folder
    • Alternatively, use command sftp saemtp.healthideas.gov.bc.ca then cd SAE_INBOUND and put ....gpg . Optionally verify size with ls -l
  5. Logon to SAE citrix desktop (sae.healthideas.gov.bc.ca)
    Open SAE File Retrieval Inbound from desktop icon
    Search *.GPG and MUST DOUBLE CLICK to highlight to select
    Select destination as \\semillon\Groups\PopData_Direct_Access\INBOUND or (P:\INBOUND)
    "Operation complete!" window will popup
  6. Decrypt GPG file with Kleopatra (key is already installed)

NOTE: If decryption fails without prompting for passphrase, sae may have been reset again. Read this: https://wiki.popdata.bc.ca/popdata/Setting_up_your_Windows_VM_for_transfers

  1. Right mouse click on the imported file -> Decrypt and verify -> Check thc checkbox "Input file is an archive; unpack with..." menu -> Passphrase for "popdatamohsae (Popdata key for popdata redzone to mohsae transfer)" [not same as sae to redzone] & OK
    Delete GPG file
    CHECK if resulting files (in new folder by same name as GPG file) are zipped. If so, use 7z at Windows command prompt (i.e. >"c:program files\7-zip\7zip.exe" x cohort.zip) The 7z password is embedded in create_dl.pl script on Franklin (look at the very bottom of file).
  2. Delete duplicated temp files (linux: use shred
    mydesktop:/downloadedpath/*.GPG
    cartier2:/data/transfer/outgoing-sftp/data/cohort.zip
  3. Done!

1. Locate the file and copy them to our own home directory 2. Login to george2 3. cp /home/msimpson/TRANSFER/REDZONE/moveme.zip /home/rnorden/TRANSFER/REDZONE (copy it to my home OR ask Mike to run the transfer script) 4. cd /home/rnorden/TRANSFER/ -> run the transfer script called "SEND_IT_TO_REDZONE.sh"

  • "cohort.zip" will be transferred to franklin:/data/transfer/redzone/rnorden/

5. login to franklin 6. create a new directory under franklin:/data/transfer/outgoing-data/ -> i.e. I created "cohort" directory for this case. 7. mv /data/transfer/redzone/rnorden/cohort.zip /data/transfer/outgoing-data/cohort 8. cd /data/transfer/outgoing-data 9. ./create_dl.pl (run create.pl script) ->...sftp?: leave it blank -> file or directory: cohort (select the folder to be transferred) ...create?: y -> ...send files?: y

  • create_dl.pl will encrypt the entire folder of cohort (only one file inside) and generate encryption key. The file name is now "cohort.zip"
  • create_dl.pl will then transfer "cohort.zip" to cartier2:/data/transfer/outgoing-sftp/data
  • create_dl.pl will also generate the encryption key file under cartier2:/data/transfer/outgoing-notify/data/

9.5. cartier2:/data/transfer/outgoing-notify/data -> sh sunset-cohort.sh (to remove these files in 30 days)

  • in some cases sunset-xxx.sh is not created. you need to remove manually

10. Download "cohort.zip" (actual file) and "cohort.txt.zip" (encryption key) from cartier2 to local PC ("cohort.email" is not necessary with the SAE transfer)

  • scp rnorden@cartier2.popdata.bc.ca:/data/transfer/outgoing-notify/data/cohort* /home/rnorden/Download
  • scp rnorden@cartier2.popdata.bc.ca:/data/transfer/outgoing-sftp/data/cohort* /home/rnorden/Download

11. 7z x cohort.txt.zip with RLU standard password -> decryption key -> print out and give it to Mike 12. At my local PC open FileZilla

13. Logon to SAE citrix desktop

  • Open SAE File Retrieval Inbound from desktop icon
  • Username/password -> Show Files on SFTP Server button -> double click on "cohort.zip" to select
  • Browse and select destination as z:\INBOUND\MSimpson (Z:\\semillon\Group\PopDataDirectAccess\)
  • "Operation complete!" window will popup

15 Delete duplicated temp files

    • mydesktop:/downloadedpath/cohort.*
    • george2:/home/rnorden/TRANSFER/REDZONE/cohort.zip
    • franklin:/data/transfer/redzone/rnorden/cohort.zip (if you cp instead of mv)
    • franklin:/data/transfer/outgoing-data/cohort (<= this is direcotry. There is data inside so use "rm -rf")
    • cartier2:/data/transfer/outgoing-sftp/data/cohort.zip
    • cartier2:/data/transfer/outgoing-notify/data/cohort.txt.zip
    • cartier2:/data/transfer/outgoing-notify/data/cohort.email

Done!<strike/>

<strike>12. At my computer, open https://sae-mtp.hlth.gov.bc.ca (Globalscape on local machine; login with external login) MUST go inside INBOUND folder for the upload. Upload "cohort_1.zip" 13. At my windows VM or use Denis's citrix for linux version (https://wiki.popdata.bc.ca/popdata/Software-citrix-SAE) launch SAE Globalscape thru https://dts.gov.bc.ca

  • Desktop -> SAE_ThirdParty -> run downloaded ICA file -> log in to the Citrix desktop
  • Access https://sae-mtp.hlth.gov.bc.ca (use different account from above)
  • The file is in the INBOUND folder. Download and move it to \\semillon\Groups\PopDataDirectAccess\INBOUND\MSimpson\

SAE to Popdata RedZone data transfer

Mike requested SAE: z:\PopDataDirectAccess\OUTBOUND\popdatarz~xxx.gpg (\\semillon\Groups) transfer to defuca

  • NOTE: GPG files are already encrypted by Brent's method. We don't need to encrypt it again while transferring files for them.
  1. At my windows VM or use Denis's citrix for linux version (Software-citrix-SAE) launch NetScaler Gateway thru https://sae.healthideas.gov.bc.ca (If passwd expired goto: https://www.pwchange.gov.bc.ca/
  2. Desktop -> SAE_ThirdParty -> run downloaded ICA file -> log in to the Citrix desktop
  3. Locate and move the file "popdatarz~xxx.gpg" into P:\OUTBOUND folder
  4. Open SAE Transfer File Outbound application from shortcut icon on the desktop
    At the SAE Zip-Transfer Tool, choose the source folder as P:\OUTBOUND\ (DG creates a folder with the same name as the file, and puts the 2 files into it) username (no IDIR\ prefix) -> Start button to transfer
  5. Popup message "Complete!" means successful. The transfer zips and renames the file as RNORDEN_YYYYMMDDxxxxxx.ZIP
    If fail, look in CMD.EXE window for error message. Ex: Error changing directory to 'DLAPLA_o/OUTBOUND\
    • The first time, need to use sftp (FileZilla) and create folder /OUTBOUND and /SAE_INBOUND
  6. Open FileZilla and connect with -> Host: sftp://saemtp.healthideas.gov.bc.ca, Username: rnorden, Port: 22 -> Download RNORDEN_YYYYMMDDxxxxxx.ZIP from OUTBOUND folder to local PC ->
    • (using sftp dlapla_o@saemtp.healthideas.gov.bc.ca from terminal window also works
  7. Move the file to cartier2 -> scp /home/rnorden/Download/RNORDEN_YYYYMMDDxxxxxx.ZIP rnorden@cartier2:/data/transfer/incoming-data/newdata
    • After successful transfer to cartier2, use "shred" to securely overwrite then delete the file from your workstation.
  8. Move the file to franklin -> cd /data/transfer/incoming-data _> ./GET_IT.sh -> cd /data/transfer/incoming-data/newdata -> mv RNORDEN_YYYYMMDDxxxxxx.ZIP /data/transfer/incoming-data/defuca
  9. Move the file to defuca -> cd to /data/transfer/incoming-data -> sudo ./getit -> cd /data/transfer/incoming-data/defuca
  10. Do the normal NEWDATA transfer process from here **
    • Unzip RNORDEN_YYYYMMDDxxxxxx.ZIP without password
    • Run Brent's special script if file has GPG extension:
    • sudo /popdata/bin/unsae popdatarz~xxx.gpg
    • Unzip with 7zip (no password) & clamscan
  11. Then checkin file (eci command) with the NEWDATA procedure.
  12. 2018 Apr - a new added encryption in the franklin:/data/transfer/outgoing-data/create_dl.pl script.
  13. check at "deal with encrypted crosswalk"
  14. "7z -p4SANDSonly x $faname"
  15. Delete duplicated copies
    • mydesktop:/downloadedpath/RNORDEN_xxxxxxxxx.zip
    • cartier:/data/transfer/incoming-data/newdata/RNORDEN_xxxxxxxx.zip
    • franklin:/data/transfer/incoming-data/defuca/RNORDEN_xxxxxxxx.zip (if any)
    • defuca:/data/transfer/incoming-data/defuca/RNORDEN_xxxxxxxx.zip (if any)

Done!

SRE Transfers - Unusual

SRE to Red Zone (Franklin/Defuca)

  • This can occassionallly happen if already-released data is to be re-used. DSU wil be generating new identifiers before re-release.
  • 2016-09: RN updated this instruction with a special request from HELP to transfer files from noyon to franklin Ticket#2016090910000035
    1. groupinfo to locate the files. yellowzone->redzone transfer can be done from all sre servers.
    2. On sre server, move the date to /data/transfer/incoming-sre/%currentservername%/
      • For example noyon:/data/transfer/incoming-sre/noyon/
      • Don't forget to change the group to datamaster (chgrp -R datamaster filenames)
    3. Login to Franklin
      • cd /data/transfer/incoming-sre
      • ./getit noyon/
      • Now file(s) are moved to franklin:/data/transfer/incoming-sre/noyon
      • Don't forget to change the group/ownership (chgrp 11-c03 filenames / chown tark-11-c03 filename OR setperm)
    4. If you need to move the files from franklin to defuca, move them to franklin:/data/transfer/incoming-data/defuca/
      • Login to Defukca
      • cd /data/transfer/incoming-data
      • sudo .getit
      • Now file(s) are moved to defuca:/data/transfer/incoming-data/defuca
    • See section below "Move data to Defuca

MDI upload to helpred on Franklin

  • HELP staff periodically want data transferred to their redzone file share "helpred" on Franklin from their Ubuntu linux server "svr1.help.ubc.ca" [142.103.81.191] maintained by UBC IT and behind a UBC IT firewall.
  • They should upload via https://secure.popdata.bc.ca/upload/ .
  • browser "lynx" is incompatible with that server, but Firefox (installed 2014-12-16 by Barb & Denis) worked for Barb over ssh connection "mobaxterm", which also provides X11 tunnelling and graphical service.
  • Following standard upload (via Sullivan=secure.popdata and Cartier2), PopData staff transfer it to Franklin and finally move the files to /helpdata/helpdata/newdata/`date "%Y-%m-MDI"` and leave HELP staff to process the data from there.

Request from Barb: Can you please transfer /home/emdi/MDIexport.zip to the HELP REDZONE

scp -i /home/davidg/.ssh/dg davidg@mdi.ubc.ca:/home/bbgoldin/cheq_2018.zip ./Downloads

EDI SRE transfer for HELP

  • From HELP redzone storage on Franklin (Y:\redzone) to SRE project 11-s01 on Hubbard
  • Request arrives via OTRS ticket system with a title "EDI Data transfer"
    • groupinfo 11-s01 (to check the destination server, probably Hubbard)
  • Login to Franklin to confirm the location of the files: probably /helpdata/transfer/redzone/lchen/
    • mv or cp files and/or folder to /helpdata/transfer/outgoing-sre/hubbard/
    • ../sendit hubbard
  • Login to Hubbard and cd to /data/transfer/outgoing-sre/hubbard/
    • ls (to confirm the successful transfer)
  • Move to a folder named like /data/sre/11-s01/DATA/2019-12-31
  • Set permission to owner root, group 11-s01, readable by group: setperms 11-s01 FOLDER
  • Obsolete permissions when Tav wanted write access: chown -R tark-11-s01:11-s01 FOLDER (set owner:group); chmod -R u=rwX,g=rX,o-rwx FOLDER (set permission)
    • NOT chown tark-11-s01 filename (Tav no longer in HELP)

HELP transfer (franklin -> defuca -> Christina -> SRE release)

  • This request comes periodically. Samples: Ticket#2016101410000013, Ticket#2016081510000151
  • Request arrives via OTRS ticket system with a title "Data transfer to SRE"
  • The HELP files are in franklin (examples: franklin:/helpdata/helpdata/ or /helpdata/transfer/redzone/lchen) so IT has to move them to defuca (for Christina)
  • ((IT team does this part))
  • At franklin: mv %file% /data/transfer/incoming-data/defuca
  • At defuca:/data/transfer/incoming-data -> sudo ./getit
  • check out & check in process by newdata command etc
  • Email DAU
  • Remove duplicate file from franklin.
  • ((Christina does this part))
  • DAU will notify Christina about the file in defuca.
  • Christina will check the file and give "go" to DAU.
  • ((IT does this part))
  • DAU will send SRE Release Request to move the data from defuca to SRE.
  • Do the usual SRE Release (move to SRE server)
  • Notify HELP and DAU.

Secure Upload

How to use

  • See also Rlu Processes&Procedures Secure_Upload/Download
  • The RLU will NOT create a user account using PDS. They will send us a request to create an account. We add a new user account by editing the file on Sullivan /etc/upload-pws.txt . First field is login name, second is password with | in between. It reads them all in order, so the last one for a given name is used (in other words, you don't need to check for duplicates).
    • The account doesn't secure anything, it just prevents spam uploads. There is no way for an external user to download a file from the secure upload site. Logic is at /home/www/secure/upload/index.pl .
  • Tell the user to go to https://secure.popdata.bc.ca/upload and fill in:
    • login (created above)
    • password
    • Select file to upload
    • send to: jim
    • comments
  • When a user uploads a file, mail is sent to upload@popdata.bc.ca which creates a ticket in OTRS.
  • Data files MUST be encrypted. RLU has a standard process (email) for requesting that uploads are encrypted. File_Zipping_and_Encryption_Instructions_for_Researchers
  • In the event the researcher does NOT encrypt the upload, then all copies must be deleted. See below "Deleting transferred data that arrived via secure upload".
  • Meanwhile on server Sullivan, /home/www/secure/upload/index.pl first
    • saves a copy under /home/tmp/CGItemp#####
    • moves it to /home/upload/ under name "pd-upload-{USERNAME}-{YYYYMMDDHHMMSS}.{EXT}"
    • then copies that via SMB to /data/transfer/incoming-data/newdata/ on Cartier2 .
    • In the case where "send to" is "Better at Home", email is sent to 'megan.engelhardt@ubc.ca' and data is transferred to special SRE storage area for 14-s06 /data/sre/14-s06/DATA/uploads/ . Note that the files should be encrypted XLSX files.
      • Unfortunately of ~16 transfers per week, one volunteer data provider forgets to select "Better at home" and the data ends up mixed with "incoming-data" stream to RedZone. To redirect, copy the files to Fraser as above, and delete them from Cartier2 .../newdata/
  • Unfortunately, there is no feedback to the sender about the success of authentication until the upload is finished!!! For example Firefox might just say "Sending request to secure.popdata.bc.ca ..." for 45 minutes (1.5G @ 0.5M/s), then "Your name and password... Please try again". New users are advised to try first with a small file.
  • Log file /var/log/upload/upload.log writes error messages as they occur ("Failed password.." "No file.." "Unable to run rsync", "Unable to open...") and finally "Received file $filename ..." "Sending email...".
  • Cleanup on Sullivan: /usr/local/bin/remove-old-uploads.sh called Tuesday 6am from crontab to delete uploads older than 1 week.

Transferring data to Red Zone

Login cartier2 as root

  • file is in /data/transfer/incoming-data/newdata
    • copy and paste the encryption key in the text file in the same directory.
vi /data/transfer/incoming-data/newdata/dg.txt
    • Copy over the Anti-Virus definition files to Defuca to update them.
cp /var/lib/clamav/*.c?d /data/transfer/incoming-data/newdata

Transfer to franklin See RedZone#Yellow_Folders

  • Login franklin as root
cd /data/transfer/incoming-data
./GET_IT.sh
  • Wait for new data appears into this directory: /data/transfer/incoming-data/newdata then move data into the correct directory for transfering to Defuca.
cd newdata (all files and directories from corresponding directory on Cartier2 should be there)
ls
mv filename.zip ../defuca
mv *.c?d ../defuca

Move data to Defuca

  • Login to mackenzie then ssh to Defuca but NOT as root.
cd /data/transfer/incoming-data
sudo ./getit
  • The data will now be transferred to defuca into this directory: /data/transfer/incoming-data/defuca
  • Create destination directory (now encrypted) using Brent's custom commands (see below),
  • then "check out" the folder to enable you to write to it.

Run electronic check-out script

newdata 15-081 (project name)
*this "newdata" script was written by Jim and it does:
     -------------------------------------------------------------------------------------------------------------- 
     | cd /popdata/projects/incoming                                                                                   |
     | ls -l   (to check existing directories)                                                                    |
     | emkdir recvd_2013-06-13_project-number   (the date the researcher uploaded it - append the project number) |
     | cd recvd_2013-06-13_project-number                                                                         |
     | eco recvd_2013-06-13_project-number "new data incoming"   (a short comment to explain what you are doing)  |
     --------------------------------------------------------------------------------------------------------------
  • Then move data, and cd to your working directory if you need to unzip the data file.
cd /popdata/projectsco/davidg/recvd_2013-06-13_project-number 
sudo mv -i /data/transfer/incoming-data/defuca/pd-upload-04-015-201306* .

Decrypt & unzip files (RLU will send it to us in an email)

7z x filename (to decrypt)
     example of prompted description: "file uploaded by 'school' for McLeod 13-202", or "Pnet transfer downloaded by Jim"

Check if the file has unzipped,then delete .7z file Scan files with antivirus tool ** Update first **

sudo mv /data/transfer/incoming-data/defuca/*.c?d /var/lib/clamav
clamscan *

Run electronic check-in script

  • When finished unzip, "check in" the data using Brent's custom command. This should also set necessary permissions for DSU staff.
cd ..  (back into /popdata/projectsco/davidg)
eci recvd_2013-06-13_project-number

tell Data Access (RLU) about it (dataaccess@popdata.bc.ca)

Data has been moved to Defuca and decrypted:
/popdata/projects/incoming/recvd_2018-06-03_project-number/

remove the duplicates from cartier2 and franklin

franklin: /data/transfer/incoming-data/defuca/ (rm old files)
cartier2: /data/transfer/incoming-data/newdata/ (rm old files)
sullivan: /home/upload/ (rm old files)

Deleting transferred data that arrived via secure upload

  • In the process of upload to Cartier2 via Sullivan then transfer to Defuca via Franklin, multiple copies of an upload get strewn around. Sometimes the data steward or PopData's privacy officer require all copies of some data to be deleted.
  • Consider using "shred --remove" instead of "rm" to overwrite before deleting.
  • The uploaded file gets stored to "pd-upload-USERNAME-YYMMDDHHMMSS.EXT", with the original name recorded in companion file "pd-upload-USERNAME-YYMMDDHHMMSS.readme.txt"
  • First in the YellowZone network
    • Sullivan: /home/upload/ (first hop in secure upload - requires manual deletion)
    • Cartier2: /data/transfer/incoming-data/newdata/ (second hop in secure upload - requires manual deletion)
  • Next in the Redzone network:
    • Franklin: /data/transfer/incoming-data (or subdirectory defuca/ : manual yellowfolder transfer; possibly removed after successful transfer)
    • Defuca: /data/transfer/incoming-data (and maybe /space/raw/incoming/recvd_YYYY-MM-DD/ )
    • At this point the data is usually extracted from the encrypted archive, and passed on to DSU staff.
  • Special case: "saved" archive copies.
    • 2015-08-31 Dl and KG agree on this one: The file to be deleted should be replaced with a file with suffix ".DELETED" containing an explanation, for example "Deleted for inappropriate content, replacement on the way. -- Denis Ticket#2015080500000055"
    • Franklin: /data/saved/incoming-data/newdata/YYMMDDHHMMSS/ (timestamp of transfer always different from timestamp of upload)
    • Franklin: /data/saved/incoming-data/newdata/YYMMDDHHMMSS/ (an extra copy unless deleted from Cartier2 after transfer))
    • Defuca: /data/saved/incoming-data/defuca/YYMMDDHHMMSS/

MOHS RP&B / MSP / Pharmacare

Done on cartier (will prompt for a passphrase, general one with recycle) They have two sites, on development, and one production, depending which user name you use.

  • log onto cartier from the yellow zone
  • run the following commands as root (replace ubcuat with ubc for production site)
# You must be root
sudo bash
su -
cd /data/transfer/incoming-data/newdata
mkdir recvd_`date +%Y-%m-%d`  (command "recvd" does this)
cd recvd_`date +%Y-%m-%d`
# For the test server
# sftp -o IdentityFile=/root/rpbkey3 ubcuat@ftp.healthideas.hnet.bc.ca
# For the production server
sftp -o IdentityFile=/root/rpbkey3 ubc@ftp.healthideas.hnet.bc.ca

# use ls to identify other directories to transfer
cd rpb
get *
quit
  • move them to franklin
  • Run these commands on franklin from the red zone
cd /data/transfer/incoming-data
./GET_IT.sh
# wait
cd newdata
cd recvd_`date +%Y-%m-%d`
ls
  • send to defuca (make sure you have premission in /popdata/raw/rpb, otherwise send to your home directory, then move it as root)
scp -r recvd_`date +%Y-%m-%d` jim@defuca:/popdata/raw/rpb/rpb/
  • set permissions on defuca
cd /popdata/raw/rpb/rpb
chown -R datamaster:mohs_rpb recvd_`date +%Y-%m-%d`
chmod -R 770 recvd_`date +%Y-%m-%d`
chmod 2770 recvd_`date +%Y-%m-%d`
  • tell Data Services about the new data
  • remove the data from cartier and franklin

Setup as root:

ssh-keygen -f rpbkey -t rsa1
sftp ubcuat@ftp.healthideas.hnet.bc.ca
cd .ssh
get authorized_keys
exit
cat rpbkey.pub >> authorized_keys
sftp ubcuat@ftp.healthideas.hnet.bc.ca
cd .ssh
put authorized_keys
exit

MOHS not dataset pickup (project file exchanges)

Done on cartier (will prompt for a passphrase, general one with recycle) They have two sites, on development, and one production, depending which user name you use.

  • log onto cartier from the yellow zone
  • run the following commands as the user
  • reportedly Jim and Harold have accounts and hold a sftp key in their key list

sudo bash
su -
cd /data/transfer/incoming-data/newdata
mkdir recvd_`date +%Y-%m-%d`  (command "recvd" does this)
cd recvd_`date +%Y-%m-%d`
sftp JimMintha@ftpsvcs.hlth.gov.bc.ca

cd PopDataBC_MOH
get *
quit
  • move them to franklin
  • Run these commands on franklin from the red zone
cd /data/transfer/incoming-data
./GET_IT.sh
# wait
cd newdata
cd recvd_`date +%Y-%m-%d`
ls
  • send to defuca (make sure you have premission in /popdata/raw/rpb, otherwise send to your home directory, then move it as root)
scp -r recvd_`date +%Y-%m-%d` jim@defuca:/popdata/raw/rpb/rpb/
  • set permissions on defuca
cd /popdata/raw/rpb/rpb
chown -R datamaster:mohs_rpb recvd_`date +%Y-%m-%d`
chmod -R 770 recvd_`date +%Y-%m-%d`
chmod 2770 recvd_`date +%Y-%m-%d`
  • tell Data Services about the new data
  • remove the data from cartier and franklin

MOHS THE INSTRUCTIONS BELOW ARE DATED SEE ABOVE

MOHS uses sftp to transfer pgp encrypted files.

Steps:

  • reset password on jchoy account on cartier: passwd jchoy
  • temp remove securID on cartier. Edit /etc/pam.d/sshd change lines to be: (reversing them)
@include common-auth
#auth required pam_securid.so
  • add jchoy to the exception list in /etc/security/access
+:root cron:LOCAL
-:ALL EXCEPT admins andrew bryony rdempsey jchoy pulse:ALL
#+:ALL:ALL
  • temporarily allow all outside ssh in /etc/hosts.allow add line:
sshd: ALL
  • enable the firewall line for access to "powerline" 137.82.131.1 via ssh
  • check if you can ssh jchoy@powerline.chspr.ubc.ca from outside
  • send details to contact person
  • phone contact person with password
  • wait for file to arrive
  • transfer file to redzone
  • after file arrived:
    • change /etc/pam.d/sshd back
    • remove the extra line from /etc/hosts.allow
    • remove jchoy from /etc/security/access
    • disable the firewall rule for powerline (if noone else is using it)
    • reset jchoy's password to something random
  • decrypt the file as root on defuca
gpg --load-extension idea CMONP.RPB.UBC.Y2015.pgp

Vital Stats

  • connect with sftp to their machine from cartier (must be cartier)
  • Cartier is on the yellow zone
cd /home/shares/transfer/incoming-data/newdata
mkdir recvd_`date +%Y-%m-%d`
cd recvd_`date +%Y-%m-%d`
sftp univbc@sftp.vs.gov.bc.ca  (was univbc@vslftp.hlth.gov.bc.ca)
ls
get *
exit
  • move them to franklin
  • Run these commands on franklin from the red zone
cd /data/transfer/incoming-data
./GET_IT.sh
# wait
cd newdata
cd recvd_`date +%Y-%m-%d`
ls
  • decrypt the data
  • decrypt-vs.sh as an alternative executes the commands below
for i in *.enc
do
    openssl smime -decrypt -in $i -inkey client_key.pem -recip client.cert -out $i.txt
done
  • send to defuca (make sure you have premission in /popdata/raw/vsa, otherwise send to your home directory, then move it as root)
scp -r recvd_`date +%Y-%m-%d` jim@defuca:/popdata/raw/vsa/
  • set permissions on defuca
cd /popdata/raw/vsa
chown -R datamaster:vsa_births recvd_`date +%Y-%m-%d`
chmod -R 770 recvd_`date +%Y-%m-%d`
chmod 2770 recvd_`date +%Y-%m-%d`
  • Remove the files from VS servers (adjust the list of file as apropriate
sftp univbc@vslftp.hlth.gov.bc.ca
ls
rm ubc*
exit
  • tell Data Services about the new data
  • remove the data from cartier and franklin

GECKO Project

  • lisa sends email that the data is ready in redzone franklin:/data/transfer/outgoing-sre
  • transfer data on franklin
cd /data/transfer/outgoing-sre
./sendit
  • on fraser in the yellow zone
cd /home/outgoing-sre/data
cd GECKO
mv * /data/sre/Data/GECKO
cd /data/sre/Data/GECKO
chown lchen:gecko *
chmod 640 *

Uploading Encrypted Data to Ministry via SFTP

  • DSU releases the data to the SFTP target
  • go to franklin in redzone, as root or datamaster
# cd /data/transfer/outgoing-data
  • run the create_dl.pl script
# ./create_dl.pl
Short name of person/org downloading (blank for sftp): 
File or Directory: vsa~09-006~cohort1~csian~20110706_162939~alow
Generate Encryption Key? (y/n): y
Generated Key: asdfjalsadfkjhuflawuneflawuiehnflwaiuehfawliuehf
Creating 09-006~cohort1~20110706_162939.zip

Compressing  09-006~cohort1~20110706_162939/docs/vs_xwalk_layout.txt
Compressing  09-006~cohort1~20110706_162939/vx_xwalk.dat

Everything is OK
Moving 09-006~cohort1~20110706_162939.zip to sftp transfer
Creating notification
Send the files? (y/n): y
Files sent
  • go to cartier and as root
# cd /data/transfer/outgoing-sftp/data
# ls -l
-rw-r--r-- 1 root root  168805 Jul  9 23:49 09-006~cohort1~20110706_162939.zip
# cd /data/transfer/outgoing-notify/data
# ls -l
-rw-r--r-- 1 root root 154 Jul  9 23:49 09-006~cohort1~20110706_162939.notify
  • The .notify file should be emailed to the RLU it looks like:
# cat 09-006~cohort1~20110706_162939.notify
File 09-006~cohort1~20110706_162939.zip has been transfered by sftp

Encryption Key is: asdfjalsadfkjhuflawuneflawuiehnflwaiuehfawliuehf
  • The zip file needs to be uploaded to the ministry sftp site.
# sftp univbc@sftp.vs.gov.bc.ca
put 09-006~cohort1~20110706_162939.zip
  • for the ministry of health, they have two sftp hosts. I believe the one for uploading is:
Usernames - JimMintha, HaroldYip, BrentHills                                                                                                                                                                                     
Host - ftpsvcs.hlth.gov.bc.ca                                                                                                                                                                                                    

I have made some changes to the folder structure on our SFTP server, ftpsvcs.hlth.gov.bc.ca                                                                                                                                      
When you login, you will see 2 folders:                                                                                                                                                                                          
                                                                                                                                                                                                                                 
PopDataBC_PNet - this contains the existing folders you have been working with                                                                                                                                                   
PopDataBC_MOH - this is a new folder for a new info sharing process between PopDataBC and MOH staff Holly Groves and Gay Corbett                                                                                                 
                                                                                                                                                                                                                                 
If you run into any issues when accessing our server, or have any questions, let me know.                                                                                                                                        
                                                                                                                                                                                                                                 
Thanks,                                                                                                                                                                                                                          
                                                                                                                                                                                                                                 
                                                                                                                                                                                                                                 
Mike Botrakoff                                                                                                                                                                                                                   
Service Coordinator                                                                                                                                                                                                              
System Services                                                                                                                                                                                                                  
Health Sector IM/IT Division                                                                                                                                                                                                     
Ministry of Health                                                                                                                                                                                                               
Main Floor, 1515 Blanshard Street, Victoria, BC    V8W 3C8                                                                                                                                                                       
Phone:  250.952.2425          Cell:  250.812.3898                                                                                                                                                                                
  • the other site is: ubc@ftp.healthideas.hnet.bc.ca
  • Lastly remove the .zip and .notify file from cartier


WorkSafe NB SFTP

https://eft.ws-ts.nb.ca
  • Suhail/Robert Macpherson sends request to upload/download data to WorkSafe NB SFTP server.
  • Accounts expires every few months (or weeks?). Need to contact Heather for the valid account info each time.
  • WorkSafeNB Tech Contact: Heather Ruttan 506-632-2203

Preparing a file for Secure Download

  • DSU releases the data to the DL target
  • go to franklin in redzone, as root or datamaster
# cd /data/transfer/outgoing-data
# ls -l
# cd into dl folder
cat password.txt (then copy the password)
7z x filename.7z (we need to unencrypt this file otherwise it becomes double-encrypted for the end user, after running create_dl.pl script)
Delete the password file and the zipped file.
  • run the create_dl.pl script
# ./create_dl.pl
Short name of person/org downloading (blank for sftp): 11-c06
File or Directory: dl~11-c06~cohort1~csian~20110806_162939~alow (copy from above and paste here)
Generate Encryption Key? (y/n): y (answer yes except for special cases)
Generated Key: asdfjalsadfkjhuflawuneflawuiehnflwaiuehfawliuehf
Creating 09-006~cohort1~20110706_162939.zip

Compressing  09-006~cohort1~20110706_162939/docs/vs_xwalk_layout.txt
Compressing  09-006~cohort1~20110706_162939/vx_xwalk.dat

Everything is OK
Moving 09-006~cohort1~20110706_162939.zip to dl transfer
Creating notification
Send the files? (y/n): y (always answer yes)
Files sent
* The prompt will return immediately but the process will still be running (especially if it's a big project). 
* You need to wait till it's finished before processing another file for the same project/shortname.
* It is finished when the zip file is created on cartier2 in yellow zone. See next step.
* Monitor this directory until it creates the final .zip file.
  • go to cartier2 and as root
# cd /data/transfer/outgoing-dl/users/11-c06
# ls -l
-rw-r--r-- 1 root root  168805 Jul  9 23:49 11-c06~cohort1~20110806_162939.zip
# cd /data/transfer/outgoing-notify/data
# ls -ltr
-rw-r--r-- 1 root root 319 Jan 26 15:36 11-C06~cohort1~20110806_162939.email
-rw-r--r-- 1 root root 380 Jan 26 15:36 11-C06~cohort1~20120124_125519.txt.zip
  • The .email file should be emailed to the RLU it looks like:
# cat 11-C06~cohort1~20110806_162939.email
Hi RLU:

Attached is the encrypted zip file (using the standard RLU password)
with the location of the file 11-C06~cohort1~20120124_125519.zip to download
and the encryption key for the user of the file.

To download the file the user will need the following:
(communicated by phone)

Login: 11-c06
Password: kodn5iyh
  • the .email file is the body of the email to send to the RLU.
  • the .txt.zip file is the encryption key & location. This file is encypted and should be attached to the email. The RLU has the key to decrypt it.
  • Run on local machine to transfer txt.zip file(s) for example:
scp davidg@cartier2.popdata.bc.ca:/data/transfer/outgoing-notify/data/03-027*.txt.zip /home/davidg/Documents/
    • If releasing more than one D/L for the same shortname/study name, only the last download .htaccess password will work. Adjust the emails to the RLU accordingly. **
  • Run the shell script that will delete the files after 30 days.
sh sunset-11-C06~cohort1~20110806_162939.sh

Testing Secure Downloads

* Modified 2016-06-29 RN
* This process can be tested by following procedure:
* 7z x 14-034~recvd_2016-05-31_14-034~20160629_144559.txt.zip (from desktop, unzip the downloaded and encrypted zip file)
* Enter RLU standard password
* It will create a text file in the same directory, i.e. 14-034~recvd_2016-05-31_14-034~20160629_144559.txt 
* cat 14-034~recvd_2016-05-31_14-034~20160629_144559.txt (copy the downloadable URL link)
* i.e. https://download.popdata.bc.ca/users/17-c01/17-C01~um_motherid20170111~20170118_102028.zip
* Open the link in the browser with user name : project name & password: encryption key in the generated text file.
* Download complete!

Changing .htaccess password

* Find the location of the .htpassword file ie:  /data/transfer/outgoing-dl/users/16-024
* Run this command: htpasswd -c -b .htpasswd username password (username is normally the project number ie: 16-024).

Expired Secure Downloads - procedure

  • If the researcher waits more than a month, the zip file is deleted by the "sunset..." script.
  • You can find a copy on Franklin, in /data/saved/outgoing-dl/users/(20170301150322 for example)/projectnumber
  • Do this to copy it back to cartier2 so it is accessible for the researcher to download.
# cd /data/transfer/outgoing-dl
# cp /data/saved/outgoing-dl/users/20170301150322/projectnumber/file.zip ./users
# ./sendit
  • The file will be on Cartier2 in /data/transfer/outgoing-dl/users
  • Copy it into the correct projectnumber subfolder (file needs 644 permissions).