IDO Catalyst export review

From popdata
Jump to: navigation, search
Back to Secure Research Environment <- SRE-in-out-software

See also

  • IDD_Catalyst_Projects DAU 1-page /Tim Choi
  • Project_Numbering DAU standard for naming project mentions YY-g##.
  • IDO Contacts, Deadlines, projects, Data sets, acronyms (not expanded), meetings (1300 lines) / Tavinder and Melissa. Linked from DSU "Integrated Data Office"


  • Catalyst is a joint project between BC Government ministriens (Citizen Services ??) IDO (a.k.a. IDD , DIP=Data Innovation Program, ...) and PopData (formerly Tav) for prototyping the use of SRE environment for studies by government staff.
  • Projects 18-g* require special procedure for manual inspection of all export.
    • 2019-10-16 OCWA (Output Checker Workflow App) replaces Yellowfolder for Export and Import. From SRE: https://ocwa.opdata.bc.ca ???
    • Report any problems to Brittany.Decker@gov.bc.ca (administrator) , Brad.Payne@gov.bc.ca (developer)
  • Currently 18-g01 "MacKenzie IDD/CYMH", 18-g02 "Wilmer IDD-Education", 18-g03 "Warburton (IDD)", 18-g04 "Wilmer (IDD)" are hosted on file server Hubbard.
  • Export submissions for review will be copied to project 18-g99; members of that "project" will inspect files, and pass them on to server Fraser for export.
  • See outputchecking_interim_instructions18Oct16-D.docx in //Gilbert/Alfresco/Systems & Security/SRE+RTL+SRTL/Docs/Catalyst/ . That document was sent to IDD staff.

OCWA processes

OCWA help for users

See \\Gilbert\Alfresco\Systems & Security\SRE+RTL+SRTL\Software\OCWA\

  • Code Sharing Instructional Guide for Project Team Members v1.3.docx (intra-SRE code exchange)
  • OCWA Instructional Guide for Project Team Members v2.4.docx (export instructions)

OCWA help for administrators

  • PDS group "exporter" apparently allows OCWA access.
    • 2019-10-30 Of the 84 *-g* accounts, The following 23 are not members of "exporter"
elarson-19-g03 jchow-18-g01 jhawkins-18-g99 jrobson-19-g01 ljones-19-g01 ltedds-19-g01 mrussell-19-g01 mstabile-19-g01 mwilkins-19-g01 nfortin-19-g01 rbruce-19-g01 rclements-19-g01 rgray-19-g01 rhamdi-19-g01 rkneebone-19-g01 rshaban-19-g01 salbers-19-g03 scott-19-g01 shazlitt-19-g01 shazlitt-19-g03 srouse-19-g03 tbagai-19-g03 zwicker1-19-g01

OCWA troubleshooting

  • 2019-10-30 still waiting . ocwa has been having trouble, and plans for maintenance are unclear.
    developer Aidan Cope <aidan.cope@gmail.com> got IT staff to email log files.
    otrs#2019082210000015
  • 2019-10-25 #2019102510000069 blank OCWA window with MS Edge
  • 2019-08-22 #2019082210000015 — OCWA redeploy various recipes
    Manually re-run nightly 2am OCWA-redeploy cron job
    • ssh ocwa-vm
    • sudo -u idduser /bin/bash
    • crontab -l
    • (run the full command that appears in the crontab)
      • 2020-03-17 Crontab for idduser dated 2019-06-05: 0 2 * * * /home/idduser/popdata-infra/workspaces/production/cron_deployment/deploy.sh idduser ocwa production/prod_ocwa
    Collect as root process info and logs into subdirectory "diagnose"; put them as tar archive in Aidan's home directory
    • mkdir diagnose
    • df -k > diagnose/df
    • docker ps > diagnose/docker-ps
    • tar -cvzf diagnose/docker-logs.tar.gz /home/docker/containers/*/*.log
    • tar -cf /home/acope/diagnose.tar diagnose
    • chown acope /home/acope/diagnose.tar
    • rm -rf diagnose

Flow of export data before OCWA

  • SRE user (ex: harry-18-g01) in Catalyst category places copy of files in TRANSFER/EXPORT_FROM_SRE folder, clicks "EXPORT-IT" which creates a trigger directory
  • On server (Hubbard) yellowfolder.pl recognizes Catalyst project review request, runs catalyst-export-submit
    2018-09-19/DL yellowfolder+.pl in Fraser:/usr/local/src/YellowFolders/ not tested
  • catalyst-export-submit then:
    1. moves files to project 18-g99 folder (Ex: REVIEW/harry-18-g01_181231.235959/ ),
      and changes permission (read-only by group 18-g99)
    2. and creates companion folder APPROVED/harry-18-g01_181231.235959), using timestamp of request by reasercher
    3. and sends notification (email) to reviewers in project 18-g99 (email list <catalyst-reviewers@popdata.bc.ca> ).
    • In general there may be multiple export requests (possibly more than one from same user) in the queue to be reviewed and exported.
    • We may consider keeping a copy in SRE transfer archive of files submitted.
    • IDO people have requested no automated scanning at this time.
  • Reviewer can have conversation with researcher about file contents.
  • As each file gets reviewed, if approved it gets moved to APPROVED sub-folder.
  • To export an approved sub-folder, reviewer drags the folder's icon onto the EXPORT_IT script (or Windows shortcut).
  • SRE script export-reviewed.py creates a trigger directory by the same name (ex: harry-18-g01_181231.235959)
  • 2018-09-19/DL Fraser:/usr/local/src/YellowFolder/ [almost done]. Debug in Fraser: 99-t01/APPROVED
    Glitch with trigger naming convention - directory name != owner .
  • On server (Hubbard) yellowfolder.pl recognizes Catalyst project, runs catalyst-export
  • catalyst-export basically duplicates yellowfolder action :
    1.  ? automatic scan and block based on different criteria ?
  • Reviewer (catalyst-export ?) notifies researcher that export is ready for pickup.

Steps to enable review + export

  • [Done] Update email list catalyst-reviewers@popdata.bc.ca (currently just Jim and Denis) Julie Hawkins (jhawkins-18-g99) <Julie.Hawkins@gov.bc.ca>
  • Unblock all 18-g0* projects

Notes in ticket [Ticket#2018102310000011 — 18-g02 - transfer permissions]

  • I have restored permissions on all the TRANSFER directories for 18-g0* (owner root; chmod a+rx) and instead changed ownership to "root" of the *PORT*SRE/*-18-g0* (permission remains u=rwx) because the second is easier to see and safer.
  • To restore normal owner for TRANSFER/EXPORT_FROM_SRE (directory owner should match directory name)
    cd /data/sre/EXPORT_FROM_SRE && find *-18g0* -maxdepth 0 -ok chown \{} \{} \;
  • To delete all the "inspect-all" flag files
    cd /data/saved/EXPORT_FROM_SRE
    find *18-g* -ipath '*/.options/inspect-all' -ok rm \{} \;
  • To list all the inspect-all flag files
    yf-restrict list

Automated warnings and blocking

  • SUBMIT: back-end program catalyst-export-submit.pl
    • derived from yellowfolder.pl on Fraser, but uses separate configuration file to specify size triggers and unwanted file types.
    • has code for examining files, records list of files, warnings, and violations (reasons why file should be blocked), but does no blocking.
      The list of all files is included, as well as the warn / violation are included (could be excluded) in the automatic email to reviewers.
  • EXPORT: catalyst-export.pl
    • derived from yellowfolder.pl on Fraser, but uses separate configuration file to specify size triggers and unwanted file types.
    • has code for examining files, records list of files, warnings, and violations (reasons why file should be blocked), but does no blocking.
      The list of all files is included in the automatic email to reviewers; other messages are excluded (could be included) .

Import function

  • Request: 2019-01-17 process Catalyst imports like any other project.
    Add "if" clause to skip special processing for projects 18-g*


How it works? What do you need to add new users?

  • This is Ryoko's memo
  • 2018-10-23 DL wrote to Tav and Tim: the new review system for Catalyst users in IDD 18-g0* projects is already integrated into the regular SRE yellowfolders system. It supports export (via reviewers), and blocks imports. Before things run smoothly for Export for these Catalyst projects, we need reviewers to be ready. One reviewer was added to 18-g99 project on October 2 - Martin Monkman (mmonkman-18-g99) He has not been added to the email list catalyst-reviewers@popdata.bc.ca .
i have restored permissions on all the TRANSFER directories for 18-g0*  (owner root; chmod a+rx)
and instead changed ownership to "root" of the *PORT*SRE/*-18-g0* (permission remains u=rwx)
because the second is easier to see and safer.

To restore normal owner  (directory owner should match directory name)
  find /data/sre/*PORT*SRE -maxdepth 1 -mindepth 1 -iname '*18-g*'  -ok chown \{} \{} \;

To delete all the "inspect-all" flag files
  cd /data/saved
  find *PORT*/*18-g* -ipath '*/.options/inspect-all'  -ok rm \{} \;

To list all the inspect-all flag files
  yf-restrict list
Tim
  from Tav and Denis, the export review software is in place, but until somebody is ready yet to perform review of exports, all new users must still get blocked.
The procedure has been slightly changed:
For new user (ex: U=testuser-18-g02)
  chown root /data/sre/*PORT*/$U
  yf-restrict $U in "IDO"
  yf-restrict $U out "IDO"
From: Denis Laplante <denis.laplante@popdata.bc.ca>
Sent: Tuesday, November 6, 2018 10:56 AM
To: Catalyst-Reviewers@popdata.bc.ca
Cc: Lawrance, Greg J CITZ:EX; Ark, Tavinder (TARK_O); Decker, Brittany CITZ:EX; MacKenzie, Dan CITZ:EX
Subject: PopData SRE "Catalyst" project - review for export function

Welcome Julie to the email list Catalyst-Reviewers@PopData.bc.ca  (CC'd
to 4 non-members)

Julie only please confirm receipt so we know email list is working for you.

When researchers in the SRE projects request for a set of files to be
exported after review, an email to this list will be automatically
generated as the files are placed in subfolder R:\1-REVIEW in the SRE
remote desktop environment for reviewers, members of project 18-g99
(currently jhawkins, mmonkman).
Currently the only members of the email list are:
julie.hawkins@gov.bc.ca , denis.laplante@popdata.bc.ca, Jim@popdata.bc.ca

Up to a few minutes ago the folder TRANSFER was blocked from access (as
noted in Greg's email), now unblocked.

To clarify our different ways of naming, on the PopData side there are 2
"projects"
18-g01 "MacKenzie" , members cmarrville, dmackenzie
18-g02 "Wilmer", members  bwilmer, dhosick, dmackenzie, jfukuiinnes,
mboozarjomehri, nnabavi, skhan
Please let me know which one you call "Education" and which is "CYMH"

Finally please note that I only work 2 5-hour days per week, generally
Tue + Wed, but this week it's Mon+Tue.
In general queries are best sent to sre@podpata.bc.ca (all IT staff get
a copy).

-- Denis Laplante -- <denis.laplante@popdata.bc.ca> -- 604-822-9914 --
---  Population Data BC support for Secure Research Environment    ---
Brittany, Julie,
  just to let you know that
when "Catalyst" researchers submit exports for review, the server-side program "catalyst-export-submit.pl" scans all files for the same criteria as for normal SRE users, and notifies PopData IT staff via our OTRS ticket system, but does not do any blocking.  It's understood that your criteria and processes are independant of ours.
I have updated our Windows-side tool "scan-studyids.py" , see bottom paragraph of   https://my.popdata.bc.ca/html/SRE/working/file_transfers.html#blocking
It reports for this review request arichardson-18-g01_2019-02-05_15.36.24
 cpp/co_occur_educ.cpp:10:std::string ignore_s("s123456789"); // presumably the null-studyid
 py/check_linkage.py:38:        if w[-1]!="s123456789":
[123456789 has been substituted for the studyID].

I also see that a second reviewer Martin Monkman <martin.monkman@gov.bc.ca>  
was added 2018-10-02 to jhawkins in the 18-g99 "project", but has never logged in.
Let us know when you want any email added to the email list <catalyst-reviewers@popdata.bc.ca>

-- Denis Laplante -- <denis.laplante@popdata.bc.ca> -- 604-822-9914 --
---  Population Data BC support for Secure Research Environment    ---
-- Working Tuesdays and Wednesdays --
  • Above indicates reviewers must be manually added - how?
  • It also indicates that there is a email list catalyst-reviewers@popdata.bc.ca and the new reviewers has to be added manually