Monitoring

From popdata
Jump to: navigation, search

System Monitoring

Syspulse

http://syspulse.popdata.bc.ca/

  • runs on cabot
  • /etc/init.d/syspulse start|stop|restart
  • config file /home/syspulse/syspulse.cfg -> /etc/syspulse/syspulse.cfg
    • After changing, syspulse will notice timestamp and re-read (after running check_config.pl /home/syspulse/syspulse.cfg)
  • syspulse folders logs and output need compressing from time to time. See /usr/local/bin/compress-syspulse-logs.sh (new versionj in /home/syspulse)
    • Note that all device logs will get re-created as needed, except "syspulse.log" (open at start), "syspulse.out" (open in /etc/init.d/syspulse)

Logs

  • The Systems Support Technician reviews filtered log files from all the servers on a daily basis (working days) using this webpage: admin.popdata.bc.ca/logs AND adminrz.popdata.bc.ca/logs
    • Modify filter script to conservatively add more filters, to help filter out unimportant info.
    • On admin.popdata.bc.ca/logs (+ adminrz), check the link "All w/Reason" to confirm that the filters are not removing pertinent log items. Do this once weekly with all servers.
    • Check for supicious log items or attempts at unauthorized entry into the system.
      • Repetitive attempts at logging in by any user are flagged. All logins by S&S staff are also noted.
      • Any remote connections by root or non-kernel processes run by root are also scrutinized.
      • System errors that could result in performance/security problems are flagged.
      • Known system work/troubleshooting by Jim will be noted in the log notes to record this activity.
  • Logs are collected on cabot: /home/logs
  • Logs themselves are kept for more than two years, and log review notes are kept for two years.
  • These logs are accessible only to the Systems and Security Manager and the systems staff with Administrator privileges on the network.

Moving Data

Data entering and leaving the Red Zone is strictly monitored and controlled.

  • SRE Release - The release of approved data to the researcher via the SRE. (Services/transfer#SRE_Transfers)
    • Only Systems and Security (S&S) staff can move the data into the SRE for the researcher.
    • This process is logged and the logs are collected by syslog process to be reviewed by Systems Support Tech.
  • Secure Uploads/Downloads
    • Data upload is the main point of entry, see Services/transfer.
    • Both are automated or semi-automated processes.
    • An email is sent to OTRS queu "Transfers"for each secure upload.
    • If any of this data is coming from/going into the red zone, it goes through a monitoring/logging process similar to "Yellow Folders" described below.
    • A note of the time, category and user is made in /var/log/yellowfolder.log . See /usr/local/sbin/scan-yellowfolder-log.pl
    • A copy of all files is ke[t under /data/saved/{CATEGORY}/{USER}
    • Services/transfer#Transferring_data_to_Red_Zone
  • Offsite Backups - Services/Backups/Offsite
    • Iron Mountain is scheduled to pick up backup drives on the first Thursday of each month.
    • Systems Security Tech copies Full backups from all servers onto encrypted disk drives. Then stores them in a locked transport case.
    • Iron Mountain arrives the first Thursday of each month and the drive transport case is handed over by Systems Support Tech.
    • The current location of all backup drives are recorded in the wiki, updated monthly by the System Support Technician.
  • Researcher Transfers (in and out of the SRE)
    • An automated process called Yellow Folders is used to transfer data in or out of the red zone. RedZone#Yellow_Folders
    • S&S has set up an automated process to monitor file transfers (filetype, size and various other criteria)
    • The system warns researchers or blocks file transfers according the the restriction criteria and a copy of the data is made for review by S&S staff.
    • An email is also sent to OTRS ticket system (category SRE) and the transferred data is reviewed with possible consultation with Privacy Officer.
    • A note of the time, category and user is made in /var/log/yellowfolder.log . See /usr/local/sbin/scan-yellowfolder-log.pl
    • A copy of all files is ke[t under /data/saved/{CATEGORY}/{USER}
    • When investigating a possible incident, typically Denis does:
      • On Fraser visit the "saved" directory in question, and inspect the names and sizes of files involved in the transfer attempt.
        • If the files are small and obviously docuentation or program files, the case is closed.
        • Otherwise binary data files can be explored using the "strings" program that extracts sequences of sensible-looking text from the unreadable binary stuff. Typically statistical data files include a list of field headers, and much of the data is also in text form. The filed names can reveal a lot about the data. This is done most conveniently by a command such as "strings FILENAME | head -1000" inside editor "emacs". When exploring whether data is about individuals (rather than aggregated statistics), consider the range of distinct values in a column.
        • If examination in Unix command-line context is inadequate, the files can be copied to an administrator's personal SRE account, e.g. /data/sre/users/dlaplante/DATA/{USER}-{PROJECT}-{DATESTAMP}
      • For SAS or SPSS or other files, I usually ask someone in DSU
  • Ministry of Health Secure File Transfers
    • The ministry has their own SFTP server. Only the head of Systems an Security can log into this site and download data.

Physical Access

  • Systems and Security Manager reviews fob access logs weekly.
  • Any time there is an after-hours entry into the Population Data ofices (the Red zone) the Systems and Security Manager will follow up with the person to determine their reason for entry.