Pathways Paper

Due: September 2021 March 2022 (?)

Tools:

• Format to be AsciiDoc (.adoc)
• AsciiDoc plugin for Visual Studio Code: asciidoctor.asciidoctor-vscode
• Rendering to PDF: see below

Two ways to position the paper:

1. A solution for OpenDRR Pathways - How the platform supports the project.
2. A solution for Open Science - How the platform enables open science from a principled perspective. Demonstrated via an implementation for OpenDRR Pathways. TO BE AN OPENFILE

Title ideas

• OpenDRR Platform, an infrastructure for open science (for risk modelling/hazards)
• Open Science Platform for Disaster Risk Reduction

TOC ideas

• Summary - Joost

• Introduction to OpenDRR (mention why opendrr platform was needed/created before Intro to openDRR) - Joost w/Murray/Sahar/Phil

• Platform Requirements - Joost

  • Open Science (R5 (Docker), FAIR)
  • Community
  • Transparency

• Architecture - Will, Drew,

  • Skip CGP, and OpenQuake, FGP, otherwise speak to components in slide 16 of URBC deck
  • High-level details
  • GitHub:
    • version control, CI, project management, issues, websites (i.e. GitHub Pages), etc.
    • See GEM deck for ideas
    • How the repo is structured and why
  • Containerization/reproducibility (Docker) > speaks to R5 and FAIR
  • API's - specifically the ESRI REST API which requirement BC Govt.

• Lessons learned - Anthony, Joost

  • Model changes constant, flexibility should have been addressed at the outset
  • Open science is hard in the GC - policies not clear
  • Public/Private repos (Protected B)
  • Test-driven development would have been a good idea, but...
  • Project outgrew the resources
  • Organic growth, initial ideas/concepts

• Future development - Drew, Will, Anthony

  • CI/CD additions
  • AWS Cloud Formation Template
  • Updating the models (every 5yrs) - maintenance
  • Adding additional models (flood, fire, tsunami, etc.)

---

Brainstorming

• State our problem and solution domain/scope: Geoscience and Geophysics; Disaster Risk Reduction.
  • Starting with Earthquake scenarios
  • Flooding (coastal and others?)
  • Wildfires
  • ... etc.
• Tools used: OpenQuake, Hazus, Tableau, QGIS (ArcGIS, Esri?), MS Office; Docker, Python, PostgreSQL with PostGIS, Elasticsearch, Kibana; Jekyll, Asciidoctor, (Hugo?), WordPress; Leaflet JS, GCWeb, WET-BOEW; AWS (EC2, Lambda, RDS (Aurora?))
• Resource requirements
  • Some resource-intensive processes include: OpenQuake, loading CSV into PostgreSQL database using Python (solvable); SSD vs HDD; CPU and RAM requirement, etc.
• Reproducible builds:
  • R5 principles
  • Reproducible Builds — a set of software development practices that create an independently-verifiable path from source to binary code, and see how e.g. Debian and Arch Linux are doing
  • What we are facing:
    • OpenQuake stable release vs. latest development snapshot with slight incompatibilities
    • Python versions related to the above; Python libraries (requirements.txt) and their compatibilities. (affects OpenQuake runs; opendrr-api; Docker image for running Python)
    • underlying OS (Ubuntu 16.04 vs 18.04 vs 20.04); different distros (e.g. CentOS); different Debian versions in Docker image
    • the current need to set up AWS EC2 instances and their environments manually...
      • Potential solution: GeoscienceAustralia/aws-openquake: Python (Boto3) script which installs, runs and downloads results from seismic hazard and risk software (OpenQuake Engine) on Amazon Web Services EC2.
  • Progress (new versions) and reproducibility: How to get both. :-)
• Concurrent builds
  • Amazon services, EC2 launch template, etc.?
• Storage strategies (?)
  • Git LFS, Amazon S3, etc.
• Backup strategies (User Story: Disaster Recovery):
  • Backing up all our GitHub repos to Amazon S3; see Issues #8, #87, #88, #89
  • Having a parallel mirror on GitLab.com and/or on government servers?
• Security
  • Protecting GitHub tokens, for example
  • Security protocols/policies compliance within the Government of Canada
• Open Science, Open Data (open/transparent process)
  • How open? Balancing transparency and confidentiality. Accountability.
  • Aggregating raw data to avoid privacy-sensitive data being exposed
  • Potential issues/liabilities of making draft scientific papers available to the public before peer review: to soon, too premature?
• Free-and-open-source software and commercial software
  • Partnership between Global Earthquake Model (GEM) Foundation and Geological Survey of Canada (NRCan, the Government of Canada) on OpenQuake is a very good example of how the open-source software model works, how it is beneficial to both parties and to the world
• How can we make our work available for anyone to copy and use for other purposes for the greater good? (feasibility, efforts; documentation, promotion, etc.)

Rendering final output formats

HTML

• Hugo with its AsciiDoc support (with Asciidoctor)? :-)
• Jekyll AsciiDoc Plugin
• Quire and its Community Showcase. Some support for interactive maps based on Hugo. Does Quire support AsciiDoc?

Some examples, with automatic Table of Contents on the side, and with built-in search:

• ReSpec Documentation used by W3C
• DocuAPI theme with Hugo, see demo

PDF

How to convert asciidoc to pdf? - Stack Overflow

1. AsciiDoc → Asciidoctor.js → HTML → wkhtmltopdf → PDF
   (Ctrl+Shift+P, select "AsciiDoc: Export document as PDF", from Visual Studio Code: asciidoctor.asciidoctor-vscode)
2. AsciiDoc → _Asciidoctor PDF → PDF
   (Ctrl+Shift+P, select "AsciiDoc: Export document as PDF", from Visual Studio Code: asciidoctor.asciidoctor-vscode, with setting "Use_asciidoctorpdf" turned on)
3. Asciidoctor PDF (using Prawn)
4. AsciiDoc → Asciidoctor or AsciiDoc → DocBook → DocBook XSL and Apache FOP → PDF
   (asciidoctor/asciidoctor-fopub: A portable DocBook-to-PDF build command that wraps DocBook XSL and Apache FOP)
5. AsciiDoc → Asciidoctor or AsciiDoc → DocBook → dblatex → PDF
6. If Quire works, use its PDF generation method? (with PrinceXML?)

To do: To test and compare.