Skip to content

Sprint 40

Jump to bottom
Anthony Fok edited this page Aug 30, 2021 · 28 revisions

Due by August 26th, 2021

Scrums: Aug 5 | Aug 10 • Aug 12 | Aug 17 • Aug 19 | Aug 24

Sprint Goals:

  • Finalize views & support edits/revisions (Will)
  • Publish FGP records (Joost)
    • Send revised data files and MXD's to FGP (Will, Phil)
      • Physical Exposure - waiting on updates before pushing changes to Charles at FGP > Sprint 41
      • Social Vulnerability
      • Multi-Hazard
      • PSRA
    • Review revised FGP data layers (Murray, Phil, Tiegan, Joost, Will)
      • Physical Exposure
      • Social Vulnerability
      • Multi-Hazard
      • PSRA
    • Prep indicator tables (next to final) for FGP documentation resources (Murray, Phil, Joost)
    • Fitness for purpose statement (Phil) DRAFT for Murray & Tiegan See issue
    • Disclaimer (Phil) See issue
    • Prep legend files (MXD's) for FGP data (Phil) See issue
      • Physical Exposure
      • Social Vulnerability
      • Multi-Hazard
      • PSRA
  • Final run of the national model (PSRA) (Tiegan) - in progress > Sprint 41
  • Merge CQL feature branch into pygeoapi (Joost)(https://github.com/OpenDRR/pygeoapi/issues/7)
  • Kibana sample visualizations (Drew) - seeking input from others, 5 done > Sprint 41
  • Draft Exposure OpenFile, discussion and validation sections (Phil) - almost done, final review next week.
  • Project coordination mapping (Phil, Joost) - with Phil, to meet (set up) with Malaika and Nicky
  • OpenDRR API pipeline tweaking (Anthony's backlog):
    • Flexible CSV header loading and verification (Anthony, 50% complete?) Issue model-factory#53
    • GitHub Actions for CI (Anthony)
    • Pipeline optimization (Anthony)
    • Migrate from Backblaze to S3 (Anthony, 50% complete?)
  • Revised SCM to Stephen and Arash - gather feedback (Joost) - Joost to set up meeting this week with Stephen and Arash
  • Update add_data.sh - ES loading refactoring (Drew, Anthony, Damon) - We need a user story for this with tasks
  • GeoJSON maps for the earthquakes repo (Damon)
  • Jekyll sample app for GitHub pages (Damon, Joost)
  • Add bilingual support for pygeoapi - (Joost) > Sprint 41
  • Convert JSON to YML in data repo - (Damon) > Sprint 41
  • Automatic milestones (Anthony)

Demos:

  • GitHub pages demo (Damon)
  • Demo map previews on data site (Joost) > Sprint 41 add attributes to map View with human readable alias
  • Pipeline optimization overview (Anthony)
  • Kibana demo (Drew)

Scrum Notes:

Thursday, August 5, 2021

  • Joost

    • working on pygeoapi enhancements
      • bilingual
      • records API (aka metadata)
      • CQL support

  • Will

    • Finished 6/7 tasks from OpenDRR/opendrr-data-store#53. Updated in current dev branch in model-factory as well. Started a PR for revised source file in openquake-inputs waiting for Phil's approval.
    • Will continue on my other sprint goals (global fabric hexbin aggregation / investigate canada-srm2 repo). Also helping with a internal review on Tiegan's new paper (developing a retrofit scheme for canada's national seismic risk profile)

  • Anthony

    • exploring GitHub CLI for migrating issues to new milestones

  • Damon

Tuesday, August 10, 2021

  • Will

    • Thursday: Git update on Slack (?)
    • Friday: Reviewing Tiegan's paper on retrofit scheme
    • Reviewing some of the updates for physical exposure: Phil has approved the PR, but still QC'ing some of the physical exposure pieces; may need my help to write some queries and do more QC'ing of the data. Phil is to meet with Murray today to go over the details.
    • To continue reviewing canada-srm2 repo, hopfully today or tomorrow

  • Damon

    • Finished with the maps stuff for now: everything working and looking good; going on Jekyll page
    • Been working on bringing information from earthquake-scenarios onto GitHub Pages, and try to make that automatically update every time
    • Tiegan has new a scenario to the folder, so I started implementing a parser (?) yesterday, and continuing today

  • Anthony

    • Updated and sent résumé (finally)
    • Started writing a test bash script which uses gh (GitHub CLI) to move issues to a new milestone, see OpenDRR/opendrr-platform#126
    • [non-OpenDRR] Update Debian’s Hugo package to v0.81.0 (and dependent packages) in anticipation of the end of freeze when Debian 11 (bullseye) is scheduled to be released on August 14, 2021

Thursday, August 12, 2021

  • Damon

    • Making pretty good progress on the GitHub Pages stuff.
      • Got the Python script that's auto-generating the HTML that gets added into it
      • To discuss with Will to sort out the naming of some of the GeoPackage files for the automated script

  • Will

    • Have still been comparing the new canada-srm2 repo; mostly, the changes are very minor.
      • One of the ebRisk files have an additional indicator to be added, so I think change needs to be made to the PSRA_copyTables.py (?) script to incorporate that one indicator.
      • Some of the "Retrofit 2" is not "Retrofit 1", which is an easy fix, but there's only a few files that actually been updated in that repo, so it's not the whole Canada, i.e. not ready yet, so not going to bother changing the Python scripts yet, but have been updating [SQL] scripts on my side to account for the r2 "Retrofit 2" to "Retrofit 1" change
      • Had an 1½-hour meeting with Phil yesterday to do some QC on the physical exposure datasets. Looks like there's a few more items that we have to adjust and fix, so I'm gonna go ahead and continue those revisions today. That also requires some updating of the source files in the openquake-input repo; one of those files needs to be updated with some new updates for the daytime population
      • Been continuing to work on the hex grids for the GEM global fabric (Joost); to continue working on that probably next week

  • Anthony

Tuesday, August 17, 2021

  • Damon:

    • Finished at least the baseline of the stuff I was working on for the GitHub Pages; finished with most of the stuff that I talked to Joost before
    • Going to start poking around in GitHub Actions because of the need to link between repos in order to get the GitHub Pages working properly and updating properly
    • Going to try to rewrite something I did in python in a Bash script just for ease of use

  • Will:

    • Since last week:
      • Going over revisions with the physical exposure pieces with Phil
      • Had a meeting, similar details on some more indicator changes, that now off my plate; Phil is now looking over the reviews
      • Went over some of Tiegan's papers, so i'm going to put in my comments today in PNT(??)
    • To do:
      • To continue on working on my sprint goals for the rest of this week
      • To connect with Tiegan on some of the questions she raised yesterday on the ESRi(?) scenario lists/indicators

  • Tiegan:

    • My number one priority is the National Model; had some failed runs
    • Hypothetically, the new plan is: instead of splitting the regions into economic regions, I wrote a script to split it into the census division geometry, which is, instead of 76 units to divide the country, 293 units.
    • Had some test runs going last night that seemed to be working okay when I signed off for the night, so I'll check on those this morning.
    • I have Will's GeoPackages from some of the scenarios that I started going through yesterday, and I'll keep working on that today just to verify that everything, after having gone through the DSRA presets(??), still looks correct, and double check all the numbers.

  • Anthony:

    • Fix a bug where the inability to handle bot messages caused a crash in my proof-of-concept Python-based Slack history capture tool.
    • Non-OpenDRR work (sorry!):
      • Now that Debian 11 (bullseye) was released on August 14, I got busy with the start of the development cycle for the future Debian 12 (bookworm):
        • Uploading held new upstream packages during the 6-month freeze
        • Communication with fellow Debian contributors, look at Go programming and font packages, etc.

Thursday, August 19, 2021

  • Damon:

    • Tinkering: Got a GitHub Action that copies file from one repo to another

  • Will:

    • Still working on the corrections or some QA/QC with Phil on the physical exposure data set. We just had another revision yesterday (Aug 18). We're trying to troubleshoot some questionable source or some for the population _______ in it, so we're just trying to figure that out and working through that.
    • Finished the review for Tiegan's paper; submitted that yesterday to her.
    • Later today: to meet with Tiegan, Phil and Murray to go over some of the ESRA scenarios. There are some indicators that she wants to go over and need to adjust, so there's going to be changes for that piece coming as well.

  • Anthony:

    • Non-OpenDRR work (sorry!):
      • [Debian] Start of Debian 12 (bookworm) development cycle: Fix bugs and add features to dh-make-golang to allow fellow Debian Developers/Maintainers to package gh, see #951374
        • Migrated from .travis.yml to GitHub Actions workflow for CI tests (migration to travis-ci.com failed to work). Pros: better integration thus easier to use in the long run. Cons: Lost access to arm64 and ppc64le architecture runners.
      • [Chinese school] Discovered https://hambaanglaang.hk/, an inspiring and innovative open-source effort on creating Cantonese stories books in printed and audiovisual formats with software tools and fonts too. (CC BY 4.0)
    • To do:
      • Regain my access to AWS in participation of Tiegan's call for help on getting OpenQuake to run in parallel for upcoming PSRA calculations.

Anthony asked for more details about Tiegan's call for help, to which Will explained:

  • The previous PSRA calculations were at the economic region level: 76 regions
  • The new PSRA calculations are much finer at the census division level: 293 census divisions across all Canada
  • It sounds like it might take a few days to run each census division. So, running these calculations concurrently can save a lot of time
  • Tiegan has two machines set up right now; would be great if we can spin up like a dozen or so high-end machines.
  • Suggest the use of a template of an image with Python and OpenQuake pre-installed
  • Suggested to talk to our resident AWS expert David Lemieux for ideas on what to do

Tuesday, August 24, 2021

  • Joost

    • Today:
      • To set up meetings with the FGP Cloud Team on the review of the revised security control matrix for our infrastructure
      • To start in on drafting on a deck that I need to present to the GEM (Global Earthquake Model) Steering Committee Meeting which is on September 1.
    • Tomorrow:
      • To restart my work on pygeoapi:
        • I did get the bilingual support into our version, but apparently there's another PR: it needs to be merged into main branch of pygeoapi proper:
        • The interface element is to be translated automatically, so right now I'm providing metadata in French and English, and pygeoapi is spitting out the metadata in the languages being requested, so that all works
        • But what's not working is the interface isn't being translated automatically, and there's a separate PR for that
        • To review that PR and see how mature it is, and to get Tom's advice on whether or the PR should be merged untested or not

  • Damon

    • Mostly tinkering here and there, and looking into random things
    • Completed most of the assigned tasks, but need to go over them with Joost to see where they are at, and need more direction to continue all the most the stuff i was doing is
      • Joost: Yeah, we'll go through that at our meeting later today. I'm interested in seeing the progress you've made.

  • Anthony

    • What I did:
      • Thu Aug 19: Got my AWS access restored! Many thanks to David Lemieux who filed a request on my behalf
      • Mon Aug 23: Finished catching up Scrum notes
      • Seeked help for VPN access (in progress): laptop being re-imaged with Windows 10 as prerequisite (Thanks Raviraj and Kurt!)
      • [Non-OpenDRR work, sorry!]
    • To do:
      • [Non-OpenDRR] Finish up dh-make-golang and get 0.5.0-1 released
      • Get back to my many long-overdue Sprint goals 🤞
      • To seek and discuss solutions for concurrent OpenQuake runs at the census-division level PSRA on AWS
    • Sharing (shameless plug):

Discussion on meeting notes for scrum:

  • New in this Sprint 40
  • Don't want to create more work for all: Scrum is supposed to be lightweight, and there is no need for everyone to come into wiki and prepopulate the entries. We can leave it for the Scrum Master to take notes for everyone.
  • But it is very useful, especially for any team members who have been away to be kept informed on news and progress without having to ping everyone
  • Joost is all for it, and would be willing to try something new, and something not previously done before, unless anyone is adverse to it
  • To discuss at Planning next week: To start doing rotation on Scrum Master duties for the rest of the fiscal year so that everyone will get their hand at running planning sessions, review sessions and scrum meetings
    • To rotate on a per-sprint or a couple sprints basis? Let's talk about it at Planning next week
  • Sidenote: Anthony has been recording the meetings and using cloud-based speech-to-text service to help with the note-taking

Sprint Planning | Sprint Review
Wikis: data | model-factory | opendrr-api | opendrr | python-env | riskprofiler-cms

Clone this wiki locally