Meeting Notes

7/20/20 - meeting with Barbara and Brian

Timeline:

• OAI feed from Pachamama stack sometime between January and July 2021
• Presentation on stack selection September 2nd PAD Tech strategy meeting
  • End-to-end harvest for collection 466 (or other small-medium Nuxeo collection with text and image files)
  • Some sort of UI for the elasticsearch?

12/3/19

• Update on Supplejack, Combine, Ingest3 investigation
  • Supplejack dependencies, install instructions, docker setup: http://digitalnz.github.io/supplejack/start/dependencies.html, http://digitalnz.github.io/supplejack/start/installation-walk-through-by-example.html, http://digitalnz.github.io/supplejack/start_docker/docker-setup.html
• Priority of making a docker container for the existing harvester? Debugging work

11/4/19

This week:

• Affirm the requirements
• Start investigation and evaluation of Combine, Ingest3, Supplejack

In one month have settled on a strategy for moving forward. Do a Tech X to present our evaluations of Combine, Ingest3, Supplejack, roll our own harvester with Glue/Spark.

Resources:

• 50% time devoted to building the new system as much as possible, pending how work supporting the existing harvester comes up.

AWS Glue for managing Spark jobs Figure out the portability of the Akara stuff; testing mapping logic outside of Akara - running raw json through the mappers If we can someone replicate Akara, then we might be able to save those mappings/enrichment items

From a usability perspective, prioritize ease of debugging, de-prioritize (but keep in mind) workflow ui. Assume advanced skill operator for our one year goal.

10/31/19 Inaugural Group Meeting

Attendees: Amy, Barbara, Brian, Adrian, Christine, Matthew, Lisa, Gabriela

Agenda:

• Goal of the meeting: decide where we should be in a year and how to get there; determine November goals
  • Be able to harvest new collections end-to-end by this time next year and set up to migrate the existing data
• Scope and timeline of the project?? - We'd prefer to jump into an agile implementation process rather than going through a lengthy pilot project/planning phase.
• What are our resources? How much time can we devote to this? - Roughly 50% over the span of the next year, given other responsibilities (ASpace, deep harvesting, Calisphere maintenance, python3 migrations, etc.)
• Meeting Process & Frequency - Barbara and Amy will have stand ups twice a week starting in November for the first few months to kick start the project. This group will meet every two weeks? three weeks? to review goals and progress - is that too much? Maybe identify whether a particular meeting will be architectural or usability focused and then people can choose whether or not to come.
  • Architectural considerations (Brian, Matthew)
  • Usability considerations (Gabriela, Adrian, Matthew)
• Issues Tracking & Documentation - Use GitHub issues and GitHub wiki for project management tools

10/8/19

Attendees: Amy W, Barbara H

• meetings every 2 weeks with Amy, Barbara, Brian, Adrian, Christine?, Matthew?, Lisa? Gabriela?
• should we keep meeting notes private? no
• go into project mode starting Oct 22
• inaugural meeting 10/22. Set agenda
• Scope and timeline of the project?? Do we want to have a plan by the end of this fiscal year? and then implement next fiscal year?
• Next step is to install. Amy and I will try to do this locally before 10/22 meeting:
  • ingest3
  • combine
  • supplejack
• Then, run small sample OAI-PMH feed through all 3 (or as many as we were able to get installed)

To do by 10/22:

• put notes on github repo wiki (if public is ok) or elsewhere
• create meeting agenda
• Amy and Barbara to install combine and supplejack locally