Skip to content

Work Session Summaries

Jump to bottom
stranger80 edited this page Apr 2, 2020 · 224 revisions

02.04.2020 - Yagna Integration Team catchup

Agenda:

  • Review of Kanban
  • Intro for Artur Zawlocki

24.03.2020 - Arch workshop - Yagna Integration Team Kickoff

Agenda:

Outcomes:

18.03.2020 - Arch workshop - Yagna Integration Team Kickoff prep

Homework:

  • MB to prepare kickoff session for the Yagna Integration Team.
  • Agenda:
    • Intro into Lightweight Golem concepts
      • Integrator's Whitepaper - as a "map" for learning
      • Yagna technical concept doc - as indication of architecture and components
      • Walkthrough of the repo
      • Integration Testing framework intents and requirements
        • Automated setup - leverage eg. Docker/docker-compose
          • IPFS performance testing framework - ask KubaKucharski for link again
        • Functional scenarios
          • Scenarios require REST API and CLI
          • Probably Python preferred? Consider JS framework to enable in-browser test execution?
          • Clear and readable definition of test scenarios
          • Stepped execution
        • Scaling & performance testing
      • Kick-off tasks
        • Open the repo, learn to run the whole setup of current Yagna (incl TestBed)
        • Research Test Frameworks for REST and CLI
        • Propose patterns for Test Scenario maintenance
        • Negotiaite issue raising-fixing interface to Yagna team (and agree priority of issue resolution)

05.03.2020 - Arch workshop - QA Approach brainstorm

Homework:

  • MB Propose a skeleton for "Lightweight Golem Whitepaper for Integrators"

Outcomes:

QA Aspects:

  • Test categories:
    • Unit tests
    • Aut. Integration tests
    • Performance tests (pref. automated)
    • Penetration tests???
    • Network protocol tests
      • Functional correctness
      • Performance (eg. DDoS resilience????)
  • Security audits
    • Payment scenarios (double spend, anti-sybil mechanisms...)
    • Golem protocol security
    • ExeUnit module security
  • UX audits
    • CLI
    • APIs
    • SDK
      • Docs
      • Samples
      • API adapters for different platforms/languages
  • QA Process
    • Code Review
    • Continuous Integration - yes/no?
    • Issue flow & resolution priority (ie. need to assign resource to address pending)

Re. tooling & automation ideas

  • Docker usage
    • Docker compose as platform to enable test scaling (when applicable, ie. when proper net and market implementations exist)
  • JS test framework?
    • can be ran in Node.js or
    • can be ran from browser, with some UI (as interactive testing/troubleshooting)
    • Unit Test management tool - for test suite management
  • Python test framework?
    • Jupiter Notebook - useful tool for interactive activities

27.02.2020 - Arch workshop - Security walkthrough 4 - Payment API - Simple GNT Implementation

Agenda:

  • SGX-112 -> DB, AW, PR, PCh, MB, Borys

20.02.2020 - Arch workshop - Security walkthrough 3 - Payment API

Agenda:

  • SGX-112 -> AW, PR, PCh, MB, Borys

18.02.2020 - Arch workshop - Yagna sync

Agenda:

Homework:

  • MB: find out what is ExeUnit wrapper???

30.01.2020 - Arch workshop - Security walkthrough 2 - SDP & Infrastructure review

Agenda:

  • SGX-86 -> PR, PCh, MB, Borys

23.01.2020 - Arch workshop - Security walkthrough 1 - Yagna Host architecture + Identity API

Agenda:

  • SGX-85 task -> MB, PCh, Borys

Outcomes:

21.01.2020 - Arch workshop - attempt daemon demo

Homework:

  • MB run end to end

    • run net MK1 router cargo run --example ya_sb_router -l NOTE: must be run on non-standard port, see cmdline above

    • run 1 daemon set YAGNA_BUS_PORT= set YAGNA_HTTP_PORT= cargo run --bin yagna service run -d

    • run 2 daemon set YAGNA_BUS_PORT= set YAGNA_HTTP_PORT= cargo run --bin yagna service run -d

    • generate 2 api keys (Req & Prov)

    • play with sqlite client to explore the db???

    • play with curl to run CreateActivity()???

    • modify .NET clients to use api keys?

      • how to propagate the negotiated agreement to the yagna daemon store???

    • start GolemMarketMock

Agenda:

  • SGX-85 task -> MB, PCh, Borys

16.01.2020 - Arch workshop - ...

Homework:

  • Witek is still working on Provider Agent mock using old Market API specs. Maybe he should switch to Market API client code and update to latest 1.4.2 specs, and then continue with Provider mock.

Agenda:

  • MB individual tasks:

    • Collaborate on Payment API yaml specs
    • Drive the API unified convention agreement
      • Error info structures
      • Status code approaches
        • what status codes do we return and when?
    • Collaborate on Yagna Daemon integration - from QA point of view
      • Organize the test/functional scenarios
      • Explore existing command space and specify/execute scenarios on Win
      • Actually execute scenarios on code being developed
      • ...potentially identify room for automation

    ...potential tasks:

    • P2P approach in LTG

      • how can we enable yagna in browser???
      • NAT traversal
      • HTTP3 vs webRTC ?

    • Golem/Yagna network health monitoring

      • "Sentry"

    • SDK approach strategy

      • what should be included in SDK?
      • levels of SDK?

Notes:

  • Re API conventions:
    • Error response structures:
      • Call it Error, include code and message
      • Market API - OK?
      • Activity API - rename ErrorMessage to Error, add 'code' field, get rid of ProblemDetails
      • Payment API - rename ErrorMessage to Error, add 'code' field.

14.01.2020 - Arch workshop - Yagna Sprint 3 demo dryrun

Agenda:

  • Yagna Sprint 3 demo dryrun

09.01.2020 - Arch workshop - High level Yagna overview

Agenda:

  • Review and adjust Market API 1.4.2

Homework:

  • (DONE) MB: Modify Market API sequence diagram to indicate WaitForApproval() vs Confirm().
  • MB: Adjust the .NET Core Requestor Client to the Yagna Daemon (post Sprint 3)

07.01.2020 - Arch workshop - High level Yagna overview

Agenda:

  • Run first session of "High level Yagna overview" for the purposes of thread modeling (JIRA SGX-82)

02.01.2020 - Arch workshop

Agenda:

  • Confirm Sprint 3 task assignments
  • Review Market API yaml proposed changes

Homework:

  • MB: Edit BB04.1 sequence diagram - move the Agreement(sign) call from CreateAgreement() to ConfirmAgreement()
  • MB: Implement changes in testbed
    • POST agreement - returns agreement Id (as er yaml 1.3.1)
  • PCh: Modify Market API yaml and adjust client code:
    • CreateAgreement() operation - only requires ProposalId and ExpirationDate on input. (ie rollback the changes proposed in 1.3.0)
    • GET agreement - return Agreement body by Agreement Id (to both Requestor and Provider)
  • MF: Activity API Yaml changes: on Provider side, change PUT state & PIT usage to GET state & GET usage.

19.12.2019 -

Agenda:

  • Review ERD

Homework:

  • (Someone): Provide a communication diagram of full Activity lifecycle to illustrate the sequence of communication between: Activity API Impl (Provider), Provider Agent, ExeUnit, GSB, Activity API Impl (Req)

  • PR: Discuss with KubKon the rpc message format to communicate the State and Usage from ExeUnit to...where?

  • MB: Wait for MF to publish the Activity API impl, to make sure I understand the impl correctly

  • MB: Update the ER diagram with comments below

  • MB: Develop the sqlite script to represent the datamodel

  • MB: Write markdown article to specify the datamodel table&column semantics

  • MB: commit into core/persistence module

ERD Remarks

  • Agreement state is required. (Created, Sent, Canceled, Closed, ...)
  • Subscription and Proposal are not needed. negotiation can be in-memory. So remove Proposal, and move relevant info into Agreement.
  • Agreement parts in JSON format are required (after Proposal object removal).
    • Keep 4 fields Demand constr, Demand Props, Offer constr, Offer props,
    • Signatures in binary format are required.
    • Need to define schema of agreement artifact so that the signatures can be calculated
      • as 3 subsequent fields (3 subsequent signatures)
    • Change all decimals to text fields

NOTE: SQLite does not have decimal integer types. decimal(18,0) could be implemented only as string. NOTE: Debit Notes and Payments are subject to spaprate discussion.

10.12.2019 - Arch workshop -

Agenda:

  • Review additional modules/components (as identified in discussions so far, as they must be included in Technical)
    • Identity/Security concepts - authentication of Daemon's REST API? how is node's keyset generated? how is it managed? stored?
    • Golem Daemon Persistence layer (ER model?)
    • Configuration Mgr module?

NOTE: Convert the above into JIRA items into backlog

Homework:

  • MB To propose a draft of ER model for Daemon Persistence layer
    • Assume separation of datamodel between API segments (Market vs Activity vs Payment)

5.12.2019 - Arch workshop

Agenda:

  • Alignment and adjustment of projected scope of 1st 3 sprints of Yagna project, realigned the scope of MarketAPI Mk0, GolemNet Mk1.

3.12.2019 - Arch workshop -

Agenda:

  • Reconciliation and rework of Activity API swagger yaml

28.11.2019 - Arch workshop -

Agenda:

  • UX concepts in early stage
    • Installation of Golem components?
    • CLI use cases

Homework:

  • MB Merge all side branches in golem-architecture to 'draft', and then move all docs into a hierarchy in yagna/Wiki, plus update links in Technical Concept GDoc.
  • MB how to indicate the "interop" library as explicitly separate component to Tehcnical COncept
  • PCh Update the https://github.com/golemfactory/golem-architecture/blob/draft/docs/market-api/market-api-mk1-central-exchange.md to clearly indicate what is being built as Mk1 (ie. client-side lib/crate to communicate with API REST endpoints)
    • TODO change the Mk1 module/crate name from market-api to stg generic?
  • MF Update the GolemNET yaml (DONE)
  • MB merge above PullRequest and add the mock implementaiton of router
  • PR continue the client-side code of GSB communication
  • PR share the CLI use cases list
  • MB Add CLI module and use cases list to the Technical concept mention

26.11.2019 - Arch workshop - ExeUnit early reqs briefing

Agenda:

Homework:

  • PR/MF+PM: fill LTG JIRA...?
  • MF update GolemNet Hub API specs
  • MB implement GolemNet Hub Mk1 in GolemCLientMockAPI

21.11.2019 - Arch workshop - LTG cont

Agenda:

Homework:

  • Filip/PR - Fill the README.md in yagna repo to indicate the purpose of all modules in hierarchy.
  • AW/PR: digest the service-bus crate placeolder, describe the gsb client design pattern
  • AW to reach out to Daniel and agree/build the CI&auto-build setup for yagna repo (so that it is up before GSB code starts getting commited)
    • cargo test
    • cargo fmt
    • cargo doc
    • ...?
  • MB: reach out to KubKon and invite for Tue to give update.
  • MB review MF's pull request with GolemNet Hub API proposal (eg look at authentication proposed).
  • MB review the Payment API state (for the mtg on Tue)
  • MF Digest the Activity API specs in order to come up with default implementation concept.
  • PR brief the 'eth experts" with the basics of Payment API concepts

19.11.2019 - Arch workshop - LTG cont

Agenda:

Homework:

  • PCh - setup repo structure (according to latest conventions ;) ):
    • must have golem-daemon executable module
    • service-bus crate placeholders?
    • ...others
  • PCh - digest the Market API Mk1 impl (as router to MarketAPI TestBed, with JWT assignment)
    • How should the JWT be constructed?
  • MB: Raise the requirement for LTG security consulting (ongoing) by ITL
    • List the security review tasks in LTG plan
    • Raise request for weekly participation in Arch workshops (for catch-up, reporting, and scheduling individual security activities)
  • PR: To raise with the need to address in SDP (at least mention as placeholders):
    • Control of project dependency upgrades (ie. no spontaneous, non-consulted dependency updates).
    • Versioning approach (how is versioning controlled, who assigns version numbers)
      • Versioning matrix
  • KK to liaise with MF and AW to discuss the ExeUnit interfaces (with GSB and Agent)
  • MF work with Filip to define GolemNet API
  • PR/MF/PCh/Filip - lay groundwork for dev activities

15.11.2019 - Arch workshop - "Lightweight Golem" technical design cont

Agenda:

  • Recap the ExeUnit concept and provide briefing for a potential owner of the ExeUnit thread of tasks
  • Review the tech concepts of GSB API registrations (as prepared by MF/PR)

Homework:

  • AW Read Activity API Specs
  • MB On Tuesday, when availability of KubKon is known, pass the message to AW (if Kubkon not available - AW to dig into ExeUnit design. If KubKon picks ExeUnit, AW to pick Activity API impl design)
  • MB Study PUB/SUB model implementation in NNG - can I extend the MarketApiTestBed with simulated GSB API endpoint??? is it possible???
  • MB/PCh Review the MarketAPI TestBed status - how to use in Market API Mk1?
  • MF Catch-up with Filip on GolemNet API - it is important to have a persisted draft of GolemNet API operations.
  • PR

12.11.2019 - Arch workshop - "Lightweight Golem" technical design cont

Agenda:

  • Brainstorm the structure of Requestor Client App

Outcomes:

  • The draft WBS/task plan has been created. How do we progress with getting resources?

Homework:

  • MB Review the nanomsg
  • MB Add QA-related tasks/effort into plan
  • MB Supplement the tech doc with today sessions outcomes (re Requestor App)
  • MF/PR Brainstorm the tech details of GSB API registration and ExeUnit communication
  • MB/MF - rename SRAPI to "Golem Service Bus API"

07.11.2019 - Arch workshop - "Lightweight Golem" technical design cont

Agenda:

  • Describe ExeUnit interfaces and interactions with Golem Daemon and Provider Agent

Homework:

05.11.2019 - Arch workshop - "Lightweight Golem" technical design cont

Agenda:

  • Review of Provider Agent thoughts
    • Sample Offers for WASM
  • Recap the ExeUnit <-> Requestor communication
    • Discuss the ExeUnit interface and its interactions with other parties (Provider Agent, Activity API, etc.)

Homework:

  • On Offer props -> PR: adjust the sample

    • MB -> update Standards with new props
      • MB: find the standard property defining Payment Cap, or add one if doesn't exist, add soft limit and hard limit (reaching soft limit triggers a debit note, reaching hard limit kills the Activity (destroyActivity()) )

  • PR: to continue working on the Agent model & description: https://github.com/golemfactory/golem-architecture/blob/prekicki/ltg/projects/ltg/ltg-provider-agent.md

  • PR/MF: Prepare the Service Relaying API (Golem Service Bus API) specs to describe the "Service Registration" features in Golem Daemon to enable ExeUnit to listen to messages sent via Activity API

31.10.2019 (Wed) - Arch workshop - "Lightweight Golem" technical design

Agenda:

  • Identify 1st level components, their responsibility and relationships.
    • High-level component diagram with dependencies?
  • Label the identified components with "risk level" and propose the sequence of attack.
  • Decide on content & layout of component specs, including:
    • Indication of scope of responsibility
    • Specification of interfaces (methods/callbacks, or messages received vs. messages sent)
    • Dependencies on other components
    • Sequence diagrams of basic flows

Homework:

  • MB: Start the Lightweight Golem Tech Design Spec doc
  • PR: Attack the "Design Golem Provider Agent" - to give better indication of what's inside, responsibilitites, interfaces.
  • MF: Review Activity API Protocol Specs, thinking about WASM ExeUnit design.

30.10.2019 (Wed) - Arch workshop - "Lightweight Golem" kickoff

Agenda:

  • First workshop on Lightweight Golem scope & plan of attack
  • Objective: to build some understanding of task breakdown, parties involved and plan of Arch involvement
    • For consideration: Do we have a dependency of new GNT smartcontract logic on the proposed Payment API? Should we complete/stabilize Payment API before the final design of new GNT contract? Include the task to design the architectural and coding patterns, pattern repo?

Outcomes:

  • We start with a broader forum (including Marek F.) to put down first draft of technical design, and decomposition onto components & interfaces.

24.10.2019 - Arch workshop - resumed

Agenda:

  1. Prepare workshop retrospective summary & potential actions a) Workshop prep

    • moments of pressure and negative emotions

      • everyone had good intentions,
      • "I don't want to go through this again"
      • milestones help to achieve focus... ...but how to focus with minimal stress as side effect?
      • overcommitment is double-destructive
        • better to raise early "I won't do" than accept and not deliver
        • overcommitment partially due to missed estimates
      • could have been prevented if we took some risk and started preps early
      • contributed to communication chaos

    • "Crazy" Devcon submission

      • when submitting the topic, we were thinking of different content!
      • ...the topic has been amended in order to "win the slot"... suddenly focus was shifted to WASM
      • we were waiting until mid Aug for acceptance - doing nothing for the workshop content
        • even though we knew we really had nothing ready when we submitted
      • ultimately when we received acceptance - the main goal to avoid embarassment - and we have succeeded :)

    • WASM in GU is almost a side-effect of the workshop preps.

      • ...but this required a change of project plans (GU in Brass integration

    • gwasm-runner was designed and built

      • an exercise in API design
      • a long-running effect - a tool which is part of Golem ecosystem, and is being finalized (with Brass backend in progress)

    • perceived impact of Devcon preps on other projects

      • it escalated quickly to involve almost everyone on the floor!
      • seems all other projects had stopped...
      • ...but did they, really?

    • workshop preps yielded one of most cost-effective products, as in 6 weeks you burn a limited amount of effort

    • hardware design and provisioning

      • two PCs were overkill? a few laptops would be sufficient
      • design and specs were partially erratic (non-identical PCs were ordered, WiFi cards were insufficient)
      • all in all - multiple redundancies were effective
      • ordering was done fairly efficiently
      • Y2 support was spot on!

    • using Docker for dev tools was a good choice

    • last minute updates are bad, but one in particular:

      • last minute UI updates (auto-updates of GU status) caused unexpected issues with page refresh which impacted the live presentation (ironic, because if we didn't update this, there would less pressure during the execution)

    • cracker challenge was time wasted

      • overall? or if it was done differently, would it yield better outcome?
      • overcomplicated?
      • not communicated?
b) Workshop exec
  - live coding demo by Kubkon was great, and (if he wants) should be an asset in other Golem demos
  - checklists are good ;) formal checklist of venue setup would be useful
  - would a different timeslot be a better choice?
    - we wouldnt make it with hardware setup
    - only demo
    - ...but more people
  - no sound amplification! no mics! worth having own equipment?
  - funny "grumpy" interaction between presenters ;)
  - turning the lights off was a bad choice!
  - recording of the session - failed!
  - stick to the plan

c) Next steps
  - Run assessment of selected team member's reflections of Devcon prep collaboration (or lack thereof)
    - to make amends where sensible
  - Complete gwasm-runner Brass backend
  - Complete hardware inventory
  - Use KubKon (esp. live coding with commentary) for conferences, talks, demos, workshops
  - Use hardware, content and scenario and run 1-2 meetups, on our conditions/venue
    - TODO: redesign workshop to fit in our preferences - how much more effort required? 
      *Provide effort estimation by End Nov 2019*
    - Do in Warsaw, more focus on Rust coding - will potentially attract Rust developers, who may be needed fairly soonish.
  1. Share the summary with Board and obtain guidelines re actions

22.10.2019 - Arch workshop - resumed

Agenda:

  • Enumerate all open topics
    1. (HighP) GNT Token migration - (w/Gegez)
    2. Followup after devcon, workshop review and next steps???
      • MB: Devcon workshop postmortem planned (3 parts: wrapup/summary, sharing with Board, communication of summary to open team on Wednesday session)
    3. (HighP) "Lightweight Golem" a) Building Blocks - Payment API specs, Onboarding API specs b) Market&Activity API protocol testbed - test the concept and interface of ExeUnit c) Market API PoC on Golem CLI d) ITL -> Graphene -> SGX-enabled native app ExeUnit
      • we still need guidance re sequence of "sample app integrations" - to define optimal path to increasing Graphene coverage (input needed from V?) e) ITL -> WASM in SGX?
      • This does not depend on Graphene
      • Is this a good task for ITL?
    4. (MidP) Golem Unlimited future
      • GU refactoring/"rearchitecturing"
      • Standalone product?

Homework:

12.9.2019 - Arch workshop

Agenda:

  • DeVcon update

Homework:

5.9.2019 - Arch workshop

Agenda:

  • DeVcon update
  • WireLine tele-conference

Homework:

  • MB: Prepare slidedeck scaffolding including:
    • Lightning talk
    • Workshop scenario background
    • Workshop wrap-up

3.9.2019 - Arch workshop - DeVCon workshop

Agenda:

  • Review and update Devcon workshop action plan

Homework:

  • MB: Rework the DevCon Workshop Action Plan into table format.

29.8.2019 - Arch workshop - DeVCon workshop

Agenda:

Homework:

7.8.2019 - Arch workshop - Graphene workshop

Agenda:

  • Graphene in Golem - Roadmap

6.8.2019 - Arch workshop - Graphene workshop preparation

Agenda:

  • Prepare for workshop with ITL

01.8.2019 - Arch workshop - Payment API Continued

Agenda:

  • Review current state of Payment API proposal.

Outcomes:

  • We rework the model of the Payment API:
    • We introduce a concept of DebitNote - issued by Provider to request Payment, before, after or during the execution of Agreement. A DebitNote does not include "incremental" Amount, instead it notifies about Total aggregated Amount Due. At the end of Activity the Provider issues a closing Invoice, which includes Amount for the whole service.
    • Note: in "after" payment - no Debit Notes are required. In "before" payment, a single Debit Note is issued upfront, and Invoice at closing. In "pay as you golem" - Debit Notes are issued in parallel with Activity execution, Invoice issued at the end. In "fire-and-forget" model - the Invoice is issued upfront.

Homework:

  • MB: to rework Payment API documentation to take the above into account.
  • MB: To add "Golem protocol interactions with licenses" into Licensing Patterns doc
  • MB: To add License implementation examples to Licensing Patterns doc
  • MB: Add standard property to indicate payment platform required (add sample values to indicate flat ethereum, Plasma?, Concent)

30.7.2019 - Arch workshop - Payment API Continued

Agenda:

  • PR absent. Chat with LG on Payment API proposals and negative amount invoices.

Outcomes:

  • Added Payment API scenario where Payment Platform can be explicitly selected and changed during Requestor/Provider interaction.
  • As an implication - this may mean Concent can be integrated in the Golem generic model as a Payment Platform implementation.

Homework:

  • MB: Add content to Payment API docs (as per TODOs)
  • MB: To add "Golem protocol interactions with licenses" into Licensing Patterns doc
  • MB: To add License implementation examples to Licensing Patterns doc
  • MB: Add standard property to indicate payment platform required (add sample values to indicate flat ethereum, Plasma?, Concent)

25.7.2019 - Arch workshop - Payment API Continued

Agenda:

  • Follow-up on Payment API

18.7.2019 - Arch workshop - Payment API Continued

Agenda:

  • Follow-up on Payment API

Outcomes:

  • Updated Payment API specs draft

Homework:

  • MB: Add content to Payment API docs (as per TODOs)
  • MB: To add "Golem protocol interactions with licenses" into Licensing Patterns doc
  • MB: To add License implementation examples to Licensing Patterns doc

16.7.2019 - Arch workshop - Licensing API Follow-up

Agenda:

  • Follow-up of Licensing API discussion

Outcomes:

  • We do not intend to implement any API. Instead we should consider and propose licensing mechanism patterns and illustrate them as recommendations.

Homework:

  • MB: To add "Golem protocol interactions with licenses" into Licensing Patterns doc
  • MB: To add License implementation examples to Licensing Patterns doc

11.7.2019 - Arch workshop - Payment API Continued

Agenda:

  • Capture fundamental concepts of Payment API

Outcomes:

10.7.2019 - Arch workshop - Payment API Kickoff

Agenda:

  • Capture fundamental concepts of Payment API

10.7.2019 - Arch presentation - Activity API Intro

Agenda:

  • Activity API overview session in front of a broader audience (Shifted from 26.06.)

05.07.2019 - Arch workshop - Licensing in Golem architecture

Agenda:

  • AS, LG to join and debate licenses and their impact on Golem architecture.

Outcomes:

02.07.2019 - Arch workshop - Devcon 5 workshop planning

Agenda:

  • Review devcon 5 abstract proposals

Outcomes:

  • MB/PCh to review and update abstract proposals
  • MB to prepare bio input

27.6.2019 - Arch workshop - Plan for summer

Agenda:

  • Plan the activities for holidays (as 5 weeks are upcoming with incomplete quorum)

Outcomes:

  • PR away on holidays for 2 weeks
  • MB+PCh keep pushing for Payment API draft
    • Payment API should imply some standards of payment mechanism, ie. Payment transaction must indicate invoice id or activity id.
  • MB+PCh - we don't think we should talk to ITL without PR
  • PR is expected to leave the refactored golem-client 0.2 with market api facade integrated. Could PCh try to keep going?

25.6.2019 - Arch workshop - Inventory of open items

Agenda:

  • Reschedule the Activity API Intro session
  • Review the open strands, update trellos and review of code repos

Outcomes:

  • PR preparing golem-client for production release (golem-client 0.1)
  • PR integrating market api mock (from golem-market) into golem-client
    • prepare bootstrap for Requestor & Provider trait APIs, which can then be handed over to others to fill with implementation
  • PR watching SimpleTaskAPI to observe the implementation of Docker generic environment - when ready this shall be adapted to be the provider for the Sample Apps (eg. transcoding)

18.6.2019 - Arch workshop - DevCon5 session proposals

Outcomes:

  • Idea for a Dev workshop to demonstrate:
    • "Deploy your own distributed application on Golem MainNet (alternatively: TestNet) to acquire GNT tokens" ;)
    • Topic and abstract required by Friday 21.06. ...

Homework:

  • PCh: to propose draft of topic & abstract of "Deploy app on Golem Unlimited and earn crypto".
  • MB: to review, translate and provide feedback.
  • PR: to propose draft of topic & abstract on "Security aspects of Golem generic architecture"
  • All: consider the sample app choice for the "Deploy app" workshop

13.6.2019 - Arch workshop - Activity API -> Golem Brass

Agenda:

  • Activity API -> Golem Brass operations - make a indicative note

Outcomes:

Homework:

  • MB: synchronize ActivityAPI REST swagger def with specs doc (add Collect)
  • MB: extract Activity API swagger and publish in repo (golem-client) - create branch: feature/golem-apis-poc
  • PR: refactor golem-market mock to merge into golem-client

12.6.2019 - Weekly Demo - Market API

Agenda:

  • Demonstrate sample simple req app operating on Market API mock.

6.6.2019 - Arch workshop -

Agenda:

  • Review golem-client status - stable REST API (swagger) expected. Review outcomes of Windows compilation attempts.
    • Request: in the Market API mock - implement Demand and Offer body as combination of properties and constraints.
  • Continue with remaining bits of Activity API spec.

Outcomes:

  • Ad. market api mock:
    • Make all 'properties' fields - objects
    • Update swagger
    • Flow to be demoed:
      • Subcribe Demand, Collect (offers), Create Proposal, CreateAgreement, ApproveAgreement

Homework:

  • MB : stabilize the Activity API specs doc

4.6.2019 - Arch workshop -

  • Review golem-client status - expected to have some working mock, so that a sample .NET Core client can be generated for the purposes of Market API demo.
  • Continue with remaining bits of Activity API spec.

Outcomes

  • Reviewed the Market API mock REST/swagger.
  • Discussed the term Agreements and Online Availability SLA ideas.

Homework:

  • MB : update the Activity REST API draft with comments.
  • MB : Build https://github.com/golemfactory/golem-market : requires sqlite static lib (check github repo of Rusqlite) : Generate and run C# client (use tests as included in golem_market_api) - Note the demo app also needs to generate sample offer and pass it to mock
  • MB : Consider the term Agreements and Availability SLA ideas - prepare samples of relevant properties.

30.5.2019 - Arch workshop - Activity API spec cont

  • Review the Activity API REST draft.
  • Discuss possible strategies of communication of Golem NG to community and devs.
  • Continue with remaining bits of Activity API spec.

Outcomes:

  • The golem-client prototype (of Market/Activity API) which illustrates Demand/Offers, Market/Activity API scenarios plugged into Brass Golem Mainnet - should be a trigger of organized communication of "Architectural artifacts"
    • Shortly before the "reveal" of the poc, there should be a sequence of 'blogposts'??? prepping-up for the "reveal"
    • After the "reveal" of poc - publishing of BBxx artifacts should follow.

Homework:

  • (done) MB : split the Work Session Summaries into Quarters.

28.5.2019 - Arch workshop - Activity API spec

  • Walkthrough the implementation and clarification questions re golem-client - Market REST API.
  • Review TRANSFER command description and questions.
  • Continue with remaining bits of Activity API spec.
  • (added by PCh on behalf of @v and @reqc) Consider mid-term goals (till end of 2019) for Architecture
    • devcon workshop GU
    • golem-client
    • how to cooperate with other streams of work

Outcomes:

  • Ad. golem-client prototype - to be delivered by 3rd June:
    • updated, stable swagger of Market REST API
    • a working mock-up of basic Market API scenario (ie. mock of Market API and set of sample, illustrative API requests)

Homework:

  • MB: To come up with Activity REST API draft (for 'Control' segment) - including the ExeScript command JSON

23.5.2019 - Arch workshop -

Agenda:

  • Focus on transcoding scenario and talk-through the implementation aspects on Market API and Activity API.
    • Consider esp. the TRANSFER operation
  • Discuss TRANSFER operation in context of all ExeUnit types.
  • refresh contents and fill descriptions in architecture column https://trello.com/b/N8peqJmJ/mid-term-goals-copy

Outcomes:

  • Created PoC sample Demand/Offer for the golem-client scenario
    • MB to adjust the transcoding.md sample scenario and upload to repo
  • Discussed the ActivityAPI concepts
  • Considered the implementation of TRANSFER operation (in generic context)
    • MB to prepare sequence diagrams for TRANSFER based on examples created during session.

21.5.2019 - Arch workshop - Activity API - TRANSFER

Agenda:

  • (done offline) Consider impact of golem-client MarketAPI facade on Golem Brass and golemapp.py.

Outcomes:

  • Ad. sample use case selection:
    • Restated objectives/conditions of golem-client:
      • To demonstrate full scenario (Market API, Activity API, Payment API)
      • Must not extend Brass Provider
        • Use-case limited to batch processing
      • Must not be Blender...
    • Transcoding seems the best scenario
    • Other cases considered:
      • Dask - this would best work in interactive service model
      • Golem Continuous Integration - would imply security issues (as requires privileges to run docker within docker...)
      • GROMACS - requires clustered providers
      • Machine Learning (eg. TensorFlow)

14.5.2019 - Arch workshop -

Agenda:

Outcomes:

  • On providing "resources" for the golem-client prototype - possible models: a. Assign various team-members on part-time, but indefinitely (suitable for undefined, research tasks) b. Assign various team members on full-time, for individual tasks (optimal for well-defined tasks)

    1. (PJ+AS): To give heads-up of the golem-client initiative and intent to provide staffing assembled from existing teams, on best-effort basis (communicate on 22.05.)
    2. (MB): (After the announcement mentioned in 1.) To turn to individual teams with a "Request" for x-amount of people resources of specific profiles, either full-time or part-time

Homework:

  • MB: Sendout the ITL invite for review

10.5.2019 - Arch workshop - Prepare for ITL discussion

Agenda:

  • Review all input and content for discussion on Graphene requirements to ITL
  • Need to agree the approach to containered app 'standard layout' - do we include Clay team in this (as it already is on simple task api wbs), or do we do it ourselves separately?

Outcomes Notes ad ActivityAPI:

  • (done) DESTROY should actually be a DestroyActivity() operation.
  • (done) All others can be in ExeScript (and thus will remain confidential in Activity API scenario)
  • (done) Refine the generic sequence diagram to reflect the discussions
  • (done) Refine the SGX sequence diagram to reflect the discussions (and example table in doc)
    • Use colour coding to indicate encrypted vs nonencrypted messages are sent in SGX sequence diagram
  • (done) Refine the example table in doc (how commands are implemented in ExeUnit samples)

Some more fundamental reqs/assumptions on SGX-enabled ExeUnit:

  1. We expect that the ExeUnit must be capable of providing confidential communication between Requestor and Provider

Homework:

  • MB: (done) Integrate the comments into docs (Activity API and SGX exeunit)
  • MB: (done) Prepare a draft of "Invite" to be sent to ITL and share for review.

2.5.2019 - Arch workshop - SGX ExeUnit workshop

Agenda:

  • Prep-work for workshop with ITL
  • Need to have a rough understanding what an "SGX ExeUnit" should be capable of. Possibly expressed via ExeScript commands it supports - ITL should respond on how these commands are implemented.

Outcomes:

  • We need to design a sequence diagram illustrating the ActivityAPI scenarios to serve as illustration for defining SQL/Graphene requirements.
  • As founding assumptions we agree:
    • The control over Activity execution in exeunit shall be conducted using ExeScript statements, generated by App and sent to GolemDaemon.
    • For SGX scenario the GolemDaemon shall include logic which is SGX-aware (ExeUnit Proxy???) responsible for handling the key-exchange with enclave, and secure ExeScript forwarding (which may include some part of implementation of eg. TRANSFER operation)
      • The above is important as it allows for proper control over the "into-from enclave" secure pipeline.
  • Also for the generic ExeScript execution model:
    • Need to design how the results are returned from ExeScript commands
    • The result from ExeScript command consists of status code (OK/ERROR CODE) and free data (may be ExeUnit-type specific, eg. may include SGX-specific metadata, public key, etc.)
    • Batches of ExeScript commands return collections of Results of subsequent commands in batch.

Homework:

  • MB: (done) To integrate above comments into the ActivityAPI specs doc
  • MB: (done) To draw generic ActivityAPI illustrating the basic flow of ExeScript (DEPLOY/START/TRANSFER/DESTROY)
  • MB: To design a sequence diagram illustrating the components/modules in SGX-enabled ExeUnit, to indicate the key operations and highlight how they would work in SGX/Graphene context -> this shall become input/pre-req for discussion with ITL On Graphene requirements
  • MB: Prepare a draft of "Invite" to be sent to ITL and share for review.
  • MB: (done) Run the current GolemCLI reimplementation (need to switch to gcc build toolset?)

25.4.2019 - Arch workshop - Graphene requirements

Agenda:

  1. Define scope of workshop with ITL (with some prerequisites) so that as a result - a more specific set of Graphene requirements can be agreed. Prerequisites to be considered:
    • Set the objective as "run arbitrary binary directly in SGX enclave"
    • Probably expect some binary package format (so layout needs to be defined)
    • Which platforms/ SGX versions/ OSes are feasible?
    • Batch vs interactive service execution - how each of these could look in enclave with Graphene?
      • Attestation of enclave itself? Verification of outputs returned from enclave? Non-repudiation (esp. with external arbitrage)?
      • Protected files?
      • Network access?
    • ...

Outcomes:

  1. Can Golem receive access to any existing concept documentation for:
  • Existing Graphene Roadmap, including:
  1. It must be easy to deploy "things" in enclave
  • A non-developer should be able to run his application in enclave (no additional steps, tools, manifests...)
  1. Graphene launch is very slow currently
  • The launch of binary in enclave should be quick (how quick???)
  1. The problem of "forking"
  • As to run an executable, one needs to provide a manifest, if this executable want to spawn another executable - it would need to provide the manifest for that other executable.
  • In theory it should be possible to apply the "root manifest" to whole process hierarchy...
  • The problem of "shared memory"
  1. Platform a) Graphene is able to execute Linux executables in SGX enclaves b) It is a target for Graphene to be able to run Linux executables with WIndows as a host OS c) Graphene does not intend to enable running other OS' binaries... (ie. there is room for other projects)

Homework:

MB: Prepare a draft of "Invite" to be sent to ITL and share for review.

23.4.2019 - Arch workshop - GolemCLI rebuild exec planning

Agenda:

  1. Define initial execution plan for GolemCLI reimplementation:
  • Owner, squad
  • iterations scoping, etc.
  1. Refine Golem reqs for Graphene (see below, outcomes from 17.4.)

Outcomes:

  • General comment on Golem Factory staff structure: would a "matrix structure" be more adequate to represent the org structure via the "skill pools" (Rust devs, Python devs, QA engineers, Analysts, Researchers) vs "work streams" (Brass support, Clay dev, GU dev, etc.)?

  • rust golemcli WBS created with draft plan for Phase 1 (https://docs.google.com/spreadsheets/d/1RGGj8ArXySmYVywczyXRAqiUrUlCA9VuhA81zGSUmiA/edit#gid=0)

  • MB to liaise with ŁG to convince to delegate R&D team representative(s) to rust-golemcli

  • PJ to sanction the rust-golemcli initiative kickoff with resources suggested in WBS

  • Golem reqs re. Graphene:

    • Prepare scope of "requirement setting" session with ITL:
      • Set the objective as "run arbitrary binary directly in SGX enclave"
      • Probably expect some binary package format (so layout needs to be defined)
      • Which platforms/ SGX versions/ OSes are feasible?
      • Batch vs interactive service execution - how each of these could look in enclave with Graphene?
        • Attestation of enclave itself? Verification of outputs returned from enclave? Non-repudiation (esp. with external arbitrage)?
        • Protected files?
        • Network access?
      • ...

17.4.2019 - Arch workshop - GolemCLI rebuild concept sharing (2)

Agenda:

  • Sharing of golemcli roadmap (link) with Ola

Outcomes:

  • Question: should Simple/Basic Task API be progressed at all?

    • The answer is: yes, this needs to be progressed, however the approach to "soft launch" needs to be deferred until strategic decisions on Golem's communication are made (June 2019).
    • It would be helpful to work on Simple Task API having in mind it's compatibility with Activity APIs ExeScript (so that the Simple Task API integrations can still be plugged under generic Activity API model)

  • Question: Should the rust-golemcli be progressed and how?

    • Yes, now requires execution plan, resourcing, methodology. Can/Should be started in parallel to Simple/Basic Task API. Requires reiteration of GolemCLI scope and task defs.

  • DevCon input proposal:

    • Demo of GU integration and setup (...also involving the WASM execution env.?, rust-golemcli?)
    • WASM-related workshop

  • Golem requirements from Graphene (kickoff)

    • First goal: run a "host direct" binary in Graphene-based ExeUnit (so that it is executed in SGX enclave)
    • Ultimate goal: run "normal" Docker container in SGX enclave:
      • Issues today: Graphene uses glibc from x years ago, so it is impossible to launch "modern" images on graphene.
      • It is difficult to upgrade grpahene to recent glibc, because of changes introduced in glibc

12.4.2019 - Arch workshop - GolemCLI rebuild concept sharing

Agenda:

  • Sharing of golemcli roadmap (link)

Outcomes:

  • Question: should Simple/Basic Task API be progressed at all?

    • It makes sense to develop this changing assumptions to:
      • Do not publish the Simple Task API openly but instead invite trusted integrators to have a play and then announce 3rd party integrators, and
      • ...work on Simple Task API having in mind it's compatibility with Activity APIs ExeScript (so that the Simple Task API integrations can still be plugged under generic Activity API model)
      • Simple Task API can be published but only when we already have a stable plan to release the generic Activity API (and relevant SDK)

  • DevCon input proposal:

    • Demo of GU integration and setup...
    • ...also involving the rust-golemcli

9.4.2019 - Arch workshop - Activity API Alignment

Agenda:

  • Review Marek's notes on Simple Task API to get better understanding of it's scope - so that we can come up with better alignment.

Outcomes:

  • We have generated 4 topics which would be relevant for Golem's future:
  1. Market API - Distributed Market Protocol implementation
  2. Golem-net with following requirements:
    • Improve UX (1-port, muxing, relays...)
    • Fame points (for using and contributing to libp2p)
  3. Implementation of subsequent GU integrations/artifacts
    • Gumpi
    • Gromax
  4. Rewrite GolemCLI in Rust and make it a platform for implementing facades for *APIs
    • Market API
    • others

4.4.2019 - Arch workshop - Activity API Alignment

Agenda:

  • Invite Marek F. for alignment of first Activity API segment draft.

2.4.2019 - Arch workshop - Activity API Kick-off

Agenda:

  • Review artifacts created.
  • Review first Activity API segment draft.

Outcomes:

  • Concept of ExeScript - a simple sequential language to be executed on Provider Node on Requestor's request
    • The ExeScript operations correspond to ExeUnit interface
      • The definition/specification of an ExeUnit must include indication of how ExeScript commands are executed (what do they mean in context of the ExeUnit)
    • ExeScript commands:
      • DEPLOY
      • DESTROY
      • START
      • EXEC ???
      • STOP
      • PAUSE
      • RESUME
      • TRANSFER
        • transfer is used to transfer data to/from Provider node (effectively ExeUnit)

Homework:

  • MB Compile the notes and prepare for catch-up with MF on Thursday
Clone this wiki locally