docs: 2018 Q3 OKRs

[daviddias]

IPFS:

• Spreadsheet for IPFS 2018 Q3
• PR for 2018 Q2 for reference

[alanshaw]

Not final or worded well, just noting down a few things in my head. Will update...

• Smooth out the onboarding for devs
  • Easy to use, non buggy "first exposure" functions
    • Solid tests for files.{add,cat}(*stream) etc.
    • Review API and make changes for easier/more intuitive use
  • Better API documentation
    • Auto generated docs - jsdoc?
    • Up to date READMEs
• Reduce dev friction for collaborators
  • Speed up the tests, could be a number of things:
    • Ensure we can spawn multiple nodes in parallel (ipfsd-ctl)
    • Run the tests in parallel
    • Address bottlenecks in js-ipfs
• Address stability and reliability of js-ipfs
  • Improve interface-ipfs-core test quality
    • Isolation (do not depend on outcomes of other tests)
    • Assertions (false positives should not exist)
    • Separation of concerns (each test should test one thing)

[daviddias]

Here is my first draft with things that come to mind:

• js-ipfs has same connectivity magic feeling as go-ipfs
  • A js-ipfs browser node can find and fetch any file on any other js-ipfs browser node (make it always verifiable with tests)
  • A js-ipfs browser node can find and fetch any file on an IPFS Gateway (make it always verifiable with tests)
  • A js-ipfs daemon can find any file that is on another daemon on the Local Area Network (make it always verifiable with tests)
• js-ipfs daemon is stable
  • libp2p errors do not crash js-ipfs (treat libp2p as any external dependency, with a separate state)
• js-ipfs can handle large amounts of data
  • continue KRs from Q2
• Testing
  • sharness tests are run against js-ipfs daemon (using go-ipfs cli)

A lot of these KRs are half baked, we should only consider them finished once they are tested and verified on CI, always.

[hugomrdias]

Also just some thoughts:

• in general improve js-ipfs code for browser bundling, this is specially browser focused because for the browser some stuff needs to be handled in a specific way so bundlers can do their jobs
  • Consolidate common code patterns, setimmediate usage, requires for tree-shaking
  • track bundle size
  • make sure webpack, rollup and parcel can handle ipfs with minimal configuration
• improve the examples
  • make them codesandbox.io compatible for easy hacking
  • examples for webpack, rollup and parcel, maybe making these into tests also
  • more building blocks examples, like how to get a file but manually setup libp2p, repo, blockservice, bitswap, ipld etc. This was very very useful to me this quarter.
• cli improvements to achieve parity with go
  • error handling
  • help/description info
  • add more features
• errors in general, several conversations already happen around this topic maybe we could gather all those thoughts and make a short term plan and a mid/long term plan.
• dependencies updates, like greenkeeper but only for ipfs, libp2p packages.
• all things service worker related need to be documented and easy to integrate, sw is a powerful tool but we can't just make a standalone bundle and give to developers. Developers can only register one sw and some events can only be attached once, this brings us several integration problems.
  Right now we have at least sw-gateway and my new streaming download solution built on service-worker, more will surely come in the future. We need to understand the best solution to provide developers with the building blocks and documentation so they can build their specific service worker.

[vasco-santos]

Just small notes that I had in my head.

Improve error handling

Stability of the system, as well as performance and reliability are crucial for the developers' productivity and community engagement.

Reference:

• Taking control of Error Handling, reducing js-ipfs Daemon crashes #1406

Improve developer experience, builds and testing

Mainly improve browser related tooling/configs, decrease bundle size, decrease the time needed for running tests, mitigate timeouts during CI and consider the usage a more strict linter.

Reference:

• #228

Service worker running IPFS

This topic is something that I have been discussing with @hugomrdias for a while. As service workers are becoming more and more used across the web, we should have a "pluggable" service worker.

Basically, an application cannot have more than one service worker per scope. This way, and taking into consideration the service-worker-gateway, as well as a service worker that @hugomrdias has implemented for fetching files, we should provide a way for users to wrap several service workers in a single one. Maybe we should open an issue for discussing this further.

Ease onboarding

Decrease the size of the mountain that a new contributor has to climb for being ready to understand and contribute to the project seems an important challenge to achieve.

[daviddias]

A bit dup of libp2p/js-libp2p#207 (comment), but here it goes as it is important for this team as well.

As food for thought, I wrote up -- ipfs/team-mgmt#654 -- to remind everyone that if we miss to understand our user and use case needs, we will only be able to see the needs that we see during the development.

If I were to name the most important objective I would say right away that is getting Connectivity Magic for js-ipfs. And this doesn't mean only DHT + IPNS, there are a lot of creative things we can do with Relay, Infra discovery, Local Discovery (get MDNS from go and js to be friends), Rendezvous and more. The return to an ipfs.catin the browser should always be the file.

[daviddias]

Let's continue discussing the OKRs over the markdown, taking inspiration from the notes already shared in this thread.

I've defined 3 main objectives (rewording is welcome if you see the need) for this quarter which capture the most requested and needed features and bug fixes.

Let's shoot to get the final list of KRs by Wednesday, final review on Thursday and ship it on Friday.

[daviddias]

@alanshaw @hugomrdias @achingbrain. I went again through your list of suggestions and completed the list of KRs. Please review:

• Check if wording makes sense.
• Check if they are achievable this quarter.
• Check the ones you will want to take the lead.

I've assigned some KRs already, given that they are a continuation of previous work.

[achingbrain]

Do you think it's worth having a 'technical debt' section for things similar to the CI testing time item?

Feature parity with the go API would also be a good thing to shoot for.

[daviddias]

CI testing time is contemplated in the third objective.

What features (APIs) do you feel js-ipfs urgently needs that go-ipfs has that are not captured by the current OKRs?

[achingbrain]

What features (APIs) do you feel js-ipfs urgently needs

The biggest pain for me at the moment is the DAG API being incomplete.

Also npm-on-ipfs needs ipfs.name.resolve and ipfs.name.publish which I don't think are implemented yet although I guess that would be captured by the 'make npm-on-ipfs a thing' OKR.

[daviddias]

@achingbrain IPNS is landing this quarter by the libp2p team. Note that publishing and resolving names is for sharing the registry only, all the rest should work without it which is the bulk of the challenge.

[daviddias]

@hugomrdias I've a few questions based on your suggestions:

• Why codesandbox.io and not codepen?
• Why is it hard to use bundlers with js-ipfs? More than the two examples we have already (browserify + webpack)?

[hugomrdias]

codesandbox gives the developer an environment very similar to a local dev environment but inside a browser. You can require/import, bundle, lint, etc. It has integration with github and deploy providers like Now. Even the UI/UX is very similar to popular editors.
This enables a developer to quickly go from a prototype to a github repo ready to build the next awesome dapp in secs, and the other way around enables us to build examples on github synchronised to a static url on codesandbox from where users can fork and play around.
some references:

• https://hackernoon.com/the-future-of-codesandbox-d1ccc5aebf59
• https://twitter.com/mlthuret/status/1012266345529073664
• https://twitter.com/CompuIves/status/1012396990800564225

About bundlers, it's not about being hard to use, it's more about being as easy as possible. Both webpack and parcel provide zero config functionality and we need to make sure ipfs is supported by those default configs. Another thing is, the community is starting to run their bundlers through the complete dependency tree and not use the pre-bundled files from packages, we need to make sure ppl can run webpack/babel through ipfs complete deps tree and still work without errors. An example of this was the work i did to make ipfs work with create-react-app 2.0.

Another topic is size, do we have the best require patterns to enable the best bundle sizes from these bundlers? ex. #1414
Do all of these bundlers include node api polyfills that work as expected ?

[daviddias]

There are a couple of suggestions that I didn't include as KRs (e.g. Bundle Size, conventions) because I didn't feel that they should be a full KR but they do support other KRs, so add them as issues on the project.

[daviddias]

In this KR, the implicit things are:

• js-ipfs should have a good Conn Mgmt defaults
• Conn Mgmt should be part of the docs of js-ipfs
• There should be instructions (i.e. Tutorial) on how to apply Conn Mgmt
• This needs to be severely tested
• No browser shall crash from too many connections.

[daviddias]

In this KR, the implicit things are:

• To enable the DHT by default, a lot of testing needs to happen. Essentially, finish Awesome Endeavour: DHT Part II #856.

[daviddias]

In this KR, the implicit things are:

• Setting up a daemon that gets monitored should be easily reproducible
• It should be tested on our IPFS Infra
• The daemon should be exposed to the "elements" (rest of the network)

[daviddias]

In this KR, the implicit things are:

• The node should be able to get them either by directly connecting and/or Relay through the bootstrappers.

[alanshaw]

I'm all for this, but I think a precursor to this is application/library code stability and test quality, which would also contribute to the reliability objective.

In some cases, the tests are not flakey, the application/library code is flakey and the tests fail because of it.

In other cases the test assertions are false positives or false negatives or are not adequately testing responses or interactions.

Tests should fail when a code contribution causes an error, not when unrelated parts of our application/library code cause errors.

We're making good progress towards addressing those problems already but the important thing to note here is that we're not going to get a daemon running for 10 days without addressing the "flakey tests" problem.

I'd like to put my name down for "A Daemon runs for 10 days without a crash" and "Cut CI testing time in js-ipfs and js-ipfs-api by 50% (2x)".

I feel these are super difficult to achieve given that they're dependent on a moving target (the codebase is constantly changing) but I'm willing to give them a shot.

[hugomrdias]

What you guys think about formalising a bit more these definitions, add the " implicit things" as sub-topics defining the scope and an extra label indicating what would be a good output to validate the sub-topics.

let me give a example:

• A Daemon runs for 10 days without a crash

  • (machine image) Setting up a daemon that gets monitored should be easily reproducible
  • (CI test) It should be tested on our IPFS Infra
  • (report) The daemon should be exposed to the "elements" (rest of the network)

• Handles correctly a scenario where the number of Connections grows (through Conn Management Strategy)

  • (code) js-ipfs should have a good Conn Mgmt defaults
  • (docs) Conn Mgmt should be part of the docs of js-ipfs
  • (example) There should be instructions (i.e. Tutorial) on how to apply Conn Mgmt
  • (CI test suite) This needs to be severely tested
  • (stress test) No browser shall crash from too many connections.

This can be done in any other away, the main thing here is define scope and expected output that will validate each KR. After assignment this can be further improved in other places.

[daviddias]

@hugomrdias we will add the notes to the "Notes on writing" section so that we don't loose it.

[daviddias]

It's time to wrap this up.

@alanshaw @achingbrain @hugomrdias, ideally, each should have at least 4 KRs with two of them being a P0. Right now the KR distribution is kind of unbalanced and it is not giving you a set of problems to focus.

Also, I'm not confident that the list is as polished as it can be for the Objectives we have. Now that the Dev Meetings happened and your understanding of the whole project expanded vastly, I want you to go through these again and come up with new proposal. Please put this as your top priority until we freeze KRs next Wednesday.

[alanshaw]

I didn't appreciate the time frame on the switch to base32 until the developer meetings. I'm happy to spearhead the development work that needs to happen on the JS side of things for this. BTW this is a P0 for In Web Browsers OKRs. I'm also going to continue to be involved with the WebUI rebuild as part of the GUI WG.

[daviddias]

@hugomrdias can I tag you on this one?

[daviddias]

We desperately need a Interop Champion. See what's happening on ipfs/interop#28 (again)

@achingbrain can I task you to get the interop test harness with iptb off the ground?

[daviddias]

Now that you contribute bits to go-ipfs and all :D + you have been catching a ton of interop bugs. I believe this makes most sense.

[achingbrain]

Sure, sounds good 👍

[daviddias]

@hugomrdias are you down to take this KR?

[daviddias]

@hugomrdias are you down to take this KR?

[hugomrdias]

yes

[daviddias]

Seems that this will get dropped.

[daviddias]

@hugomrdias are you down to take this KR?

[hugomrdias]

yes

[daviddias]

OKRs for 2018 Q3 are set! Godspeed team! And may the dag be with you 🌐