Create a Workflow component

[kinow]

This is related to #87 and #75 .

@oliver-sanders showed us the JupyterLab application, which allows users to add as many notebooks as they like to a sort of dashboard panel.

This looks like an experiment worth a try - with the risk of having some performance issues due to the number of workflows open simultaneously. So I think a Workflow view could be used for that.

JupyterLab uses PhosphorJS. From their about:

PhosphorJS provides a rich set of widgets, layouts, events, and data structures. These enable developers to construct advanced, production-quality, desktop-like web applications that would be otherwise impossible using CSS alone.

In order to include it in our current application, we would need something similar to the following steps:

• Create a Workflow component, that simply loads the Phosphor main view
• Create a WorkflowTreeView component (simpler than the graph for this experiment)
• Make sure the WorkflowTreeView can be added multiple times to the WorkflowTreeView component panel - that probably means using props or some other mechanism in the component to pass the workflow we want it to display

With that I think we would be able to have something similar to JupyterLab.

[kinow]

Examples:

• Tabs
• Menu
• DockPanel
• more here

@oliver-sanders the Workflow component would simply have a template that loads the DockPanel... and each dock would be a WorkflowTreeView (for now, but later we can add graph view too). What do you think?

[oliver-sanders]

DockPanel! Awesome!

[sadielbartholomew]

While it looks powerful, I am not sure that PhosphorJS is the right way to approach creating our UI frame components (panels, tabs, menus etc,). Would it not be much simpler to use Vue component(s) designed for this purpose, so we can create a solution most customised to our own requirements, & so it/they are simpler to incorporate being Vue-based?

From a quick look at the Awesome Vue listing, I can find some components that do very similar to that shown for PhosphorJs. For instance, there is a huge list of possibilities for the UI layout & some more for tabbing. Lots of great features are included in some/most: responsivity, drag, drop, . Some decent examples for are:

• Vue grid layout: about & "basic" demo
• Autoreponsive Vue: about & demo
• Vue tabs: about & "basic" demo
• v-chaceli: about & demo

Actually, I was hoping to be able to have a go at implementing something for the UI frame components, using one the above. How far along are you on this @kinow? If you haven't started, can I take over, as I have some spare time? Otherwise don't worry, I can find something else. Thanks.

[kinow]

It would be grand to have you working on this Sadie! And no need to follow what I described here. It was what I found when looking at JupyterLabs, but if we have something done in Vue.js, then it might be even easier.

Thanks for stepping in!
Bruno

[kinow]

See comment from @oliver-sanders in the linked PR above:

Should be really easy to implement, the field workflow.status field is either running or held:

query {
 workflows{
   status
 }
}

[kinow]

In the #301 PR for GScan component, I'm using Material Design icons. If others approve the design, and using icons instead of SVGs, then this issue won't be necessary.

[oliver-sanders]

If others approve the design, and using icons instead of SVGs, then this issue won't be necessary.

???

[hjoliver]

In the #301 PR for GScan component, I'm using Material Design icons. If others approve the design, and using icons instead of SVGs, then this issue won't be necessary.

Did you accidentally put this comment on the wrong issue @kinow?

[kinow]

Ops, wrong issue!

[kinow]

See comment from @oliver-sanders in the linked PR above:

Should be really easy to implement, the field workflow.status field is either running or held:

query {
 workflows{
   status
 }
}

Wrong issue as well 😄

[kinow]

The PhosphorJS project has been archived, and is not maintained any more. Jupyter has started lumino fork to maintain it and use it in JupyterLab.

For now it should be possible to use the phosphor docs while experimenting with lumino.

[kinow]

Also spent some time looking at the project referenced by @sadielbartholomew (thanks!). Here's a comparison table:

Project	Vue?	Npm?	Supports drag & drop	Tabs?	Widget Resize?	Dynamic Widget?	License	Contributors	Last commit
https://github.com/jbaysolutions/vue-grid-layout	☑	☑	☑	☐	☑	☐	MIT	33	01/11/19
https://github.com/xudafeng/autoresponsive-vue	☑	☑	☐	☐	☐	☐	MIT	1	14/02/19
https://github.com/cristijora/vue-tabs	☑	☑	☐	☑	☐	☐	MIT	7	17/05/18
https://gitlab.com/shellyBits/v-chacheli	☑	☐	☑	☐	☑	☐	MIT	2	08/04/18
https://phosphorjs.github.io/	☐	☑	☑	☑	☑	☑	BSD-3	19	06/11/19
https://github.com/jupyterlab/lumino	☐	☑	☑	☑	☑	☑	BSD-3 (?)	20	20/11/19

• Dynamic Widget is not a good title. That's a feature where widgets can be freely moved around, attaching to the sides of the canvas. I think vue-grid-layout may have it too, like lumino.

Four can be discarded, IMO.

• PhosphorJS - is archived, not maintained.
• https://gitlab.com/shellyBits/v-chacheli - was never published to NPM, so we would have to keep an eye on commits, and install manually. Its last commit is also from over a year ago.
• https://github.com/cristijora/vue-tabs - Last commit over a year ago.
• https://github.com/xudafeng/autoresponsive-vue - Maintained by a single developer so far, also doesn't provide many features.

That would leave us with two options, lumino and vue-grid-layout. Both have pros and cons.

lumino

A fork by the Jupyter team from the archvied PhosphorJS project. It is being maintained mainly for JupyterLab.

The first issue is that the project is new, so there are no docs. We can probably rely on the PhosphorJS documentation for now.

PhosphorJS' main developer has left the project. So there is no guarantee on when issues will be addressed (they are going to be busy for a while with the new project bootstrap), nor if they will have the know-how of the old project internals.

Finally, there is a risk that the code is contaminated by Jupyter-specific code.

The pros is that even though the project is not a vue component, it should be fairly easy to create a wrapper to vue-ifies it. And it checks most boxes for the features we need.

Its main advantage is that is has several components that interact with each other. So we can have a dynamic canvas (the widgets area is dynamic, where a widget can occupy 1 unit of size, or more), widgets can be resized, and it can also have tabs.

Other solutions will require us to wire everything up together, forcing us to maintain more code.

vue-grid-layout

Not as powerful as lumino, but provides very similar functionality. Its learning curve is way less steep than lumino's. Adding it to the project should be fairly straightforward.

Its only issue is that it doesn't support tabs. Not sure if it would be easy to add tabs. Vue has other components that provide tabs, but we would have to "sync" the vue-grid-layout with the tabs, meaning that we would have to listen to events, propagate them, and make sure everything works well together.

Conclusion

For now, I think the simplest is vue-grid-layout for a quick experiment. Where we have a simple canvas like their demo. It would start empty, and users could then add a Tree or a Graph to the workflow area/canvas.

Any objection? This issue will be probably split into other issues, and have more follow up issues. @sadielbartholomew feel free to chime in and volunteer if you would like to work on anything related to this issue.

It is not high priority. My next priority will be mutations, but I'm working on it while other PR's are reviewed 🙂

[oliver-sanders]

Thanks for doing this, the table is really helpful.

I think my preference would be Lumino as it gives us everything we need out of the box. My thinking would be that Vue support isn't important for this and trying to get an interchangeable tabbing system working with "dynamic widgets" is pretty darned hard.

If Lumino is difficult and we want to use Vue-Grid as a stepping stone ...

[kinow]

master...kinow:try-lumino

After I finished testing the subscriptions/websockets this morning, I had a go at Lumino. This branch contains a DockPanel (lumino class), with the other changes necessary to match the design sketch - i.e. workflow/:name now shows a Workflow.vue view, instead of Tree.vue, it is using our GraphQL utility code, even loads style for lumino.

With little documentation for both Lumino and Phosphor, I got stuck now where I need to create a Widget (lumino class) and add it to the DockPanel. As this is just an experiment, I am trying to add a widget showing a Tree component for the current workflow.

The current issue is that it is not clear to me how I get a Vue component, and convert/wrap/integrate into a Lumino widget. But once that's done, in theory, we should have a working prototype that resembles JupyterLab 🤞

[kinow]

At the moment it displays the Workflow.vue view, with no content, but - what's good - with no console errors either.

[kinow]

(the GIF doesn't show my mouse cursor, but I am clicking on Add View link on the top right corner several times to add the widget/tree)

Branch updated, a few console errors now that I am trying to understand how to avoid (they are from Lumino, not from Vue or Cylc UI).

Already spent some time (30mins) overriding the Lumino default CSS theme to look more like the design sketches.

Once I have fixed the console errors, it will be missing:

• use - probably? - a modal dialog to let user choose between the different components available (i.e. graph & tree for now) then add the new widget for that component
• attach the component to the widget... have a few ideas for that, but will have to spend some time testing them to see whether they work or not, and pros&cons
• performance/memory... we will end up with a part of the application that is managed by lumino, not entirely by Vue.js, which means that we must be careful to sync whenever a tab/widget is removed, we must remove all related Vue.js stuff; ditto for when a view is removed/navigated out, as we also need to make sure that the lumino stuff has been correctly disposed
• test, test, test :-) no idea if it will work on mobile, if it won't affect the performance of our components, etc

But right now I would say it's near 50% done?

[kinow]

Ready for a draft PR.

[bollwyvl]

Thanks for chiming in over on the lumino docs: sorry we haven't been more responsive, but taking on another upstream is a bit of a challenge for the already heads-down-trying-to-get-a-major-version-out JupyterLab team.

Basically, the only magic in wrapping insert-framework-hotness-here in a Widget is a DOM node you plan to manage, e.g.

node = document.createElement('div')
FrameworkDoYourThing(node)
widget = new Widget({node: node})

or using your framework to use the node it gets.

widget = new Widget({node: node})
FrameworkDoYourThing(widget.node)

(though you'd probably do them inside a subclass, etc.).

It pretty much gets out of the way after that, though you might want to overload some lifecycle methods like dispose and onResize.

There isn't a lot of in-the-wild examples of wrapping lumino/JupyterLab components inside other app frameworks, as especially @lumino/application:Application (which owns things like keyboard shortcuts, commands, etc.) not unreasonably wants to own the window.

Here's the deepest I've delved into Lumino/JupyterLab/Vue, which was picked up by the community to move a bit further (though it kinda looks like that would make a lot of Vue objects, but then, I don't really understand how vue works).

Another approach is to extend the JupyterLab application itself, which isn't as insane as it sounds especially if you don't plan on supporting third-party extensions. Indeed, it's possible to write plugins that work for any lumino application. A good starting point is the notebook example which is kind of the minimum testable product for reusing lab components.

Hopefully we'll be able to get something moving on the docs front, soon!

[kinow]

Hi @bollwyvl ! Thanks for the detailed reply. I will go through it again in two weeks once the summer break is over and I will start working on our Vue component that is using Lumino (still in a PR, but looks like we will go down that route).

Oh, and I thought I recognized your GitHub username. Looks like this is your second time helping us, thanks for this and also for the graphql-ws pull request. Even though that's pending, we have incorporated your code in our code base for now, until they finish the review.

So in theory, once everything is working, we should have that code helping to send the data to Vue.js & ApolloClient, which puts the data in the store for Vue to react and then create/update components, which will be Lumino widgets too. I will have to take a look at your explanation and links to see if the approach I did is similar to what others have out there.

[bollwyvl]

Funny little world! Enjoy the holiday! I confess utter ignorance of your product, of course, but as you may guess, if there's any overlap between you're users and jupyter users, I'd also recommend investigating getting your novel client pieces into the jupyterlab client as an extension... Especially if you have a well-constrained data file format. This could span from a custom editor, a live preview, a renderer or a jupyter widget... Or a custom lab app. While everyone likes to control their own window, the serendipity of having multiple features can often yield something even more powerful. The server piece looks more heavy duty... But given the pr referenced above, if you're already on the event loop, you could be running right inside the notebook server (or kernel), if that has any value (even for development or training).

…

On Sat, Dec 28, 2019, 01:17 Bruno P. Kinoshita ***@***.***> wrote: Hi @bollwyvl <https://github.com/bollwyvl> ! Thanks for the detailed reply. I will go through it again in two weeks once the summer break is over and I will start working on our Vue component that is using Lumino (still in a PR, but looks like we will go down that route). Oh, and I thought I recognized your GitHub username. Looks like this is your second time helping us, thanks for this and also for the graphql-ws pull request <graphql-python/graphql-ws#25>. Even though that's pending, we have incorporated your code in our code base for now, until they finish the review. So in theory, once everything is working, we should have that code helping to send the data to Vue.js & ApolloClient, which puts the data in the store for Vue to react and then create/update components, which will be Lumino widgets too. I have to take a look at your explanation and links to see if the approach I did is similar to what others have out there. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#90?email_source=notifications&email_token=AAALCRCONDGM5S6JDV5XQFLQ23VQLA5CNFSM4HZXXE72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEHYDJWY#issuecomment-569390299>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAALCRGYA6LU7SZLEWAI2PDQ23VQLANCNFSM4HZXXE7Q> .

[kinow]

We have a way to start our app in "offline" mode, without the need for the server. I've deployed the version with Lumino here: https://kinow.github.io/#/workflows/one

If you click on "Add View" on the right top corner, and choose "Tree" (Graph is slightly broken ATM), that will add a new Vue component/Lumino Widget.

The difference from the real, non-offline mode, is that this Vue.js app is accessed through a Tornado application. And this Tornado application is actually imitating JupyterHub's single-notebook app.

Our workflow service is composed of several parts, one being the UI Server, which is basically this Tornado app running behind JupyterHub. Once logged in, users should initiate a WebSocket sending GraphQL queries to the UI Server (which is replying it using your code), that will serve workflows data (it aggregates data from workflow servers).

So I am not sure if a JupyterLab extension would work for us. We have a UI that has other parts, where the Lumino widgets will be just a part of it - though one of the most important parts. Although what you are suggesting could work as well... instead of a complete Vue.js app, we could create some JupyterLab extensions instead, and perhaps we would have a similar UI.

I guess it depends on how much control we need over the UI, and how much JupyterLab can be extended. Do you know if it is easy to control authorization in JupyterLab? e.g. we will need to limit what users see, what actions they can execute in some of the widgets/components. CC @oliver-sanders (who had the idea to use Lumino and copy JupyterLab).

Good food for thought @bollwyvl, we will have a developer meetup in New Zealand in about two months. I will try to explore more this idea. Thanks!

[bollwyvl]

Very exciting stuff! Obviously no pressure on any of these fronts, just trying to be helpful, but I'm always look for ways to poke open source tools into supporting rich integration into Lab/Hub vs just "iframing it"/"shelling out"!

offline mode

If users can ever interact with data that is like what is in the cool offline demo, then a MimeRenderer might fit the bill. If the data is live, widgets can go further, and can support both lab/classic, though it's tricky: luckily there's help.

Our workflow service is composed of several parts, one being the UI Server, which is basically this Tornado app running behind JupyterHub. Once logged in, users should initiate a WebSocket sending GraphQL queries to the UI Server (which is replying it using your code), that will serve workflows data (it aggregates data from workflow servers).

Integrating with Hub is always exciting! But is really the value-add of the Jupyter ecosystem, if you have users that at some point will be consuming your data programmatically: it's no fun to build import/export mechanisms to lossy formats like csv/excel or (gasp) word if they then have to turn around and write a parser to get them into a dataframe. Giving users a (insert kernel here)-compatible client, and being able to use their credentials in-kernel is amazing, e.g. jupyterlab-dask, especially if the magic of hub takes care of all the auth business. Specifically on the GraphQL front, I've wanted to build a generalized GraphQL subscription widget, but haven't gotten around to it.

I guess it depends on how much control we need over the UI, and how much JupyterLab can be extended.

The sky is pretty much the limit: I usually suggest folk start with a MainAreaWidget (as described in the extension tutorial), then adding some sidebar panels for things that sort-of quack like a file browser (perhaps a WorkflowList) in your case. From your offline demo, i'd see these wrappers basically allowing your UI to exist in lab:

• always on:
  • CylcSidebar (pretty much what you already have there, would appear next to the file browser/command palette, or you can default it to the right-hand side)
• main area widgets, based on user activities
  • CylcDashboard (you could also put this on the Launcher)
  • CylcWorkflows (these could also be reflected in the Running tab, along with kernels and terminals, soon)
  • CylcWorkflow
    • nesting dockpanels in dockpanels is generally frowned upon, but hey, if you need it, you need it, and it works just fine... though you need to be careful about if you don't want dock tabs to be draggable out of the inner panel and out to the "main" dockpanel.

In addition to the above, you'd be able to add new things to the main menu, the status bar, command palettes, keyboard shortcuts, per-user settings, (multiple) workspace restoration/import/export, etc. In addition, external links that you want to keep as iframes can be trivially directly in the dockpanel (though navigation is somewhat impeded, as is the ability to create bookmarks, etc).

You can eventually go way beyond what is in core, and selectively disable or replace existing things... but at some point, if you disable too much stuff (basically @jupyterlab/notebook and its dependencies) then the scientist will lose the agency to do familiar, kernel-based work or effectively use installed labextensions... at which point, it's not worth the headache of integrating into the Lab build chain. But folk really love stuff like widgets, bokeh, plotly, panel, etc. and making super-custom stuff work with those is a pretty tremendous chore, as everything moves so fast.

Do you know if it is easy to control authorization in JupyterLab?

Assuming the GraphQL frontend is a hub-managed service, the client would likely already have the right tokens set from their initial login. The Lab component would likely be able to use those creds to open the connection. Another approach is to move some of the complexity to inside each user's spawned notebook server, as is done on jupyterlab-github. Then you can use the "standard" ServiceManager methods, etc. and essentially proxy through to your "real" service. This can also make testing a little easier, as it's generally easier to test python-side stuff.

As to selectively disabling/enabling features with the same build, I'd usually try to model it in the GraphQL layer: a pattern I have had to implement is a whoami query which returns a User type, which includes a simple Capability type which is used elsewhere to, for example, hide a delete button here, add an admin button there. This model can eventually be extended to support mutations, if you are an admin.

Happy to keep ruminating! Excited to see what you folks come up with!

[kinow]

A lot of things to go through and read/learn/experiment. Thanks heaps @bollwyvl ! Will wait for others to return from summer holidays too and chime in. In the meantime might have time to try some of what you suggested.

I am also involved with a possible GSoC for next year with Jenkins & ML, where we are trying to port a plugin (active choices) to JupyterLab extensions, then drive Jenkins (or other workflow runners) in the background. Might drop by Jupyter's GitHub/Gitter later to ask for help, but might use some of your suggestions there as well (discussions open in gitter too).

Oh, about that offline mode, I forgot to mention that's used only for development. So instead of needing the hub, the UI Server, Python, etc, it basically mocks the response from the GraphQL so developers can quickly see what components will look like. So I used it just so you could take a look without having to install anything :-) But MimeRenderer looks interesting! Another link to the bookmarks to read later.

Thanks again!!

[oliver-sanders]

While everyone likes to control their own window, the serendipity of having multiple features can often yield something even more powerful.

We had considered offering cut-down components of our framework as Jupyterlab extensions, the idea of orchestrating the entire UI via Jupyterlab hadn't occurred.

In terms of interaction between Jupyterlab and Cylc, users run workflows with Cylc, these workflows spit out data which they may want to poke around with interactively, some are already using the Jupyter environment for this.

It would be nice to streamline the development => workflow => analysis process, Jupyterlab provides a promising platform for this. Some benefits:

• Built-in file editor with lots of extensions e.g. keybindings for vim, sublime, etc.
• Terminal emulator for institutions without ssh access to workflow hosts.
• Potential new home for rosie go / suite management, for ideas see

My initial thinking is that implementing the Cylc UI via Jupyterlab would result in a reduction in functionality and usability from the Cylc perspective, this would outstrip functionality gains mainly for people running larger numbers of workflows. Despite similar appearances there are fundamental differences in what we are trying to achieve, but it's worth experimenting, I might be wrong and it would be a very interesting union if it could be pulled off.

Some of the hurdles we would need to get past (off the top of my head):

• Data files:

  • Especially if you have a well-constrained data file format

  • Cylc workflows are not files, they are running processes. We initiate a client to talk to those processes through which we request the data we require.
  • Cylc workflows are more like kernels than files, we can issue commands, start, stop and restart them.

• Tabbing and workflows:

  • The Cylc UI is workflow orientated whereas the Jupyterlab UI is file/kernel orientated.
  • In the Cylc UI the "workflows list" isn't just a list like a file browser, it is also a tabbing mechanism.
  • This is because Cylc effectively gives you a dashboard for the current workflow made up of multiple views all relating to that workflow.
  • So in Jupyterlab speak, when you select a workflow you are taken to a new "workspace", within which you can open views on that workflow (along with other stuff).
  • [I think] with Jupyterlab there is a 1:1 mapping between a "kernel" (e.g. Python) and a "view" (the resulting tab), we need 1:many (if going down the kernel route).
  • Just to complicate it we have some emerging ideas for multi-workflow "workspaces" too.

• Centralisation of Cylc data provision:

  • Users have multiple "views" open for each workflow.
  • These views are different representations of the same workflow so have different but overlapping data requirements.
  • To handle this we have a central data store and a subscription system. This means we only request, process and store data once making these multiple views efficient.

• File browser:

  • Workflows generate files, these files are associated with nodes in the workflow (e.g. jobs).
  • Just to make it fun these files potentially lie across multiple filesystems on multiple remote platforms. Cylc knows where these files are and how to copy/stream them.
  • In the current GTK UI users can select a job, select a file and open it in a file viewer. So we would need to be able to create new Jupyterlab "tabs" from within existing ones. Hopefully easy.

Some avenues for exploration (off the top of my head):

• Could we add "workflows" as a top-level Jupyterlab feature where selecting a workflow takes you to a new "jupyterlab" workspace?
  • This might give us the workspace management we have been looking for.
  • We could set the file-browser location to the local directory of the running suite.
• Perhaps we could roll our own Cylc flavoured Jupyterlab with a more workflow-orientated vibe.
  • Roll-your-own or wrapper solutions are a bit of a last resort.
• Perhaps it's enough to just link through to a Jupyterlab instance in another browser tab.
  • Cylc is a complex system in its own right and not well suited to being a smaller component in a larger system.
  • Jupyterlab is visually complex with much functionality not relevant to Cylc (e.g. the menu bar) which would complicate the Cylc UI.
  • Perhaps integration rather than incorporation is a better route?

[bollwyvl]

Lots to chew on there. I'll start with what I can see on my screen while replying, and work my way backwards...

Could we add "workflows" as a top-level Jupyterlab feature

Sure: in Lab parlance, this might be a WorkflowRegistry (there is presently a DocumentRegistry). Instead of being driven by the Jupyter Contents REST API, it would be driven by your GraphQL endpoint. You could drive whatever you want (even more DocumentRegistrys) off this, either as sidebars or main area tabs, and take advantage of all the MIME renderers (data grid, image viewers, etc) for those eventual files.

where selecting a workflow takes you to a new "jupyterlab" workspace?

Depends on what you mean by "workspace"... a lab workspace incorporates all of the sources/sizes of open tabs, the state of the sidebars, and a few other things. But you could also default to "single document mode" Ctrl-or-⌘+Shift+D, and let each workflow act as a workspace, controlling its own DockPanel of many tabs, and never really present the lab many-tab view. But if you did need many workflows in the future, you're ready to go... it would just take some doing to prevent views from "escaping" from a workflow DockPanel into the main (or each others!) DockPanel.

We could set the file-browser location to the local directory of the running suite.

Right: each suite could act like its own file browser. An examples of this is jupyterlab-github or jupyterlab-irods or jupyterlab-girder, all of which offer a very predictable ui (good up to some thousands of files in a given directory). If you're going much higher than that, you'll probably want to use the lumino/datagrid, which can handle billions/trillions of rows. But unifying all of them under a single (server-side) ContentsManager and (client-side) DocumentRegistry might be possible as well (if the user had their own files).

Perhaps we could roll our own Cylc flavoured Jupyterlab with a more workflow-orientated vibe.

Yes. In fact, shipping a "gold master" cyclab is a great idea: this is what Lab does, itself. Here's a pretty aggressive one we do for supporting acceptance testing/robot process automation, but hardly the worst I've done. At build time (on a dev/CI machine), you'd do all the npm/webpack stuff, put that into a whl/tar.gz with setuptools (and/or tar.bz2 with conda-build) and most of the time nothing would ever have to change on your scientists boxes/containers. If you had a particularly adventurous user, they'd be able to cyclabextension install whatever off npm, and everything should still work: Lab is pretty pedantic about not letting you install incompatible versions.

Perhaps it's enough to just link through to a Jupyterlab instance in another browser tab.

Sure! Like I said, I'm just trying to build some 🌉, and help users downstream that might want to use Jupyter + X (where X is cylc, here), and some of your needs sound suspiciously like you might be able to cash in on what lumino/lab have already put together over the last few years.

Jupyterlab is visually complex with much functionality not relevant to Cylc (e.g. the menu bar) which would complicate the Cylc UI.

Many of the bells and whistles can be stripped away, either for real by (notionally) cyclabextension disableing them (they won't be included in the build), or cosmetically, through a CSS Theme (sometimes just by changing a CSS3 variable). The latter approach, hidden behind an "expert mode" would allow the adventurous scientist to make use of these other features.

[bollwyvl]

File browser:

• Just to make it fun these files potentially lie across multiple filesystems on multiple remote platforms. Cylc knows where these files are and how to copy/stream them.

No problem: either feed them yourself to ContentsModels in JS/TS, or implement a ContentsManager on the python side. If you already have an evented source (e.g. over gql) any views will be kept in sync.

• In the current GTK UI users can select a job, select a file and open it in a file viewer. So we would need to be able to create new Jupyterlab "tabs" from within existing ones. Hopefully easy.

Yeah, this would be a "command" invocation, commands.execute('filebrowser:open-path', {path: response.path}).

[bollwyvl]

Centralisation of Cylc data provision:

• Users have multiple "views" open for each workflow.

Throughout lab, there is the model/renderer pattern: you build/wrap an evented model, and then any number of views can subscribe to it, primarily via lumino signals. You can see this in lab by right clicking on just about any tab, and selecting "New view for ..." to get a new tab with the same model. Sometimes this is taken further, where a single part of a view can be pulled out into it's own view (e.g. a notebook output can become its own tab). For the most part (at least in core, sometimes i get lazy on extensions) these view placements will persist between page reloads, and the models will all still be linked.

• These views are different representations of the same workflow so have different but overlapping data requirements.

Yup, just don't read what you don't need. The only real gotcha is if a model must be interpreted as entirely different model serializations: we run into this with JSON-as-text and JSON-as-JavaScirpt-Objects: non-trivial to keep in sync, as text can sometimes not be JSON, even if you say it is.

• To handle this we have a central data store and a subscription system. This means we only request, process and store data once making these multiple views efficient.

See WorkflowRegistry.

[bollwyvl]

Tabbing and workflows:

• The Cylc UI is workflow orientated whereas the Jupyterlab UI is file/kernel orientated.

As mentioned above, Lab is more model oriented, but folks are usually interested in models of... files and kernels :P

In the Cylc UI the "workflows list" isn't just a list like a file browser, it is also a tabbing mechanism.

This is because Cylc effectively gives you a dashboard for the current workflow made up of multiple views all relating to that workflow.

As mentioned above, you can have many views (tabs) of the same model (be it a kernel, file or... other).

So in Jupyterlab speak, when you select a workflow you are taken to a new "workspace", within which you can open views on that workflow (along with other stuff).

See thoughts about "single document mode" above. You can hide complexity until you need it...

[I think] with Jupyterlab there is a 1:1 mapping between a "kernel" (e.g. Python) and a "view" (the resulting tab), we need 1:many (if going down the kernel route).

As mentioned above, you can have many (separate) views into the same model, and one view can encompass multiple models. An example: a code editor might have a text model of a Python source file, but be tied to a model of a running kernel. That kernel model can be shared with a running notebook (which has an evented JSON model) and a running console (which doesn't really have a file, but sort of does).

Just to complicate it we have some emerging ideas for multi-workflow "workspaces" too.

This is where the lumino/lab approach starts to make sense, as we've already fought a lot of these dragons.

[bollwyvl]

Data files:

Especially if you have a well-constrained data file format
Cylc workflows are not files, they are running processes. We initiate a client to talk to those processes through which we request the data we require.

This was meant more of, "are you using something that is schema constrained"? If you're not stuffing huge, unpredictable stuff into strings, GraphQL makes is basically impossible to not be well-structured.

Cylc workflows are more like kernels than files, we can issue commands, start, stop and restart them.

Right.

[oliver-sanders]

@bollwyvl thanks for all this info! I think we've got some digging to do now, particularly with Labs models, "single document mode" and try to figure out how users might want to bridge between Cylc workflows and other Labs components.

[kinow]

Thanks indeed @bollwyvl ! I've got a huge list of things to read about, but will start by going through some of your links, and trying to play with existing widgets, and maybe create a small one too.

[bollwyvl]

While everyone likes to control their own window, the serendipity of having multiple features can often yield something even more powerful.

We had considered offering cut-down components of our framework as Jupyterlab extensions, the idea of orchestrating the entire UI via Jupyterlab hadn't occurred.

As long as you make your pieces small and unopinionated enough (e.g. don't be writing graphql in your mouseover event), it's unlikely you'd need to strip anything down, just not use part of it.

In terms of interaction between Jupyterlab and Cylc, users run workflows with Cylc, these workflows spit out data which they may want to poke around with interactively, some are already using the Jupyter environment for this.

Hooray! Give people a graph/table/model/analysis/database (Thing), and they'll keep asking you for another, bigger, faster, more interactive Thing linked to more kinds of Things. Give them a (self-)documented, evented, easy-to-install, sanely-versioned API client in their language of choce that can access the data underlying the Thing that they can run in a notebook, and work in/around/next to the Thing you gave them, and they will amaze you with what they can build.

It would be nice to streamline the development => workflow => analysis process, Jupyterlab provides a promising platform for this. Some benefits:

Built-in file editor with lots of extensions e.g. keybindings for vim, sublime, etc.

And if you don't like CodeMirror, there's plugins for other renderers, e.g. monaco (what powers vscode). Also emacs bindings!

Terminal emulator for institutions without ssh access to workflow hosts.

The same terminal that is in vscode. Also runs emacs!

Potential new home for rosie go / suite management, for ideas see

Yeah, dunno what those are!

My initial thinking is that implementing the Cylc UI via Jupyterlab would result in a reduction in functionality and usability from the Cylc perspective, this would outstrip functionality gains mainly for people running larger numbers of workflows.

I can't quite parse that, but Lab just offers a modular approach to building high-interactivity work environments, and happens to already offer a bunch of interesting kinds of work that can be done.

Despite similar appearances there are fundamental differences in what we are trying to achieve, but it's worth experimenting,

Indeed!

I might be wrong and it would be a very interesting union if it could be pulled off.

I'm all out of stuff to respond to, but yeah: I think the future of work lies in bringing a scientist the 80% solution in off-the-shelf, open source UI/auth/infra/vcs/data/pkgmgmt, and leaving them the tools they need to build their remaining 20%... in as close a proximity as possible.

[kinow]

@bollwyvl

Here's the deepest I've delved into Lumino/JupyterLab/Vue, which was picked up by the community to move a bit further (though it kinda looks like that would make a lot of Vue objects, but then, I don't really understand how vue works).

Thanks for the link. I had a look at the issue, and at the linked repo. That's interesting. It does the reverse of what I did. Instead of using a Widget with Vue, the repo linked shows an example of using Vue with JupyterLab. Which is still interesting, but not exactly what I was trying to achieve.

But if we move in the direction of using JupyterLab for Cylc, that might be a way to migrate our existing Vue app to within JupyterLab :-) So far I've been able to keep Vue components & Lumino widgets in sync, mainly through a central event bus. Then I capture the Lumino events, do whatever is required (e.g. destroy a tab/widget in a boxpanel) then emit another event for Vue (e.g. destroy a component that was rendered in a tab/widget).

[bollwyvl]

Sounds intense! If that component (e.g a vue wrapper for dockpanel) is sufficiently self-contained, that would make a great package, e.g lumino-vue-dockpanel, or at least an examples/vue/dockpanel which could be included in the docs and test infra. _Lumino for X_ is something that we haven't been focused on so much, but would no doubt be value-add to the community.

…

On Wed, Jan 8, 2020, 20:32 Bruno P. Kinoshita ***@***.***> wrote: @bollwyvl <https://github.com/bollwyvl> Here's the deepest I've delved into Lumino/JupyterLab/Vue, which was picked up by the community to move a bit further (though it kinda looks like that would make a lot of Vue objects, but then, I don't really understand how vue works). Thanks for the link. I had a look at the issue, and at the linked repo. That's interesting. It does the reverse of what I did. Instead of using a Widget with Vue, the repo linked shows an example of using Vue with JupyterLab. Which is still interesting, but not exactly what I was trying to achieve. But if we move in the direction of using JupyterLab for Cylc, that might be a way to migrate our existing Vue app to within JupyterLab :-) So far I've been able to keep Vue components & Lumino widgets in sync, mainly through a central event bus. Then I capture the Lumino events, do whatever is required (e.g. destroy a tab/widget in a boxpanel) then emit another event for Vue (e.g. destroy a component that was rendered in a tab/widget). — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#90?email_source=notifications&email_token=AAALCRHO4NE4R2JMXBSEJBLQ4Z5BBA5CNFSM4HZXXE72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEIOTHXI#issuecomment-572339165>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAALCRF2NJEI4CJXD6WUXH3Q4Z5BBANCNFSM4HZXXE7Q> .

[kinow]

Good idea @bollwyvl ! I've added a post-it to polish the code and submit as an example later. Not sure if that's the best code example, but sharing should give others an opportunity to either re-use, or provide feedback on better alternatives—so a win-win I think. Thanks!

[bollwyvl]

Yeah, gotta start somewhere! Just having a pr to point people towards and be found by search engines would help. Another approach may be to make a (working) cookiecutter/yeoman repo that encapsulates and tests the example. In Jupyter, we've been favoring cc, as it's simpler, but yo is more broadly used in the js crowd. If nothing else, it would motivate deciding if such things belong in the main repo, or in a contrib-examples repo... No doubt adding every framework under the sun would bloat the install/test cycle.

…

On Fri, Jan 10, 2020, 03:34 Bruno P. Kinoshita ***@***.***> wrote: Good idea @bollwyvl <https://github.com/bollwyvl> ! I've added a post-it to polish the code and submit as an example later. Not sure if that's the best code example, but sharing should give others an opportunity to either re-use, or provide feedback on better alternatives—so a win-win I think. Thanks! — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#90?email_source=notifications&email_token=AAALCRDONTNL5XKWZ2HK22LQ5AXJRA5CNFSM4HZXXE72YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEITDJBY#issuecomment-572929159>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAALCRCVZGKWUUGD5NFVHF3Q5AXJRANCNFSM4HZXXE7Q> .