Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

graph view and central data store #1108

Merged
merged 86 commits into from
Nov 28, 2022
Merged

Commits on Nov 21, 2022

  1. graph view experiment 1: get data

    * We need to use the "tree data" whether we want a tree or not because
      this is the only data window which associates nodes with their parent
      workflows directly.
    * Allow edge deltas through.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    ac271aa View commit details
    Browse the repository at this point in the history
  2. graph view experiment 2: construct a global tree

    * Construct a data tree in the workflow store to be used as the cenrtral
      data store.
    * This stores everything Cylc:
      * UID objects `~user/a/b/workflow//cycle/task/job`.
      * $namespaces
      * $edges
    * This fulfils common requirements e.g. sorting.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    176cf14 View commit details
    Browse the repository at this point in the history
  3. graph: implement the graph view

    * Add a GraphNode component.
    * Render nodes.
    * Load GraphViz (WASM).
    * Run GraphViz layout..
    * Translate nodes to match layout coordinates.
    * Render edges.
    
    Make some adjustments to the central data store:
    
    * flat index (cylcTree.$index)
    * tree (cylcTree.children)
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    d3b2ca2 View commit details
    Browse the repository at this point in the history
  4. store: refactor the store to eliminate cicrular references

    * VueX doesn't like circular references (e.g. node.parent).
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    18b1c18 View commit details
    Browse the repository at this point in the history
  5. Configuration menu
    Copy the full SHA
    9edf7fe View commit details
    Browse the repository at this point in the history
  6. graph: auto-refresh

    * Auto refresh the graph at a hardcoded rate but only when the data has
      changed.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    fbcd0e3 View commit details
    Browse the repository at this point in the history
  7. graph node: make tasks clickable

    * Jobs are not clickable yet, however, we don't have any job mutations
      yet and the graph view might want to provide a custom job overview
      here so omitting this for now.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    5d5e0d7 View commit details
    Browse the repository at this point in the history
  8. Configuration menu
    Copy the full SHA
    8377270 View commit details
    Browse the repository at this point in the history
  9. graph: add transpose option

    * Allow the graph to be transposed (i.e. switch from top-bottom to
      left-right edge orientation).
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    5c4fafd View commit details
    Browse the repository at this point in the history
  10. graph: add centre graph control

    * Add a button to centre the graph.
    * Increase the permitted zoom range.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    d8fdcfa View commit details
    Browse the repository at this point in the history
  11. task: re-implement task icon

    * The old task icon modifiers didn't render correctly when loaded into a
      larger document due to some relative transformation problem.
    * Simplified back to a regular template component.
    * Tweaked the icons a little.
    * Still needs work, the modifiers are too small!
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    c940d82 View commit details
    Browse the repository at this point in the history
  12. Configuration menu
    Copy the full SHA
    817cf92 View commit details
    Browse the repository at this point in the history
  13. workflow service: run global callback by default

    * Enable the global callback by default irrespective of which views are
      open.
    * Disable local callbacks for the central data store.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    b664721 View commit details
    Browse the repository at this point in the history
  14. task: switch to new task component

    * Switch views to use the re-implemented task component.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    92f648c View commit details
    Browse the repository at this point in the history
  15. store: add family support to the tree

    * Add a `familyTree` node under each cycle point when families are
      recieved by the store.
    * Populate this tree using `childTasks` and `ancestors` from the schema.
    * Use the flat ID for families as it is not possible to reconstruct the
      hierarchical ID for updates as the ancestors list will not have
      change so will not be present in the delta.
    * Test registration of children.
    * Fix cycle/family insertion order issue
    * Remove tasks from family tree when removed from childTasks
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    6ee0155 View commit details
    Browse the repository at this point in the history
  16. workflow service: fix query merging

    * Make queries mergable
      * In order to merge queries properly their definition names must
        match, otherwise the fragments can become isolated from the
        definitions which use them.
      * Fixed by standardising the naming of defintions so that they can
        merge cleanly.
    * Fix re-issue of queries after merging
      * Closes cylc#1110
      * The query object has some derived properties which are not updated
        when we modify it in place.
      * The ApolloClient appears to by using these properties which causes it
        to see the query as it was *before* merging rather than after.
      * To fix we re-create the query object after merge in order to ensure
        its derived properties are set correctly.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    282dc1d View commit details
    Browse the repository at this point in the history
  17. tree: fix filtering & job-details nodes

    * Remove root nodes from tree.
    * Fix job-details nodes.
    * Fix filtering.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    7d15851 View commit details
    Browse the repository at this point in the history
  18. Configuration menu
    Copy the full SHA
    769470d View commit details
    Browse the repository at this point in the history
  19. Configuration menu
    Copy the full SHA
    e8bb6ab View commit details
    Browse the repository at this point in the history
  20. Configuration menu
    Copy the full SHA
    712c862 View commit details
    Browse the repository at this point in the history
  21. Configuration menu
    Copy the full SHA
    4eb2e4f View commit details
    Browse the repository at this point in the history
  22. Configuration menu
    Copy the full SHA
    36f399b View commit details
    Browse the repository at this point in the history
  23. Configuration menu
    Copy the full SHA
    372eea5 View commit details
    Browse the repository at this point in the history
  24. gscan: convert to run off of the central data store

    * Remove local callbacks.
    * Convert subscriptions to match the pattern of the others for merging
      purposes.
    * Search and filter are broken for the moment.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    801ac43 View commit details
    Browse the repository at this point in the history
  25. Configuration menu
    Copy the full SHA
    3711326 View commit details
    Browse the repository at this point in the history
  26. Configuration menu
    Copy the full SHA
    8b0a955 View commit details
    Browse the repository at this point in the history
  27. Configuration menu
    Copy the full SHA
    6f85bda View commit details
    Browse the repository at this point in the history
  28. gscan: fix sorting & filtering

    * Adapt sort and filter mechanism to work with the central data store.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    f3b1d29 View commit details
    Browse the repository at this point in the history
  29. Configuration menu
    Copy the full SHA
    efa99c7 View commit details
    Browse the repository at this point in the history
  30. aotf: fix node data

    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    b7be28f View commit details
    Browse the repository at this point in the history
  31. Configuration menu
    Copy the full SHA
    07a0acc View commit details
    Browse the repository at this point in the history
  32. store: housekeep the store for workflow subscriptions

    * When we used local callbacks we deleted the data obtained by the
      callback when the subscription was closed.
    * Now we use a central data store we cannot rely on this mechanism so
      need to manually wipe the data the subscription requested.
    * This is a crude initial implementation which assumes that no other
      subscriptions active at the time were requesting
      cycles/families/namespaces etc from the workflow(s) that were being
      requested. At the time of writing that is an accurate assumption.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    6645669 View commit details
    Browse the repository at this point in the history
  33. task: delete unused styles and re-calibrate progress

    * Delete styles/cylc/_task (these styles now reside in the Vue
      component).
    * Fix the size of the task "hub" (the middle bit of the running icon).
    * Re-calibrate the progress indicator (which needs to be fiddled
      whenever we change its size).
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    5a054de View commit details
    Browse the repository at this point in the history
  34. graph: prevent Nanageddon

    * Nanageddon i.e. a bug where the transform matrix of the pan-zoom
      layer (the SVG group that SVGPanZoom inserts for us) gets NaN
      (not a number) values causing an error (and broken pan/zoom
      functionality).
    * Resolve this by registering the pan/zoom on the `$.nextTick` after mount
      rather than at mount time as the SVG was only initialised ~90% of the time.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    7c23fe2 View commit details
    Browse the repository at this point in the history
  35. Configuration menu
    Copy the full SHA
    ad9a6dc View commit details
    Browse the repository at this point in the history
  36. Configuration menu
    Copy the full SHA
    b730433 View commit details
    Browse the repository at this point in the history
  37. Configuration menu
    Copy the full SHA
    e5ff894 View commit details
    Browse the repository at this point in the history
  38. task: fix progress icon in webkit/safari

    * The progress icon was rotated but the point around which it was
      rotated was not specified. Safari/Webkit picked the centre of the
      object, Opera/Chromium picked the top-left corner. Hardcoded this
      into the rotation to avoid issues.
    * Closes cylc#1078
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    c61be9c View commit details
    Browse the repository at this point in the history
  39. Configuration menu
    Copy the full SHA
    cf8a682 View commit details
    Browse the repository at this point in the history
  40. Configuration menu
    Copy the full SHA
    2d46207 View commit details
    Browse the repository at this point in the history
  41. Configuration menu
    Copy the full SHA
    51e41d4 View commit details
    Browse the repository at this point in the history
  42. graphql: only stripNull for updated Deltas

    * Setting stripNull for Added deltas could cause null-like values
      to be absent in the node data resulting in undefined errors.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    3c88f3d View commit details
    Browse the repository at this point in the history
  43. tree: fix missing tasks on root family

    * Caused by subscription omitting updates for the root family
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    fb3bf1d View commit details
    Browse the repository at this point in the history
  44. graphql: remove unnecessary sorting

    * The central data store performs sorting so we don't need to bother
      the UIS to do this for us.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    69b8bb5 View commit details
    Browse the repository at this point in the history
  45. table: convert to run on central data store

    * Also add workflow icon and status columns to close cylc#1116
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    b5ba5da View commit details
    Browse the repository at this point in the history
  46. Configuration menu
    Copy the full SHA
    e1a9e06 View commit details
    Browse the repository at this point in the history
  47. graph: add toolbar and manual refresh

    * Convert clunky buttons into slick toolbar.
    * Allow the auto-refresh timer to be disabled and add a manual refresh
      option.
    * Set edge colour to match node colour.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    e114e95 View commit details
    Browse the repository at this point in the history
  48. Configuration menu
    Copy the full SHA
    4eafc93 View commit details
    Browse the repository at this point in the history
  49. store: remove old stores, callbacks and fix tests

    * Remove the old workflow store code which powered the local callbacks.
    * Remove the local callbacks which have been replaced by the central
      data store.
      NOTE: Local callbacks are still possible, however, we don't need to provide
      a pattern for this as we currently have no uses.
    * Move a function for easier testing.
    * Fix tests.
    * tests/unit/task: remove in favour of component tests
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    7391886 View commit details
    Browse the repository at this point in the history
  50. Configuration menu
    Copy the full SHA
    fddd54c View commit details
    Browse the repository at this point in the history
  51. Configuration menu
    Copy the full SHA
    6724223 View commit details
    Browse the repository at this point in the history
  52. tree: fix custom outputs and support messages

    * Custom outputs were broken with the removal of the local callbacks.
    * Before that custom outputs weren't being correctly associated with the
      jobs which created them.
    * Tied outputs into messages in order to filter out the job which
      created the outputs. Closes cylc#1129
    * Added messages in with outputs (after the filtering the default
      messages out). Closes cylc#595
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    346beb6 View commit details
    Browse the repository at this point in the history
  53. Configuration menu
    Copy the full SHA
    391ad39 View commit details
    Browse the repository at this point in the history
  54. Configuration menu
    Copy the full SHA
    88b05ef View commit details
    Browse the repository at this point in the history
  55. Configuration menu
    Copy the full SHA
    791efb1 View commit details
    Browse the repository at this point in the history
  56. Configuration menu
    Copy the full SHA
    a7a655c View commit details
    Browse the repository at this point in the history
  57. Configuration menu
    Copy the full SHA
    932fd77 View commit details
    Browse the repository at this point in the history
  58. aotf: fix node description

    * Fix the description which appears at the top of the mutation menu.
    * This currently shows the node type and status.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    00a83d6 View commit details
    Browse the repository at this point in the history
  59. graphql: remove unused fields from queries

    * Remove a coupld of fields which were never used.
    * Remove a few fields which are no longer required since the data
      store changes.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    16ec361 View commit details
    Browse the repository at this point in the history
  60. tree: sort families before tasks

    * Match the previous store implementation which sorted families before
      tasks in the tree.
    * This is similar to directory/file sort rules and makes the tree view
      easier to read.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    94e2a55 View commit details
    Browse the repository at this point in the history
  61. tree: fix erroneous filter issue

    * Fix issue where nodes could sometimes be filtered "in" when they
      shouldn't have been as a result of the filter logic following the task
      tree rather than the family tree.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    c784f41 View commit details
    Browse the repository at this point in the history
  62. Configuration menu
    Copy the full SHA
    8a73323 View commit details
    Browse the repository at this point in the history
  63. Configuration menu
    Copy the full SHA
    ebe38c8 View commit details
    Browse the repository at this point in the history
  64. mutation: remove task icon from mutation form

    * The task icon was a temporary development / debugging tool no longer
      required
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    b574a11 View commit details
    Browse the repository at this point in the history
  65. task: make the queued modifier more prominent & add expired state

    * Increase the default modifier size.
    * Remove the outline for the queued modifier and increase the size of
      the horizontal bars.
    * Fix the modifier translation to better work for a wider range of
      modifier sizes.
    * Test the task icon with different modifier sizes.
    * Add icon for the expired task state.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    0e493af View commit details
    Browse the repository at this point in the history
  66. store: tidy

    * Remove unused getter.
    * Move getTree getter (test utility) into the unit tests.
    * Comments ++;
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    1cf7973 View commit details
    Browse the repository at this point in the history
  67. misc: tidy

    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    2cc7e53 View commit details
    Browse the repository at this point in the history
  68. store: centralise tree walking functions

    * Provide an efficnent tree walking function as a vuex getter.
    * Use this to provide node lists for each of the Cylc "views".
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    3ad9edc View commit details
    Browse the repository at this point in the history
  69. graph: Apply suggestions from code review

    Co-authored-by: Ronnie Dutta <[email protected]>
    
    * package: move wasm-loader from dependencies into devDependencies.
    * graph: change edge transformation for transposed graphs
      * We use static "fudge factors" to align the node and edge layers.
      * The graph now uses separate factors for transposed and non-transposed
        graphs for cleaner layout.
    * store: remove unnecessary familyPath variable.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    08cbec6 View commit details
    Browse the repository at this point in the history
  70. guide: fix task icons

    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    b43050e View commit details
    Browse the repository at this point in the history
  71. graph node: re-jig item sizes

    * Reduce task size.
    * Increase cycle size.
    * Increase size of first job.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    c2a134f View commit details
    Browse the repository at this point in the history
  72. store: fix subscription merging issue

    * In some situations subscriptions were not being merged causing the new
      view to be missing data.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    3927ea0 View commit details
    Browse the repository at this point in the history
  73. store: fix edge removal

    * Fix issue where the $edges & $namespaces indicies were not being
      cleared when switching workflows.
    * This caused duplicate edges in the graph view.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    3e029be View commit details
    Browse the repository at this point in the history
  74. graph: prevent NaNageddon

    * Fixes an issue where we tried to mount the svgPanZoom component before
      the dimensions of the SVG document could be measured.
    * This resulted in the console error:
      `Error: <g> attribute transform: Expected number, "matrix(NaN,0,0,NaN,NaN,…"`
    * The solution is to check the dimensions of the first item in the
      document, once it exists and has non-zero dimensions the SVG is ready.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    196a45e View commit details
    Browse the repository at this point in the history
  75. graph: fix scrollbars

    * A vertial scrollbar could appear in some browsers as a result of the
      view toolbar pushing the SVG outside of the tab.
    * This makes the view toolbar a floating component to allow more space
      for the graph and prevent this issue.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    017873d View commit details
    Browse the repository at this point in the history
  76. store: fix merging of updates

    * Updates are "merged" into our reactivate data blobs.
    * This merging used to use a nested algorithm which attempted to merge
      updates in arrays.
    * This isn't what we want as our updates are atomic, e.g. if an update
      contains `{childTasks: [{'id': 'a'}, {'id': 'b'}]}`, that means please
      update childTasks to match this data NOT please change the IDs of the
      first two childTasks on the node to match these values.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    f95e3d5 View commit details
    Browse the repository at this point in the history
  77. graph: improve stability with large graphs

    * Prevent parallel layout.
      This was unlikely but possible if layout time was greater than
      refresh time (~2s).
    * Wait for nodes to be rendered before attempting to layout.
      Sometimes one or more nodes hadn't yet rendered when layout was
      attempted causing an error reading the node dimensions.
      There's no "wait for everything to render" function so we must place a
      check in a loop.
    * Wait for all edges to be rendered before centering the graph.
      Auto-centering the graph on first load sometimes failed if the graph
      took a while to render.
      Put another condition in a loop to delay centering until the SVG is
      ready.
    * Catch layout errors.
      If anything does go wrong during layout, the graph will now re-try
      next time rather than crashing.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    e26d672 View commit details
    Browse the repository at this point in the history
  78. graph: package wasm file

    * The `graphvizlib.wasm` file provided by `@hpcc-js/wasm` needs to be
      included in `dist/js`.
    * It is then loaded async by a JS script imported by the graph view.
    * At the time of writing we are using Webpack 4 which doesn't seem to
      play nicely with this approach so the best solution I can come up with
      is to symlink the wasm file into the `public/js` directory.
    * This symlink will be broken until you `yarn install`.
    * This hack might not be needed when we upgrade to Webpack 5, however,
      this might just be due to the way HPCC packages graphviz.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    ccc69dc View commit details
    Browse the repository at this point in the history
  79. graph: add offline data and first e2e test

    * Add edges into the offline data to allow the graph view to be tested
      offline.
    * Add a basic cypress test which checks the layout gets completed and the nodes
      and edges get rendered.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    ac3bf77 View commit details
    Browse the repository at this point in the history
  80. graph: allow display of edgeless graphs

    * Allow the graph view to display graphs which don't contain any edges.
    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    ac2e835 View commit details
    Browse the repository at this point in the history
  81. TreeItem: small fix

    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    62ab67a View commit details
    Browse the repository at this point in the history
  82. changelog: graph view

    oliver-sanders committed Nov 21, 2022
    Configuration menu
    Copy the full SHA
    51d67cf View commit details
    Browse the repository at this point in the history

Commits on Nov 23, 2022

  1. Configuration menu
    Copy the full SHA
    599e2cd View commit details
    Browse the repository at this point in the history
  2. Configuration menu
    Copy the full SHA
    1a0f1b8 View commit details
    Browse the repository at this point in the history
  3. Update src/components/cylc/GraphNode.vue

    Co-authored-by: Ronnie Dutta <[email protected]>
    oliver-sanders and MetRonnie authored Nov 23, 2022
    Configuration menu
    Copy the full SHA
    fbe64e6 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2022

  1. Configuration menu
    Copy the full SHA
    3d28f92 View commit details
    Browse the repository at this point in the history