Strategy to update workflows needed #3

bgruening · 2018-07-08T10:33:27Z

External workflows, e.g. the once from the training material, should be updated regularly. Or even better should be a linked against the training material. Maybe we can pull them down before running the tests as we do with data sets currently.

cat-bro · 2020-06-04T06:54:11Z

Hi @bgruening, continuing discussion from galaxyproject/training-material#1896

A way forward could be to rewrite the update_workflows function in update-training.sh so that it updates each workflow within training/* (possibly with wget https://github.com/galaxyproject/training-material/blob/master/<topic_name>/tutorials/<tutorial_name>/workflows/<workflow_name> for each individual file rather than cloning training-material). It would copy the workflows only, not the tests. This could be run by jenkins prior to running the workflow tests, with any changes committed into workflow-testing.

Prior to being able to implement this there would need to be a clean-up of the paths (in workflow-testing) so that any training/<topic_name>/<tutorial_name>/<workflow_name> is consistent with a corresponding topics/<topic_name>/tutorials/<tutorial_name>/workflows/<workflow_name> path in training-material. (for example training/statistics/machine_learning/classification/linear_SVC_classification.ga would become training/statistics/classification_regression/classification_LSVC.ga)

I could make a start on this.

bgruening · 2020-06-04T07:29:01Z

I like that, this would also save the long cloning step.

Thanks @cat-bro for working on that.

hexylena · 2020-06-04T08:26:57Z

for what it's worth: My future plan was to just merge this back into the training material. I only created the separate repo to let things move a bit faster while we were getting testing worked out.

Things seem stable enough now (and with a second site!) that we should make this a standard part of the GTN. I'm happy to help with this too, for what it's worth. It would not be too difficult to merge back + update the tests.

bgruening · 2020-06-04T08:45:00Z

We would still need this repo for other - non training workflows I think.

hexylena · 2020-06-04T08:48:02Z

Ahh I hadn't considered that use case. Would it make sense to keep both repos then? I would really love to see the training tests as close to the training materials as possible (and kept to high standards), and then this repository could give more freedom, whatever test you want to write, or so?

cat-bro · 2020-07-23T02:52:44Z

What if everything under training were kept in galaxyproject/training-material (single source of truth) and in this repo there was just a yaml listing the .ga files and test files for any workflows that had tests? Jenkins could get these with wget rather than cloning the repo. The thing I suggested in June about syncing them over would work, but even then the double-ups are problematic. Non-GTN workflows would still live here.

hexylena · 2020-07-23T04:10:24Z

That could work @cat-bro, sounds good.

bgruening · 2020-07-23T06:54:35Z

I assume then developing the workflow tests is not that intuitive. As you don't have the workflows next to the tests.
What about adding the test files to the workflows in GTN?

hexylena · 2020-07-23T07:08:02Z

What about adding the test files to the workflows in GTN?

Oh, I assumed that's what @cat-bro meant, both WFs + tests live in the GTN.

cat-bro · 2020-07-23T07:09:00Z

yes, what @hexylena said

bgruening · 2020-07-23T07:21:08Z

Cool, Having everything in GTN sounds great!

bgruening · 2020-08-03T07:57:45Z

So we need some logic to traverse GTN, find workflow-tests and run them.

What do you think about adding a Makefile that clones the GTN (only latest HEAD) and traverses the tree and somehow returns the path to the workflow-test files.

We could also keep the list of workflows (outside of GTN) in this Makefile and finally trigger a run. This would also simplify the Jenkins setup.

The current list from Jenkins is:

training/transcriptomics/rna-seq-viz-with-volcanoplot/rna-seq-viz-with-volcanoplot.ga
training/transcriptomics/rna-seq-viz-with-heatmap2/rna-seq-viz-with-heatmap2.ga
raceid3/raceid3_workflow.ga
example1/wf3-shed-tools.ga
example2/wf4-shed-tools.ga
GraphClust2/GC-lite.ga
training/transcriptomics/small_ncrna_clustering/blockclust_workflow.ga
sklearn/adaboost/adaboost.ga
sklearn/ard/ard.ga
training/variant-analysis/microbial-variants/microbial_variant_calling.ga
training/variant-analysis/dip/diploid.ga
training/variant-analysis/mapping-by-sequencing/mapping_by_sequencing.ga
training/proteomics/protein-id-sg-ps/protein-id-sg-ps.ga
training/proteomics/protein_quant_sil/protein_quant_sil.ga
training/proteomics/metaproteomics/metaproteomics.ga
training/sequence-analysis/ref-based-rad-seq/rad_seq_ref_based.ga
training/sequence-analysis/quality-control/quality_control.ga
training/sequence-analysis/mapping/mapping.ga
training/chip-seq/formation_of_super-structures_on_xi/formation_of_super_structures_on_xi.ga
training/epigenetics/methylation-seq/methylation-seq.ga
training/assembly/general-introduction/assembly-general-introduction.ga
training/assembly/unicycler-assembly/unicycler.ga
training/metagenomics/general-tutorial/amplicon.ga
training/transcriptomics/ref-based/ref_based.ga
training/transcriptomics/small_ncrna_clustering/blockclust_workflow.ga
training/statistics/classification_regression/regression_GradientBoosting.ga
training/statistics/classification_regression/classification_LSVC.ga
training/proteomics/F1000_Metaproteomics_QueryTabular/F1000_Metaproteomics_QueryTabular.ga
training/proteomics/F1000_Proteogenomics_QueryTabular/F1000_Proteogenomics_QueryTabular.ga
training/computational-chemistry/bio3danalysis/MD_Analysis_using_Bio3D.ga
training/computational-chemistry/bio3danalysis/gromacs.ga
training/metabolomics/F1000_Metabolomics_Query_Tabular_Mass_Adjustment.ga
training/statistics/machinelearning/machine_learning.ga

cat-bro · 2020-08-03T09:06:32Z

Yes, that sounds good.

Something like

git clone https://github.com/galaxyproject/training-material.git
for f in $(find 'training-material' -path '*-test.yml' ); do echo "${f/-test.yml/.ga}" >> list_of_workflows.txt; done
<script that runs the tests>
rm -rf training-material

Some of the tutorials listed above will not have equivalents in GTN (sklearn I think?) so there would need to be a separate list of these

hexylena · 2020-08-03T09:16:57Z

unnecessary, small optimisation:

find 'training-material' -path '*-test.yml' | sed 's/-test.yml/.ga' > list_of_workflows.txt

otherwise sounds good1

bgruening · 2020-08-03T09:23:07Z

Some of the tutorials listed above will not have equivalents in GTN (sklearn I think?) so there would need to be a separate list of these

Yes, I think there is a need to have still workflows here. E.g. user-worfklows that will not make it into GTN anytime soon. Or a really strange workflow that just utilize a buch of functionallity etc ...

bgruening · 2020-09-14T18:53:10Z

@cat-bro do we have a plan to move forward? Are you planning to work on this? Can we help somehow?
Next release is coming and we could test those workflows against it.

cat-bro · 2020-09-14T19:57:18Z

Hi @bgruening, this is on my trello but I keep getting distracted by other things.

I would really love to be able to test all of the workflows that run on Galaxy Australia. It would be a great comfort to people who run training sessions. A short term goal would be to have tests for as many of the workflows in GTN as possible.

Is it OK for me to update some of the GTN tutorials to use more up-to-date versions of the tools than they currently do? I think there are some that still use tool versions that break on python3, but there are also some that use tool versions from 2017/2018 that I don't believe anybody actually teaching or learning from the tutorial will be using: in generally they will be using the latest version available.

A long term goal in my mind would be to have a strategy of workflow testing that accounts for the fact that most of the time, tool versions used in tutorials will be the latest available versions and not necessarily the versions listed on the workflows. I think that the tests as they currently are are useful, but that sometimes they are testing flows that would be unlikely in a teaching setting. For example if Galaxy has grappa version 1.2a and grappa version 3.4b, the workflow might contain grappa 1.2a but in a real world teaching scenario students will be using 3.4b, if this makes any sense.

bgruening · 2020-09-14T20:02:30Z

Is it OK for me to update some of the GTN tutorials to use more up-to-date versions of the tools than they currently do?

Of course!

A long term goal in my mind would be to have a strategy of workflow testing that accounts for the fact that most of the time, tool versions used in tutorials will be the latest available versions and not necessarily the versions listed on the workflows. I think that the tests as they currently are are useful, but that sometimes they are testing flows that would be unlikely in a teaching setting. For example if Galaxy has grappa version 1.2a and grappa version 3.4b, the workflow might contain grappa 1.2a but in a real world teaching scenario students will be using 3.4b, if this makes any sense.

In theory, our training materials should all be updated to latest versions. We are just not able to do this currently. I guess one step in this direction is to inform the training author ... so have something automatic that bumps the workflow to the latest versions, runs the tests and informs the author of the training to check the (update) PR.

cat-bro · 2020-09-14T20:23:53Z

cat-bro · 2020-09-14T20:26:21Z

^ boxes are ticked if the tutorial has an equivalent working test in GTN. I think that the sequence-analysis tutorials do too but I'd need to run the tests again to be sure.

As a first step it would be good to have equivalent tests in GTN for all of these, so that no value is lost moving to running the GTN tests instead of the tests in this repo.

bgruening · 2020-10-20T08:52:34Z

@cat-bro; @malloryfreeberg has started to add several new tests to GTN in the last weeks. This is all super exciting! Thanks all!

malloryfreeberg · 2020-10-20T08:56:54Z

@cat-bro; @malloryfreeberg has started to add several new tests to GTN in the last weeks. This is all super exciting! Thanks all!

@cat-bro the list of workflows and workflow tests I've been adding to the GTN material are all referenced in this ticket: galaxyproject/training-material#1459

cat-bro · 2020-10-20T15:53:37Z

That's great! There are now more tests for training material in GTN than in this repo. There are still a few in this repo that do not have equivalent tests in GTN but they may need a bit more work and can be added to GTN over time. It's probably time to abandon the training tests in this repo and run the tests that are in GTN.

A script to run the tests could be something like

# get list of local workflows with tests (ignoring training folder)
find . \( -name '*-test.yml' ! -path './training*' \) | sed 's/^\.\///g' | sed 's/-test.yml/.ga/g' > $workflow_list

# clone training-material repo
git clone https://github.com/galaxyproject/training-material.git

# get list of training-material workflows with tests
find 'training-material' -path '*-test.yml' | sed 's/-test.yml/.ga/g' >> $workflow_list

mkdir results

cat $workflow_list | while read workflow_path; do
   ./run_galaxy_workflow_tests.sh $workflow_path
   cp tool_test_output.json results/$(sed 's/\//_/g' <<< $workflow_path).tool_test_output.json
done

# planemo merge_test_reports ...... ## merge all of the files in reports and produce on html doc

The above is untested/unfinished.

Not sure if it's best to be cloning training-material each time or to have a clone somewhere on the Jenkins server that can have git pull run on it each time.

I really like the idea of having a merged test report that could be available on Jenkins.

malloryfreeberg · 2020-10-20T18:48:16Z

@cat-bro I like your ideas, and am fully supportive of harmonising the tests and finding ways to keep them up-to-date. Let me know what would be helpful for you to support this. I'm happy to continue going through all the GTN materials and making sure all the tutorials have both a workflow and a workflow test.

bgruening · 2020-10-20T22:36:13Z

Not sure if it's best to be cloning training-material each time or to have a clone somewhere on the Jenkins server that can have git pull run on it each time.

Cloning is ok. We can use git clone --depth 1 to speed it up. @cat-bro can we have a Makefile target that runs your script and the cloneing? This would make testing locally easy but also the Jenkins integration.

This is what we do in the tools-land to merge the planemo Json reports: https://github.com/galaxyproject/tools-iuc/blob/master/.github/workflows/pr.yaml#L315

hexylena mentioned this issue Nov 7, 2018

Add test for MiModD GTN tutorial workflow #16

Merged

hexylena changed the title ~~stradegy to update workflows needed~~ Strategy to update workflows needed Jul 23, 2020

cat-bro mentioned this issue Oct 3, 2020

Some workflow test updates galaxyproject/training-material#2045

Merged

cat-bro mentioned this issue Oct 20, 2020

transcriptomics workflow tests from workflow-testing galaxyproject/training-material#2049

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strategy to update workflows needed #3

Strategy to update workflows needed #3

bgruening commented Jul 8, 2018

cat-bro commented Jun 4, 2020

bgruening commented Jun 4, 2020

hexylena commented Jun 4, 2020

bgruening commented Jun 4, 2020

hexylena commented Jun 4, 2020

cat-bro commented Jul 23, 2020

hexylena commented Jul 23, 2020

bgruening commented Jul 23, 2020

hexylena commented Jul 23, 2020

cat-bro commented Jul 23, 2020

bgruening commented Jul 23, 2020

bgruening commented Aug 3, 2020

cat-bro commented Aug 3, 2020

hexylena commented Aug 3, 2020 •

edited

Loading

bgruening commented Aug 3, 2020

bgruening commented Sep 14, 2020

cat-bro commented Sep 14, 2020

bgruening commented Sep 14, 2020

cat-bro commented Sep 14, 2020 •

edited

Loading

cat-bro commented Sep 14, 2020

bgruening commented Oct 20, 2020

malloryfreeberg commented Oct 20, 2020

cat-bro commented Oct 20, 2020

malloryfreeberg commented Oct 20, 2020

bgruening commented Oct 20, 2020

Strategy to update workflows needed #3

Strategy to update workflows needed #3

Comments

bgruening commented Jul 8, 2018

cat-bro commented Jun 4, 2020

bgruening commented Jun 4, 2020

hexylena commented Jun 4, 2020

bgruening commented Jun 4, 2020

hexylena commented Jun 4, 2020

cat-bro commented Jul 23, 2020

hexylena commented Jul 23, 2020

bgruening commented Jul 23, 2020

hexylena commented Jul 23, 2020

cat-bro commented Jul 23, 2020

bgruening commented Jul 23, 2020

bgruening commented Aug 3, 2020

cat-bro commented Aug 3, 2020

hexylena commented Aug 3, 2020 • edited Loading

bgruening commented Aug 3, 2020

bgruening commented Sep 14, 2020

cat-bro commented Sep 14, 2020

bgruening commented Sep 14, 2020

cat-bro commented Sep 14, 2020 • edited Loading

cat-bro commented Sep 14, 2020

bgruening commented Oct 20, 2020

malloryfreeberg commented Oct 20, 2020

cat-bro commented Oct 20, 2020

malloryfreeberg commented Oct 20, 2020

bgruening commented Oct 20, 2020

hexylena commented Aug 3, 2020 •

edited

Loading

cat-bro commented Sep 14, 2020 •

edited

Loading