Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fixed some CWL translator issues #628

Merged
merged 9 commits into from
Jan 19, 2025
Merged

Fixed some CWL translator issues #628

merged 9 commits into from
Jan 19, 2025

Conversation

LanderOtto
Copy link
Collaborator

@LanderOtto LanderOtto commented Dec 17, 2024

This commit fixes several CWL translator issues:

  • Fix the retrieval of the step name from the cwl-utils objects. Before this commit, the out attribute of a WorkflowStep object was always expected to be a list of strings. Instead, it should be a list of Any, following the typing hint of the cwl_utils.parser.WorkflowStep constructor. In particular, it can contain WorkflowStepOutput objects.
  • Fix the CommandOutputProcessor creation when a schema is defined. Before this commit, the CommandInputRecordSchema was not included in the RecordSchema case. Similar for EnumSchema and ArraySchema cases, respectively.
  • Fix optional inputs inside a scatter step
  • Fix the _get_path method in the CWL translator module. When the id field is explicitly defined in a CWL file, the element_id of the corresponding Python object is constructed as /path/to/file#id
  • Fix the configuration file's path stored in the DataManager. Before this commit, it was saved the path defined in the StreamFlow file, which could be a relative path
  • Fix optional inputs of type list or record. Before this commit, some workflows were failing because their input data were erroneously treated as mandatory

Copy link

codecov bot commented Dec 17, 2024

❌ 1 Tests Failed:

Tests completed Failed Passed Skipped
1989 1 1988 9
View the top 1 failed tests by shortest run time
tests/test_remotepath.py::test_download[docker]
Stack Traces | 4.05s run time
context = <streamflow.core.context.StreamFlowContext object at 0x1155a7020>
connector = <streamflow.deployment.connector.container.DockerConnector object at 0x114d95f90>
location = <streamflow.core.deployment.ExecutionLocation object at 0x114b0c200>

    @pytest.mark.asyncio
    async def test_download(context, connector, location):
        """Test remote file download."""
        urls = [
            "https://raw.githubusercontent..../streamflow/master/LICENSE",
            "https://github..../refs/tags/0.1.6.zip",
        ]
        parent_dir = StreamFlowPath(
            tempfile.gettempdir() if location.local else "/tmp",
            context=context,
            location=location,
        )
        paths = [
            parent_dir / "LICENSE",
            parent_dir / "streamflow-0.1.6.zip",
        ]
        path = None
        for i, url in enumerate(urls):
            try:
                path = await remotepath.download(context, location, url, str(parent_dir))
                assert path == paths[i]
>               assert await path.exists()
E               assert False

tests/test_remotepath.py:78: AssertionError

To view more test analytics, go to the Test Analytics Dashboard
📢 Thoughts on this report? Let us know!

@LanderOtto LanderOtto force-pushed the fix/elem-out branch 2 times, most recently from 1599d32 to 44e2c9a Compare December 18, 2024 11:17
@LanderOtto LanderOtto changed the title Fixed the retrieval of the step name in the CWL translator Fixed some CWL translator issues Dec 18, 2024
@LanderOtto LanderOtto force-pushed the fix/elem-out branch 2 times, most recently from dccecb7 to 3cb7ec9 Compare December 18, 2024 16:01
@LanderOtto LanderOtto force-pushed the fix/elem-out branch 2 times, most recently from 59eea37 to d956f9b Compare December 28, 2024 21:49
@LanderOtto LanderOtto force-pushed the fix/elem-out branch 7 times, most recently from 7840d0b to 6267b03 Compare January 4, 2025 13:45
@LanderOtto LanderOtto force-pushed the fix/elem-out branch 7 times, most recently from 5a03e0e to 6bc009b Compare January 9, 2025 18:55
 - the retrieval of the step name. Before this commit, it was expected that the `out` attribute of `WorkflowStep` was always a list of `string`. Instead, it is a list of `Any`, following the typing hint of the `cwl_utils.parser.WorkflowStep` constructor. In particular, it can have `WorkflowStepOutput` object elements.
 - the creation of the `CommandOutputProcessor` when a schema is defined. Before this commit, the `CommandInputRecordSchema` was not included in the `RecordSchema` case. Similar for `EnumSchema` and `ArraySchema` cases, respectively.
…flow, the `element_id` is: `/path/to/file#id`
… this commit, it was saved the path defined in the `StreamFlow file`, which can be a relative path
Copy link
Member

@GlassOfWhiskey GlassOfWhiskey left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@GlassOfWhiskey GlassOfWhiskey merged commit 5cad7bf into master Jan 19, 2025
32 checks passed
@GlassOfWhiskey GlassOfWhiskey deleted the fix/elem-out branch January 19, 2025 14:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants