v1.13.5

What's Changed

• Fix race conditions in the Authentication client by @rdeaton-freenome in #2635
• Update uv to 0.2.37 in default image builder by @thomasjpfan in #2687
• Input through file and pipe by @mao3267 in #2552
• Improve error message for get signed url failure by @pingsutw in #2679
• Add pythonpath "." before loading modules by @arbaobao in #2673
• Better error message for FailureNodeInputMismatch error by @pingsutw in #2693
• Remove prerelease flag with uv in image builder by @thomasjpfan in #2697
• Fix: Catch unsupported node types in map node array task by @fg91 in #2699
• Deprecation notice for pod plugin by @wild-endeavor in #2698
• Fix integration tests by @eapolinario in #2701
• Add pip_extra_index_url as a supported parameter in default builder by @thomasjpfan in #2704
• remove upper bound of plugin dependencies for flytekit-polars by @Mecoli1219 in #2514
• fix: cover 500 errors for new Harbor repos by @dylanspag-lmco in #2696
• Fix local test test_get_config_file fail by @Mecoli1219 in #2705
• Fix local test test_get_remote fail by @Mecoli1219 in #2706
• Fix local test test_saving_remote fail by @Mecoli1219 in #2707
• Default execution name should be generated in flyteadmin by @wayner0628 in #2678
• Support default values in typing.List[dataclass] and typing.Dict[str, dataclass] by @mao3267 in #2603
• Use logger.debug when numpy is not found by @thomasjpfan in #2712
• refactor(core): Update with_overrides signatures and type hints by @pingsutw in #2323
• Make task loading a part of dispatch execute by @thomasjpfan in #2711
• Adds neptune plugin for experiment tracking by @thomasjpfan in #2686
• Add Echo task by @pingsutw in #2654
• [Remote] check subworkflows for launch plan nodes by @wild-endeavor in #2714
• Improve jupyter repr and __repr__ for Flyte models by @thomasjpfan in #2647
• Improve error message for entity not found in flytekit cli by @pingsutw in #2713
• Polars lazyframe by @cosmicBboy in #2695
• unify the image spec hash function by @pingsutw in #2593
• Provide upper bound to great expectations by @thomasjpfan in #2717

New Contributors

• @mao3267 made their first contribution in #2552
• @dylanspag-lmco made their first contribution in #2696

Full Changelog: v1.13.4...v1.13.5

v1.13.4

What's Changed

• Update env variables and runtime name for default image builder by @thomasjpfan in #2670
• Copy user files that were imported by workflow in pyflyte run by @thomasjpfan in #2663
• [fix] Validate workflow input name is lowercase by @ddl-rliu in #2656
• Do not use micromamba lockfiles by @thomasjpfan in #2672
• Update uv and remove pip workaround in default image builder by @thomasjpfan in #2674
• Return explicit task execution code not found by @iaroslav-ciupin in #2659
• Replace Exceptions with specific errors by @pingsutw in #2668
• Follow FLYTE_PUSH_IMAGE_SPEC in default image builder by @thomasjpfan in #2682
• Fix docker warnings by @eapolinario in #2683
• Move UV install to after the ENV is set by @thomasjpfan in #2681
• Remove false error inside dynamic task in local executions by @pingsutw in #2675
• Create duckdb connection during execution by @thomasjpfan in #2684
• Fix None deserialization bug in dataclass outputs by @JackUrb in #2610

Full Changelog: v1.13.3...v1.13.4

v1.13.3

What's Changed

• Bump grpc receive message size by @wild-endeavor in #2640
• Raise an exception when filters' value isn't a list. by @arbaobao in #2576
• Update error message for TypeTransformerFailedError by @wayner0628 in #2648
• [Error Message] Dataclasses Mismatched Type by @Future-Outlier in #2650
• Added warning for command list and shell true by @pryce-turner in #2653
• In FlyteRemote.upload_file, pass the file object directly rather than the entire bytes buffer by @redartera in #2641
• Modify test_array_node.py to support running in python 3.8 by @eapolinario in #2652
• Handle common cases of mutable default arguments explicitly by @eapolinario in #2651
• allow a hash method to be present for numpy arrays by @demmerichs in #2649
• return exceptions when gathering by @wild-endeavor in #2657
• Correct FlyteFile docstring by @ppiegaze in #2658
• Remove pip cache after install by @thomasjpfan in #2662
• Adds validation to image_spec for parameters that are list of strings by @thomasjpfan in #2655
• Make elastic timeout configurable for HorovovJob. by @supercharleszhu in #2631
• Fix overriding of loader_args task resolver in papermill plugin by @eapolinario in #2660
• Catch all exceptions when rendering python dependencies by @thomasjpfan in #2664
• Don't check the return statement for reference_launch_plan by @pingsutw in #2665
• Bump flyteidl to 1.13.1 by @eapolinario in #2666

New Contributors

• @arbaobao made their first contribution in #2576
• @wayner0628 made their first contribution in #2648
• @demmerichs made their first contribution in #2649
• @supercharleszhu made their first contribution in #2631

Full Changelog: v1.13.2...v1.13.3

v1.13.2

What's Changed

• [Elastic/Artifacts] Pass through model card by @wild-endeavor in #2575
• Remove pyarrow as a direct dependency by @thomasjpfan in #2228
• Boolean flag to show local container logs to the terminal by @aditya7302 in #2521
• Enable Ray Fast Register by @fiedlerNr9 in #2606
• [Artifacts/Elastic] Skip partitions by @wild-endeavor in #2620
• Install flyteidl from master in plugins tests by @eapolinario in #2621
• Quality of Life: Using ParamSpec to show underlying typehinting by @JackUrb in #2617
• Support ArrayNode mapping over Launch Plans by @pvditt in #2480
• Richer printing for some artifact objects by @wild-endeavor in #2624
• ci: Add Python 3.9 to build matrix by @pingsutw in #2622
• bump idl by @wild-endeavor in #2627
• Added alt prefix head to FlyteFile.new_remote by @pryce-turner in #2601
• Feature gate for FlyteMissingReturnValueException by @pingsutw in #2623
• Remove use of multiprocessing from the OAuth client by @rdeaton-freenome in #2626
• Update codespell in precommit to version 2.3.0 by @eapolinario in #2630
• Fix Snowflake Agent Bug by @Future-Outlier in #2605
• run test_missing_return_value on python 3.10+ by @pingsutw in #2637
• [Elastic] Fix context usage and apply fix to fork method by @wild-endeavor in #2628
• Add flytekit-omegaconf plugin by @mg515 in #2299
• Adds extra-index-url to default image builder by @thomasjpfan in #2636
• reference_task should inherit from PythonTask by @pingsutw in #2643
• Fix Get Agent Secret Using Key by @Future-Outlier in #2644
• Fix Snowflake Agent Secret Bug by @Future-Outlier in #2645
• Don't call remote when --help in remote-X by @wild-endeavor in #2642

New Contributors

• @aditya7302 made their first contribution in #2521
• @JackUrb made their first contribution in #2617
• @mg515 made their first contribution in #2299

Full Changelog: v1.13.1...v1.13.2

[Beta] v1.13.2b1

from v1.13.2b0

What's Changed

• [Artifacts/Elastic] Skip partitions by @wild-endeavor in #2620
• Install flyteidl from master in plugins tests by @eapolinario in #2621
• Quality of Life: Using ParamSpec to show underlying typehinting by @JackUrb in #2617
• Support ArrayNode mapping over Launch Plans by @pvditt in #2480
• Richer printing for some artifact objects by @wild-endeavor in #2624
• ci: Add Python 3.9 to build matrix by @pingsutw in #2622
• bump idl by @wild-endeavor in #2627
• Added alt prefix head to FlyteFile.new_remote by @pryce-turner in #2601
• Feature gate for FlyteMissingReturnValueException by @pingsutw in #2623
• Remove use of multiprocessing from the OAuth client by @rdeaton-freenome in #2626
• Update codespell in precommit to version 2.3.0 by @eapolinario in #2630
• Fix Snowflake Agent Bug by @Future-Outlier in #2605
• run test_missing_return_value on python 3.10+ by @pingsutw in #2637
• [Elastic] Fix context usage and apply fix to fork method by @wild-endeavor in #2628

New Contributors

• @JackUrb made their first contribution in #2617

Full Changelog: v1.13.2b0...v1.13.2b1

v1.13.2b0

What's Changed

• [Elastic/Artifacts] Pass through model card by @wild-endeavor in #2575
• Remove pyarrow as a direct dependency by @thomasjpfan in #2228
• Boolean flag to show local container logs to the terminal by @aditya7302 in #2521
• Enable Ray Fast Register by @fiedlerNr9 in #2606

New Contributors

• @aditya7302 made their first contribution in #2521

Full Changelog: v1.13.1...v1.13.2b0

v1.13.1

Notes

flytekitplugins-kfpytorch

The distributed pytorch and distributed elastic-pytorch tasks in flytekitplugins-kfpytorch by default increase the shared memory limit by mounting an emptyDir volume with medium Memory to to /dev/shm as this is almost always required when working with torch multiprocessing (e.g. multi-processed data loader workers or local worker group in distributed training). To disable this, pass increase_shared_mem=False to task_config=PyTorch/Elastic. Elastic tasks now also set a default join timeout of 15 minutes to prevent timeouts when some worker pods require a node scale-up. This setting can be modified via task_config=Elastic(rdzv_configs{...}).

What's Changed

• Add myself to code owners of flytekit-kf-pytorch by @fg91 in #2556
• Add run_policy to ElasticTask by @bgedik in #2555
• Added entrypoint to imagespec and default builder by @pryce-turner in #2553
• Set default width if fail to get terminal size by @pingsutw in #2558
• Disable rich traceback for papermill plugin by @pingsutw in #2559
• fix datetime in eager workflow by @novahow in #2541
• Fix CSS with Flyte decks by @thomasjpfan in #2565
• Fix FlyteDirectory on Azure by @Tom-Newton in #2564
• Default image builder: Creates HOME if it does not exists by @thomasjpfan in #2562
• Removes isodate requirement by @thomasjpfan in #2568
• Read FLYTE_SDK_DEV_LOGGING_LEVEL from env by @pingsutw in #2571
• print native input/output when cache hit by @pingsutw in #2567
• Fix: Set OMP_NUM_THREADS by default in Elastic by @fellhorn in #2569
• Use logging level in env for default dev logger by @pingsutw in #2572
• Bump certifi from 2023.7.22 to 2024.7.4 in /tests/flytekit/integration/remote/mock_flyte_repo/workflows by @dependabot in #2566
• Allow for flytekit version to be specified in default image builder by @thomasjpfan in #2563
• Removes jinja2 dependency by @thomasjpfan in #2570
• Improve error when missing type annotations in task/workflow by @pingsutw in #2549
• Rename databricks task type by @pingsutw in #2574
• remove upper bound of plugin dependencies for flytekit-sqlalchemy by @Mecoli1219 in #2517
• Remove flytekitplugins-deck-standard in Dockerfile by @Future-Outlier in #2582
• Bump certifi from 2024.2.2 to 2024.7.4 by @dependabot in #2581
• handle existing sagemaker deployments gracefully by @samhita-alla in #2400
• Use /opt/micromamba for default image builder by @thomasjpfan in #2578
• Improve error message for missing return by @pingsutw in #2551
• Override Dataclass Serialization/Deserialization Behavior for FlyteTypes by mashumaro by @Future-Outlier in #2554
• Fix FlyteDirectory docs rendering by @ppiegaze in #2584
• Make openai_organization Optional in OpenAI plugins by @Future-Outlier in #2585
• Adds comet-ml plugin by @thomasjpfan in #2550
• Replace super(AsyncAgentBase) with super(DatabricksAgent) by @pingsutw in #2590
• add kubernetes package to official docker image again by @flixr in #2589
• Feat: Improve UX of pytorch-elastic plugin by configuring reasonable defaults by @fg91 in #2543
• Improved date parsing in CLI with meta variables by @kumare3 in #2595
• truncate sagemaker agent outputs and automate idempotence token handling by @samhita-alla in #2588
• [BUG] support setting extended resources for array node map tasks by @pvditt in #2592
• Fix DataClass Json Schema Error for get literal type method by @Future-Outlier in #2587
• Sagemaker dict determinism by @samhita-alla in #2597
• refactor(core): Enhance return type extraction logic by @pingsutw in #2598
• Feat: Make exception raised by external command authenticator more actionable by @fg91 in #2594
• Fix: Properly re-raise non-grpc exceptions during refreshing of proxy-auth credentials in auth interceptor by @fg91 in #2591
• validate idempotence token length in subsequent tasks by @samhita-alla in #2604
• Add nvidia-l4 gpu accelerator by @eapolinario in #2608
• eliminate redundant literal conversion for Iterator[JSON] type by @samhita-alla in #2602
• [FlyteSchema] Fix numpy problems by @Future-Outlier in #2619
• add nim plugin by @samhita-alla in #2475

Full Changelog: v1.13.0...v1.13.1

v1.13.1a5

What's Changed

• add nim plugin by @samhita-alla in #2475

Full Changelog: v1.13.1a4...v1.13.1a5

v1.13.1a4

What's Changed

• [FlyteSchema] Fix numpy problems by @Future-Outlier in #2619

Full Changelog: v1.13.1a3...v1.13.1a4

v1.13.1a3

What's Changed

• add kubernetes package to official docker image again by @flixr in #2589
• Feat: Improve UX of pytorch-elastic plugin by configuring reasonable defaults by @fg91 in #2543
• Improved date parsing in CLI with meta variables by @kumare3 in #2595
• truncate sagemaker agent outputs and automate idempotence token handling by @samhita-alla in #2588
• [BUG] support setting extended resources for array node map tasks by @pvditt in #2592
• Fix DataClass Json Schema Error for get literal type method by @Future-Outlier in #2587
• Sagemaker dict determinism by @samhita-alla in #2597
• refactor(core): Enhance return type extraction logic by @pingsutw in #2598
• Feat: Make exception raised by external command authenticator more actionable by @fg91 in #2594
• Fix: Properly re-raise non-grpc exceptions during refreshing of proxy-auth credentials in auth interceptor by @fg91 in #2591
• validate idempotence token length in subsequent tasks by @samhita-alla in #2604
• Add nvidia-l4 gpu accelerator by @eapolinario in #2608

Full Changelog: v1.13.1a2...v1.13.1a3