Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Prohibit execution parameters that serialize to more than 262144 bytes in size. #6863

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Prohibit execution parameters that serialize to more than 262144 bytes in size. #6863

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions RELEASE.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@
most likely you discovered a bug and should not use an f-string in the first
place. If it is truly your intention to print the placeholder (not its
resolved value) for debugging purposes, use `repr()` or `!r` instead.
* Execution parameters that serialize to larger than 262144 bytes are now
prohibited, and will cause an error during DSL compilation.

### For Pipeline Authors

Expand Down Expand Up @@ -224,7 +226,7 @@

## Bug Fixes and Other Changes

* Support to task type "workerpool1" of CLUSTER_SPEC in Vertex AI training's
* Support to task type "workerpool1" of CLUSTER_SPEC in Vertex AI training's
service according to the changes of task type in Tuner component.
* Propagates unexpected import failures in the public v1 module.

Expand Down Expand Up @@ -2887,4 +2889,4 @@ the 1.1.x release for TFX library.

### For component authors

* N/A
* N/A
11 changes: 11 additions & 0 deletions tfx/dsl/compiler/compiler.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@
from tfx.utils import deprecation_utils
from tfx.utils import name_utils

# Maximum size of serialized parameter Value proto.
_PARAMETER_VALUE_LIMIT = 2**20


class Compiler:
"""Compiles a TFX pipeline or a component into a uDSL IR proto."""
Expand Down Expand Up @@ -485,6 +488,14 @@ def _set_node_parameters(node: pipeline_pb2.PipelineNode,
raise ValueError(
"Component {} got unsupported parameter {} with type {}.".format(
tfx_node.id, key, type(value))) from e
size = len(parameter_value.SerializeToString())
if size > _PARAMETER_VALUE_LIMIT:
raise ValueError(
"Component {} got parameter {} which is too big: it serializes to "
"{} bytes, which exceeds the limit of {}".format(
tfx_node.id, key, size, _PARAMETER_VALUE_LIMIT
)
)


def _set_node_execution_options(
Expand Down