-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
protocols and generic non-unit base classes for signal transformers. #71
base: dev
Are you sure you want to change the base?
Conversation
…r signal transformers.
In the 3rd most-recent commit, I added a WIP protocol for adaptive transformers with a In the 2nd most-recent commit, I added a WIP protocol for transformers where the core processing logic should really be asynchronous. I can't think of any in ezmsg-sigproc, but this could be desirable for transformers that do store-and-passthrough (loggers, some sinks), or transformers that require retrieving data from disk/network/etc. I think this pattern will be more useful on sources and sinks rather than transformers. |
I don't think I put this in a doctstring, but one of my requirements is that I can use these transformers in stateless (cloud) frameworks. e.g., in Bytewax this would look like: flow = Dataflow("my_flow")
inp = op.input("inp", flow, TestingSource(src_items))
keyed_inp = op.key_on("key", inp, lambda msg: str(msg["user_id"]))
some_keyed_result = op.map_value("pick_amount", keyed_inp, lambda msg: msg["txn_amount"])
downsamp_result = op.stateful_map(
"downsample_step",
some_keyed_result,
DownsampleTransformer(axis="time", target_rate=200.0).stateful_op
)
# ... Bytewax will take care of passing (state, input) to
I have 0 familiarity with Beam but that looks like it will require custom classes too. I'm starting with Bytewax because it has the friendliest API for Python users. |
In the most recent 2 commits, within the base class I use I still can't eliminate (e.g.) |
…ust like ez.Unit.
…hat don't return a value.
3e0ed20
to
8205137
Compare
2nd most-recent commit enabled using Most recent commit aded StatefulProcessor base above SignalTransformer for processors that don't return a value in their main |
See #70 for motivation.
So far in this draft PR, I added protocols and base classes for non-ezmsg-unit transformers as well as the minimal ez.Unit wrapper around the transformer. These are intended to replace the generator methods and
GenAxisArray
-child units that we have currently.I also refactored Downsample to this new design. All of its tests pass.
There's at least one imperfection here that I don't know how to solve. For example, in
Downsample
, I need to provideDownsampleSettings
both as a generic type in the class definition as well as a class variable:SETTINGS = DownsampleSettings
. I don't know a way around this because ez.Unit's init requires the concrete type. This is a minor gripe that I can live with.I'd love some feedback on this before I start converting all the other sigproc nodes. @griffinmilsap @pperanich
Also, I kept the
downsample
method for backwards compatibility though it isn't strictly a generator method anymore.