Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement gRPC log stream #8730

Merged
merged 38 commits into from
Jan 22, 2025
Merged

Implement gRPC log stream #8730

merged 38 commits into from
Jan 22, 2025

Conversation

jprochazk
Copy link
Member

@jprochazk jprochazk commented Jan 19, 2025

This PR implements Viewer-side log streaming from an in-memory storage node.

It is not currently possible to host the server either in the SDK or the viewer, so for now a separate binary exists in the re_grpc_server crate (cargo run -p re_grpc_server), that starts on localhost:1852.

Currently the URL prefix is temp://, which is a placeholder for the real URL prefix. The prefix is used by the Viewer to determine what kind of data source it is working with, so we need something that uniquely identifies the in-memory storage node. It has to be separate from existing gRPC infrastructure, because there is currently no overlap in functionality between the two.

@jprochazk jprochazk added include in changelog 🪵 Log & send APIs Affects the user-facing API for all languages remote-store remote store gRPC API labels Jan 19, 2025
Copy link

github-actions bot commented Jan 19, 2025

Web viewer built successfully. If applicable, you should also test it:

  • I have tested the web viewer
Result Commit Link Manifest
858c844 https://rerun.io/viewer/pr/8730 +nightly +main

Note: This comment is updated whenever you push a commit.

@jprochazk jprochazk force-pushed the jan/grpc-log-stream branch from 362d97a to cc6e789 Compare January 19, 2025 17:37
@emilk
Copy link
Member

emilk commented Jan 21, 2025

I added this issue for removing the temp:// thing:

Copy link
Member

@emilk emilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you tested this with the web viewer? pixi run rerun-web temp://…

crates/viewer/re_viewer/src/viewer_analytics/event.rs Outdated Show resolved Hide resolved
crates/viewer/re_viewer/src/ui/top_panel.rs Outdated Show resolved Hide resolved
crates/utils/re_smart_channel/src/lib.rs Show resolved Hide resolved
crates/utils/re_smart_channel/src/lib.rs Show resolved Hide resolved
crates/store/re_grpc_server/src/main.rs Outdated Show resolved Hide resolved
crates/store/re_data_source/src/data_source.rs Outdated Show resolved Hide resolved
crates/store/re_grpc_client/Cargo.toml Outdated Show resolved Hide resolved
crates/store/re_grpc_client/src/lib.rs Show resolved Hide resolved
crates/store/re_grpc_client/src/message_proxy/read.rs Outdated Show resolved Hide resolved
Base automatically changed from jan/grpc-log-sink to main January 22, 2025 09:41
@jprochazk jprochazk force-pushed the jan/grpc-log-stream branch from b8ee932 to a82ed71 Compare January 22, 2025 11:42
@jprochazk
Copy link
Member Author

jprochazk commented Jan 22, 2025

Ok, I have this working on web now. Took much longer than expected, because I realised half-way that I need to use the envoy proxy. envoy doesn't seem to be available through pixi, so we'll have to install it manually for the time being: https://www.envoyproxy.io/docs/envoy/latest/start/install

  1. envoy -c crates/store/re_grpc_server/envoy.yml
  2. cargo run -p re_grpc_server --release
  3. pixi run rerun-web-release and open http://localhost:9090/?url=temp://localhost:1853
  4. import rerun as rr
    rr.init("rerun_example_text_log")
    rr.connect_grpc('http://127.0.0.1:1853')
    rr.log("log", rr.TextLog("Application started.", level=rr.TextLogLevel.INFO))

This greatly complicates the usual rerun-web setup... If we no longer support websockets, we'll always have to run the proxy. I'm going to try and look into if there's a way to do our own "proxying" in the same process as re_grpc_server, otherwise this is dead on arrival...

@jprochazk
Copy link
Member Author

jprochazk commented Jan 22, 2025

I checked dataplatform and looks like I just forgot to setup tonic-web... 🤦

  1. cargo run -p re_grpc_server --release
  2. pixi run rerun-web-release and open http://localhost:9090/?url=temp://localhost:1852
  3. import rerun as rr
    rr.init("rerun_example_text_log")
    rr.connect_grpc('http://127.0.0.1:1853')
    rr.log("log", rr.TextLog("Application started.", level=rr.TextLogLevel.INFO))

@jprochazk jprochazk requested a review from emilk January 22, 2025 12:23
Copy link
Member

@emilk emilk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice!

crates/store/re_grpc_client/Cargo.toml Show resolved Hide resolved
crates/store/re_grpc_server/README.md Outdated Show resolved Hide resolved
crates/store/re_grpc_server/README.md Outdated Show resolved Hide resolved
crates/store/re_grpc_server/src/main.rs Outdated Show resolved Hide resolved
crates/viewer/re_viewer/src/web_tools.rs Outdated Show resolved Hide resolved
@jprochazk jprochazk removed the do-not-merge Do not merge this PR label Jan 22, 2025
@jprochazk jprochazk merged commit 2d5c213 into main Jan 22, 2025
33 of 34 checks passed
@jprochazk jprochazk deleted the jan/grpc-log-stream branch January 22, 2025 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
include in changelog 🪵 Log & send APIs Affects the user-facing API for all languages remote-store remote store gRPC API
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Implement gRPC log stream
2 participants