Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Migrate SDK comms to gRPC #8838

Open
wants to merge 71 commits into
base: main
Choose a base branch
from
Open

Migrate SDK comms to gRPC #8838

wants to merge 71 commits into from

Conversation

jprochazk
Copy link
Member

@jprochazk jprochazk commented Jan 28, 2025

Related

What

The goal of this PR is to completely delete re_sdk_comms and re_ws_comms. The prerequisite to this is to first migrate all of their usages over to our temporary gRPC server ("message proxy").

Step-by-step:

  • Expose gRPC variants of spawn/connect from all SDKs
  • Start a gRPC server instead of TCP server on native and connect to it by default
  • Start a gRPC server instead of the re_ws_comms server and connect to it by default
  • Remove all remaining usage of re_sdk_comms and re_ws_comms
  • Delete re_sdk_comms and re_ws_comms

I don't really know how to split this up, to be honest! There are tons of subtle but very strong dependencies between different parts of our SDKs, the Viewer, and the "server". Breaking it apart seems too difficult, and it'd also temporarily make some parts of rerun unusable, and thus untestable. So instead I'll be trying to keep individual commits somewhat sensible and reviewable.

Testing checklist

  • SDK spawn
    • C++
    • Rust
    • Python
  • SDK serve
    • C++
    • Rust
    • Python
  • rerun (default args) + SDK connect
    • C++
    • Rust
    • Python
  • rerun --web-viewer + SDK connect
    • C++
    • Rust
    • Python
  • Custom examples
  • More?

Migration guide

TODO

Unresolved questions

  • API
  • Should the http:// or temp:// prefix be necessary?

Copy link

github-actions bot commented Jan 28, 2025

Web viewer built successfully. If applicable, you should also test it:

  • I have tested the web viewer
Result Commit Link Manifest
357f243 https://rerun.io/viewer/pr/8838 +nightly +main

Note: This comment is updated whenever you push a commit.

@jprochazk
Copy link
Member Author

As a treat...

@rerun-bot full-check

Copy link

@Wumpf
Copy link
Member

Wumpf commented Jan 29, 2025

if possible we need some better error messages / warnings when spawning old viewers. This usually works fine but we're doing a hard break here that we should intercept and warn about. Right now it does this:

    ⚠ The version of the Rerun Viewer available on your PATH does not match the version of your Rerun SDK ⚠

    Rerun does not make any kind of backwards/forwards compatibility guarantee yet: this can lead to (subtle) bugs.

    > Rerun Viewer: v0.20.0 (executable: "rerun")
    > Rerun SDK: v0.22.0-alpha.1+dev

[2025-01-29T17:21:45Z INFO  re_sdk_comms::server] Hosting a SDK server over TCP at 0.0.0.0:9876. Connect with the Rerun logging SDK.
[2025-01-29T17:21:45Z ERROR re_grpc_client::message_proxy::write] Write messages call failed: status: Unknown, message: "transport error", details: [], metadata: MetadataMap { headers: {} }
[2025-01-29T17:21:45Z ERROR re_grpc_client::message_proxy::write] failed to gracefully shut down message proxy client: channel closed
[2025-01-29T17:21:45Z WARN  re_viewer::app] Failed to restore application state. This is expected if you have just upgraded Rerun versions.

@Wumpf
Copy link
Member

Wumpf commented Jan 29, 2025

pixi run rerun gives me additional log that that's not color coded and Rust coded. Will that show up in release versions?

2025-01-29 18:25:02.692 rerun[88428:15834291] +[IMKClient subclass]: chose IMKClient_Modern
2025-01-29 18:25:02.692 rerun[88428:15834291] +[IMKInputSession subclass]: chose IMKInputSession_Modern

@Wumpf
Copy link
Member

Wumpf commented Jan 29, 2025

We no longer pick up running viewers correctly it seems:

Open one command line with pixi run rerun
Open another one to start a snippet, I used ./build/debug/docs/snippets/image_send_columns (after pixi run -e cpp cpp-build-snippets)

-> the snippet will try to spawn a new viewer which in my case brings up a 0.20 viewer since this is the last one I system-installed. It then fails as before

@oxkitsune
Copy link
Contributor

pixi run rerun gives me additional log that that's not color coded and Rust coded. Will that show up in release versions?

2025-01-29 18:25:02.692 rerun[88428:15834291] +[IMKClient subclass]: chose IMKClient_Modern
2025-01-29 18:25:02.692 rerun[88428:15834291] +[IMKInputSession subclass]: chose IMKInputSession_Modern

This is macOS thing (rust-windowing/winit#3931) 😔

Copy link

github-actions bot commented Jan 30, 2025

Latest documentation preview deployed successfully.

Result Commit Link
357f243 https://landing-h4yfof25k-rerun.vercel.app/docs

Note: This comment is updated whenever you push a commit.

}

assert!(!failed, "one or more test cases failed");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: What's the benefit of just having a single assert here at the end? Won't that make it harder to debug potential problems?

Ok(Self(url.to_owned()))
}
// TODO(#8761): URL prefix
else if let Some(url) = url.strip_prefix("temp") {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: Maybe put "temp" in a const TEMP_URL_PREFIX to make refactoring later easier.

Comment on lines +161 to +178
macro_rules! test_parse_url {
($name:ident, $url:literal, error) => {
#[test]
fn $name() {
assert!(MessageProxyUrl::parse($url).is_err());
}
};

($name:ident, $url:literal, expected: $expected_http:literal) => {
#[test]
fn $name() {
assert_eq!(
MessageProxyUrl::parse($url).map(|v| v.to_http()),
Ok($expected_http.to_owned())
);
}
};
}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: I think having explicit test cases might make it easier to see what's going on a at a glance.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
🌊 C++ API C/C++ API specific include in changelog 🐍 Python API Python logging API 🦀 Rust API Rust logging API 🕸️ web regarding running the viewer in a browser
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Migrate all TCP/WS comms to gRPC
4 participants