Use native Rust support for async traits in LogExporter::export() method (11% improvement) #2374

lalitb · 2024-12-02T22:34:02Z

Changes

Continuation to #2143, more to show feasibility, performance improvements, and have discussion before raising the PR for eventual review, this PR demonstrate using async traits support in Rust exporter. The support was added in Rust v1.75, while the msrv as of now is v1.70 for otel-sdk

The change is:
Existing:

#[async_trait]
pub trait LogExporter: Send + Sync + Debug {
  async fn export(&mut self, batch: LogBatch<'_>) -> LogResult<()>;
}

PR:

pub trait LogExporter: Send + Sync + Debug {
    fn export<'a>(
        &'a mut self,
        batch: &'a LogBatch<'a>,
    ) -> impl std::future::Future<Output = LogResult<()>> + Send + 'a;
}

There is 11% improvement as seen from stress test:

Before PR:

Number of threads: 16
Throughput: 40,468,186 iterations/sec
Throughput: 40,792,480 iterations/sec
Throughput: 40,456,227 iterations/sec
Throughput: 40,455,461 iterations/sec

After this PR:

Throughput: 44,793,174 iterations/sec
Throughput: 45,321,168 iterations/sec
Throughput: 45,387,969 iterations/sec
Throughput: 45,629,648 iterations/sec
Throughput: 45,607,326 iterations/sec
Throughput: 45,303,331 iterations/sec
Throughput: 45,634,108 iterations/sec

Thanks,
Lalit

Merge requirement checklist

CONTRIBUTING guidelines followed
Unit tests added/updated (if applicable)
Appropriate CHANGELOG.md files updated for non-trivial, user-facing changes
Changes in public API reviewed (if applicable)

opentelemetry-sdk/src/export/logs/mod.rs

opentelemetry-appender-tracing/src/layer.rs

codecov · 2024-12-06T23:51:18Z

Codecov Report

Attention: Patch coverage is 27.07182% with 132 lines in your changes missing coverage. Please review.

Project coverage is 76.7%. Comparing base (80629c8) to head (36ea18a).
Report is 1 commits behind head on main.

Files with missing lines	Patch %	Lines
opentelemetry-otlp/src/exporter/http/logs.rs	0.0%	37 Missing ⚠️
opentelemetry-otlp/src/exporter/tonic/logs.rs	0.0%	28 Missing ⚠️
opentelemetry-stdout/src/logs/exporter.rs	0.0%	24 Missing ⚠️
opentelemetry-otlp/src/logs.rs	0.0%	19 Missing ⚠️
opentelemetry-sdk/src/logs/log_processor.rs	74.4%	12 Missing ⚠️
opentelemetry-appender-tracing/src/layer.rs	0.0%	10 Missing ⚠️
opentelemetry-otlp/src/exporter/http/mod.rs	0.0%	1 Missing ⚠️
opentelemetry-otlp/src/exporter/tonic/mod.rs	0.0%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #2374     +/-   ##
=======================================
- Coverage   76.8%   76.7%   -0.2%     
=======================================
  Files        122     122             
  Lines      22217   22266     +49     
=======================================
+ Hits       17082   17093     +11     
- Misses      5135    5173     +38

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

lalitb · 2024-12-07T00:05:49Z

@lalitb I think it maybe better to first check how to avoid requiring mut reference for exporting, before evaluating the results of this perf test, as that requirement has forced the need of mutex and introduced contention and hence stress tests shows way low throughput than before.

OK, have updated this PR after removing the mut self in #2380. This eliminates the need of mutex for exporter in stress test. The stress result are now similar to earlier i.e., without the inclusion of exporter. Which means, adding futures::block_on() on native async-method doesn't introduce much of latency.

Before this PR (the regression introduced by #2380 because of inclusion of exporter ~44 -> ~40):

Number of threads: 16
Throughput: 40,468,186 iterations/sec
Throughput: 40,792,480 iterations/sec
Throughput: 40,456,227 iterations/sec
Throughput: 40,455,461 iterations/sec

After this PR:

Throughput: 44,793,174 iterations/sec
Throughput: 45,321,168 iterations/sec
Throughput: 45,387,969 iterations/sec
Throughput: 45,629,648 iterations/sec
Throughput: 45,607,326 iterations/sec
Throughput: 45,303,331 iterations/sec
Throughput: 45,634,108 iterations/sec

The cons of introducing the native-async in trait would be

Bump the msrv to 1.75
The exporter interface is no longer object-safe. Which means, it can't be used in dynamic dispatch (e.g., via dyn Trait). This introduced slight complexity in otel-otlp, where dyn dispatch was used for http and tonic exporters.

Current export method could be simplified a bit for lifetime specifiers. I will look into that. But in general, open for discussion if we should bump the msrv.

cijothomas · 2024-12-10T17:28:39Z

Bump the msrv to 1.75
The exporter interface is no longer object-safe. Which means, it can't be used in dynamic dispatch (e.g., via dyn Trait). This introduced slight complexity in otel-otlp, where dyn dispatch was used for http and tonic exporters.

Based on today's SIG discussion:
1.75 bump is acceptable.
the exporter interface not being object safe is an easy to fix issue as this PR already demonstrates fixing it for OTLP.

Awaiting more feedback.

lalitb · 2024-12-13T07:33:48Z

This is ready for review now, with #2417 to bump msrv is merged

opentelemetry-appender-tracing/benches/logs.rs

opentelemetry-otlp/src/exporter/tonic/logs.rs

opentelemetry-otlp/src/logs.rs

lalitb · 2024-12-20T01:38:38Z

Have updated the description now to reflect the improvement seen through stress test, which is ~11%

opentelemetry-sdk/src/export/logs/mod.rs

cijothomas · 2024-12-20T03:49:59Z

stress/src/logs.rs

@@ -37,14 +39,15 @@ pub struct MockLogProcessor {
 impl LogProcessor for MockLogProcessor {
    fn emit(&self, record: &mut opentelemetry_sdk::logs::LogRecord, scope: &InstrumentationScope) {
        let log_tuple = &[(record as &LogRecord, scope)];
-        let _ = futures_executor::block_on(self.exporter.export(LogBatch::new(log_tuple)));
+        let log_batch = LogBatch::new(log_tuple);
+        let _ = futures_executor::block_on(self.exporter.export(&log_batch));


The overall flow looks complex..
emit gets a mutable ref to LogRecord. It is ref by design (and mut by design.)
processor, if need to batch it and send, can clone and keep the copy. When it calls the exporter, can it pass ownership the exporter? Wondering why we only pass a ref to the exporter? The processor has no reason to retain ownership anyway, so why not give it up, to the exporter?

Thinking out loud to see if this can be simplified without sacrificing perf..

This is good point. We are passing LogBatch as reference. LogBatch is a lightweight wrapper around the slice of tuple of reference to LogRecord and InstrumentationScope. Passing it by value won't change the ownership of this borrowed data (record and scope). We can change batch to be passed as value (though record and scope would still remain as reference). This can simplify the design (remove 'a need). However, reference just shows the intent better, and need to test the perf (theoretically it shouldn't affect).

ps - Need to dig the reason for passing record and scope as reference, I remember we discussed and agreed for it in earlier discussion. Here, we are only taking about passing batch as reference or value.

The processor has no reason to retain ownership anyway, so why not give it up, to the exporter?

Probably it was not very clear from my above message. Just to clarify - this PR is not changing the ownership model. The exporter always gets the reference to the LogRecord, and it clones if the LogRecord needs to be used beyond the export function lifetime. Are we now discussing the existing design?

reference just shows the intent better,

Not so sure, since we are passing a custom structure LogBatch, which is defined to contains refs only.
Or we can pass slice of refs only, avoiding LogBatch structure entirely. This is better discussed separately.

Are we now discussing the existing design?

This PR is changing existing design - previously owned LogBatch was passed, now we pass a &LogBatch only. It is not clear if that is required for using async native or it is unrelated to that.
If there is no need to change from passing LogBatch to &LogBatch to make async-native work, then lets revert that part.

If there is no need to change from passing LogBatch to &LogBatch to make async-native work, then lets revert that part.

This is reverted now.

…y-rust into log-async-trait-impl

stress/src/logs.rs

cijothomas · 2024-12-20T15:51:14Z

opentelemetry-sdk/src/export/logs/mod.rs

@@ -81,7 +79,11 @@ pub trait LogExporter: Send + Sync + Debug {
    /// A `LogResult<()>`, which is a result type indicating either a successful export (with
    /// `Ok(())`) or an error (`Err(LogError)`) if the export operation failed.
    ///
-    async fn export(&self, batch: LogBatch<'_>) -> LogResult<()>;
+    fn export(


please add a changelog entry as this affects custom exporter authors.

cijothomas

LGTM.

The perf gains are only applicable for concurrently exporting log processors that exports without any batching (like etw exporting).
For components in this repo (like OTLP), export is not in hot path and won't see perf gains.

But irrespective, this still is an improvement and also eliminates one more dependency!

cijothomas · 2024-12-20T15:55:37Z

@TommyCpp would you have time to another look?

cijothomas · 2024-12-20T16:03:47Z

Merging to unblock other works that'd otherwise have to deal with merge conflicts. Please continue reviewing, and we can address feedbacks in followups.

lalitb added 6 commits December 1, 2024 02:26

initial commit

d454536

futher changes..

3ed482d

changes..

a2aa648

initial change

8ba1173

stdout exporter

37823e7

update stress

3b4cd57

lalitb requested a review from a team as a code owner December 2, 2024 22:34

cijothomas reviewed Dec 2, 2024

View reviewed changes

opentelemetry-sdk/src/export/logs/mod.rs Outdated Show resolved Hide resolved

cijothomas reviewed Dec 3, 2024

View reviewed changes

opentelemetry-appender-tracing/src/layer.rs Outdated Show resolved Hide resolved

Merge branch 'main' into log-async-trait-impl

858cd87

lalitb mentioned this pull request Dec 4, 2024

Remove mut self reference from LogExporter::export() method. #2380

Merged

4 tasks

lalitb added 2 commits December 6, 2024 11:44

Merge branch 'main' into log-async-trait-impl

e6472bf

fix otlp

8cacf52

cijothomas added this to the 0.28.0 milestone Dec 10, 2024

lalitb mentioned this pull request Dec 12, 2024

bump MSRV to 1.75.0 for all crates #2417

Merged

4 tasks

lalitb added 2 commits December 12, 2024 21:17

Merge branch 'main' into log-async-trait-impl

dfac978

add comment

f752d50

lalitb changed the title ~~Draft to Discuss: Use native Rust support for async traits in LogExporter::export() method~~ Use native Rust support for async traits in LogExporter::export() method Dec 13, 2024

lalitb commented Dec 13, 2024

View reviewed changes

opentelemetry-appender-tracing/benches/logs.rs Show resolved Hide resolved

lalitb added the integration tests Run integration tests label Dec 13, 2024

Merge branch 'main' into log-async-trait-impl

6ac8aa0

cijothomas reviewed Dec 13, 2024

View reviewed changes

opentelemetry-otlp/src/exporter/tonic/logs.rs Outdated Show resolved Hide resolved

Merge branch 'main' into log-async-trait-impl

4924e3b

TommyCpp reviewed Dec 13, 2024

View reviewed changes

opentelemetry-otlp/src/logs.rs Outdated Show resolved Hide resolved

TommyCpp reviewed Dec 13, 2024

View reviewed changes

opentelemetry-otlp/src/logs.rs Outdated Show resolved Hide resolved

review comment

8db475c

lalitb and others added 2 commits December 16, 2024 07:55

Merge branch 'main' into log-async-trait-impl

8c34380

Merge branch 'main' into log-async-trait-impl

87c6c9d

lalitb mentioned this pull request Dec 17, 2024

Handle batch log processing in a dedicated background thread #2436

Merged

4 tasks

Merge branch 'main' into log-async-trait-impl

be7ecec

cijothomas mentioned this pull request Dec 18, 2024

feature: refactor and extend log_exporter benchmark #2143

Closed

4 tasks

lalitb and others added 8 commits December 19, 2024 10:58

Merge branch 'main' into log-async-trait-impl

5c8c644

resolve conflicts

d1f3cbb

initial commit

fa11cd7

further conflicts

67f006a

remove unwantd comments

85f653c

Merge branch 'main' into log-async-trait-impl

33b7572

merge conflict

0d5a17c

Merge branch 'main' into log-async-trait-impl

e8333f5

lalitb changed the title ~~Use native Rust support for async traits in LogExporter::export() method~~ Use native Rust support for async traits in LogExporter::export() method (11% improvement) Dec 20, 2024

lalitb added 2 commits December 20, 2024 08:44

Remove async_trait import and attribute

4e4a189

remove unused crate

7627224

cijothomas reviewed Dec 20, 2024

View reviewed changes

opentelemetry-sdk/src/export/logs/mod.rs Outdated Show resolved Hide resolved

cijothomas reviewed Dec 20, 2024

View reviewed changes

lalitb added 3 commits December 20, 2024 11:13

Merge branch 'main' into log-async-trait-impl

09f7cb8

update export to take batch by value

ddb15bf

Merge branch 'log-async-trait-impl' of github.com:lalitb/opentelemetr…

5e025d2

…y-rust into log-async-trait-impl

cijothomas reviewed Dec 20, 2024

View reviewed changes

stress/src/logs.rs Outdated Show resolved Hide resolved

lalitb added 2 commits December 20, 2024 07:38

remove temporary batch variable

158e922

keep diff minimal

36ea18a

cijothomas reviewed Dec 20, 2024

View reviewed changes

cijothomas approved these changes Dec 20, 2024

View reviewed changes

cijothomas merged commit 23f6ae2 into open-telemetry:main Dec 20, 2024
19 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Use native Rust support for async traits in LogExporter::export() method (11% improvement) #2374

Use native Rust support for async traits in LogExporter::export() method (11% improvement) #2374

lalitb commented Dec 2, 2024 •

edited

Loading

codecov bot commented Dec 6, 2024 •

edited

Loading

lalitb commented Dec 7, 2024 •

edited

Loading

cijothomas commented Dec 10, 2024

lalitb commented Dec 13, 2024

lalitb commented Dec 20, 2024

cijothomas Dec 20, 2024

lalitb Dec 20, 2024 •

edited

Loading

lalitb Dec 20, 2024

cijothomas Dec 20, 2024

lalitb Dec 20, 2024

cijothomas Dec 20, 2024

cijothomas left a comment

cijothomas commented Dec 20, 2024

cijothomas commented Dec 20, 2024

Use native Rust support for async traits in LogExporter::export() method (11% improvement) #2374

Use native Rust support for async traits in LogExporter::export() method (11% improvement) #2374

Conversation

lalitb commented Dec 2, 2024 • edited Loading

Changes

Merge requirement checklist

codecov bot commented Dec 6, 2024 • edited Loading

Codecov Report

lalitb commented Dec 7, 2024 • edited Loading

cijothomas commented Dec 10, 2024

lalitb commented Dec 13, 2024

lalitb commented Dec 20, 2024

cijothomas Dec 20, 2024

Choose a reason for hiding this comment

lalitb Dec 20, 2024 • edited Loading

Choose a reason for hiding this comment

lalitb Dec 20, 2024

Choose a reason for hiding this comment

cijothomas Dec 20, 2024

Choose a reason for hiding this comment

lalitb Dec 20, 2024

Choose a reason for hiding this comment

cijothomas Dec 20, 2024

Choose a reason for hiding this comment

cijothomas left a comment

Choose a reason for hiding this comment

cijothomas commented Dec 20, 2024

cijothomas commented Dec 20, 2024

lalitb commented Dec 2, 2024 •

edited

Loading

codecov bot commented Dec 6, 2024 •

edited

Loading

lalitb commented Dec 7, 2024 •

edited

Loading

lalitb Dec 20, 2024 •

edited

Loading