Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: add fluvio producer callback #4340

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

fraidev
Copy link
Contributor

@fraidev fraidev commented Jan 19, 2025

This PR adds a callback option for the producer.

Also prints total bytes and total records per sec at the end:

> fluvio bench producer --batch-size 16kib --num-records 300000
27244 records sent, 27244 records/sec: (138.9 KB/sec), 1.34ms avg latency, 3.69ms max latency
37800 records sent, 37800 records/sec: (192.8 KB/sec), 1.41ms avg latency, 10.63ms max latency
42028 records sent, 42028 records/sec: (214.3 KB/sec), 1.38ms avg latency, 10.63ms max latency
41552 records sent, 41552 records/sec: (211.9 KB/sec), 1.36ms avg latency, 10.63ms max latency
40138 records sent, 40138 records/sec: (204.7 KB/sec), 1.37ms avg latency, 12.28ms max latency
40868 records sent, 40868 records/sec: (208.4 KB/sec), 1.37ms avg latency, 12.28ms max latency
40488 records sent, 40488 records/sec: (206.5 KB/sec), 1.37ms avg latency, 12.28ms max latency

1.36ms avg latency, 12.28ms max latency, 1.27ms p0.50, 1.70ms p0.95, 2.54ms p0.99
300000 total records sent, 40689 records/sec: (207.5 MB/sec) 
Benchmark completed

@fraidev fraidev force-pushed the fluvio_producer_callback branch 13 times, most recently from 21775b3 to a283846 Compare January 20, 2025 04:08
@fraidev fraidev marked this pull request as ready for review January 20, 2025 04:08
@fraidev fraidev requested a review from sehz January 20, 2025 04:13
@fraidev fraidev force-pushed the fluvio_producer_callback branch from a283846 to 530de55 Compare January 20, 2025 04:14
@fraidev fraidev force-pushed the fluvio_producer_callback branch from 530de55 to c182387 Compare January 20, 2025 04:14
self.stat.start();
let time = std::time::Instant::now();
let send_out = self
self.stat.start().await;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

should remove this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Start is setting the first instant to calculate the final bandwidth at the end.

.fluvio_producer
.send(record.key, record.data.clone())
.await?;

self.stat.send_out((send_out, time));
self.stat.add_record(record.data.len() as u64).await;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

no longer need this

}
// send end
let hist = hist.lock().await;
let start_time = start_time.lock().await.expect("start time");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

start time should be tracked by callback?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is the first start time, I can rename it

// send end
let hist = hist.lock().await;
let start_time = start_time.lock().await.expect("start time");
let elapsed = start_time.elapsed();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be come from callback

@@ -118,6 +120,9 @@ pub struct TopicProducerConfig {

#[builder(default)]
pub(crate) smartmodules: Vec<SmartModuleInvocation>,

#[builder(setter(into, strip_option), default)]
pub(crate) callback: Option<SharedProducerCallback<ProduceCompletionEvent>>,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why make generic? don't think there is need to make generic

created_at: batch_metadata.created_at,
metadata: batch_metadata.clone(),
};
callback.finished(event).await.unwrap();
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remove unwrap

}

pub(crate) fn set_current_time(&mut self) {
self.start_time = Instant::now();
pub(crate) fn add_record(&mut self, bytes: u64) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be invoked by callback

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants