Spring Pulsar Template is slow compared to native pulsar producer

[chaitanya-guttula-collibra]

What is the best way to use spring pulsar template so that it is efficient in sending messages? Spring Pulsar template is almost 8 times slower than native pulsar producer. We did some performance testing for message production using Gatling with 100K & 1 million messages of 2KB. Please see the results below

1 million messages - 6 partitioned topic

• Native pulsar

  • Time taken 9 min 21 s - 561s
  • Throughput - 1782 msg/sec

• Pulsar Template

  • Time taken - 1h 9 m 16s - 4156
  • Throughput - 240 msgs/sec

Native Pulsar is 7.4 times faster

### 100k messages - 6 partitioned topic

- Native pulsar
- Time taken 1 min 1s - 61s
- Throughput - 1639 msg/sec

- Pulsar Template
- Time taken - 8 min 2s - 482
- Throughput - 207 msgs/sec

- Native pulsar is 7.9 times faster

100k messages - non partitioned topic

• Native pulsar

  • Time taken 2 min 20s - 140s
  • Throughput - 714 msg/sec

• Pulsar Template

  • Time taken - 27 min 57s - 1677
  • Throughput - 59 msgs/sec

• Native pulsar is 11.9 times faster

Please let me know if you need additional information.

[onobc]

Hi @chaitanya-guttula-collibra ,

Thanks for the detailed report. We did a comparison a while back and the performance was equal (and in some cases better when caching the same topic). We plan on putting this into a standard CI workflow so that in case something creeps in we will notice it - we do not have that workflow in place yet though.

A couple of possible causes:

1. a regression has entered and things are slower now
2. something needs to be tuned/configured/used in your setup (aka the question in your intro "What is the best way to use spring pulsar template").

Next steps

• Can you share your test harness (sample) that you are using w/ Gatling so we can take a look?
• On our side, we will re-run our perf tests and see what we see.

Sound like a plan?

[chaitanya-guttula-collibra]

Hi @onobc, Thank you for your quick response. Please find the sample used for testing.
https://github.com/chaitanya-guttula-collibra/spring-pulsar-sample-app

I also want to correct the scenario 2 that I added above.

100k messages - 6 partitioned topic

• Native pulsar

  • Time taken 1 min 1s - 61s
  • Throughput - 1639 msg/sec

• Pulsar Template

  • Time taken - 4 min 20s - 260s
  • Throughput - 384 msgs/sec

• Native pulsar is 4.2 times faster

[jonas-grgt]

@chaitanya-guttula-collibra I'm getting a 404 on that repo, could it be that it is a private one?

[chaitanya-guttula-collibra]

@jonasgeiregat Yes its a private repo. I have granted access.

[onobc]

Thanks @chaitanya-guttula-collibra - can you please add me as well? Thanks

[chaitanya-guttula-collibra]

Done

[onobc]

Hi @chaitanya-guttula-collibra ,

I am just now getting back to this - busy week.

I suggest we focus on the "partitioned w/ schema" case and ignore the other variants for now.

For the Spring Pulsar app this coresponds to:
https://github.com/chaitanya-guttula-collibra/spring-pulsar-sample-app/blob/f56747ea006af3adbe8fcffa108a77766c364c2a/app/src/main/java/org/example/controller/ProducerController.java#L58-L68

For the Native Pulsar app this corresponds to:
https://github.com/chaitanya-guttula-collibra/spring-pulsar-sample-app/blob/f56747ea006af3adbe8fcffa108a77766c364c2a/native-pulsar-producer/src/main/java/org/example/nativ/pulsar/producer/controller/NativeProducerController.java#L43-L52

When I compare the above impls, I notice that in the Native case you are not sending a payload.

String messageId = producer.newMessage(Schema.JSON(ExampleMessage.class)).sendAsync().get().toString();

vs.

String messageId = pulsarTemplate1.newMessage(new ExampleMessage()).withTopic(topicWithSchema).sendAsync().get().toString();

Do you mind adjusting this and re-running the test?

[onobc]

Hi Edgar, For some reason your comment is not showing up in the UI thread. I copied it directly into the discussion below replied - PTAL.

…

On Fri, Feb 23, 2024 at 3:37 PM Edgar Molina ***@***.***> wrote: hi, my team is facing a similar problem, i believe the issue is that when we use a producerCustomizer at the message level, just as @chaitanya-guttula-collibra <https://github.com/chaitanya-guttula-collibra> is doing in his code: mbs = pulsarTemplate1.newMessage(new ExampleMessage()).withTopic(topicWithSchema).withProducerCustomizer(producer -> { producer.enableBatching(true); producer.compressionType(CompressionType.LZ4); producer.blockIfQueueFull(true); producer.batchingMaxMessages(1_000); producer.batchingMaxPublishDelay(100, TimeUnit.MILLISECONDS); producer.batchingMaxBytes(1024 * 1024); }).sendAsync(); The sendAsync() will create a new producer per message here https://github.com/spring-projects/spring-pulsar/blob/main/spring-pulsar/src/main/java/org/springframework/pulsar/core/PulsarTemplate.java#L236 and here https://github.com/spring-projects/spring-pulsar/blob/main/spring-pulsar/src/main/java/org/springframework/pulsar/core/PulsarTemplate.java#L289 . So i guess the question is how can we globally customize the producer properties of the pulsar template (enable compression, batching, etc.) and not per message. — Reply to this email directly, view it on GitHub <#577 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AG4RTQZWTHH2DVJSTAPKDV3YVEDZ3AVCNFSM6AAAAABDIP2Z7WVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DKNZTGEZTC> . You are receiving this because you were mentioned.Message ID: <spring-projects/spring-pulsar/repo-discussions/577/comments/8573131@ github.com>

[onobc]

The sendAsync() will create a new producer per message here
https://github.com/spring-projects/spring-pulsar/blob/main/spring-pulsar/src/main/java/org/springframework/pulsar/core/PulsarTemplate.java#L236
and here
https://github.com/spring-projects/spring-pulsar/blob/main/spring-pulsar/src/main/java/org/springframework/pulsar/core/PulsarTemplate.java#L289
.

The producer's are actual cached as the "producerFactory" in your link above is a CachingPulsarProducerFactory by default.
I will double-check that the customizers are being properly included in the cache key and that it is caching as expected.

So i guess the question is how can we globally customize the producer
properties of the pulsar template (enable compression, batching, etc.) and
not per message.

Any spring.pulsar.producer.* properties that are supported by Spring Boot (via PulsarProperties config props) can be set in your application properties. Note that Spring Boot only supports a subset of these properties by design. We wanted to add properties as they are requested by the community rather than add everything.

This section in the Spring Boot ref docs talks about these properties and also the customizers you can register globally.

The PulsarTemplate relies on a PulsarProducerFactory to create the underlying Pulsar producer. Spring Boot auto-configuration also provides this producer factory, which by default, caches the producers that it creates. You can configure the producer factory and cache settings by specifying any of the spring.pulsar.producer.* and spring.pulsar.producer.cache.* prefixed application properties.

If you need more control over the producer factory configuration, consider registering one or more ProducerBuilderCustomizer beans. These customizers are applied to all created producers. You can also pass in a ProducerBuilderCustomizer when sending a message to only affect the current producer.

I will follow up once I double-check that this particular combination of the customizers is in fact being cached properly.

Thanks

[onobc]

I have verified that the producer is being cached and is only created once in both single and batch cases. The customizers are taken into account as part of the cache key. We have numerous tests in place for this case so I did not expect this to be the issue.

@chaitanya-guttula-collibra if you could please do these 3 things and report back that would be greatly appreciated.

Send user message

Let's focus on the single "send/example" endpoint (both Spring Pulsar and Native) but add the actual User object to the payload in the Spring Pulsar app.

Enable debug log for caching producer factory

In the Spring Pulsar app please enable debugging logging for the caching producer factory (add this to your yml file):

logging.level:
  org.springframework.pulsar.core.CachingPulsarProducerFactory: debug

and then re-run and look for log statements containing evicted from cache due to.

Possibly configure cache

There should only be 1 entry in the cache. However, if you see evictions (debug log above) then adjust the cache properties accordingly.

The producer cache has the following defaults:

/**
 * Time period to expire unused entries in the cache.
 */
private Duration expireAfterAccess = Duration.ofMinutes(1);

/**
 * Maximum size of cache (entries).
 */
private long maximumSize = 1000L;


/**
 * Initial size of cache.
 */
private int initialCapacity = 50;

but they can be configured via:

spring.pulsar.producer.cache:
  expire-after-access: 1m
  maximum-size: 1000
  initial-capacity: 50

Thanks,
Chris

[chaitanya-guttula-collibra]

Thank you Chris for your response. I am so sorry about the miss on not sending message with native pulsar producer, I might have overlooked it. Please see my finding that you requested.

Enable debug log for caching producer factory

I enabled the logging and I did see only one cache eviction statement for spring pulsar template after completion of the run.

Send user message

I sent the message for both spring pulsar template and native pulsar producer and with new run both spring pulsar and native producer seems to be performing similarly.

100k messages - non partitioned topic

• Native pulsar
  • Time taken 1 min - 60s
  • Throughput - 1666 msg/sec
• Spring Pulsar Template
  • Time taken - 1 min 1s - 61
  • Throughput - 1639 msgs/sec

100k messages - 6 partitioned topic

• Native pulsar
  • Time taken 1 min - 60s
  • Throughput - 1666 msg/sec
• Spring Pulsar Template
  • Time taken - 1 min 1s - 61
  • Throughput - 1639 msgs/sec

1 million messages - 6 partitioned topic

• Native pulsar
  • Time taken 9 min 42 s - 582s
  • Throughput - 1718 msg/sec
• Spring Pulsar Template
  • Time taken - 9 min 44s - 584
  • Throughput - 1712 msgs/sec

[onobc]

No worries @chaitanya-guttula-collibra . This is exactly what we saw in our previous perf. tests. Glad it is still behaving the same. We are going to set up a scheduled performance test to make sure no regressions creep in over time.

Thanks for brining this up and let us know if you have any further questions.

[onobc]

The outcome of the discussion was the the PulsarTemplate was performing on par w/ the native Pulsar producer. Closing as resolved.