Zilla and Volt Active Data (VoltDB) TOPICS Integration

[biplabvoltdb]

When exploring Zilla and Volt Topic integration, I ran around and issue where I was not able to send messages from Zilla to Volt topics.

I am using Volt's Opaque TOPIC

The errors in the Volt Topics side were below :

2024-04-12 09:24:30,868   TRACE [nioEventLoopGroup-3-2] TOPICS: Received request from /127.0.0.1:63984: RequestHeaderData(requestApiKey=32, requestApiVersion=0, correlationId=0, clientId='zilla') DescribeConfigsRequestData(resources=[DescribeConfigsResource(resourceType=2, resourceName='mqtt-sessions', configurationKeys=[cleanup.policy, max.message.bytes, segment.bytes, segment.index.bytes, segment.ms, retention.bytes, retention.ms, delete.retention.ms, min.compaction.lag.ms, max.compaction.lag.ms, min.cleanable.dirty.ratio])], includeSynonyms=false, includeDocumentation=false)

2024-04-12 09:24:30,870   WARN  [nioEventLoopGroup-3-2] TOPICS: DESCRIBE_CONFIGS on mqtt-sessions (2) requests keys [cleanup.policy, max.message.bytes, segment.bytes, segment.index.bytes, segment.ms, retention.bytes, retention.ms, delete.retention.ms, min.compaction.lag.ms, max.compaction.lag.ms, min.cleanable.dirty.ratio]

2024-04-12 09:24:30,873   TRACE [nioEventLoopGroup-3-2] TOPICS: Responding to /127.0.0.1:63984 clientId='zilla' with: ResponseHeaderData(correlationId=0) DescribeConfigsResponseData(throttleTimeMs=0, results=[DescribeConfigsResult(errorCode=0, errorMessage='', resourceType=2, resourceName='mqtt-sessions', configs=[DescribeConfigsResourceResult(name='cleanup.policy', value='compact', readOnly=false, isDefault=false, configSource=1, isSensitive=false, synonyms=[], configType=0, documentation='')])])

2024-04-12 09:24:30,880   TRACE [nioEventLoopGroup-3-2] TOPICS: Received request from /127.0.0.1:63984: RequestHeaderData(requestApiKey=3, requestApiVersion=5, correlationId=1, clientId='zilla') MetadataRequestData(topics=[MetadataRequestTopic(topicId=AAAAAAAAAAAAAAAAAAAAAA, name='mqtt-retained')], allowAutoTopicCreation=false, includeClusterAuthorizedOperations=false, includeTopicAuthorizedOperations=false)

2024-04-12 09:24:30,983   TRACE [nioEventLoopGroup-3-2] TOPICS: Responding to /127.0.0.1:63984 clientId='zilla' with: ResponseHeaderData(correlationId=1) MetadataResponseData(throttleTimeMs=0, brokers=[MetadataResponseBroker(nodeId=0, host='BIPLABs-MacBook-Pro.local', port=9092, rack=null)], clusterId='database', controllerId=0, topics=[MetadataResponseTopic(errorCode=0, name='mqtt-retained', topicId=AAAAAAAAAAAAAAAAAAAAAA, isInternal=false, partitions=[MetadataResponsePartition(errorCode=0, partitionIndex=0, leaderId=-1, leaderEpoch=-1, replicaNodes=[0], isrNodes=[0], offlineReplicas=[])], topicAuthorizedOperations=-2147483648)], clusterAuthorizedOperations=-2147483648)

As you can see above Volt's topic is receiving the request and responding a bunch of information back to the Zilla's Kafka producer client.
The question is what is the Zilla's Kafka producer client expecting as a response and why is Client not happy with the response messages, so that Volt Topics can receive the Event sent.

I was using the below to try and test the integration : https://docs.aklivity.io/zilla/latest/concepts/kafka-proxies/http-proxy.html

My ultimate goal is to use the MQTT to Kafka proxy, but since that didn't turn out to work I tried the above HTTP PROXY and it seems even that has a problem sending the messages to VOLT TOPICS.

Request you to kindly help me answer and help me fix the issue.

I have been taking to AJ at Zilla to get some help and he has some context about my problem, please reach out to him incase I am not able to respond on time.

Document Link about Volt Topics : https://docs.voltactivedata.com/UsingVoltDB/exporttopics.php#:~:text=VoltDB%20topics%20are%20composed%20of,optional%20set%20of%20keys%20values.

[jfallows]

Thanks for posting this @biplabvoltdb I spoke with AJ to get the context too.

Can you please provide your setup and config steps to reproduce so we can verify the behavior locally to diagnose?

Many thanks in advance.

[biplabvoltdb]

Hey John,
Thanks for responding.
Volts Topic feature is not opensourced.
So if you want Volt's setup I will have to arrange a trail license for you.
It might take a few days.
I will you the setup as soon as possible after an internal discussion with Volt side.

[biplabvoltdb]

Would it be ok if i can have the setup running on aws ?

[jfallows]

No problem @biplabvoltdb accesing VoltDB running on AWS should work great for us to diagnose!

[biplabvoltdb]

Hi John,
I have created a setup on GCP. Would be able to ping me on slack one to one so that I can share the credentials with you

[jfallows]

Thanks @biplabvoltdb the GCP testing has been very helpful.

Please review #970 to confirm that meets your requirements for MQTT QoS 1 publish.