KAFKA-18206: EmbeddedKafkaCluster must set features

[brandboat]

related to KAFKA-18206, set features in EmbeddedKafkaCluster in both streams and connect module, note that this PR also fix potential transaction with empty records in sendPrivileged method as transaction version 2 doesn't allow this kind of scenario.

Committer Checklist (excluded from commit message)

• Verify design and implementation
• Verify test coverage and CI build status
• Verify documentation (including upgrade notes)

[mjsax]

note that this PR also fix potential transaction with empty records in sendPrivileged method as transaction version 2 doesn't allow this kind of scenario.

Should we split this into it's own PR?

[brandboat]

Should we split this into it's own PR?

Hi @mjsax, sorry I forgot to mention that this was handling in a different issue, here is the PR: #18448

[jolshan]

We have merged 18448 so we should be able to proceed here :)

[brandboat]

I'm not sure why, but it seems like a lot of tests have become more flaky than before in this PR. One failed test in CI (https://github.com/apache/kafka/actions/runs/12810813472) caught my attention: MirrorConnectorsIntegrationTransactionsTest #testOffsetTranslationBehindReplicationFlow. The error message is:

java.lang.RuntimeException: java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.InvalidTxnStateException: The producer attempted a transactional operation in an invalid state.

I expected this failed test should to be mitigated after merging PR #18448, so I'll spend some time investigating this.

[brandboat]

This #18189 (comment) seems due to TV_2 return incorrect error, and should be handled correctly in #18604.

[jolshan]

@brandboat I just merged my PR and will restart the tests here. You may need to rebase, but we can try as is for now.