v1.9.0

confluent-kafka-python v1.9.0

v1.9.0 is a feature release:

• OAUTHBEARER OIDC support
• KIP-140 Admin API ACL support

Fixes

• The warnings for use.deprecated.format (introduced in v1.8.2) had its logic reversed, which result in warning logs to be emitted when the property was correctly configured, and the log message itself also contained text that had it backwards. The warning is now only emitted when use.deprecated.format is set to the old legacy encoding (True). #1265
• Use str(Schema) rather than Schema.to_json to prevent fastavro from raising exception TypeError: unhashable type: 'mappingproxy'. (@ffissore, #1156, #1197)
• Fix the argument order in the constructor signature for AvroDeserializer/Serializer: the argument order in the constructor signature for AvroDeserializer/Serializer was altered in v1.6.1, but the example is not changed yet. (@DLT1412, #1263)
• Fix the json deserialization errors from _schema_loads for valid primitive declarations. (@dylrich, #989)

confluent-kafka-python is based on librdkafka v1.9.0, see the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.

v1.8.2

confluent-kafka-python v1.8.2

v1.8.2 is a maintenance release with the following fixes and enhancements:

• IMPORTANT: Added mandatory use.deprecated.format to ProtobufSerializer and ProtobufDeserializer.
  See Upgrade considerations below for more information.
• Python 2.7 binary wheels are no longer provided.
  Users still on Python 2.7 will need to build confluent-kafka from source and install librdkafka separately, see README.md for build instructions.
• Added use.latest.version and skip.known.types (Protobuf) to the Serializer classes. (Robert Yokota, #1133).
• list_topics() and list_groups() added to AdminClient.
• Added support for headers in the SerializationContext (Laurent Domenech-Cabaud)
• Fix crash in header parsing (Armin Ronacher, #1165)
• Added long package description in setuptools (Bowrna, #1172).
• Documentation fixes by Aviram Hassan and Ryan Slominski.
• Don't raise AttributeError exception when CachedSchemaRegistryClient constructor raises a valid exception.

confluent-kafka-python is based on librdkafka v1.8.2, see the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.

Note: There were no v1.8.0 and v1.8.1 releases.

Upgrade considerations

Protobuf serialization format changes

Prior to this version the confluent-kafka-python client had a bug where nested protobuf schemas indexes were incorrectly serialized, causing incompatibility with other Schema-Registry protobuf consumers and producers.

This has now been fixed, but since the old defect serialization and the new correct serialization are mutually incompatible the user of confluent-kafka-python will need to make an explicit choice which serialization format to use during a transitory phase while old producers and consumers are upgraded.

The ProtobufSerializer and ProtobufDeserializer constructors now both take a (for the time being) configuration dictionary that requires
the use.deprecated.format configuration property to be explicitly set.

Producers should be upgraded first and as long as there are old (<=v1.7.0) Python consumers reading from topics being produced to, the new (>=v1.8.2) Python producer must be configured with use.deprecated.format set to True.

When all existing messages in the topic have been consumed by older consumers the consumers should be upgraded and both new producers and the new consumers must set use.deprecated.format to False.

The requirement to explicitly set use.deprecated.format will be removed in a future version and the setting will then default to False (new format).

v1.7.0

v1.7.0 is a maintenance release with the following fixes and enhancements:

• Add error_cb to confluent_cloud.py example (#1096).
• Clarify that doc output varies based on method (@slominskir, #1098).
• Docs say Schema when they mean SchemaReference (@slominskir, #1092).
• Add documentation for NewTopic and NewPartitions (#1101).

confluent-kafka-python is based on librdkafka v1.7.0, see the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.

v1.6.1

v1.6.1

v1.6.1 is a feature release:

• KIP-429 - Incremental consumer rebalancing support.
• OAUTHBEARER support.

Fixes

• Add return_record_name=True to AvroDeserializer (@slominskir, #1028)
• Fix deprecated schema.Parse call (@casperlehmann, #1006).
• Make reader schema optional in AvroDeserializer (@97nitt, #1000).
• Add **kwargs to legacy AvroProducer and AvroConsumer constructors to
  support all Consumer and Producer base class constructor arguments, such
  as logger (@venthur, #699).
• Add bool for permanent schema delete (@slominskir, #1029).
• The avro package is no longer required for Schema-Registry support (@jaysonsantos, #950).
• Only write to schema cache once, improving performance (@fimmtiu, #724).
• Improve Schema-Registry error reporting (@jacopofar, #673).
• producer.flush() could return a non-zero value without hitting the specified timeout.

confluent-kafka-python is based on librdkafka v1.6.1, see the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.

v1.6.0

v1.6.0

v1.6.0 is a feature release with the following features, fixes and enhancements:

• Bundles librdkafka v1.6.0 which adds support for Incremental rebalancing,
  Sticky producer partitioning, Transactional producer scalabilty improvements,
  and much much more. See link to release notes below.
• Rename asyncio.py example to avoid circular import (#945)
• The Linux wheels are now built with manylinux2010 (rather than manylinux1)
  since OpenSSL v1.1.1 no longer builds on CentOS 5. Older Linux distros may
  thus no longer be supported, such as CentOS 5.
• The in-wheel OpenSSL version has been updated to 1.1.1i.
• Added Message.latency() to retrieve the per-message produce latency.
• Added trove classifiers.
• Consumer destructor will no longer trigger consumer_close(),
  consumer.close() must now be explicitly called if the application
  wants to leave the consumer group properly and commit final offsets.
• Fix PY_SSIZE_T_CLEAN warning
• Move confluent_kafka/ to src/ to avoid pytest/tox picking up the local dir
• Added producer.purge() to purge messages in-queue/flight (@peteryin21, #548)
• Added AdminClient.list_groups() API (@messense, #948)
• Rename asyncio.py example to avoid circular import (#945)

confluent-kafka-python is based on librdkafka v1.6.0, see the librdkafka release notes
for a complete list of changes, enhancements, fixes and upgrade considerations.

v1.5.0

Confluent's Python client for Apache Kafka

v1.5.0

v1.5.0 is a maintenance release with the following fixes and enhancements:

• Bundles librdkafka v1.5.0 - see release notes for all enhancements and fixes.
• Documentation fixes
• Dockerfile examples
• List offsets example

confluent-kafka-python is based on librdkafka v1.5.0, see the librdkafka release notes for a complete list of changes, enhancements, fixes and upgrade considerations.

v1.4.2

Confluent's Python client for Apache Kafka

v1.4.2 is a maintenance release with the following fixes and enhancements:

• Fix produce/consume hang after partition goes away and comes back,
  such as when a topic is deleted and re-created (regression in v1.3.0).
• Consumer: Reset the stored offset when partitions are un-assign()ed.
  This fixes the case where a manual offset-less commit() or the auto-committer
  would commit a stored offset from a previous assignment before
  a new message was consumed by the application
• Seed random number generator used for random broker selection.
• Multiple calls to Consumer.close will not raise RunTimeError (@mkmoisen , #678)
• Module lookup failures raise ImportError (@overstre #786)
• Fix SchemaRegistryClient basic auth configuration (@blown302 #853)
• Don't send empty credentials to SR in Authorization Header (#863)
• miscellaneous test cleanup and enhancements (@nicht #843 ,#863)

confluent-kafka-python is based on librdkafka v1.4.2, see the librdkafka v1.4.2 release notes for a complete list of changes, enhancements, fixes and upgrade considerations.

v1.4.0

Confluent's Python client for Apache Kafka

v1.4.0 is a feature release:

• KIP-98: Transactional Producer API
• KIP-345: Static consumer group membership (by @rnpridgeon)
• KIP-511: Report client software name and version to broker
• Generic Serde API (experimental)
• New AvroSerializer and AvroDeserializer implementations including configurable subject name strategies.
• JSON Schema support (For Schema Registry)
• Protobuf support (For Schema Registry)

confluent-kafka-python is based on librdkafka v1.4.0, see the librdkafka v1.4.0 release notes for a complete list of changes, enhancements, fixes and upgrade considerations.

Transactional Producer API

Release v1.4.0 for confluent-kafka-python adds complete Exactly-Once-Semantics (EOS) functionality, supporting the idempotent producer (since v1.0.0), a transaction-aware consumer (since v1.2.0) and full producer transaction support (v1.4.0).

This enables developers to create Exactly-Once applications with Apache Kafka.

See the Transactions in Apache Kafka page for an introduction and check the transactions example.

Generic Serializer API

Release v1.4.0 introduces a new, experimental, API which adds serialization capabilities to Kafka Producer and Consumer. This feature provides the ability to configure Producer/Consumer key and value serializers/deserializers independently. Previously all serialization must be handled prior to calling Producer.produce and after Consumer.poll.

This release ships with 3 built-in, Java compatible, standard serializer and deserializer classes:

Name	Type	Format
Double	float	IEEE 764 binary64
Integer	int	int32
String	Unicode	bytes*

* The StringSerializer codec is configurable and supports any one of Python's standard encodings. If left unspecified 'UTF-8' will be used.

Additional serialization implementations are possible through the extension of the Serializer and Deserializer base classes.

See avro_producer.py and avro_consumer.py for example usage.

Avro, Protobuf and JSON Schema Serializers

Release v1.4.0 for confluent-kafka-python adds support for two new Schema Registry serialization formats with its Generic Serialization API; JSON and Protobuf. A new set of Avro Serialization classes have also been added to conform to the new API.

Format	Serializer Example	Deserializer Example
Avro	avro_producer.py	avro_consumer.py
JSON	json_producer.py	json_consumer.py
Protobuf	protobuf_producer.py	protobuf_consumer.py

Security fixes

Two security issues have been identified in the SASL SCRAM protocol handler:

• The client nonce, which is expected to be a random string, was a static string.
• If sasl.username and sasl.password contained characters that needed escaping, a buffer overflow and heap corruption would occur. This was protected, but too late, by an assertion.

Both of these issues are fixed in this release.

Enhancements

• Bump OpenSSL to v1.0.2u
• Bump monitoring-interceptors to v0.11.3

Fixes

General:

• Remove unused variable from README example (@qzyse2017, #691)
• Add delivery report to Avro example (#742)
• Update asyncio example companion blog URL (@filippovitale, #760)

Schema Registry/Avro:

• Trim trailing / from Schema Registry base URL (@IvanProdaiko94 , #749)
• Make automatic Schema registration optional (@dpfeif, #718)
• Bump Apache Avro to 1.9.2[.1] (@RyanSkraba, #779)
• Correct the SchemaRegistry authentication for SASL_INHERIT (@abij, #733)

Also see the librdkafka v1.4.0 release notes for fixes to the underlying client implementation.

v1.3.0

Confluent's Python client for Apache Kafka

confluent-kafka-python is based on librdkafka v1.3.0, see the librdkafka v1.3.0 release notes for a complete list of changes, enhancements, fixes and upgrade considerations.

This is a feature release adding support for KIP-392 Fetch from follower, allowing a consumer to fetch messages from the closest replica to increase throughput and reduce cost.

Features

• KIP-392 - Fetch messages from closest replica / follower (by @mhowlett)
• Python 3.8 binary wheel support for OSX and Linux. Windows Python 3.8 binary wheels are not currently available.

Enhancements

• New example using python3 and asyncio (by @mhowlett)
• Add warnings for inconsistent security configuration.
• Optimizations to hdr histogram (stats) rollover.
• Print compression type per message-set when debug=msg
• Various doc fixes, updates and enhancements (@edenhill , @mhowlett)

Fixes

• Fix crash when new topic is not created. (@mostafa Razavi,#725)
• Fix stringer/repr for SerializerError class(@ferozed, #675)
• Fix consumer_lag in stats when consuming from broker versions <0.11.0.0 (regression in librdkafka v1.2.0).
• Properly handle new Kafka-framed SASL GSSAPI frame semantics on Windows (#2542). 
This bug was introduced in v1.2.0 and broke GSSAPI authentication on Windows.
• Fix msgq (re)insertion code to avoid O(N^2) insert sort operations on retry (#2508).
 The msgq insert code now properly handles interleaved and overlapping message range inserts, which may occur during Producer retries for
 high-throughput applications.
• Fix producer insert msgq regression in v1.2.1 (#2450).
• Upgrade builtin lz4 to 1.9.2 (CVE-2019-17543, #2598).
• Don't trigger error when broker hostname changes (#2591).
• Less strict message.max.bytes check for individual messages (#993).
• Don't call timespec_get() on OSX (since it was removed in recent XCode) by @maparent .
• LZ4 is available from ProduceRequest 0, not 3 (fixes assert in #2480).
• Address 12 code issues identified by Coverity static code analysis.

v1.2.0

Confluent's Python client for Apache Kafka

confluent-kafka-python is based on librdkafka v1.2.0, see the librdkafka v1.2.0 release notes for a complete list of changes, enhancements, fixes and upgrade considerations.

• Transaction aware consumer (isolation.level=read_committed) implemented by @mhowlett.
• Sub-millisecond buffering (linger.ms) on the producer.
• Improved authentication errors (KIP-152)

Consumer-side transaction support

This release adds consumer-side support for transactions.
In previous releases, the consumer always delivered all messages to the application, even those in aborted or not yet committed transactions. In this release, the consumer will by default skip messages in aborted transactions.
This is controlled through the new isolation.level configuration property which
defaults to read_committed (only read committed messages, filter out aborted and not-yet committed transactions), to consume all messages, including for aborted transactions, you may set this property to read_uncommitted to get the behaviour of previous releases.
For consumers in read_committed mode, the end of a partition is now defined to be the offset of the last message of a successfully committed transaction (referred to as the 'Last Stable Offset').
For non-transactional messages there is no change from previous releases, they will always be read, but a consumer will not advance into a not yet committed transaction on the partition.

Upgrade considerations

• linger.ms default was changed from 0 to 0.5 ms to promote some level of batching even with default settings.

New configuration properties

• Consumer property isolation.level=read_committed ensures the consumer will only read messages from successfully committed producer transactions. Default is read_committed. To get the previous behaviour, set the property to read_uncommitted, which will read all messages produced to a topic, regardless if the message was part of an aborted or not yet committed transaction.

Enhancements

• Cache FastAvro schema for improved Avro Serialization/Deserialization (@BBM89, #627)
• Protocol decoding optimization, increasing consume performance.
• Add CachedSchemaRegistry docs (@lowercase24 , #495)

Fixes

General:

• Rate limit IO-based queue wakeups to linger.ms, this reduces CPU load and lock contention for high throughput producer applications. (#2509)
• SSL: Use only hostname (not port) when valid broker hostname (by Hunter Jacksson)
• SSL: Ignore OpenSSL cert verification results if enable.ssl.certificate.verification=false (@salisbury-espinosa)
• SASL Kerberos/GSSAPI: don't treat kinit ECHILD errors as errors (@hannip)
• Refresh broker list metadata even if no topics to refresh (#2476)
• Correct AdminClient doc (@lowercase24, #653)
• Update Avro example to be compliant with csh (@andreyferriyan , #668)
• Correct Avro example typo (@AkhilGNair, #598)

Consumer:

• Make pause|resume() synchronous, ensuring that a subsequent poll() will not return messages for the paused partitions.
• Consumer doc fixes (@hrchu , #646, #648)

Producer:

• Fix message timeout handling for leader-less partitions.
• message.timeout.ms=0 is now accepted even if linger.ms > 0 (by Jeff Snyder)