Skip to content

Latest commit

 

History

History
47 lines (38 loc) · 3.53 KB

README.md

File metadata and controls

47 lines (38 loc) · 3.53 KB

Kafka Connect Weaviate Connector

kafka-connect-weaviate is a Kafka Connector sink connector for loading data to any Weaviate cluster.

This is currently a Work In Progress connector, use it at your own risk.

Configuration

Parameter Description Default value
weaviate.connection.url "Weaviate connection URL, should be following the format <scheme>://<host>:<port> http://localhost:8080
weaviate.grpc.url Weaviate GRPC connection URL localhost:50051
weaviate.grpc.secure Weaviate GRPC TLS secured connection, set to True to enable TLS encryption false
weaviate.auth.scheme Authentication mechanism to use to connect to Weaviate, could be NONE, API_KEY or OIDC_CLIENT_CREDENTIALS NONE
weaviate.api.key User API Key if API Key authentication mechanism is used
weaviate.oidc.client.secret User OIDC client secret if OIDC authentication mechanism is used
weaviate.oidc.scopes OIDC client scope if OIDC authentication mechanism is used
collection.mapping Mapping between Kafka topic and Weaviate collection ${topic}
document.id.strategy Java class returning the document ID for each record io.weaviate.connector.idstrategy.NoIdStrategy
vector.strategy Java class returning the document embedding for each record io.weaviate.connector.vectorstrategy.NoVectorStrategy
vector.field.name Field name containing the embedding (used only for FieldVectorStrategy)

Example of configuration

{
  "connector.class": "io.weaviate.connector.WeaviateSinkConnector",
  "topics": "test",
  "weaviate.connection.url": "http://weaviate:8080",
  "weaviate.grpc.url": "weaviate:50051",
  "collection.mapping": "weaviate_${topic}",
  "value.converter": "org.apache.kafka.connect.json.JsonConverter",
  "value.converter.schemas.enable": false
}

Quickstart

docker-compose up -d
sleep 5
python examples/create_collection.py weaviate_test
curl -i -X PUT localhost:8083/connectors/weaviate/config -H 'Content-Type:application/json' --data @examples/weaviate-simple-sink.json
echo '{"string": "Hello World", "number": 1.23 }' |  docker-compose exec -T kafka kafka-console-producer --bootstrap-server localhost:9092 --topic test