Namespaced KafkaChannel dispatcher has RBAC issues

[aliok]

Describe the bug
As the time of writing, I only tried with 0.17.7.

When I create a namespaced KafkaChannel (i.e. with annotation eventing.knative.dev/scope: namespace), I see following:

kafka-ch-dispatcher-7cb455959f-996vq dispatcher E1106 11:16:48.094504       1 leaderelection.go:331] error retrieving resource lock knative-eventing/kafkachannel-dispatcher.knative.dev-eventing-contrib-kafka-channel-pkg-reconciler-dispatcher.reconciler.00-of-01: leases.coordination.k8s.io "kafkachannel-dispatcher.knative.dev-eventing-contrib-kafka-channel-pkg-reconciler-dispatcher.reconciler.00-of-01" is forbidden: User "system:serviceaccount:ns02:kafka-ch-dispatcher" cannot get resource "leases" in API group "coordination.k8s.io" in the namespace "knative-eventing"

Expected behavior
No error

To Reproduce

• Install Strimzi
• Create namespace config for KafkaChannel:

apiVersion: v1
kind: ConfigMap
metadata:
  name: config-kafka
  namespace: ns02
data:
  bootstrapServers: my-cluster-kafka-bootstrap.kafka:9092

• Since there's a bug that KafkaChannel doesn't get the bootstrapServers info from local configmap (see KafkaChannel doesn't respect the scope annotation when creating the topic #1189), also edit the config-kafka configmap in knative-eventing:

bootstrapServers: my-cluster-kafka-bootstrap.kafka:9092

• Create KafkaChannel:

apiVersion: messaging.knative.dev/v1beta1
kind: KafkaChannel
metadata:
  name: kafkachannel
  namespace: ns02
  annotations:
    eventing.knative.dev/scope: namespace
spec:
  numPartitions: 10
  replicationFactor: 1

• Create PingSource that sinks to the KafkaChannel:

apiVersion: sources.knative.dev/v1alpha2
kind: PingSource
metadata:
  name: source
  namespace: ns02
spec:
  schedule: "*/1 * * * *"
  jsonData: '{"message": "Thanks for doing Knative Tutorial 02"}'
  sink:
    ref:
      apiVersion: messaging.knative.dev/v1beta1
      kind: KafkaChannel
      name: kafkachannel

Knative release version
0.17.7, maybe future versions have the same problem, haven't tried yet.

Additional context
Add any other context about the problem here such as proposed priority

[lionelvillard]

The problem lies in the leader elector builder which creates leases in the system namespace. The fix in pkg is quite big due to import cycle.

The dispatcher is not meant to run in leader-election mode and AFAIK it does run in this mode by accident, not by choice. So let's turn this off sooner than later.

[aliok]

For reference:

• Fix for master: [consolidated] Disable KafkaChannel dispatcher leader election knative-extensions/eventing-kafka#173
• Fix for 0.17: disable leader election (#173) #1663