[ISSUE #50] Feature add rocketmq exporter to nameserver

README.md

@@ -14,6 +14,7 @@
   - [Prepare Volume Persistence](#prepare-volume-persistence)
     - [Prepare HostPath](#prepare-hostpath)
     - [Prepare Storage Class of NFS](#prepare-storage-class-of-nfs)
+  - [Define Your RocketMQ Exporter](#define-your-rocketmq-exporter)
   - [Define Your RocketMQ Cluster](#define-your-rocketmq-cluster)
   - [Create RocketMQ Cluster](#create-rocketmq-cluster)
   - [Verify the Data Storage](#verify-the-data-storage)
@@ -153,6 +154,205 @@ nfs-client-provisioner-7cf858f754-7vxmm   1/1     Running   0          136m
 rocketmq-operator-564b5d75d-jllzk         1/1     Running   0          108s
 ```
 
+### Define Your RocketMQ Exporter 
+You can use the rocketmq exporter to get metrics information of the RocketMQ cluster. These metrics can be captured by prometheus, and can use grafana to graphically display metrics information . At the same time, when an abnormality happens, an alarm email can be triggered.
+
+1. Check the file `rocketmq_v1alpha1_rocketmq_exporter_cluster.yaml` in the example directory, for example:
+```
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: broker-config
+data:
+  # BROKER_MEM sets the broker JVM, if set to "" then Xms = Xmx = max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
+  BROKER_MEM: " -Xms2g -Xmx2g -Xmn1g "
+  broker-common.conf: |
+    # brokerClusterName, brokerName, brokerId are automatically generated by the operator and do not set it manually!!!
+    deleteWhen=04
+    fileReservedTime=48
+    flushDiskType=ASYNC_FLUSH
+    # set brokerRole to ASYNC_MASTER or SYNC_MASTER. DO NOT set to SLAVE because the replica instance will automatically be set!!!
+    brokerRole=ASYNC_MASTER
+
+---
+apiVersion: rocketmq.apache.org/v1alpha1
+kind: Broker
+metadata:
+  # name of broker cluster
+  name: broker
+spec:
+  # size is the number of the broker cluster, each broker cluster contains a master broker and [replicaPerGroup] replica brokers.
+  size: 1
+  # nameServers is the [ip:port] list of name service
+  nameServers: ""
+  # replicaPerGroup is the number of each broker cluster
+  replicaPerGroup: 0
+  # brokerImage is the customized docker image repo of the RocketMQ broker
+  brokerImage: apacherocketmq/rocketmq-broker:4.5.0-alpine-operator-0.3.0
+  # imagePullPolicy is the image pull policy
+  imagePullPolicy: Always
+  # resources describes the compute resource requirements and limits
+  resources:
+    requests:
+      memory: "2048Mi"
+      cpu: "250m"
+    limits:
+      memory: "12288Mi"
+      cpu: "500m"
+  # allowRestart defines whether allow pod restart
+  allowRestart: true
+  # storageMode can be EmptyDir, HostPath, StorageClass
+  storageMode: EmptyDir
+  # hostPath is the local path to store data
+  hostPath: /tmp/data/rocketmq/broker
+  # scalePodName is [Broker name]-[broker group number]-master-0
+  scalePodName: broker-0-master-0
+  # env defines custom env, e.g. BROKER_MEM
+  env:
+    - name: BROKER_MEM
+      valueFrom:
+        configMapKeyRef:
+          name: broker-config
+          key: BROKER_MEM
+  # volumes defines the broker.conf
+  volumes:
+    - name: broker-config
+      configMap:
+        name: broker-config
+        items:
+          - key: broker-common.conf
+            path: broker-common.conf
+  # volumeClaimTemplates defines the storageClass
+  volumeClaimTemplates:
+    - metadata:
+        name: broker-storage
+      spec:
+        accessModes:
+          - ReadWriteOnce
+        resources:
+          requests:
+            storage: 8Gi
+        selector:
+          matchLabels:
+            app: broker-storage-pv
+---
+apiVersion: rocketmq.apache.org/v1alpha1
+kind: NameService
+metadata:
+  name: name-service
+spec:
+  # size is the the name service instance number of the name service cluster
+  size: 1
+  # nameServiceImage is the customized docker image repo of the RocketMQ name service
+  nameServiceImage: apacherocketmq/rocketmq-nameserver:4.5.0-alpine-operator-0.3.0
+  # imagePullPolicy is the image pull policy
+  imagePullPolicy: Always
+  # hostNetwork can be true or false
+  hostNetwork: true
+  #  Set DNS policy for the pod.
+  #  Defaults to "ClusterFirst".
+  #  Valid values are 'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'.
+  #  DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy.
+  #  To have DNS options set along with hostNetwork, you have to specify DNS policy
+  #  explicitly to 'ClusterFirstWithHostNet'.
+  dnsPolicy: ClusterFirstWithHostNet
+  # resources describes the compute resource requirements and limits
+  resources:
+    requests:
+      memory: "512Mi"
+      cpu: "250m"
+    limits:
+      memory: "1024Mi"
+      cpu: "500m"
+  # storageMode can be EmptyDir, HostPath, StorageClass
+  storageMode: EmptyDir
+  # hostPath is the local path to store data
+  hostPath: /data/rocketmq/nameserver
+  exporter:
+    enabled: true
+    env:
+      - name: WEB_TELEMETRY_PATH
+        value: /metrics
+      - name: NAMESRV_ADDR
+        value: 127.0.0.1:9876
+    image: apacherocketmq/rocketmq-exporter:latest
+    imagePullPolicy: Always
+    resources:
+      requests:
+        memory: "512Mi"
+        cpu: "250m"
+      limits:
+        memory: "1024Mi"
+        cpu: "500m"
+  # volumeClaimTemplates defines the storageClass
+  volumeClaimTemplates:
+    - metadata:
+        name: namesrv-storage
+      spec:
+        accessModes:
+          - ReadWriteOnce
+        storageClassName: rocketmq-storage
+        resources:
+          requests:
+            storage: 1Gi
+```
+which defines the RocketMQ name server cluster and the broker cluster. We have added an optional exporter field to the NameService. Through this field, we can inject a rocketmq-exporter as a sidecar into each nameserver in the cluster. 
+
+* exporter.env contains metrics path and nameserver address
+
+The configuration of prometheus can refer to:
+```
+- job_name: general-logic-pods
+  honor_timestamps: true
+  scrape_interval: 30s
+  scrape_timeout: 10s
+  metrics_path: /metrics
+  scheme: http
+  kubernetes_sd_configs:
+  - role: pod
+  relabel_configs:
+  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
+    separator: ;
+    regex: "true"
+    replacement: $1
+    action: keep
+  - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
+    separator: ;
+    regex: (.+)
+    target_label: __metrics_path__
+    replacement: $1
+    action: replace
+  - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
+    separator: ;
+    regex: ([^:]+)(?::\d+)?;(\d+)
+    target_label: __address__
+    replacement: $1:$2
+    action: replace
+  - source_labels: [__meta_kubernetes_pod_host_ip]
+    separator: ;
+    regex: (.*)
+    target_label: instance
+    replacement: $1:9100
+    action: replace
+  - separator: ;
+    regex: __meta_kubernetes_pod_label_(.+)
+    replacement: $1
+    action: labelmap
+  - source_labels: [__meta_kubernetes_namespace]
+    separator: ;
+    regex: (.*)
+    target_label: kubernetes_namespace
+    replacement: $1
+    action: replace
+  - source_labels: [__meta_kubernetes_pod_name]
+    separator: ;
+    regex: (.*)
+    target_label: kubernetes_pod_name
+    replacement: $1
+    action: replace
+```
+
+
 ### Define Your RocketMQ Cluster
 
 RocketMQ Operator provides several CRDs to allow users define their RocketMQ service component cluster, which includes the Name Server cluster and the Broker cluster.

deploy/crds/rocketmq_v1alpha1_nameservice_crd.yaml

@@ -63,6 +63,9 @@ spec:
               items:
                 type: object
               type: array
+            exporter:
+              description: Exporter defines rocketmq metrics
+              type: object
           required:
           - size
           - nameServiceImage

example/rocketmq_v1alpha1_rocketmq_exporter_cluster.yaml

@@ -0,0 +1,157 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements.  See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License.  You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: broker-config
+data:
+  # BROKER_MEM sets the broker JVM, if set to "" then Xms = Xmx = max(min(1/2 ram, 1024MB), min(1/4 ram, 8GB))
+  BROKER_MEM: " -Xms2g -Xmx2g -Xmn1g "
+  broker-common.conf: |
+    # brokerClusterName, brokerName, brokerId are automatically generated by the operator and do not set it manually!!!
+    deleteWhen=04
+    fileReservedTime=48
+    flushDiskType=ASYNC_FLUSH
+    # set brokerRole to ASYNC_MASTER or SYNC_MASTER. DO NOT set to SLAVE because the replica instance will automatically be set!!!
+    brokerRole=ASYNC_MASTER
+
+---
+apiVersion: rocketmq.apache.org/v1alpha1
+kind: Broker
+metadata:
+  # name of broker cluster
+  name: broker
+spec:
+  # size is the number of the broker cluster, each broker cluster contains a master broker and [replicaPerGroup] replica brokers.
+  size: 1
+  # nameServers is the [ip:port] list of name service
+  nameServers: ""
+  # replicaPerGroup is the number of each broker cluster
+  replicaPerGroup: 0
+  # brokerImage is the customized docker image repo of the RocketMQ broker
+  brokerImage: apacherocketmq/rocketmq-broker:4.5.0-alpine-operator-0.3.0
+  # imagePullPolicy is the image pull policy
+  imagePullPolicy: Always
+  # resources describes the compute resource requirements and limits
+  resources:
+    requests:
+      memory: "2048Mi"
+      cpu: "250m"
+    limits:
+      memory: "12288Mi"
+      cpu: "500m"
+  # allowRestart defines whether allow pod restart
+  allowRestart: true
+  # storageMode can be EmptyDir, HostPath, StorageClass
+  storageMode: EmptyDir
+  # hostPath is the local path to store data
+  hostPath: /tmp/data/rocketmq/broker
+  # scalePodName is [Broker name]-[broker group number]-master-0
+  scalePodName: broker-0-master-0
+  # env defines custom env, e.g. BROKER_MEM
+  env:
+    - name: BROKER_MEM
+      valueFrom:
+        configMapKeyRef:
+          name: broker-config
+          key: BROKER_MEM
+  # volumes defines the broker.conf
+  volumes:
+    - name: broker-config
+      configMap:
+        name: broker-config
+        items:
+          - key: broker-common.conf
+            path: broker-common.conf
+  # volumeClaimTemplates defines the storageClass
+  volumeClaimTemplates:
+    - metadata:
+        name: broker-storage
+      spec:
+        accessModes:
+          - ReadWriteOnce
+        resources:
+          requests:
+            storage: 8Gi
+        selector:
+          matchLabels:
+            app: broker-storage-pv
+---
+apiVersion: rocketmq.apache.org/v1alpha1
+kind: NameService
+metadata:
+  name: name-service
+spec:
+  # size is the the name service instance number of the name service cluster
+  size: 1
+  # nameServiceImage is the customized docker image repo of the RocketMQ name service
+  nameServiceImage: apacherocketmq/rocketmq-nameserver:4.5.0-alpine-operator-0.3.0
+  # imagePullPolicy is the image pull policy
+  imagePullPolicy: Always
+  # hostNetwork can be true or false
+  hostNetwork: true
+  #  Set DNS policy for the pod.
+  #  Defaults to "ClusterFirst".
+  #  Valid values are 'ClusterFirstWithHostNet', 'ClusterFirst', 'Default' or 'None'.
+  #  DNS parameters given in DNSConfig will be merged with the policy selected with DNSPolicy.
+  #  To have DNS options set along with hostNetwork, you have to specify DNS policy
+  #  explicitly to 'ClusterFirstWithHostNet'.
+  dnsPolicy: ClusterFirstWithHostNet
+  # resources describes the compute resource requirements and limits
+  resources:
+    requests:
+      memory: "512Mi"
+      cpu: "250m"
+    limits:
+      memory: "1024Mi"
+      cpu: "500m"
+  # storageMode can be EmptyDir, HostPath, StorageClass
+  storageMode: EmptyDir
+  # hostPath is the local path to store data
+  hostPath: /data/rocketmq/nameserver
+  exporter:
+    enabled: true
+    annotations:
+      prometheus.io/path: /metrics
+      prometheus.io/port: "5557"
+      prometheus.io/scrape: "true"
+    env:
+      - name: WEB_TELEMETRY_PATH
+        value: /metrics
+      - name: ROCKETMQ_VERSION
+        value: V4_3_2
+      - name: NAMESRV_ADDR
+        value: 127.0.0.1:9876
+    image: miaolinjie/rocketmq-exporter:latest
+    imagePullPolicy: Always
+    resources:
+      requests:
+        memory: "512Mi"
+        cpu: "250m"
+      limits:
+        memory: "1024Mi"
+        cpu: "500m"
+  # volumeClaimTemplates defines the storageClass
+  volumeClaimTemplates:
+    - metadata:
+        name: namesrv-storage
+      spec:
+        accessModes:
+          - ReadWriteOnce
+        storageClassName: rocketmq-storage
+        resources:
+          requests:
+            storage: 1Gi