问题描述

使用 Apache Flink 连接支持 Apache Kafka的Azure Event Hub后，由于消费端的Consumer Group是动态创建，在门户页面和Service Bus Explorer工具中均无法查看，所以如何来查看 Kafka consumer group呢？可以参考以下内容：

 

使用前提

将 Apache Flink 与适用于 Apache Kafka 的 Azure 事件中心配合使用： https://docs.azure.cn/zh-cn/event-hubs/event-hubs-kafka-flink-tutorial

在Azure中创建一个支持Kafka的Event Hub（标准层及以上）

下载示例代码：git clone https://github.com/Azure/azure-event-hubs-for-kafka.git 

修改 producer.config 和  consumer.config

 

操作步骤

If customer has an active group, they can check consumer lag (which indicate what data has been consumed) through Kafka's own kafka-consumer-groups.sh command (https://kafka.apache.org/documentation/#basic_ops_consumer_lag)

Make sure you setup Java JDK on your machine. then download the Kafka binary downloads from (https://kafka.apache.org/downloads) - you just need the binary downloads, no need to get the source unless you intend to modify the source code that the script uses.

from command prompt, navigate to your downloaded bin directory - e.g. F:\\kafka_2.13-3.0.0\bin\windows (for windows shell script).

Create a properties file that has the connection string for your namespace. For example you can create a config.properties file that has the following.

bootstrap.servers=.servicebus.windows.net:9093

security.protocol=SASL_SSL

sasl.mechanism=PLAIN

sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required username="$ConnectionString" password="";

 

After that just run through the command as describe in https://kafka.apache.org/documentation/#basic_ops_consumer_lag to see your information. sample below. Note that currently, this only works if the group is an active group meaning it has active consumers running for the group. this command does not work for EventHub service if the group is inactive.

F:\kafka_2.13-3.0.0\bin\windows>.\kafka-consumer-groups.bat --bootstrap-server .servicebus.windows.net:9093 --command-config F:\kafka_2.13-3.0.0\bin\windows\config.properties --list

$hmlam-4

F:\kafka_2.13-3.0.0\bin\windows>.\kafka-consumer-groups.bat --bootstrap-server .servicebus.windows.net:9093 --command-config F:\kafka_2.13-3.0.0\bin\windows\config.properties --describe --group $hmlam-4

GROUP TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID

$hmlam-4 hmlam-eh-2 1 52648 52651 3 hmlam-cbn04-001.servicebus.windows.net:c:$hmlam-4:I:perf_client#1-c391de052f2446ff85fbf6899160a236 0.0.0.0 perf_client#1

$hmlam-4 hmlam-eh-2 0 52932 52946 14 hmlam-cbn04-001.servicebus.windows.net:c:$hmlam-

 

目前遇见的问题

当根据以上步骤设置好，并且同时运行了生产者代码和消费组代码，可是却无法获取到kafka group信息

 

（TODO: 问题还在进一步的研究中)

 

相关文章

评论可见，请评论后查看内容，谢谢！！！评论后请刷新页面。