Stack Overflow Asked by Asad Iqbal on October 14, 2020
Flink-kafka-consumer has two types of consumers e.g. :
Both of these consumer hierarchies extend same FlinkKafkaConsumerBase class. I wonder what is the design decision behind maintaining the numbered classes? When should we use one or the other?
I noticed that classes with numbered suffix have some more features (E.g. ratelimiting).
EDIT: (following link has stopped working as this file was removed in master branch).
https://github.com/apache/flink/blob/master/flink-connectors/flink-connector-kafka-0.10/src/main/java/org/apache/flink/streaming/connectors/kafka/FlinkKafkaConsumer010.java#L77
The versioned Kafka consumers (and producers) are built against those versions of the Kafka client, and are intended to each be used with those specific versions of Kafka. The unversioned connectors -- FlinkKafkaConsumer and FlinkKafkaProducer -- are built using the universal client library and are compatible with all versions of Kafka since 0.10.
Note that the versioned consumers and producers for Kafka 0.8 and 0.9 were removed in Flink 1.11, and the versions for 0.10 and 0.11 will be removed in Flink 1.12 (https://issues.apache.org/jira/browse/FLINK-19152).
EDIT:
In some cases it is enough to simply allow the backpressure from the sink to throttle the source. But in other cases (e.g., multiple sources) that may not work well enough.
You will find some discussion about this, and code for doing your own rate limiting, at http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Kafka-Rate-Limit-in-FlinkConsumer-td36417.html.
Answered by David Anderson on October 14, 2020
Get help from others!
Recent Questions
Recent Answers
© 2024 TransWikia.com. All rights reserved. Sites we Love: PCI Database, UKBizDB, Menu Kuliner, Sharing RPP