kafka scripts in "bin" directory fail with "Address already in use" (running in kubernetes)

Question

I have a kafka set up running in minikube. It's up and running, and I'm able to produce messages into it and consume them back out.

However, when I'm logging in to one the brokers (using kubectl exec if relevant) and try to get some information about its state using one of the scripts in bin, I get an exception Address already in use:

root@kafka-0:/kafka# bin/kafka-consumer-groups.sh  --list --bootstrap-server kafka:9092
Exception in thread "main" java.lang.reflect.InvocationTargetException
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:513)
    at java.instrument/sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:525)
Caused by: java.net.BindException: Address already in use
    at java.base/sun.nio.ch.Net.bind0(Native Method)
    at java.base/sun.nio.ch.Net.bind(Net.java:461)
    at java.base/sun.nio.ch.Net.bind(Net.java:453)
    at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:227)
    at java.base/sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:80)
    at jdk.httpserver/sun.net.httpserver.ServerImpl.bind(ServerImpl.java:134)
    at jdk.httpserver/sun.net.httpserver.HttpServerImpl.bind(HttpServerImpl.java:54)
    at io.prometheus.jmx.shaded.io.prometheus.client.exporter.HTTPServer.<init>(HTTPServer.java:145)
    at io.prometheus.jmx.shaded.io.prometheus.jmx.JavaAgent.premain(JavaAgent.java:54)
    ... 6 more
FATAL ERROR in native method: processing of -javaagent failed
Aborted (core dumped)

A quick look using strace showed that it is trying to bind to port 80. I have no idea why that is even necessary for a script that is meant only to obtain data.

The same error happens on all scripts in the kafka bin directory.

Any ideas regarding on how to fix will be a great help

yuranos · Answer

Your javaagent is making use of 2 env vars which are conflicting with bin console scripts. Those env vars are KAFKA_OPTS and JMX_PORT.
What helped for me for confluent container(confluentinc/cp-server:5.5.1) is unsetting both of them when entering a running container(via docker exec)
unset KAFKA_OPTS
unset JMX_PORT

This will not impact a running broker because those vars have session scope.

Lam Hoang Viet · Answer

In my experience, the reason is that JMX exporter will be started under port that already used by Kafka server.

To double-check, you can execute command ps -ef | grep kafka | grep jmx_prometheus, then see the result whether it show like this -javaagent:./jmx_prometheus_javaagent-0.3.1.jar=9083:/usr/hdp/current/kafka-broker/prometheus/kafka.yaml.

It is not like above, please skip my post :), otherwise let's continue

Almost scripts inside "$KAFKA_HOME/bin" will load file "$KAFKA_HOME/conf/kafka-env.sh" and init env variables, one of them is "KAFKA_OPTS" - this option defines JMX port. And then you execute scripts, they will start up javaagent under JMX port ==> raise issue as the below

Caused by: java.net.BindException: Address already in use

How to fix:
I edit a little bit inside code, set KAFKA_OPTS="" before executing java command  and it works.

Abhishek Mishra · Answer

In the command instead of --bootsrap-server can you use --broker-list

kafka scripts in "bin" directory fail with "Address already in use" (running in kubernetes)

3 Answers

Add your own answers!

Ask a Question