How Can I Reduce Delay When Sending File Contents to Kafka Producer Using Script?

Question

I want to write a bash script to constantly add new files to a folder and use cat to send the file contents to a Kafka producer with:

    cat $FILEPATH |$KAFKA_HOME/bin/kafka-console-producer.sh --bootstrap-server localhost:9092 --topic test

But it generally takes more than 4 seconds to finish the execution. I am wondering if there's any way to write a script to automatically add new messages to a Kafka producer with less delay.

I also tried to echo the contents of the file to the producer instead, but the delay still exists. So, I think it has something to do with the producer command.

OneCricketeer · Accepted Answer · 2024-07-20 01:43:14Z

0

The JVM needs to startup, yes, but that shouldn't take very long. You could try installing kcat command instead, but you should first try without using cat

kafka-console-producer ... < $FILEPATH

Otherwise, you're running the producer command for each line of the file Useless use of cat?

automatically add new messages to a Kafka producer

This doesn't require a shell script. You can use Kafka Connect, Filebeat, Fluentbit, etc to watch a directory of files automatically and produce their lines to Kafka

answered Jul 20, 2024 at 1:43

OneCricketeer

193k20 gold badges146 silver badges276 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

lanny Over a year ago

Thanks, @OneCricketeer! I tried replacing cat with < but it still takes more than 4 sec to execute the command. What I actually want to do here is pipe my auto generated data into a Kafka topic. I tried to first add data to a Flink table using the filesystem connector, then sink the table to another table with a Kafka connector. I found that the data can be shown in a Kafka consumer in real-time. So I'm thinking about giving up on piping data directly from files since what matters is to pipe data into the topics. Do you know any other way to add auto-generated test data into a Kafka topic?

OneCricketeer Over a year ago

I don't have experience with Flink. You can try the options I listed, if you have files. If you just want to generate some random data, take a look at ShadowTraffic

Collectives™ on Stack Overflow

How Can I Reduce Delay When Sending File Contents to Kafka Producer Using Script?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related