Install Kafka on Linux 2016-03-08 06:19
Kafka is a high-throughput distributed publish-subscribe messaging system that can handle real-time data feed. In this series tutorials I will show you how to use Kafka step by step. The first one is how to install Kafka on Linux. This tutorial assumes there is no existing Kafka or ZooKeeper data on your computer.
Download Kafka and un-tar it. (when I write this blog 0.9 is the latest)
tar -xzf kafka_2.11-0.9.0.0.tgz cd kafka_2.11-0.9.0.0
Start Zookeeper and Kafka
Zookeeper is required for running Kafka. If you do not install Zookeeper you can use the convenience script packaged with kafka to get a quick-and-dirty single-node ZooKeeper instance. Open a terminal and type the command as bellow.
Open the second terminal type the command as bellow to start Kafka
My virtual machine is 1G RAM, I get following error message
Java HotSpot(TM) 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000bae00000, 1073741824, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 1073741824 bytes for committing reserved memory. # An error report file with more information is saved as: # /root/kafka_2.11-0.9.0.1/hs_err_pid5976.log
I change the maximum heap size (-Xmx)
export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M"
run Kafka again, success;)
Create a topic
Open the third terminal to create a topick named
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
Kafka save log in different server by
partitions parameter. You can divide a topic in multi-partition. The more partitions in your topic the more consumers can use.
Send some messages
we can send some message in command line client.
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test This is a message This is another message
Start a consumer
we can also use command line consumer to get the message. Open the fourth terminal and type the command.
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
now you can see the message.
This is a message This is another message
Type some words in the third terminal you can get them in fourth terminal. There is a sample producer(the third terminal) and a consumer(the fourth terminal) in this demo.