bigdata-02-kafka

2018-03-01 11:24:18来源:oschina作者:moon888人点击

分享

下载:http://kafka.apache.org/downloads 安装:tar -zxvf kafka_2.11-0.9.0.0.tgz -C ~/app


配置环境变量vi ~/.bash_profile


export KAFKA_HOME=/Users/liujinjie/app/kafka_2.11-0.9.0.0export PATH=$KAFKA_HOME/bin:$PATH


使环境变量生效source ~/.bash_profile


~~~~~~~~~~~~~~~~~~~~~单节点单broker的部署(一个机器上配置一个broker)~~~~~~~~~~~~~~~~~~~~~~


修改配置文件 $KAFKA_HOME/config/server.propertiesbroker.id=0# 一个kafka一个实例listeners=PLAINTEXT://:9092host.name=localhostlog.dirs=/Users/liujinjie/app/tmp/kafka-logszookeeper.connect=localhost:2181


官方启动文档 http://kafka.apache.org/quickstart


kafka-server-start.sh USAGE: /Users/liujinjie/app/kafka_2.11-0.9.0.0/bin/kafka-server-start.sh [-daemon] server.properties [--override property=value]*


1、启动 kafka-server-start.sh -daemon $KAFKA_HOME/config/server.properties


2、创建topic kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic hello_topic


3、发送消息 kafka-console-producer.sh --broker-list localhost:9092 --topic test-topic-180216-01


4、消费消息


(1)kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test-topic-180216-01


(2)kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning


--from-beginning 每次都从头消费


# 老版本的写法 kafka-console-consumer.sh --zookeeper localhost:2181 --topic test-topic-180216-01


5、查看描述 kafka-topics.sh --describe --zookeeper localhost:2181


返回 Topic:hello_topic PartitionCount:1 ReplicationFactor:1 Configs:Topic: hello_topic Partition: 0 Leader: 0 Replicas: 0 Isr: 0


6、查看所有topic


kafka-topics.sh --list --zookeeper localhost:2181

7、根据topic名称查看内容


kafka-topics.sh --describe --zookeeper localhost:2181 --topic hello_topic

~~~~~~~~~~~~~~~~~~~~~~单节点多broker部署(一个机器上部署多个broker)~~~~~~~~~~~~~~~~~~~~~~~~~~~~~


cd $KAFKA_HOME/conf cp server.proerties server-1.properties cp server.proerties server-2.properties cp server.proerties server-3.properties


vi server-1.propertiesbroker.id=1listeners=PLAINTEXT://:9093log.dirs=/Users/liujinjie/app/tmp/kafka-logs-1


vi server-2.propertiesbroker.id=2listeners=PLAINTEXT://:9094log.dirs=/Users/liujinjie/app/tmp/kafka-logs-2


vi server-1.propertiesbroker.id=3listeners=PLAINTEXT://:9095log.dirs=/Users/liujinjie/app/tmp/kafka-logs-3


启动 kafka-server-start.sh -daemon $KAFKA_HOME/config/server-1.properties &


kafka-server-start.sh -daemon $KAFKA_HOME/config/server-2.properties &


kafka-server-start.sh -daemon $KAFKA_HOME/config/server-3.properties &


创建topic kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 3 --partitions 1 --topic my-replicated-topic


看topic详情kafka-topics.sh --describe --zookeeper localhost:2181 --topic my-replicated-topic


返回 Topic:my-replicated-topic PartitionCount:1 ReplicationFactor:3 Configs:Topic: my-replicated-topic Partition: 0 Leader: 1 Replicas: 1,3,2 Isr: 1,3,2


PartitionCount:1 分区数量是1 ReplicationFactor:3副本系数是3 Leader: 1 broker-id=1是leader Replicas:1,3,23个副本 Isr: 1,3,2活着的3个


启动生产者 kafka-console-producer.sh --broker-list localhost:9093,localhost:9094,localhost:9095 --topic my-replicated-topic


消费消息 kafka-console-consumer.sh --zookeeper localhost:2181 --topic my-replicated-topic


Now let's test out fault-tolerance. Broker 1 was acting as the leader so let's kill it:


> ps aux | grep server-1.properties 7564 ttys0020:15.91 /System/Library/Frameworks/JavaVM.framework/Versions/1.8/Home/bin/java... > kill -9 7564


最新文章

123

最新摄影

闪念基因

微信扫一扫

第七城市微信公众平台