'KafkaOffsetMonitor'에 해당되는 글 2건

  1. 2017.09.27 ARTIK MQTT + Raspberry Pi Apache Kafka Cluster Bridge [P014]
  2. 2017.06.08 Raspberry Pi Apache Kafka Cluster [P013]

ARTIK MQTT + Raspberry Pi Apache Kafka Cluster Bridge [P014]



* Arduino MQTT Client to ARTIK MQTT Broker


* ARTIK MQTT Broker & Raspberry Pi Kafka Cluster


* Run MQTT-Kafka-Bridge


* Kafka Console Consumer


* KafkaMonitor



https://www.youtube.com/watch?v=JIEIv-pEFFs


*GitHub : https://github.com/rdiot/rdiot-p014.git


* Parts

- Arduino UNO

Ethernet W5100 Shield (W5100) [B004]

LCD1602 I2C (LCD1602) [D016]

Photo Resistor Module (KY-018) [S002]

Temperature and humidity DHT22 (DHT22) [S063]

TPM-300 Air Quality Module (TPM-300) [S092]

Samsung ARTIK 5 (ARTIK5-V0.5) [B023]

Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 3ea

- USB Power Supply

- Ethernet Hub



* Contents

1.  ARTIK MQTT Start 

- mqtt broker (1883)

$ mosquitto &


2. Add MQTT Topics in the node-red

- node-red (1880)

$ node-red &


3. Arduino MQTT Client 

- source

https://github.com/rdiot/rdiot-p014/blob/master/arduino_mqtt_client.ino



4. MQTT Kakfa Bridge
- source

https://github.com/rdiot/rdiot-p014/blob/master/MqttToKafkaReConn.java

- maven 

    <!--  mqtt-client -->

<dependency>

    <groupId>org.fusesource.mqtt-client</groupId>

    <artifactId>mqtt-client</artifactId>

    <version>1.14</version>

</dependency>

<!-- kafka -->

<dependency>

<groupId>org.apache.kafka</groupId>

<artifactId>kafka_2.9.1</artifactId>

<version>0.8.2.1</version>

</dependency>


- run script

https://github.com/rdiot/rdiot-p014/blob/master/mqtt-kafka-bridge.sh

$ java -jar mqtt-kafka-bridge-0.0.1-SNAPSHOT.jar 

$ nohup java -jar mqtt-kafka-bridge-0.0.1-SNAPSHOT.jar > output.txt &


5. Kafka Console Consumer 

- run script

https://github.com/rdiot/rdiot-p014/blob/master/consumer.sh

$ /data1/kafka/kafka/bin/kafka-console-consumer.sh --zookeeper kafka-pi-01:2181,kafka-pi-02:2181,kafka-pi-03:2181 --topic $1


- run command

$ ./consumer.sh temperature,humidity,cds,airGrade

$ ./consumer.sh temperature


Posted by RD IoT RDIoT

댓글을 달아 주세요

Raspberry Pi Apache Kafka Cluster [P013]

 




https://www.youtube.com/watch?v=dAVwsc1Zhfo


  

* Parts

- Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 3ea

- USB Power Supply

- Ethernet Hub

 

* Contents

- Installation & Setup

1. JDK

$ sudo apt-get install openjdk-8-jdk 


2. Kafka

$ cd 

$ mkdir kafka

$ wget http://apache.tt.co.kr/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz

$ tar zxvf kafka_2.11-0.10.1.0.tgz


3. hosts

$ vi /etc/hosts

192.168.0.11 kafka-pi-01 

192.168.0.12 kafka-pi-02 

192.168.0.13 kafka-pi-03 


4. Config Zookeeper 

$ vi config/zookeeper.properties (added)

initLimit=5 

syncLimit=2 

server.1=kafka-pi-01:2888:3888 

server.2=kafka-pi-02:2888:3888 

server.3=kafka-pi-03:2888:3888


$ mkdir /tmp/zookeeper (each server)

$ echo 1 > /tmp/zookeeper/myid 

$ echo 2 > /tmp/zookeeper/myid 

$ echo 3 > /tmp/zookeeper/myid 


5. Config Kafka

$ vi config/server.properties (each server 1,2,3)

broker.id=1 

zookeeper.connect=kafka-pi-01:2181,kafka-pi-02:2181,kafka-pi-03:2181



* Demo Test Script

1. Zookeeper Cluster Start

/home/dragon/kafka/kafka/bin/zookeeper-server-start.sh -daemon /home/dragon/kafka/kafka/config/zookeeper.properties


2. Kafka Cluster Start

/home/dragon/kafka/kafka/bin/kafka-server-start.sh -daemon /home/dragon/kafka/kafka/config/server.properties


3. Create topics

/home/dragon/kafka/kafka/bin/kafka-topics.sh --create --zookeeper kafka-pi-01:2181 --replication-factor 3 --partitions 5 -topic rdtest


4. Producer 

/home/dragon/kafka/kafka/bin/kafka-console-producer.sh --broker-list kafka-pi-01:9092 --topic rdtest


5. Consumer

/home/dragon/kafka/kafka/bin/kafka-console-consumer.sh --zookeeper kafka-pi-01:2181 --topic rdtest --from-beginning


6. Start KafkaOffsetMonitor

java -cp KafkaOffsetMonitor-assembly-0.2.1.jar \

com.quantifind.kafka.offsetapp.OffsetGetterWeb \

--zk kafka-pi-01:2181,kafka-pi-02:2181,kafka-pi-03:2181 \

--port 8080 \

--refresh 10.seconds \

--retain 2.days &


7. Monitor Kafka 

http://192.168.0.11:8080


8. How to solve the "map failed" problem in the 32bit raspberry pi.

- edit kafka-run-class.sh 

- replace the KAFKA_JVM_PERFORMANCE_OPTS as follows

# JVM performance options

if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then

  #KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true"

  KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true"

fi


- add KAFKA_HEAP_OPTS under 1G

export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M"



Posted by RD IoT RDIoT

댓글을 달아 주세요