RDIoT Demo

MQTT + Kafka + Raspberry Pi ElasticSearch Cluster [P016]

https://www.youtube.com/watch?v=coKGRMDqAJk

* GitHub : https://github.com/rdiot/rdiot-p016

* Parts

- Arduino UNO

- Ethernet W5100 Shield (W5100) [B004]

- LCD1602 I2C (LCD1602) [D016]

- Photo Resistor Module (KY-018) [S002]

- Temperature and humidity DHT22 (DHT22) [S063]

- TPM-300 Air Quality Module (TPM-300) [S092]

- Samsung ARTIK 5 (ARTIK5-V0.5) [B023]

- Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 3ea : for Kafka Cluster

- Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 4ea : for ElasicSearch Cluster

- USB Power Supply

- Ethernet Hub

* Contents

1. Raspberry Pi ElasticSearch Cluster Setup

- version : elasticsearch 2.4.6

- master config
 : https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-master-01.yml

- data node config 

 : https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-data-01.yml

 : https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-data-02.yml

 : https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-data-03.yml

2. logstash configuration

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-airGrade.conf

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-cds.conf

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-humidity.conf

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-temperature.conf

3. start logstash script example

#!/bin/sh

export LS_HEAP_SIZE="500m"

/data1/logstash/logstash/bin/logstash -f /data1/logstash/logstash-kafka-elasticsearch-temperature.conf -l logstash.log &

4. Kafka logstash Monitoring

5. ElasticSearch data node shard

6. Grafana Setup

- Install in Raspberry Pi

$ sudo apt-get install libfontconfig

$ curl -L https://github.com/fg2it/grafana-on-raspberry/releases/download/v4.0.1/grafana_4.0.1-1480722482_armhf.deb -o /tmp/grafana_4.0.1-1480722482_armhf.deb

$ sudo dpkg -i /tmp/grafana_4.0.1-1480722482_armhf.deb

$ sudo vi /etc/grafana/grafana.ini

# The ip address to bind to, empty will bind to all interfaces

http_addr = 192.168.0.20

$ sudo systemctl enable grafana-server

$ sudo systemctl restart grafana-server

- Connect Grafana WebAdmin

http://92.168.0.20:3000

id/pwd : admin/admin

- Add Data Sources : http://192.168.0.20:3000/datasources/new

- edit data source examples

7. Grafana Dashboard 

- Kafka-Sensor-Monitoring

 : Temperature, Humidity, Cds, AirGrade

MQTT + Kafka + Amazon ElasticSearch Service [P015]

https://www.youtube.com/watch?v=-8B8Yp52XMY

* GitHub : https://github.com/rdiot/rdiot-p015

* Parts

- Arduino UNO

- Ethernet W5100 Shield (W5100) [B004]

- LCD1602 I2C (LCD1602) [D016]

- Photo Resistor Module (KY-018) [S002]

- Temperature and humidity DHT22 (DHT22) [S063]

- TPM-300 Air Quality Module (TPM-300) [S092]

- Samsung ARTIK 5 (ARTIK5-V0.5) [B023]

- Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 3ea : for Kafka Cluster

- USB Power Supply

- Ethernet Hub

* Contents

1. Amazon ElasticSearch Service Setup

 - Define domain

- Configure cluster

- Set up access

- Review

- Amazon Elasticsearch Service dashboard

2. ES Information example 

- ElasticSearch Endpoint : https://search-rdiot-aws-es-2yolnmmxbjghreiywdbh4yrsay.ap-northeast-2.es.amazonaws.com

- Kibana URL : https://search-rdiot-aws-es-2yolnmmxbjghreiywdbh4yrsay.ap-northeast-2.es.amazonaws.com/_plugin/kibana/

3. MQTT - Kafka - Bridge (json)

- source : https://github.com/rdiot/rdiot-p015/blob/master/MqttToKafkaReConn.java

- result 

{"time": 1511884443315,"value":724}

{"time": 1511884444405,"value":724}

{"time": 1511884445488,"value":723}

{"time": 1511884446579,"value":724}

{"time": 1511884447657,"value":726}

4. install logstash : tested logstash-2.4.1 

$ cd /data1/logstash

$ wget https://download.elastic.co/logstash/logstash/logstash-2.4.1.tar.gz

$ tar zxvf logstash-2.4.1.tar.gz

$ ln -s logstash-2.4.1 logstash

$ sudo apt-get install ant texinfo openjdk-8-jdk build-essential

$ git clone https://github.com/jnr/jffi.git

$ export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-armhf"

$ cd jffi

$ ant jar

$ sudo cp build/jni/libjffi-1.2.so /data1/logstash/logstash/vendor/jruby/lib/jni/arm-Linux

5. logstash config : tested logstash-2.4.1 

- source : https://github.com/rdiot/rdiot-p015/blob/master/logstash-kafka-aws-es.conf

input {

   kafka {

   zk_connect => "kafka-pi-01:2181"

   group_id => "logstash"

   topic_id => "cds"

   consumer_threads => 2

   decorate_events => true

  }

}

filter {

  json {

    source => "message"

  }

  date {

    match => [ "time", "UNIX"]

    target => "time_new"

  }   

}

output {

  elasticsearch {

    index => "test-%{+YYYY.MM.dd}"

    hosts => ["https://search-rdiot-aws-es-2yolnmmxbjghreiywdbh4yrsay.ap-northeast-2.es.amazonaws.com"]

    codec => json

  }

  stdout {

    codec => "rubydebug"

  }

}

6. Amazon ElasticSearch Query Sample

7. Kibana : Search & Chart 

ARTIK MQTT + Raspberry Pi Apache Kafka Cluster Bridge [P014]

* Arduino MQTT Client to ARTIK MQTT Broker

* ARTIK MQTT Broker & Raspberry Pi Kafka Cluster

* Run MQTT-Kafka-Bridge

* Kafka Console Consumer

* KafkaMonitor

https://www.youtube.com/watch?v=JIEIv-pEFFs

*GitHub : https://github.com/rdiot/rdiot-p014.git

* Parts

- Arduino UNO

- Ethernet W5100 Shield (W5100) [B004]

- LCD1602 I2C (LCD1602) [D016]

- Photo Resistor Module (KY-018) [S002]

- Temperature and humidity DHT22 (DHT22) [S063]

- TPM-300 Air Quality Module (TPM-300) [S092]

- Samsung ARTIK 5 (ARTIK5-V0.5) [B023]

- Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 3ea

- USB Power Supply

- Ethernet Hub

* Contents

1.  ARTIK MQTT Start 

- mqtt broker (1883)

$ mosquitto &

2. Add MQTT Topics in the node-red

- node-red (1880)

$ node-red &

3. Arduino MQTT Client 

- source

https://github.com/rdiot/rdiot-p014/blob/master/arduino_mqtt_client.ino

4. MQTT Kakfa Bridge
- source

https://github.com/rdiot/rdiot-p014/blob/master/MqttToKafkaReConn.java

- maven 

    <!--  mqtt-client -->

<dependency>

    <groupId>org.fusesource.mqtt-client</groupId>

    <artifactId>mqtt-client</artifactId>

    <version>1.14</version>

</dependency>

<!-- kafka -->

<dependency>

<groupId>org.apache.kafka</groupId>

<artifactId>kafka_2.9.1</artifactId>

<version>0.8.2.1</version>

</dependency>

- run script

https://github.com/rdiot/rdiot-p014/blob/master/mqtt-kafka-bridge.sh

$ java -jar mqtt-kafka-bridge-0.0.1-SNAPSHOT.jar 

$ nohup java -jar mqtt-kafka-bridge-0.0.1-SNAPSHOT.jar > output.txt &

5. Kafka Console Consumer 

- run script

https://github.com/rdiot/rdiot-p014/blob/master/consumer.sh

$ /data1/kafka/kafka/bin/kafka-console-consumer.sh --zookeeper kafka-pi-01:2181,kafka-pi-02:2181,kafka-pi-03:2181 --topic $1

- run command

$ ./consumer.sh temperature,humidity,cds,airGrade

$ ./consumer.sh temperature

Raspberry Pi Apache Kafka Cluster [P013]

https://www.youtube.com/watch?v=dAVwsc1Zhfo

* Parts

- Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 3ea

- USB Power Supply

- Ethernet Hub

* Contents

- Installation & Setup

1. JDK

$ sudo apt-get install openjdk-8-jdk 

2. Kafka

$ cd 

$ mkdir kafka

$ wget http://apache.tt.co.kr/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz

$ tar zxvf kafka_2.11-0.10.1.0.tgz

3. hosts

$ vi /etc/hosts

192.168.0.11 kafka-pi-01 

192.168.0.12 kafka-pi-02 

192.168.0.13 kafka-pi-03 

4. Config Zookeeper 

$ vi config/zookeeper.properties (added)

initLimit=5 

syncLimit=2 

server.1=kafka-pi-01:2888:3888 

server.2=kafka-pi-02:2888:3888 

server.3=kafka-pi-03:2888:3888

$ mkdir /tmp/zookeeper (each server)

$ echo 1 > /tmp/zookeeper/myid 

$ echo 2 > /tmp/zookeeper/myid 

$ echo 3 > /tmp/zookeeper/myid 

5. Config Kafka

$ vi config/server.properties (each server 1,2,3)

broker.id=1 

zookeeper.connect=kafka-pi-01:2181,kafka-pi-02:2181,kafka-pi-03:2181

* Demo Test Script

1. Zookeeper Cluster Start

/home/dragon/kafka/kafka/bin/zookeeper-server-start.sh -daemon /home/dragon/kafka/kafka/config/zookeeper.properties

2. Kafka Cluster Start

/home/dragon/kafka/kafka/bin/kafka-server-start.sh -daemon /home/dragon/kafka/kafka/config/server.properties

3. Create topics

/home/dragon/kafka/kafka/bin/kafka-topics.sh --create --zookeeper kafka-pi-01:2181 --replication-factor 3 --partitions 5 -topic rdtest

4. Producer 

/home/dragon/kafka/kafka/bin/kafka-console-producer.sh --broker-list kafka-pi-01:9092 --topic rdtest

5. Consumer

/home/dragon/kafka/kafka/bin/kafka-console-consumer.sh --zookeeper kafka-pi-01:2181 --topic rdtest --from-beginning

6. Start KafkaOffsetMonitor

java -cp KafkaOffsetMonitor-assembly-0.2.1.jar \

com.quantifind.kafka.offsetapp.OffsetGetterWeb \

--zk kafka-pi-01:2181,kafka-pi-02:2181,kafka-pi-03:2181 \

--port 8080 \

--refresh 10.seconds \

--retain 2.days &

7. Monitor Kafka 

http://192.168.0.11:8080

8. How to solve the "map failed" problem in the 32bit raspberry pi.

- edit kafka-run-class.sh 

- replace the KAFKA_JVM_PERFORMANCE_OPTS as follows

# JVM performance options

if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then

  #KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true"

  KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true"

fi

- add KAFKA_HEAP_OPTS under 1G

export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M"

Raspberry Pi GPIO Ruler [B180]

https://www.youtube.com/watch?v=tJtQQQQIlWI

* Specs

This GPIO Ruler is the ideal quick reference for RPi.GPIO hacking on the Raspberry Pi in Python has all the most commonly used RPi.GPIO code, which can be hard to remember
can be used as a ruler (cm/inches)
fits easily in a pencil case, toolbox or desk tidy

Package including :

1pc*RPI GPIO Ruler

Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088]

https://youtu.be/lETtwwq2iiY

* Specs

Broadcom BCM2837 chipset running at 1.2 GHz

64-bit quad-core ARM Cortex-A53

802.11 b/g/n Wireless LAN

Bluetooth 4.1 (Classic & Low Energy)

Dual core Videocore IV® Multimedia co-processor

1 GB LPDDR2 memory

Supports all the latest ARM GNU/Linux distributions and Windows 10 IoT

microUSB connector for 2.5 A power supply

1 x 10/100 Ethernet port

1 x HDMI video/audio connector

1 x RCA video/audio connector

4 x USB 2.0 ports

40 GPIO pins

Chip antenna

DSI display connector

microSD card slot

Dimensions: 85 x 56 x 17 mm

Raspberry Pi 2 Model B (RASPBERRY-PI-2-B) [B016]

https://youtu.be/3ahuUZExtF0

* Specs

A 900MHz quad-core ARM Cortex-A7 CPU

1GB RAM

4 USB ports

40 GPIO pins

Full HDMI port

Ethernet port

Combined 3.5mm audio jack and composite video

Camera interface (CSI)

Display interface (DSI)

Micro SD card slot

VideoCore IV 3D graphics core