MQTT + Kafka + Raspberry Pi ElasticSearch Cluster [P016]

MQTT + Kafka + Raspberry Pi ElasticSearch Cluster [P016]

https://www.youtube.com/watch?v=coKGRMDqAJk

* GitHub : https://github.com/rdiot/rdiot-p016

* Parts

- Arduino UNO

- Ethernet W5100 Shield (W5100) [B004]

- LCD1602 I2C (LCD1602) [D016]

- Photo Resistor Module (KY-018) [S002]

- Temperature and humidity DHT22 (DHT22) [S063]

- TPM-300 Air Quality Module (TPM-300) [S092]

- Samsung ARTIK 5 (ARTIK5-V0.5) [B023]

- Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 3ea : for Kafka Cluster

- Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 4ea : for ElasicSearch Cluster

- USB Power Supply

- Ethernet Hub

* Contents

1. Raspberry Pi ElasticSearch Cluster Setup

- version : elasticsearch 2.4.6

- master config
 : https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-master-01.yml

- data node config 

 : https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-data-01.yml

 : https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-data-02.yml

 : https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-data-03.yml

2. logstash configuration

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-airGrade.conf

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-cds.conf

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-humidity.conf

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-temperature.conf

3. start logstash script example

#!/bin/sh

export LS_HEAP_SIZE="500m"

/data1/logstash/logstash/bin/logstash -f /data1/logstash/logstash-kafka-elasticsearch-temperature.conf -l logstash.log &

4. Kafka logstash Monitoring

5. ElasticSearch data node shard

6. Grafana Setup

- Install in Raspberry Pi

$ sudo apt-get install libfontconfig

$ curl -L https://github.com/fg2it/grafana-on-raspberry/releases/download/v4.0.1/grafana_4.0.1-1480722482_armhf.deb -o /tmp/grafana_4.0.1-1480722482_armhf.deb

$ sudo dpkg -i /tmp/grafana_4.0.1-1480722482_armhf.deb

$ sudo vi /etc/grafana/grafana.ini

# The ip address to bind to, empty will bind to all interfaces

http_addr = 192.168.0.20

$ sudo systemctl enable grafana-server

$ sudo systemctl restart grafana-server

- Connect Grafana WebAdmin

http://92.168.0.20:3000

id/pwd : admin/admin

- Add Data Sources : http://192.168.0.20:3000/datasources/new

- edit data source examples

7. Grafana Dashboard 

- Kafka-Sensor-Monitoring

 : Temperature, Humidity, Cds, AirGrade