RDIoT Demo

AWS IoT Core + Raspberry Pi + AWS IoT Device SDK for Java [P021]

https://www.youtube.com/watch?v=xhVIT6XJ01s

GitHub : https://github.com/rdiot/rdiot-p021

* Parts

- Raspberry Pi2

- Temperature and humidity DHT11 sensor module (KY-015) [S008]

* Contents

1. Sample Test (basic publish)

- install maven3 in pi

- download maven3 : https://fossies.org/linux/misc/apache-maven-3.5.3-bin.tar.gz/

$ tar zxvf apache-maven-3.5.3-bin.tar.gz

$ export PATH=/home/pi/AWSIoT/apache-maven-3.5.3/bin:$PATH

- com.amazonaws.services.iot.client.sample.pubSub.PublishSubscribeSample

- download root ca certificate and install aws device sdk for java

- run pub/sub sample 

$ unzip http://connect_device_package.zip

$ ./start.sh

mvn exec:java -pl aws-iot-device-sdk-java-samples -Dexec.mainClass="com.amazonaws.services.iot.client.sample.pubSub.PublishSubscribeSample" -Dexec.args="-clientEndpoint <prefix>.iot.<region>.amazonaws.com -clientId <unique client id> -certificateFile <certificate file> -privateKeyFile <private key file>"

[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 04:15 min

[INFO] Finished at: 2018-06-01T13:43:37Z

[INFO] ------------------------------------------------------------------------

Running pub/sub sample application...

[INFO] Scanning for projects...

[INFO] Inspecting build with total of 1 modules...

[INFO] Installing Nexus Staging features:

[INFO]   ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin

[INFO] 

[INFO] -----------< com.amazonaws:aws-iot-device-sdk-java-samples >------------

[INFO] Building aws-iot-device-sdk-java-samples 1.1.1

[INFO] --------------------------------[ jar ]---------------------------------

[INFO] 

[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ aws-iot-device-sdk-java-samples ---

Cert file:../RaspberryPi.cert.pem Private key: ../RaspberryPi.private.key

Jun 01, 2018 1:43:59 PM com.amazonaws.services.iot.client.core.AwsIotConnection onConnectionSuccess

INFO: Connection successfully established

Jun 01, 2018 1:43:59 PM com.amazonaws.services.iot.client.core.AbstractAwsIotClient onConnectionSuccess

INFO: Client connection active: sdk-java

1527860639952: >>> hello from blocking publisher - 1

1527860639952: >>> hello from non-blocking publisher - 1

1527860639973: <<< hello from blocking publisher - 1

1527860639989: <<< hello from non-blocking publisher - 1

1527860640953: >>> hello from non-blocking publisher - 2

1527860640964: >>> hello from blocking publisher - 2

1527860640995: <<< hello from non-blocking publisher - 2

1527860641042: <<< hello from blocking publisher - 2

1527860641953: >>> hello from non-blocking publisher - 3

1527860641979: <<< hello from non-blocking publisher - 3

2. Develop

- Requirements : +java1.7, aws-iot-device-sdk-java 1.1.1, Jackson 2.x, Paho MQTT client for Java 1.1.x, pi4j

- Install the SDK using Maven

 <dependency>

    <groupId>com.amazonaws</groupId>

    <artifactId>aws-iot-device-sdk-java</artifactId>

    <version>1.1.1</version>

</dependency>

<dependency>

    <groupId>com.amazonaws</groupId>

    <artifactId>aws-iot-device-sdk-java-samples</artifactId>

    <version>1.1.1</version>

</dependency>

- source : https://github.com/rdiot/rdiot-p021/blob/master/src/main/java/com/rdiot/awsiot/App.java

3. Setup AWS IoT Core 

4. Package Download 

5. AWS IoT Rule Setup 

- Rule Name : email_notification

- Description : when temperature is over 30 degrees.

- Rule Query : SELECT temperature as t, humidity as h FROM 'sensor/dht11' WHERE temperature > 30

- Actions 
 1) SNS : sendEmail 

 2) Republish : sensor/dht11/over30

6. run application

$ java -Dpi4j.linking=dynamic -jar awsiot-0.0.1-SNAPSHOT.jar (run.sh)

7. Monitoring

8. Action Check over 30 degrees

- republish to sensor/dht11/over30

- notification sns : email

- Reference

GitHub AWS IoT Device SDK for Java : https://github.com/aws/aws-iot-device-sdk-java/blob/master/README.md

API Document : http://aws-iot-device-sdk-java-docs.s3-website-us-east-1.amazonaws.com/

Sample Code : https://github.com/aws/aws-iot-device-sdk-java/blob/master/aws-iot-device-sdk-java-samples/src/main/java/com/amazonaws/services/iot/client/sample/pubSub/PublishSubscribeSample.java

MQTT + Kafka + Raspberry Pi ElasticSearch Cluster [P016]

https://www.youtube.com/watch?v=coKGRMDqAJk

* GitHub : https://github.com/rdiot/rdiot-p016

* Parts

- Arduino UNO

- Ethernet W5100 Shield (W5100) [B004]

- LCD1602 I2C (LCD1602) [D016]

- Photo Resistor Module (KY-018) [S002]

- Temperature and humidity DHT22 (DHT22) [S063]

- TPM-300 Air Quality Module (TPM-300) [S092]

- Samsung ARTIK 5 (ARTIK5-V0.5) [B023]

- Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 3ea : for Kafka Cluster

- Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 4ea : for ElasicSearch Cluster

- USB Power Supply

- Ethernet Hub

* Contents

1. Raspberry Pi ElasticSearch Cluster Setup

- version : elasticsearch 2.4.6

- master config
 : https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-master-01.yml

- data node config 

 : https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-data-01.yml

 : https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-data-02.yml

 : https://github.com/rdiot/rdiot-p016/blob/master/elasticsearch_es-pi-data-03.yml

2. logstash configuration

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-airGrade.conf

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-cds.conf

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-humidity.conf

- https://github.com/rdiot/rdiot-p016/blob/master/logstash-kafka-elasticsearch-temperature.conf

3. start logstash script example

#!/bin/sh

export LS_HEAP_SIZE="500m"

/data1/logstash/logstash/bin/logstash -f /data1/logstash/logstash-kafka-elasticsearch-temperature.conf -l logstash.log &

4. Kafka logstash Monitoring

5. ElasticSearch data node shard

6. Grafana Setup

- Install in Raspberry Pi

$ sudo apt-get install libfontconfig

$ curl -L https://github.com/fg2it/grafana-on-raspberry/releases/download/v4.0.1/grafana_4.0.1-1480722482_armhf.deb -o /tmp/grafana_4.0.1-1480722482_armhf.deb

$ sudo dpkg -i /tmp/grafana_4.0.1-1480722482_armhf.deb

$ sudo vi /etc/grafana/grafana.ini

# The ip address to bind to, empty will bind to all interfaces

http_addr = 192.168.0.20

$ sudo systemctl enable grafana-server

$ sudo systemctl restart grafana-server

- Connect Grafana WebAdmin

http://92.168.0.20:3000

id/pwd : admin/admin

- Add Data Sources : http://192.168.0.20:3000/datasources/new

- edit data source examples

7. Grafana Dashboard 

- Kafka-Sensor-Monitoring

 : Temperature, Humidity, Cds, AirGrade

MQTT + Kafka + Amazon ElasticSearch Service [P015]

https://www.youtube.com/watch?v=-8B8Yp52XMY

* GitHub : https://github.com/rdiot/rdiot-p015

* Parts

- Arduino UNO

- Ethernet W5100 Shield (W5100) [B004]

- LCD1602 I2C (LCD1602) [D016]

- Photo Resistor Module (KY-018) [S002]

- Temperature and humidity DHT22 (DHT22) [S063]

- TPM-300 Air Quality Module (TPM-300) [S092]

- Samsung ARTIK 5 (ARTIK5-V0.5) [B023]

- Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 3ea : for Kafka Cluster

- USB Power Supply

- Ethernet Hub

* Contents

1. Amazon ElasticSearch Service Setup

 - Define domain

- Configure cluster

- Set up access

- Review

- Amazon Elasticsearch Service dashboard

2. ES Information example 

- ElasticSearch Endpoint : https://search-rdiot-aws-es-2yolnmmxbjghreiywdbh4yrsay.ap-northeast-2.es.amazonaws.com

- Kibana URL : https://search-rdiot-aws-es-2yolnmmxbjghreiywdbh4yrsay.ap-northeast-2.es.amazonaws.com/_plugin/kibana/

3. MQTT - Kafka - Bridge (json)

- source : https://github.com/rdiot/rdiot-p015/blob/master/MqttToKafkaReConn.java

- result 

{"time": 1511884443315,"value":724}

{"time": 1511884444405,"value":724}

{"time": 1511884445488,"value":723}

{"time": 1511884446579,"value":724}

{"time": 1511884447657,"value":726}

4. install logstash : tested logstash-2.4.1 

$ cd /data1/logstash

$ wget https://download.elastic.co/logstash/logstash/logstash-2.4.1.tar.gz

$ tar zxvf logstash-2.4.1.tar.gz

$ ln -s logstash-2.4.1 logstash

$ sudo apt-get install ant texinfo openjdk-8-jdk build-essential

$ git clone https://github.com/jnr/jffi.git

$ export JAVA_HOME="/usr/lib/jvm/java-8-openjdk-armhf"

$ cd jffi

$ ant jar

$ sudo cp build/jni/libjffi-1.2.so /data1/logstash/logstash/vendor/jruby/lib/jni/arm-Linux

5. logstash config : tested logstash-2.4.1 

- source : https://github.com/rdiot/rdiot-p015/blob/master/logstash-kafka-aws-es.conf

input {

   kafka {

   zk_connect => "kafka-pi-01:2181"

   group_id => "logstash"

   topic_id => "cds"

   consumer_threads => 2

   decorate_events => true

  }

}

filter {

  json {

    source => "message"

  }

  date {

    match => [ "time", "UNIX"]

    target => "time_new"

  }   

}

output {

  elasticsearch {

    index => "test-%{+YYYY.MM.dd}"

    hosts => ["https://search-rdiot-aws-es-2yolnmmxbjghreiywdbh4yrsay.ap-northeast-2.es.amazonaws.com"]

    codec => json

  }

  stdout {

    codec => "rubydebug"

  }

}

6. Amazon ElasticSearch Query Sample

7. Kibana : Search & Chart 

ARTIK MQTT + Raspberry Pi Apache Kafka Cluster Bridge [P014]

* Arduino MQTT Client to ARTIK MQTT Broker

* ARTIK MQTT Broker & Raspberry Pi Kafka Cluster

* Run MQTT-Kafka-Bridge

* Kafka Console Consumer

* KafkaMonitor

https://www.youtube.com/watch?v=JIEIv-pEFFs

*GitHub : https://github.com/rdiot/rdiot-p014.git

* Parts

- Arduino UNO

- Ethernet W5100 Shield (W5100) [B004]

- LCD1602 I2C (LCD1602) [D016]

- Photo Resistor Module (KY-018) [S002]

- Temperature and humidity DHT22 (DHT22) [S063]

- TPM-300 Air Quality Module (TPM-300) [S092]

- Samsung ARTIK 5 (ARTIK5-V0.5) [B023]

- Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 3ea

- USB Power Supply

- Ethernet Hub

* Contents

1.  ARTIK MQTT Start 

- mqtt broker (1883)

$ mosquitto &

2. Add MQTT Topics in the node-red

- node-red (1880)

$ node-red &

3. Arduino MQTT Client 

- source

https://github.com/rdiot/rdiot-p014/blob/master/arduino_mqtt_client.ino

4. MQTT Kakfa Bridge
- source

https://github.com/rdiot/rdiot-p014/blob/master/MqttToKafkaReConn.java

- maven 

    <!--  mqtt-client -->

<dependency>

    <groupId>org.fusesource.mqtt-client</groupId>

    <artifactId>mqtt-client</artifactId>

    <version>1.14</version>

</dependency>

<!-- kafka -->

<dependency>

<groupId>org.apache.kafka</groupId>

<artifactId>kafka_2.9.1</artifactId>

<version>0.8.2.1</version>

</dependency>

- run script

https://github.com/rdiot/rdiot-p014/blob/master/mqtt-kafka-bridge.sh

$ java -jar mqtt-kafka-bridge-0.0.1-SNAPSHOT.jar 

$ nohup java -jar mqtt-kafka-bridge-0.0.1-SNAPSHOT.jar > output.txt &

5. Kafka Console Consumer 

- run script

https://github.com/rdiot/rdiot-p014/blob/master/consumer.sh

$ /data1/kafka/kafka/bin/kafka-console-consumer.sh --zookeeper kafka-pi-01:2181,kafka-pi-02:2181,kafka-pi-03:2181 --topic $1

- run command

$ ./consumer.sh temperature,humidity,cds,airGrade

$ ./consumer.sh temperature

Arduino MQTT Client & ARTIK MQTT Broker [P012]

https://www.youtube.com/watch?v=L44LwGBY9vw

*GitHub : https://github.com/rdiot/rdiot-p012.git

* Parts

- Arduino UNO

- Ethernet W5100 Shield (W5100) [B004]

- LCD1602 I2C (LCD1602) [D016]

- Photo Resistor Module (KY-018) [S002]

- Temperature and humidity DHT22 (DHT22) [S063]

- TPM-300 Air Quality Module (TPM-300) [S092]

- Samsung ARTIK 5 (ARTIK5-V0.5) [B023]