DynamoDB Stream + Amazon ES (ElasticSearch, Kibana) [P024]




* Parts

- Raspberry Pi 

- AWS Lambda Java Project in Ecplise


* Contents

1. Upload sensor data to json file using the Amazon S3 Rest API

Pi4J + Amazon S3 REST API + Amazon Athena [P019]



2. Amazon Lambda S3 Trigger (node.js) : S3toDynamoDB

- source code : https://github.com/rdiot/rdiot-p022/blob/master/S3toDynamoDB/index.js

'use strict';

console.log('RDIoT S3toDynamoDB Loading post function');

var AWS = require('aws-sdk'); 

var s3 = new AWS.S3();

var dynamo = new AWS.DynamoDB.DocumentClient();


exports.handler = function(event, context, callback) {

    console.log('Received event:', JSON.stringify(event, null, 2));

    var bucket = event.Records[0]['s3']['bucket']['name'];

    var en = event.Records[0]['eventName'];

    var et = event.Records[0]['eventTime'];

    var key = event.Records[0]['s3']['object']['key'];

    var sip = event.Records[0]['requestParameters']['sourceIPAddress'];


    var params1 = {Bucket: bucket, Key: key}; 

    s3.getObject( params1, function(err,data) {

        if(err) {

            console.log(err.stack);

            callback(err);

        } else {

            var value = data.Body.toString('ascii')

            console.log(data);

            console.log("Raw text:\n" + value);


            var params2 = {

                TableName: "pi-sensor",

                Item:{

                    "id": et,

                    "event" : en,

                    "bucket": bucket,

                    "key" : key,

                    "sip": sip,

                    "value" : value

                }

            };

            console.log("Gettings IoT device details...");


            //S3 to DynamoDB

            dynamo.put(params2, function(err, data) {

                if(err) {

                    console.error("Unable to post devices. Error JSON:", JSON.stringify(err, null, 2));

                    context.fail();

                } else {

                    console.log("keepet data:", JSON.stringify(data, null, 2));

                    context.succeed('success post');

                }


            });

        }

    });

}


3. AWS DynamoDB Monitoring




4. Create IAM Role

- It must have basic Amazon ES, DynamoDB, and Lambda execution permissions

- create custom role name : LambdaDynamoDBtoES


{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Action": [

        "es:ESHttpPost",

        "es:ESHttpPut",

        "dynamodb:DescribeStream",

        "dynamodb:GetRecords",

        "dynamodb:GetShardIterator",

        "dynamodb:ListStreams",

        "logs:CreateLogGroup",

        "logs:CreateLogStream",

        "logs:PutLogEvents"

      ],

      "Resource": "*"

    }

  ]

}


5. Setup Amazon ES (Elasticsearch + Kibana)

- Define domain

- Configure cluster

- Set up access

- Review

- Service dashboard

6. Amazon Lambda DynamoDB Stream Trigger (java) : DynamoDBtoES




7. Develop AWS Lambda Java Project in Eclipse

- New Project : AWS Lambda Java Project


- New AWS Maven Project : awsLambdaDynamoDBStreamtoAmazonES

 : Class Name: LambdaFunctionHandler

 : Input Type : Dynamodb Event


- Check the items of DynamoDB when running pi4j


- Check the insert event message in the Cloud Watch Log 

 : public Integer handleRequest(DynamodbEvent event, Context context) {

   context.getLogger().log("Received event: " + event);

{ApproximateCreationDateTime: Mon Aug 13 12:07:00 UTC 2018,Keys: {id={S: 2018-08-13T12:07:12.024Z,}},NewImage: {bucket={S: rdiot-test,}, id={S: 2018-08-13T12:07:12.024Z,}, sip={S: 211.245.211.252,}, event={S: ObjectCreated:Put,}, value={S: {"name":"temperature","value":33.0},}, key={S: pi_dht11_2018813079.json,}},SequenceNumber: 187352800000000003101012958,SizeBytes: 175,StreamViewType: NEW_AND_OLD_IMAGES}



- Key Code
for (DynamodbStreamRecord record : event.getRecords()) {
            context.getLogger().log(record.getEventID());
            context.getLogger().log(record.getEventName());
            context.getLogger().log(record.getDynamodb().toString());
            switch (record.getEventName()) {
            case "INSERT":
            case "MODIFY":
            // insert, update code area forAmazon  Elastic Search 
            break;
            case "REMOVE":
            // none
            break;
           
            }
    }

- Reference
https://docs.aws.amazon.com/ko_kr/amazondynamodb/latest/developerguide/Streams.Lambda.html
https://docs.aws.amazon.com/AWSJavaScriptSDK/latest/AWS/ES.html
https://github.com/elastic/elasticsearch-js

https://docs.amazonaws.cn/en_us/elasticsearch-service/latest/developerguide/es-aws-integrations.html#es-aws-integrations-dynamodb-es

https://docs.aws.amazon.com/ko_kr/elasticsearch-service/latest/developerguide/aes-dg.pdf

Posted by RD IoT RDIoT

댓글을 달아 주세요

AWS IoT Core + Raspberry Pi + AWS IoT Device SDK for Java [P021]





https://www.youtube.com/watch?v=xhVIT6XJ01s


GitHub : https://github.com/rdiot/rdiot-p021


* Parts

- Raspberry Pi2

Temperature and humidity DHT11 sensor module (KY-015) [S008]


* Contents

1. Sample Test (basic publish)

- install maven3 in pi

- download maven3 : https://fossies.org/linux/misc/apache-maven-3.5.3-bin.tar.gz/

$ tar zxvf apache-maven-3.5.3-bin.tar.gz

$ export PATH=/home/pi/AWSIoT/apache-maven-3.5.3/bin:$PATH


- com.amazonaws.services.iot.client.sample.pubSub.PublishSubscribeSample

- download root ca certificate and install aws device sdk for java

- run pub/sub sample 

$ unzip http://connect_device_package.zip

$ ./start.sh

mvn exec:java -pl aws-iot-device-sdk-java-samples -Dexec.mainClass="com.amazonaws.services.iot.client.sample.pubSub.PublishSubscribeSample" -Dexec.args="-clientEndpoint <prefix>.iot.<region>.amazonaws.com -clientId <unique client id> -certificateFile <certificate file> -privateKeyFile <private key file>"


[INFO] BUILD SUCCESS

[INFO] ------------------------------------------------------------------------

[INFO] Total time: 04:15 min

[INFO] Finished at: 2018-06-01T13:43:37Z

[INFO] ------------------------------------------------------------------------


Running pub/sub sample application...

[INFO] Scanning for projects...

[INFO] Inspecting build with total of 1 modules...

[INFO] Installing Nexus Staging features:

[INFO]   ... total of 1 executions of maven-deploy-plugin replaced with nexus-staging-maven-plugin

[INFO] 

[INFO] -----------< com.amazonaws:aws-iot-device-sdk-java-samples >------------

[INFO] Building aws-iot-device-sdk-java-samples 1.1.1

[INFO] --------------------------------[ jar ]---------------------------------

[INFO] 

[INFO] --- exec-maven-plugin:1.6.0:java (default-cli) @ aws-iot-device-sdk-java-samples ---

Cert file:../RaspberryPi.cert.pem Private key: ../RaspberryPi.private.key

Jun 01, 2018 1:43:59 PM com.amazonaws.services.iot.client.core.AwsIotConnection onConnectionSuccess

INFO: Connection successfully established

Jun 01, 2018 1:43:59 PM com.amazonaws.services.iot.client.core.AbstractAwsIotClient onConnectionSuccess

INFO: Client connection active: sdk-java

1527860639952: >>> hello from blocking publisher - 1

1527860639952: >>> hello from non-blocking publisher - 1

1527860639973: <<< hello from blocking publisher - 1

1527860639989: <<< hello from non-blocking publisher - 1

1527860640953: >>> hello from non-blocking publisher - 2

1527860640964: >>> hello from blocking publisher - 2

1527860640995: <<< hello from non-blocking publisher - 2

1527860641042: <<< hello from blocking publisher - 2

1527860641953: >>> hello from non-blocking publisher - 3

1527860641979: <<< hello from non-blocking publisher - 3


- Check the message in the AWS IoT Console



- check the subscription, topic name : sdk/test/java



2. Develop

- Requirements : +java1.7, aws-iot-device-sdk-java 1.1.1, Jackson 2.x, Paho MQTT client for Java 1.1.x, pi4j


- Install the SDK using Maven

 <dependency>

    <groupId>com.amazonaws</groupId>

    <artifactId>aws-iot-device-sdk-java</artifactId>

    <version>1.1.1</version>

</dependency>

<dependency>

    <groupId>com.amazonaws</groupId>

    <artifactId>aws-iot-device-sdk-java-samples</artifactId>

    <version>1.1.1</version>

</dependency>


- maven :  https://github.com/rdiot/rdiot-p021/blob/master/pom.xml

- source : https://github.com/rdiot/rdiot-p021/blob/master/src/main/java/com/rdiot/awsiot/App.java


3. Setup AWS IoT Core 


4. Package Download 


5. AWS IoT Rule Setup 

- Rule Name : email_notification

- Description : when temperature is over 30 degrees.

- Rule Query : SELECT temperature as t, humidity as h FROM 'sensor/dht11' WHERE temperature > 30

- Actions 
 1) SNS : sendEmail 

 2) Republish : sensor/dht11/over30


6. run application

$ java -Dpi4j.linking=dynamic -jar awsiot-0.0.1-SNAPSHOT.jar (run.sh)



7. Monitoring




8. Action Check over 30 degrees

- republish to sensor/dht11/over30

- notification sns : email



- Reference

GitHub AWS IoT Device SDK for Java : https://github.com/aws/aws-iot-device-sdk-java/blob/master/README.md

API Document : http://aws-iot-device-sdk-java-docs.s3-website-us-east-1.amazonaws.com/

Sample Code : https://github.com/aws/aws-iot-device-sdk-java/blob/master/aws-iot-device-sdk-java-samples/src/main/java/com/amazonaws/services/iot/client/sample/pubSub/PublishSubscribeSample.java



Posted by RD IoT RDIoT

댓글을 달아 주세요

Call server API for multiple sensor information [P002]


 

https://www.youtube.com/watch?v=6oUOoBihOfo


*GitHubhttps://github.com/rdiot/rdiot-p002.git

 

* Parts

- Arduino Uno R3

- Ethernet W5100 Shield (W5100) [B004]

- LCD1602 (HD44780) [D002]

- Temperature and humidity DHT11 sensor module (KY-015) [S008]

- Button

 

* Contents

- Collect multiple (two temperature / humidity) sensor values on one sensor and load them into the server

Posted by RD IoT RDIoT

댓글을 달아 주세요