RDIoT Demo

DynamoDB Stream + Amazon ES (ElasticSearch, Kibana) [P024]

* Parts

- Raspberry Pi 

- AWS Lambda Java Project in Ecplise

* Contents

1. Upload sensor data to json file using the Amazon S3 Rest API

- Pi4J + Amazon S3 REST API + Amazon Athena [P019]

2. Amazon Lambda S3 Trigger (node.js) : S3toDynamoDB

- source code : https://github.com/rdiot/rdiot-p022/blob/master/S3toDynamoDB/index.js

'use strict';

console.log('RDIoT S3toDynamoDB Loading post function');

var AWS = require('aws-sdk'); 

var s3 = new AWS.S3();

var dynamo = new AWS.DynamoDB.DocumentClient();

exports.handler = function(event, context, callback) {

    console.log('Received event:', JSON.stringify(event, null, 2));

    var bucket = event.Records[0]['s3']['bucket']['name'];

    var en = event.Records[0]['eventName'];

    var et = event.Records[0]['eventTime'];

    var key = event.Records[0]['s3']['object']['key'];

    var sip = event.Records[0]['requestParameters']['sourceIPAddress'];

    var params1 = {Bucket: bucket, Key: key}; 

    s3.getObject( params1, function(err,data) {

        if(err) {

            console.log(err.stack);

            callback(err);

        } else {

            var value = data.Body.toString('ascii')

            console.log(data);

            console.log("Raw text:\n" + value);

            var params2 = {

                TableName: "pi-sensor",

                Item:{

                    "id": et,

                    "event" : en,

                    "bucket": bucket,

                    "key" : key,

                    "sip": sip,

                    "value" : value

                }

            };

            console.log("Gettings IoT device details...");

            //S3 to DynamoDB

            dynamo.put(params2, function(err, data) {

                if(err) {

                    console.error("Unable to post devices. Error JSON:", JSON.stringify(err, null, 2));

                    context.fail();

                } else {

                    console.log("keepet data:", JSON.stringify(data, null, 2));

                    context.succeed('success post');

                }

            });

        }

    });

}

3. AWS DynamoDB Monitoring

4. Create IAM Role

- It must have basic Amazon ES, DynamoDB, and Lambda execution permissions

- create custom role name : LambdaDynamoDBtoES

{

  "Version": "2012-10-17",

  "Statement": [

    {

      "Effect": "Allow",

      "Action": [

        "es:ESHttpPost",

        "es:ESHttpPut",

        "dynamodb:DescribeStream",

        "dynamodb:GetRecords",

        "dynamodb:GetShardIterator",

        "dynamodb:ListStreams",

        "logs:CreateLogGroup",

        "logs:CreateLogStream",

        "logs:PutLogEvents"

      ],

      "Resource": "*"

    }

  ]

}

5. Setup Amazon ES (Elasticsearch + Kibana)

- Define domain

- Configure cluster

- Set up access

- Review

- Service dashboard

6. Amazon Lambda DynamoDB Stream Trigger (java) : DynamoDBtoES

https://docs.amazonaws.cn/en_us/elasticsearch-service/latest/developerguide/es-aws-integrations.html#es-aws-integrations-dynamodb-es

https://docs.aws.amazon.com/ko_kr/elasticsearch-service/latest/developerguide/aes-dg.pdf

Raspberry Pi Camera 5MP Night Version (SEN0184) [S280]

https://www.youtube.com/watch?v=HhDAtAHRuQ8

* Specs

Camera: 5 million pixels

Photosensitive Module: OV5647

Camera Parameters:

CCD size: 1/4 inches

Aperture (f): 1.8

Focal Length: 3.6MM (adjustable)

Diagonal: 75.7 degrees

Sensor best PIX: 1080p

4 Screw Holes

Can be used in a fixed position

3.3V external power supplies

Supports access to the infrared light or fill light

Dimension: 25mm x 24mm

* Contents

- Capture still image

$ raspistill -t 30000 -o rdiot.jpg

- Capture video

$ raspivid -t 10000 -o video.h264

- Capture Night Test

- Capture Day Test

- Download Capture File Sample : 

- Reference 

 : RASPICAM COMMANDS : https://www.raspberrypi.org/documentation/usage/camera/raspicam/README.md

Amazon Polly TTS(Text to Speech) [P017]

https://www.youtube.com/watch?v=1ehU_bueWdQ

GitHub : https://github.com/rdiot/rdiot-p017.git

* Specs

Amazon Polly is a service that turns text into lifelike speech, allowing you to create applications that talk, and build entirely new categories of speech-enabled products. Amazon Polly is a Text-to-Speech service that uses advanced deep learning technologies to synthesize speech that sounds like a human voice.

With dozens of lifelike voices across a variety of languages, you can select the ideal voice and build speech-enabled applications that work in many different countries.

* Contents

- Install AWS CLI in Raspberry Pi

$ wget https://s3.amazonaws.com/aws-cli/awscli-bundle.zip

$ unzip awscli-bundle.zip 

$ sudo ./install -i /usr/local/aws -b /usr/local/bin/aws

- config AWS CLI

$ aws configure

- java amazon polly library and text file

- run amazon polly application

- binary library download : polly.tar.gz

- maven dependency 

 : https://github.com/rdiot/rdiot-p017/blob/master/pom.xml

<dependency>

<groupId>com.amazonaws</groupId>

<artifactId>aws-java-sdk-polly</artifactId>

<version>1.11.77</version>

</dependency>

<!-- https://mvnrepository.com/artifact/com.googlecode.soundlibs/jlayer -->

<dependency>

<groupId>com.googlecode.soundlibs</groupId>

<artifactId>jlayer</artifactId>

<version>1.0.1-1</version>

</dependency>

- Souce Code

 : https://github.com/rdiot/rdiot-p017/blob/master/PollyDemo.java

package com.rdiot.polly;

import java.io.BufferedReader;

import java.io.FileInputStream;

import java.io.IOException;

import java.io.InputStream;

import java.io.InputStreamReader;

import com.amazonaws.ClientConfiguration;

import com.amazonaws.auth.DefaultAWSCredentialsProviderChain;

import com.amazonaws.regions.Region;

import com.amazonaws.regions.Regions;

import com.amazonaws.services.polly.AmazonPollyClient;

import com.amazonaws.services.polly.model.DescribeVoicesRequest;

import com.amazonaws.services.polly.model.DescribeVoicesResult;

import com.amazonaws.services.polly.model.OutputFormat;

import com.amazonaws.services.polly.model.SynthesizeSpeechRequest;

import com.amazonaws.services.polly.model.SynthesizeSpeechResult;

import com.amazonaws.services.polly.model.Voice;

import javazoom.jl.player.advanced.AdvancedPlayer;

import javazoom.jl.player.advanced.PlaybackEvent;

import javazoom.jl.player.advanced.PlaybackListener;

public class PollyDemo {

private final AmazonPollyClient polly;

private final Voice voice;

//private static final String SAMPLE = "Polly Test";

private static String SAMPLE;

public PollyDemo(Region region) {

// create an Amazon Polly client in a specific region

polly = new AmazonPollyClient(new DefaultAWSCredentialsProviderChain(), 

new ClientConfiguration());

polly.setRegion(region);

// Create describe voices request.

DescribeVoicesRequest describeVoicesRequest = new DescribeVoicesRequest().withLanguageCode("en-US");

// Synchronously ask Amazon Polly to describe available TTS voices.

DescribeVoicesResult describeVoicesResult = polly.describeVoices(describeVoicesRequest);

voice = describeVoicesResult.getVoices().get(0);

}

public static String roadLocalFile(String filepath) {

    String readFile= "";

    try {

        BufferedReader in = new BufferedReader(new InputStreamReader(new FileInputStream(filepath),"UTF-8"));

        String s;

        while ((s = in.readLine()) != null) {

            readFile+= s;

        }

        in.close();

    } catch (IOException e) {

        System.err.println(e);

        System.exit(1);

    }

    return readFile;

}

public InputStream synthesize(String text, OutputFormat format) throws IOException {

SynthesizeSpeechRequest synthReq = 

new SynthesizeSpeechRequest().withText(text).withVoiceId(voice.getId())

.withOutputFormat(format);

SynthesizeSpeechResult synthRes = polly.synthesizeSpeech(synthReq);

return synthRes.getAudioStream();

}

public static void main(String args[]) throws Exception {

SAMPLE = roadLocalFile("./text.txt");

//create the test class

PollyDemo helloWorld = new PollyDemo(Region.getRegion(Regions.AP_NORTHEAST_2));

//get the audio stream

InputStream speechStream = helloWorld.synthesize(SAMPLE, OutputFormat.Mp3);

//create an MP3 player

AdvancedPlayer player = new AdvancedPlayer(speechStream,

javazoom.jl.player.FactoryRegistry.systemRegistry().createAudioDevice());

player.setPlayBackListener(new PlaybackListener() {

@Override

public void playbackStarted(PlaybackEvent evt) {

System.out.println("#################################################################");

System.out.println("RD IoT Amazon Polly TTS");

System.out.println("#################################################################");

System.out.println("Playback started");

System.out.println("-----------------------------------------------------------------");

System.out.println("TTS : " + SAMPLE);

}

@Override

public void playbackFinished(PlaybackEvent evt) {

System.out.println("-----------------------------------------------------------------");

System.out.println("Playback finished");

System.out.println("#################################################################");

}

});

// play it!

player.play();

}

}

USB 2.0 Microphone Speech MIC Audio Adapter Driver [S278]

https://www.youtube.com/watch?v=Zw0uAvTD8n0

* Specs

Brand Name: ANBES

Set Type: Single Microphone

Communication: Wireless

Transducer: Condenser Microphone

Use: Computer Microphone

Model Number: USB Microphone

Polar Patterns: Omnidirectional

Package: No

Style: Computer Microphone

Item Name: Computer Microphone

* Contents

- Check Device

$ lsusb

$ dmesg

- Setup 

$ sudo vi /usr/share/alsa/alsa.conf

defaults.ctl.card 0 => 1

defaults,pcm, card 0 => 1

- Setup Volume

$ alsamixer 

- Recording

$ arecord -D plughw:1,0 -c 1 -f S16_LE -r 44100 test.wav

- Play 

$ aplay test.wav

SunFounder RAB 5 in 1 Breadboard [B193]

* Specs

1) Compatibale with Raspberry Pi 3 Model B,2 Model B and 1 Model B+. 

2) For any Arduino board whose mounting holes are the same with For Arduino UNO and Mega 2560 can be fixed upon this Holder with screws. 

3) Be able to fix the 400 points and 800 points self-adhesive breadboard.

4) There are M3 fixing holes in the middle which can be used to fix the RAB Holder upon your experiment table.

5) Size:200*135mm ->7.87inch * 5.31inch.

- Technical Details Introduction

RAB Holder, also called Raspberry Pi, For Arduino and Breadboard Holder. Presumably you can know its function from the name. It can ensure the security of your demoboard and avoid placing the demoboard on the desk freely to cause short cut. The RAB Holder also simplifies your experiment when you are building a complex circuit. There are three zones on the RAB Holder and each has a name on it. We have made some slots in the zone that used to fix the Raspberry Pi to make the use more comfortable.

A screwdriver is provided for you to install the Raspberry Pi and for Arduino board onto the holder with screws. 

- Package Contents

2x M3*10 Screw 

4x M2.5*6 Screw 

6x M3*6 Screw 

1x RAB Holder

1 x Screwdriver

Raspberry Pi Apache Kafka Cluster [P013]

https://www.youtube.com/watch?v=dAVwsc1Zhfo

* Parts

- Raspberry Pi 3 B Model (RASPBERRY-PI-3-B) [B088] x 3ea

- USB Power Supply

- Ethernet Hub

* Contents

- Installation & Setup

1. JDK

$ sudo apt-get install openjdk-8-jdk 

2. Kafka

$ cd 

$ mkdir kafka

$ wget http://apache.tt.co.kr/kafka/0.10.1.0/kafka_2.11-0.10.1.0.tgz

$ tar zxvf kafka_2.11-0.10.1.0.tgz

3. hosts

$ vi /etc/hosts

192.168.0.11 kafka-pi-01 

192.168.0.12 kafka-pi-02 

192.168.0.13 kafka-pi-03 

4. Config Zookeeper 

$ vi config/zookeeper.properties (added)

initLimit=5 

syncLimit=2 

server.1=kafka-pi-01:2888:3888 

server.2=kafka-pi-02:2888:3888 

server.3=kafka-pi-03:2888:3888

$ mkdir /tmp/zookeeper (each server)

$ echo 1 > /tmp/zookeeper/myid 

$ echo 2 > /tmp/zookeeper/myid 

$ echo 3 > /tmp/zookeeper/myid 

5. Config Kafka

$ vi config/server.properties (each server 1,2,3)

broker.id=1 

zookeeper.connect=kafka-pi-01:2181,kafka-pi-02:2181,kafka-pi-03:2181

* Demo Test Script

1. Zookeeper Cluster Start

/home/dragon/kafka/kafka/bin/zookeeper-server-start.sh -daemon /home/dragon/kafka/kafka/config/zookeeper.properties

2. Kafka Cluster Start

/home/dragon/kafka/kafka/bin/kafka-server-start.sh -daemon /home/dragon/kafka/kafka/config/server.properties

3. Create topics

/home/dragon/kafka/kafka/bin/kafka-topics.sh --create --zookeeper kafka-pi-01:2181 --replication-factor 3 --partitions 5 -topic rdtest

4. Producer 

/home/dragon/kafka/kafka/bin/kafka-console-producer.sh --broker-list kafka-pi-01:9092 --topic rdtest

5. Consumer

/home/dragon/kafka/kafka/bin/kafka-console-consumer.sh --zookeeper kafka-pi-01:2181 --topic rdtest --from-beginning

6. Start KafkaOffsetMonitor

java -cp KafkaOffsetMonitor-assembly-0.2.1.jar \

com.quantifind.kafka.offsetapp.OffsetGetterWeb \

--zk kafka-pi-01:2181,kafka-pi-02:2181,kafka-pi-03:2181 \

--port 8080 \

--refresh 10.seconds \

--retain 2.days &

7. Monitor Kafka 

http://192.168.0.11:8080

8. How to solve the "map failed" problem in the 32bit raspberry pi.

- edit kafka-run-class.sh 

- replace the KAFKA_JVM_PERFORMANCE_OPTS as follows

# JVM performance options

if [ -z "$KAFKA_JVM_PERFORMANCE_OPTS" ]; then

  #KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSClassUnloadingEnabled -XX:+CMSScavengeBeforeRemark -XX:+DisableExplicitGC -Djava.awt.headless=true"

  KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+DisableExplicitGC -Djava.awt.headless=true"

fi

- add KAFKA_HEAP_OPTS under 1G

export KAFKA_HEAP_OPTS="-Xmx256M -Xms128M"

Raspberry Pi GPIO Ruler [B180]

https://www.youtube.com/watch?v=tJtQQQQIlWI

* Specs

This GPIO Ruler is the ideal quick reference for RPi.GPIO hacking on the Raspberry Pi in Python has all the most commonly used RPi.GPIO code, which can be hard to remember
can be used as a ruler (cm/inches)
fits easily in a pencil case, toolbox or desk tidy

Package including :

1pc*RPI GPIO Ruler

7 inch 1024*600 TFT LCD with Driver Board [D053]

https://www.youtube.com/watch?v=RPg8WPzc5UM

* Specs

- Features: 

Power input: DC 5V~12V, 2A, 6~7W;

Signal input, 2 AV + VGA +HDMI (HDMI 1.1);

Physical Resolution: 1024 x 600; Resolution Range: 640 x 480 ~ 1600 x 1200 (adjustable

Language support: Chinese (simple), Chinese (traditional), English, Japanese, Korea, Spanish, French, German, Italian, Portuguese;

Control, OSD;

Viewing angle: L70 ° R70 ° U50 ° D60;

Panel type, TFT; Contrast: 500:1;

Brightness: 220cd/m2;

Response time: 20ms;

Background light: LED;

Working temperature: -20 ~ 80;

Interface: 50 PIN Digital (TTL) TCON;

- Package Includes:

1 x 7-inch LCD screen

1 x LCD Driver board

1 x White cable (connected to the key board)

1 x Long green key board

1 x USB to DC5.5 power line