BigData

DEMO_INDEX 2016. 12. 22. 14:51

RD IoT Demo Index > BigData

Youtube BigData Movies : Link

    Reference Site

     

    * Open Data

        - Open Source Data Portal CHAN : https://ckan.org

        - USA data.gov : https://catalog.data.gov/dataset

        - World Bank : http://datacatalog.worldbank.org

        - KOR DB Pedia: http://lod.data.go.kr

        - KOR data portal : http://data.go.kr

        - KOR Seoul open data plaza : http://data.seoul.go.kr

        - KOR RTWQI : http://koreawqi.go.kr

        - Yahoo Finance : https://finance.yahoo.com

            : ex) Download Facebook : https://finance.yahoo.com/quote/FB/history?period1=1330268400&period2=1488121200&interval=1d&filter=history

        - Google Finance : https://www.google.com/finance     

     

    * Data Format

        - XML

        - JSON

        - BSON

        - MessagePack : https://msgpack.org/

        - Protobuf (Google Protocol Buffers) : https://developers.google.com/protocol-buffers/

          : gRPC : https://www.grpc.io/


    * Big Data Collection

        - Logstash : https://www.elastic.co/products/logstash

            : Codec plugins : https://www.elastic.co/guide/en/logstash/2.4/codec-plugins.html

            : Kafka input : https://www.elastic.co/guide/en/logstash/2.4/plugins-inputs-kafka.html

            : Kafka output : https://www.elastic.co/guide/en/logstash/2.4/plugins-outputs-kafka.html

            : ElasticSearch output : https://www.elastic.co/guide/en/logstash/2.4/plugins-outputs-elasticsearch.html

        - Fluentd : http://www.fluentd.org

        - Splunk

            : Splunk Docs : http://docs.splunk.com/Documentation/Splunk

            : Download Splunk Enterprise : https://www.splunk.com/en_us/download/splunk-enterprise.html

            : Download Splunk Universial Forwarder : https://www.splunk.com/en_us/download/universal-forwarder.html

        - Flume  : https://flume.apache.org/

        - Chukwa : http://incubator.apache.org/chukwa

        - Scribe : https://github.com/facebookarchive/scribe

        - Sqoop (RDB to HDFS) : http://sqoop.apache.org/

            : Sqoop Docs : http://sqoop.apache.org/docs/1.4.6/index.html

            : Download Sqoop1 Latest stable release 1.4.6 : http://www.apache.org/dyn/closer.lua/sqoop/1.4.6

            : Download Sqoop2 Lastest cut 1.99.7 : http://www.apache.org/dyn/closer.lua/sqoop/1.99.7  

        - OpenAPI

     

    * MQ (Message Queuing)

        - JMS(Java Message Service)

            : JSR 914: Java Message Service (JMS) API : http://jcp.org/en/jsr/detail?id=914

            : Java Message Service 1.1 specification: http://java.sun.com/products/jms/docs.html
            : Open Message Queue: http://mq.java.net/
            : JSR 322: Java EE Connector Architecture 1.6: http://jcp.org/en/jsr/summary?id=322 

        - AMQP(Advaned Message Queuing Protocol) 

            : AMQP Wiki : https://en.wikipedia.org/wiki/Advanced_Message_Queuing_Protocol
            : AMQP Protocol : https://www.amqp.org/

            : AMQP Specs 0-9-1 (pdf) : http://www.rabbitmq.com/resources/specs/amqp0-9-1.pdf

        - MQTT(Message Queue Telemetry Transport) 

        - ZeroMQ(0MQ) : http://zeromq.org

        - ActiveMQ : http://activemq.apache.org

        - RabbitMQ (AMQP 0-9-1) : http://www.rabbitmq.com

            : Download RabbitMQ Server : http://www.rabbitmq.com/download.html

            : Get Started : http://www.rabbitmq.com/getstarted.html

            : erlang : http://www.erlang.org

            : Spring AMQP : http://projects.spring.io/spring-amqp

            : Mule AMQP : http://www.mulesoft.org/connectors/amqp-connector

            : Beetle : https://github.com/xing/beetle

        - Kestrel : https://github.com/twitter-archive/kestrel
        - Apache Kafka : http://kafka.apache.org

            : Kafka data structures in Zookeeper : https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper

            : Kafka Producer API : http://kafka.apache.org/documentation/#producerapi

            : Kafka Customer API : http://kafka.apache.org/documentation/#consumerapi  

            : Kafka Offset Monitor : http://quantifind.github.io/KafkaOffsetMonitor/

        - RabbitMQ vs Kafka : https://prismoskills.appspot.com/lessons/System_Design_and_Big_Data/Chapter_12_-_RabbitMQ_vs_Kafka.jsp

        - Exploring Message Brokers: RabbitMQ, Kafka, ActiveMQ, and Kestrel : https://dzone.com/articles/exploring-message-brokers

        - Amazon SQS (Amazon Simple Queue Service) : https://aws.amazon.com/sqs/

            : Amazon SQS Developer Guide : http://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/Welcome.html

       

    * ELK Stack

        - ElasticSearch (Realtime RESTful search and analytics engine) : https://www.elastic.co/en/products/elasticsearch

        - Logstach (server-side data processing pipeline) : https://www.elastic.co/en/products/logstash 

        - Kibana (Visalize & Explore) : https://www.elastic.co/products/kibana

        - Amazon ES (Elasticsearch Service) 

            : Amazon Gettting Started : http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/es-gsg.html

            : Amazon ES Dev Guide : http://docs.aws.amazon.com/elasticsearch-service/latest/developerguide/what-is-amazon-elasticsearch-service.html

     

    * DW (Data Warehouse)

        - Amazon Redshift : https://aws.amazon.com/redshift/

            : Amazon Redshift Database Developer Guide : http://docs.aws.amazon.com/ko_kr/redshift/latest/dg/welcome.html

            : Download Redshift JDBC Driver : http://docs.aws.amazon.com/redshift/latest/mgmt/configure-jdbc-connection.html#download-jdbc-driver

            : Download SQL Workbench/J for connecting Redshift : http://www.sql-workbench.net/downloads.html

     

    * ETL (Extraction Transformation Load)

        - Amazon Data Pipeline : https://aws.amazon.com/datapipeline/

           : Amazon Data Pipeline Developer Guide : http://docs.aws.amazon.com/datapipeline/latest/DeveloperGuide/what-is-datapipeline.html

     

     * NoSQL

        - nosql-database.org : http://nosql-database.org/  

      1) In-Memory Key-Value Store

        - Memcached : http://memcached.org/

            : libevent (an event notification library) : http://www.monkey.org/~provos/libevent/

        - Repcached (memcached + replication) : http://repcached.lab.klab.org/

        - Redis : https://redis.io/

            : Redis Commands : https://redis.io/commands

            : Redis CLI (redis-cli) : https://redis.io/topics/rediscli

            : Download Redis Desktop Manager : https://redisdesktop.com/download

            : Redis Java Client 

              + Jedis : https://github.com/xetorthio/jedis/wiki

              + Lettuece : https://lettuce.io/

              + Redisson : https://redisson.org/

            : Redis Monitoring 

              + redis-stat : https://github.com/junegunn/redis-stat

              + redmon : https://github.com/steelThread/redmon

              + RedisLive : https://github.com/nkrode/RedisLive

              + redis-faina : https://github.com/facebookarchive/redis-faina

        - Amazon ElastiCache (Redis, Memcached) :  https://aws.amazon.com/elasticache/

            : Amazon ElastiCache User Guide : http://docs.aws.amazon.com/AmazonElastiCache/latest/UserGuide/WhatIs.html

            : Download ElastiCache Memcached Cluster Client LibraryforJava : https://elasticache-downloads.s3.amazonaws.com/ClusterClient/Java/latest

            : Memcached Wiki : https://github.com/memcached/memcached/wiki

            : Download redis-cli : http://download.redis.io/redis-stable.tar.gz

      2) Key-Value Store 

        - Tokyo Tyrant : http://fallabs.com/tokyotyrant/

      3) Document Database 

       - Amazon DynamoDB : https://aws.amazon.com/dynamodb/

            : Amazon Dynamo DB Developer Guide : http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/index.html

            : Download DynamoDBLocal : https://s3-ap-northeast-1.amazonaws.com/dynamodb-local-tokyo/dynamodb_local_latest.tar.gz

            : Dynamo DB Stremas : https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Streams.html

            : SQLite DB Browser Tools : https://github.com/sqlitebrowser/sqlitebrowser/releases

      4) Colunmnar  Database

       - Apache Cassandra : http://cassandra.apache.org/

            : Download Cassandra : http://cassandra.apache.org/download/

            : Cassandra Wiki : https://wiki.apache.org/cassandra/

            : Cassandra Query Language(CQL) v3.4.0 : http://cassandra.apache.org/doc/old/CQL-3.0.html

            : DataStax DevCenter for Cassandra : https://academy.datastax.com/downloads?dxt=DX#Studio

            : NoSQL Manager for Cassandra Tool (30days) : https://www.mongodbmanager.com/cassandra

      5) Graph  Database

        -  Neo4j Database : https://neo4j.com


    * Notification

        - AWS SES(Amazon Simple Email Service)

            : AWS SES Developer Guide : http://docs.aws.amazon.com/ses/latest/DeveloperGuide/Welcome.html

            : Send an Email Throught Using the AWS SDKforJava : http://docs.aws.amazon.com/ses/latest/DeveloperGuide/send-using-sdk-java.html

        - AWS SNS(Amazon Simple Notification Service) : http://docs.aws.amazon.com/sns/latest/dg/welcome.html

        - GCM (Google Cloud Messaging) 

        - FCM (Firebase Cloud Messaging) : the new version of GCM : https://firebase.google.com/docs/cloud-messaging

     

    * Query Engine

        - Presto (Distributed SQL Query Engine for Big Data) : https://prestodb.io/

        - Amazon Athena (Serverless Query Service - Amazon S3) : https://aws.amazon.com/athena/

     

    * Apache Spark 

        - Apache Spark : http://spark.apache.org

        - Download Apache Spark : http://spark.apache.org/downloads.html

        - Download Scala IDE for Eclipse : http://scala-ide.org/download/sdk.html

        - Download IntelliJ IDEA : https://www.jetbrains.com/idea/download/

        - Submit Application(spark-submit) : https://spark.apache.org/docs/latest/submitting-applications.html

        - Apache Spark Streaming

            : Spark Streaming Programming Guide : http://spark.apache.org/docs/latest/streaming-programming-guide.html

            : DStream : http://spark.apache.org/docs/latest/streaming-programming-guide.html#discretized-streams-dstreams

        - sparkjava (spark+java+web framework CRUD) : http://sparkjava.com/

            : GitHub : https://github.com/perwendel/spark

        - Spark Cassandra Connector : https://github.com/datastax/spark-cassandra-connector


    * Apache Storm

        - Apache Storm : http://storm.apache.org/


    * R

        - R (cran) : https://cran.r-project.org

        - R Manual : https://cran.r-project.org/manuals.html 

        - R bloggers : https://www.r-bloggers.com/

        - Cookbook for R Graphs(ggplot2) : http://www.cookbook-r.com/Graphs/

        - Download R 3.4.0 for Windows : https://cran.r-project.org/bin/windows/base/R-3.4.0-win.exe

        - RStudio : https://www.rstudio.com/products/RStudio/

            : Download RStudio Desktop : https://download1.rstudio.org/RStudio-1.0.143.exe

     

    * H2O

        - H2O.ai : https://www.h2o.ai/


    * Data Mining Tools

        - Weka :  https://sourceforge.net/projects/weka/

        - Knime :  https://www.knime.com/

           

    * Data Analytics Visualization

        - Apache Zeppelin (web-based notebook): http://zeppelin.apache.org/

        - Grafana : http://grafana.com/ 

        - D3(Data-Driven Documents) : https://d3js.org/  

        - Cytoscape (Network Graph) : http://www.cytoscape.org/download.php

        - Gephi (Network Graph) : https://gephi.org/users/download/

     

    저작자표시

    'DEMO_INDEX' 카테고리의 다른 글

    Project Demo  (0) 2017.02.20
    IoT AI  (0) 2016.11.28
    Reference  (0) 2016.08.31
    IoT H/W Platform  (0) 2016.08.31
    IoT Sensor  (0) 2016.08.31
    Posted by RDIoT
    |

    티스토리툴바