Fluentd Plugins for CouchDB, Amazon SQS/SNS
6 likes6,023 views
An Introduction to Fluentd Plugins for Couchdb,Amazon SQS/SNS. at "Fluentd Meetup in Tokyo." http://www.zusaar.com/event/193104 #fluentd
![[AWS Dev Day] 앱 현대화 | AWS Fargate를 사용한 서버리스 컨테이너 활용 하기 - 삼성전자 개발자 포털 사례 - 정영준...](https://cdn.slidesharecdn.com/ss_thumbnails/appmodernizationawsfargate-190930044252-thumbnail.jpg?width=560&fit=bounds)
Ad
More Related Content
What's hot (10)
Similar to Fluentd Plugins for CouchDB, Amazon SQS/SNS (20)
Ad
Recently uploaded (20)
Ad
Fluentd Plugins for CouchDB, Amazon SQS/SNS
- 1. An Introduction to ... Fluentd plugins for CouchDB, Amazon SQS, Amazon SNS @ixixi
- 2. $ whoami @ixixi (Yudai Odagiri) Engineer of Future Architect, Inc. Hadoop Ustreamer CORSS, MongoTokyo2012, , GraphDB , Twitter , MongoDB ...... Ustream
- 3. Agenda CouchDB? fluent-plugin-couch demonstration Amazon SQS? fluent-plugin-sqs Amazon SNS? fluent-plugin-sns next features...
- 4. CouchDB? Relax!
- 5. CouchDB? Document-oriented database MVCC (MultiVersion Cuncurrency Control) Auto versioning (No “update”) RESTfull HTTP JSON APIs
- 6. CouchDB? Small footprint run on mobile devices ( android / iOS ) Multi-master Replication Replication (via HTTP)
- 7. CouchDB? Written in Erlang Not only DB Server, but also Application & Web Server. Built-in web GUI “Futon”
- 9. CouchDB MapReduce User-defined query structured as M/R MapReduce View (index) view index is built lazily View Definition (MapReduce) Result
- 10. CouchDB MapReduce Incremental view index update “possible to use M/R to query huge partitioned clusters in realtime, instead of having to wait for a whole M/R job to complete or having stale, occasionally updated indexes.” ~Damien Katz http://damienkatz.net/2008/02/incremental_map.html implementation details { omitted } this is Lightning Talk :-P see : http://horicky.blogspot.com/2008/10/couchdb- implementation.html
- 12. fluent-plugin-couch fluentd (buffered) output plugin store fluent-event as CouchDB Document to CouchDB install $ gem install fluent-plugin-couch <match couch.**> configuration type couch database fluent host ixixi.cloudant.com port 80 # for Basic Authentication (optional) user ixixi password ********** # Other buffer configurations.... </match>
- 13. Demo count tweets in public timeline (use Streaming API) group by timezone and language in real-time. M/R view update (incremental) bulk insert Streaming API fluentd flush interval => 5sec. buffer
- 14. fluent-plugin-sqs
- 15. why need queue? to stop and update receiver application to consider the state of receiver servers Receiver servers ( on which heavy processes are running) may not be able to deal with messages. fluentd
- 16. Amazon SQS Amazon Simple Queue Service (SQS) Distributed Queue Service since 2006 ( the oldest service of AWS ) Automatic Replication in data centers High Availability High Scalability Secure
- 17. why SQS? Queue Server requires high availability Generally, high availability queue server is very expensive Amazon SQS is inexpensive $0.000001( per request ) free tier (~100,000 requests/month) $~0.201 /GB (Data transfer out)
- 18. fluent-plugin-sqs fluentd (buffered) output plugin store fluent-event to Amazon SQS install $ gem install fluent-plugin-sqs configuration <match sqs.**> type sqs aws_key_id {AWS_KEY_ID} aws_sec_key {AWS_SEC_KEY} queue_name fluent-queue sqs_endpoint sqs.ap-northeast-1.amazonaws.com # Other buffer configurations.... </match>
- 19. fluent-plugin-sns
- 20. Amazon SNS Amazon Simple Notification Service (SNS) multi-protocol e-mail,email(json),http(s),SMS,SQS(!!) Installation not required push-base architecture high availability very inexpensive
- 21. Amazon SNS Manager,Publisher,Subscriber Model manager makes “topic” and set subscribers to “topic(s)” Publisher sends message to SNS topic, then SNS pushes message to subscribers
- 22. Amazon SNS Subscriber Publisher e-mail SNS fluentd SMS Subscriber Topic A send HTTP Subscriber Topic B Manager Subscriber
- 23. Amazon SNS Publisher(->fluentd) is not Manager. Should be written many subscriber information in “fluent.conf” ? ---- No.
- 24. fluent-plugin-sns fluentd output plugin send fluent-event to Amazon SQS install $ gem install fluent-plugin-sns configuration <match sns.**> type sns aws_key_id {AWS_KEY_ID} aws_sec_key {AWS_SEC_KEY} sns_topic_name = fluentd-topic sqs_endpoint sqs.ap-northeast-1.amazonaws.com sns_subject_key title </match>
- 26. next feature...? fluent-plugin-couch “update document” e.g.) To handle events such as updating the status of the server. can refer to previous documents. ∵ CouchDB is MVCC based , automatic versioning. fluent-plugin-sqs input-plugin fluent-plugin-sns customize formatting & multi formatting
- 27. thank you!
Editor's Notes
- #2: \n
- #3: \n
- #4: \n
- #5: &#x5148;&#x9031;CouchConfTokyo&#x304C;&#x3042;&#x3063;&#x305F;.(100&#x4EBA;&#x898F;&#x6A21;)2&#x5272;&#x3050;&#x3089;&#x3044;&#x5916;&#x56FD;&#x4EBA;&#x3060;&#x3063;&#x305F;&#x3089;&#x3057;&#x3044;.&#x65E5;&#x672C;&#x3067;&#x306F;Mongo&#x304C;&#x5727;&#x5012;&#x3057;&#x3066;&#x3044;&#x308B;&#x611F;&#x304C;&#x3042;&#x308B;&#x304C;&#x3001;&#x4E16;&#x754C;&#x3067;&#x306F;CouchDB&#x3082;&#x7D50;&#x69CB;&#x4F7F;&#x308F;&#x308C;&#x3066;&#x3044;&#x308B;.\n\n
- #6: \n
- #7: \n
- #8: \n
- #9: \n
- #10: \n
- #11: \n
- #12: \n
- #13: \n
- #14: \n
- #15: \n
- #16: \n
- #17: \n
- #18: \n
- #19: \n
- #20: \n
- #21: \n
- #22: \n
- #23: \n
- #24: \n
- #25: \n
- #26: \n
- #27: \n
- #28: \n