CloudCamp Chicago - Big Data & Cloud May 2015 - All Slides
2 likes1,050 views
The May 2015 CloudCamp "unconference" focused on "Big Data and Cloud" About CloudCamp: the event features short lightning talks, an "unpanel" with audience participation and questions, and small breakout clusters around beers and pizza. Hosted by Cohesive Networks at TechNexus. Slides for the night's Lightning Talks: "Big Data without Big Infrastructure" - Dan Chuparkoff, VP of Product at Civis Analytics @Chuparkoff "Simplicity, Storytelling and Big Data" - Craig Booth, Data Engineer at Narrative Science @craigmbooth "Spark: A Quick Ignition" - Matthew Kemp, Architect of Things at Signal @mattkemp "Building warehousing systems on Redshift" - Tristan Crockett, Software Engineer at Edgeflip @thcrock Join us next time. Register at cloudcampchicago.eventbrite.com
![#!/bin/python
regex = re.compile('[%s]' % re.escape(string.punctuation))
def word_count(sc, in_file_name, out_file_name):
sc.textFile(in_file_name)
.map(lambda line: regex.sub(' ', line).strip().lower())
.flatMap(lambda line: [
(word, 1) for word in line.split()
])
.reduceByKey(lambda a, b: a + b)
.map(lambda (word, count): '%s,%s' % (word, count))
.saveAsTextFile(out_file_name)
Example: Word Count](https://image.slidesharecdn.com/cloudcampmay2015master-150511183725-lva1-app6891/85/CloudCamp-Chicago-Big-Data-Cloud-May-2015-All-Slides-43-320.jpg)
![#!/bin/python
regex = re.compile('[%s]' % re.escape(string.punctuation))
def word_count(sc, in_file_name, out_file_name):
sc.textFile(in_file_name)
.map(lambda line: regex.sub(' ', line))
.map(lambda line: line.strip())
.map(lambda line: line.lower())
.flatMap(lambda line: line.split())
.map(lambda word: (word, 1))
.reduceByKey(lambda a, b: a + b)
.map(lambda (word, count): '%s,%s' % (word, count))
.saveAsTextFile(out_file_name)
Example: Alternate Word Count](https://image.slidesharecdn.com/cloudcampmay2015master-150511183725-lva1-app6891/85/CloudCamp-Chicago-Big-Data-Cloud-May-2015-All-Slides-44-320.jpg)
![#!/bin/bash
text=$(cat ${1} | tr "[:punct:]" " " |
tr "[:upper:]" "[:lower:]")
parsed=(${text})
for w in ${parsed[@]}; do echo ${w}; done | sort | uniq -c
A (Bad) Shell Version](https://image.slidesharecdn.com/cloudcampmay2015master-150511183725-lva1-app6891/85/CloudCamp-Chicago-Big-Data-Cloud-May-2015-All-Slides-47-320.jpg)
Ad
More Related Content
What's hot (16)
Viewers also liked (15)
Ad
Similar to CloudCamp Chicago - Big Data & Cloud May 2015 - All Slides (20)
Ad
More from CloudCamp Chicago (20)
Recently uploaded (20)
CloudCamp Chicago - Big Data & Cloud May 2015 - All Slides
- 1. CloudCamp Chicago “Big Data and Cloud” #cloudcamp @CloudCamp_CHI Sponsored by Hosted by
- 2. Emcee Margaret Walker Cohesive Networks Tweet: @CloudCamp_Chi #cloudcamp #cloudcamp @CloudCamp_CHI Sponsored by Hosted by
- 3. … sponsored by you! William Knowles - Evident.io Adam Kallish - IBM Craig Hancock - HealthEngine Brandon Pittman -VMware Chuck Mackie - Maven Wave Partners Brad Foster - Maven Wave Partners Kim Neuwirth - Narrative Science PiaOpulencia - Narrative Science JimStiller - CloudTechnology Partners Networks Brian Lickenbrock - EY
- 4. 6:00 pm Introductions 6:05 pm: Lightning Talks "Big Data without Big Infrastructure" - Dan Chuparkoff,VP of Product at Civis Analytics @Chuparkoff "Simplicity, Storytelling and Big Data" - Craig Booth, Data Engineer at Narrative Science @craigmbooth "Spark:A Quick Ignition" - Matthew Kemp,Team Lead & Engineer of Things at Signal @mattkemp "Building warehousing systems on Redshift" - Tristan Crockett, Software Engineer at Edgeflip @thcrock 7:00 pm: Unpanel 7:45 pm: Unconference / Networking, drinks and pizza Agenda #cloudcamp @CloudCamp_CHI Sponsored by Hosted by
- 5. "Big Data without Big Infrastructure" Dan Chuparkoff VP of Product at Civis Analytics Tweet: @Chuparkoff #cloudcamp #cloudcamp @CloudCamp_CHI Sponsored by Hosted by
- 6. @chuparkoff BIG Data without BIG Infrastructure Dan Chuparkoff VP of Product Civis Analytics
- 7. @chuparkoff Big Data without Big Infrastructure Civis is an easy-to-use, incredibly extensible data science platform in the cloud for teams who want to make great data-driven decisions to drive their organizations forward. I work at Civis
- 8. Big Data without Big Infrastructure@chuparkoff “The ability to use the data that you’ve built up in the past to inform & improve what you’re going to do in the future.” Big Data at Civis Analytics
- 9. @chuparkoff Big Data without Big Infrastructure Data science is too damn hard have a report every day that says what happened yesterday? apply predictive modeling to improve my customer retention? to use data from my past to improve acquisition in the future? Why can’t I… ? ? ?
- 10. @chuparkoff Big Data without Big Infrastructure Everyone’s story • Aggregate • Unify • Explore • Optimize • Share • Automate
- 11. Big Data without Big Infrastructure@chuparkoff Where should we start? Cloud OnPrem vs.
- 12. @chuparkoff Big Data without Big Infrastructure Civis Analytics uses AWS
- 13. @chuparkoff Big Data without Big Infrastructure • No hardware costs and infinitely scalable • Safety and security of AWS • Automatic backups to multiple data centers • Access from any computer with an internet connection
- 14. @chuparkoff Big Data without Big Infrastructure Redshift S3 EC2 DynamoDB RDS EMR
- 15. @chuparkoff Big Data without Big Infrastructure
- 16. @chuparkoff Big Data without Big Infrastructure
- 17. @chuparkoff Big Data without Big Infrastructure
- 18. @chuparkoff Big Data without Big Infrastructure Civis data streams aggregate data from virtually any source. Get all pf your data together in one place. Aggregate From data to activation
- 19. @chuparkoff Big Data without Big Infrastructure Next, Civis’ intelligent matching algorithms link data in disparate data stores. No matter where your data starts, Civis helps you build a unified data repository. Unify From data to activation
- 20. @chuparkoff Big Data without Big Infrastructure Explore and transform the data in a fast analytics database. Explore From data to activation
- 21. @chuparkoff Big Data without Big Infrastructure Build powerful predictive models and easily score results with the Civis platform’s advanced modeling engine. This is the heart of data- driven decision making! Optimize From data to activation
- 22. @chuparkoff Big Data without Big Infrastructure Create, automate, & share reports across your team. Empower your entire organization to move forward with precision. Share From data to activation
- 23. @chuparkoff Big Data without Big Infrastructure When tomorrow comes there’s no need to reinvent the wheel. Civis let’s you automate and schedule from start to finish, so you can get back to pushing boundaries. Automate From data to activation
- 24. @chuparkoff Big Data without Big Infrastructure Big Data + the Cloud + AWS helps Civis Analytics turn an analyst into a data scientist & a data scientist into a team of data scientists. Thanks!
- 25. "Simplicity, Storytelling and Big Data" Craig Booth Data Engineer at Narrative Science Tweet: @craigmbooth #cloudcamp #cloudcamp @CloudCamp_CHI Sponsored by Hosted by
- 26. Simplicity, Storytelling & Big Data Craig Booth What I Wish I Knew About Big Data On Day One.
- 27. My Background data driven science 30+ journal articles; complex analytics on 10s of TB of data data powered storytelling
- 29. lumière léger
- 32. Credit: Josh Bloom Henrik Brink of wise.io
- 33. “…more than 2000 hours of work in order to come up with the final combination of 107 algorithms that gave them this prize” Xavier Amatriain and Justin Basilico, Netflix
- 34. “We evaluated some of the new methods offline but the additional accuracy gains that we measured did not seem to justify the engineering effort needed to bring them into a production environment.” Xavier Amatriain and Justin Basilico, Netflix Explainability Im plem entability AccuracyC an Icom m unicate results? H ow longw illittake m e to build? C an Itolerate som e errors?
- 35. "Spark:A Quick Ignition" Matthew Kemp Team Lead & Engineer of Things at Signal Tweet: @mattkemp #cloudcamp #cloudcamp @CloudCamp_CHI Sponsored by Hosted by
- 36. Spark: A Quick Ignition Matthew Kemp
- 37. Provides distributed processing Main unit of abstraction is the RDD Can be used with frameworks like Mesos or Yarn Supports Java, Python and Scala https://spark.apache.org/ What is Spark?
- 38. Can be created from… Files or HDFS In memory iterable Cassandra or SQL tables Transformations Lazily create a new RDD from an existing one Actions Usually return a value, force computation of RDD Resilient Distributed Dataset
- 41. Sample Text Spark Example Spark Shell Shell Example Gists
- 42. Example: Word Count flatMap()input reduceBy Key() map() outputmap()
- 43. #!/bin/python regex = re.compile('[%s]' % re.escape(string.punctuation)) def word_count(sc, in_file_name, out_file_name): sc.textFile(in_file_name) .map(lambda line: regex.sub(' ', line).strip().lower()) .flatMap(lambda line: [ (word, 1) for word in line.split() ]) .reduceByKey(lambda a, b: a + b) .map(lambda (word, count): '%s,%s' % (word, count)) .saveAsTextFile(out_file_name) Example: Word Count
- 44. #!/bin/python regex = re.compile('[%s]' % re.escape(string.punctuation)) def word_count(sc, in_file_name, out_file_name): sc.textFile(in_file_name) .map(lambda line: regex.sub(' ', line)) .map(lambda line: line.strip()) .map(lambda line: line.lower()) .flatMap(lambda line: line.split()) .map(lambda word: (word, 1)) .reduceByKey(lambda a, b: a + b) .map(lambda (word, count): '%s,%s' % (word, count)) .saveAsTextFile(out_file_name) Example: Alternate Word Count
- 45. $ pyspark ... Using Python version 2.7.2 (default) SparkContext available as sc. >>> from word_count import word_count >>> word_count(sc, 'text.txt', 'text_counts') Running the Example
- 46. a,23 able,1 about,6 above,1 accept,1 accuse,1 ago,2 alarm,2 all,7 although,1 always,2 an,1 The Results From Spark and,26 anger,1 another,1 any,2 anyone,1 arches,1 are,1 arm,1 armour,1 as,7 assistant,2 ...
- 47. #!/bin/bash text=$(cat ${1} | tr "[:punct:]" " " | tr "[:upper:]" "[:lower:]") parsed=(${text}) for w in ${parsed[@]}; do echo ${w}; done | sort | uniq -c A (Bad) Shell Version
- 48. 23 a 1 able 6 about 1 above 1 accept 1 accuse 2 ago 2 alarm 7 all 1 although 2 always 1 an The Results From the Shell 26 and 1 anger 1 another 2 any 1 anyone 1 arches 1 are 1 arm 1 armour 7 as 2 assistant ...
- 49. Our Use Case distinct() 3rd party 3rd party distinct() join() join() union() distinct() foreach() 1st party
- 50. Questions?
- 52. "Building warehousing systems on Redshift" Tristan Crockett Software Engineer at Edgeflip Tweet: @thcrock #cloudcamp #cloudcamp @CloudCamp_CHI Sponsored by Hosted by
- 53. Redshift: Lessons Learned Tristan Crockett – Software Engineer, Edgeflip
- 54. Basics ● Analytical database ● PostgreSQL with column storage engine ● Automatic Data compression ● No traditional indexes; specify a sort key (how are records in the table sorted?) and distribution key (which node will house a record?)
- 55. My Work with Redshift ● Data warehouse for Facebook user feeds and related app data ● Inputs – RDS (MySQL) – DynamoDB – Facebook ● Stats – ~2TB of compressed data – Two main tables, ~5bil and ~25bil rows respectively
- 56. Advantages / Disadvantages ● Fast at copying data in from S3 ● Fast at computing aggregate/analytical functions over an entire table ● Decent at intra-db operations (create table as select, insert into select) ● Most everything else is slow ● Without traditional indexes, table design isn't as flexible
- 57. Lessons/Tips ● Optimize load size (1 MB to 1 GB per file) ● Compress input ● Upsert when needed, and always vacuum ● Don't populate tables with 'CREATE TABLE AS' if you like compression (which you do) ● To avoid complicated joins, consider computing single-table aggregates and join on the results
- 58. Upsert
- 59. Keep an Eye on Compression
- 62. Un-panel Discussion volunteer to join the panel & ask questions from the floor! #cloudcamp @CloudCamp_CHI Sponsored by Hosted by
- 63. Unconference Small groups & discussions, network Pizza’s almost here! #cloudcamp @CloudCamp_CHI Sponsored by Hosted by