Data Scientist's Daily Life
Download as PPTX, PDF•14 likes•5,487 views
A data scientist's daily life involves collecting and storing large amounts of data from various sources, preprocessing and analyzing the data using tools like Linux, SQL databases, Python and R, and applying machine learning algorithms like clustering, classification, and regression to derive insights. The data scientist must effectively manage terabytes of data and choose the appropriate machine learning techniques and algorithms to gain knowledge from big data in an efficient and intelligent manner. Visualization tools are then used to showcase the findings and insights discovered.
Data Scientist's Daily Life
- 1. DATA SCIENTIST’S DAILY LIFE BRYAN YANG 2015.09
- 2. A B O U T M E • Blog Bryan的行銷研究及資料分析筆記 http://bryannotes.blogspot.tw • Group Spark.TW
- 3. A G E N D A • Data scientist? • Big data and data scientist • Data scientist’s Toolbox • Data is the biggest
- 6. F R O M B A C K E N D T O F R O N T E N D https://doubleclix.wordpress.com/2012/12/15/what-or-who-is-a-data-scientist/
- 7. WHAT IS BIG DATA?
- 8. WHERE DO THE DATA COME FROM • Web Log data • Machine data • Transactional data • Social media data • …
- 11. A WEB SERVICE RECEIVE THE LOG DATA MORE THEN 50G PER DAY TOTAL SPACE USED LAST THREE MONTH ：4500G TOTAL SPACE USED LAST ONE YEAR ：18,000G(17.6T)
- 12. • Data Storage/ Backup • 2T/per HDD • How to save the data MORE than 2T? • $0.3 USD/per gigabyte • Pay 900 USR for KEEPING data but do nothing else. • Read/Write Speed • Read: 131.6 MB/s / Write 131.4MB/s • Spend 393s(6 min) reading just ONE day data. • Large number of transactions immediately
- 14. H A D O O P A N D H D F S http://www.fraudtechwire.com/f-level-guide-to-hadoop-hdfs/
- 17. – D I S T R I B U T E D A L G O R I T H M 「The world will change，when data is distributed」
- 18. M A P R E D U C E http://www.milanor.net/blog/?p=853
- 19. https://chamibuddhika.wordpress.com/2012/02/26/joins-with-map-reduce/
- 21. P E R F O R M A N C E O F H A D O O P ? • Not good, but at least can run. • Count 86,389,084 rows/per day in 39 sec. (64G ram, E5 8core * 2/per node * 10) • How about 39sec * 30days ?
- 22. BEFORE ANALYTIC…
- 23. E X T R A C T T R A S F O R M L O A D m/e-university/data-warehouse-etl-toolkit-tutorial-201/surrounding-the-requirements-1
- 27. L I N U X • The best server choice • Free and freedom • Easy to control system • Easy data processing • Hadoop is based on Linux
- 29. P O W E R F U L S H E L L S C R I P T
- 30. S Q L D A T A B A S E • MySql, Postgresql, Hive, MongoDB(NOSQL) • Standard SQL Language • Store and Manage data
- 31. R E L A T I O N A L D A T A B A S E
- 32. T A B L E R E L A T I O N https://cloudant.com/blog/foundbites-data-model-relational-db-vs-nosql-on-cloudant/
- 34. S Q L S Y N T A X
- 35. R & P Y T H O N • Basic Analysis Tools • Easy to Learn • Many Packages
- 38. • Example • http://bryannotes.blogspot.tw/2014/08/r-ptt- wantedsocial-network-analysis.html • http://bryannotes.blogspot.tw/2014/10/python-k- means-script.html
- 39. E T C … • Excel • Google Analytics • Visualisation tools (tableau) • Web Crawler • Version control management (git) • ETL and job scheduling tools (jenkins) • …
- 40. D A T A I S T H E B I G G E S T
- 41. – J O S H W I L L S “Person who is better at statistics than any software engineer and better at software engineering than any statistician.”
- 42. S T A T I S T I C
- 43. W H Y D O W E N E E D M A C H I N E L E A R N I N G ? • Clustering 這些人可以分成幾類 • Classification 哪個人屬於哪一類？ • Regression 某個事件發生或某人屬於哪類的機率是多少？ • Dimensionality reduction 降維
- 44. C L U S T E R I N G http://simplystatistics.org/2014/02/18/k-means-clustering-in-a-gif/ source http://humble-developer.blogspot.tw/2011/01/kmeans-clustering-algorithm-part-1.html
- 45. C L A S S I F I C A T I O N http://letsmakerobots.com/content/tcs3200-color-sensor-with-k-nearest-neighbor-classification-algorithm
- 47. L O G I S T I C R E G R E S S I O N https://www.coursera.org/instructor/andrewng
- 48. C O S T F U N C T I O N https://www.coursera.org/instructor/andrewng
- 49. O V E R F I T T I N G https://www.coursera.org/instructor/andrewng
- 50. OH MY GOD! HOW TO CHOOSE IT
- 51. M A C H I N E L E A R N I N G A L G O R I T H M N http://amueller.github.io/sklearn_tutorial/
- 52. S T A T I S T I C V S M L S T A T T I S T I C M A C H I N E L E A R N I N G F O C U S O N U N D E R S T A N D I N G D A T A I N T E R M S O F M O D E L S F O C U S O N T H E A N A L Y S I S O F L E A R N I N G A L G O R I T H M S I N T E R P R E T A B I L I T Y , H Y P O T H E S I S T E S T I N G G R E A T E R F O C U S O N P R E D I C T I O N
- 53. S Y S T E M A T I C S A N D A U T O M A T I O N http://www.slideshare.net/CetasAnalytics/cetas-e-baymeetupprezofinal
- 55. SHOW YOUR DATA AND FINDINGS
- 60. THE REAL CASE
- 61. HOW TO START?
- 62. • Codecademy http://www.codecademy.com/ Include kinds of programming language, i.e. python, JavaSrtipt, even shell script and sql • Coursera http://www.codecademy.com/ Famous self-learning MOOC website.