Spark working with a Cloud IDE: Notebook/Shiny Apps
2 likes752 views
The document provides an overview of IBM's Data Science Experience, highlighting its features such as tutorials, community collaboration, and support for various programming languages like Scala, Python, and R. It emphasizes the platform's integration with tools like Apache Spark, RStudio, and Shiny apps, aimed at enhancing data analysis and model deployment. Additionally, it outlines the accessibility of data sources and the ability to connect with a variety of cloud and on-premises databases.
![© 2015 IBM Corporation17
Legal Disclaimer
• © IBM Corporation 2014. All Rights Reserved.
• The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained
in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are
subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing
contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and
conditions of the applicable license agreement governing the use of IBM software.
• References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or
capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to
future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by
you will result in any specific sales, revenue growth or other results.
• If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete:
Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will
experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage
configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here.
• If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete:
All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs
and performance characteristics may vary by customer.
• Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM
Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server).
Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your
presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in
your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International
Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both.
• If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete:
Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other
countries.
• If you reference Java™ in the text, please mark the first use and include the following; otherwise delete:
Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both.
• If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete:
Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both.
• If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete:
Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States
and other countries.
• If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete:
UNIX is a registered trademark of The Open Group in the United States and other countries.
• If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete:
Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of
others.
• If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta
Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration
purposes only.](https://image.slidesharecdn.com/dsx-meetup-blocpower-170517184339/85/Spark-working-with-a-Cloud-IDE-Notebook-Shiny-Apps-17-320.jpg)
Spark working with a Cloud IDE: Notebook/Shiny Apps
- 1. © 2015 IBM Corporation IBM Data Science Experience Overview
- 2. © 2015 IBM Corporation2 Contents § About me… § Data Science Experience § Community - Fork and Share § Open Source - RStudio - Shiny Web App Framework § IBM Value-Add - IBM Analytics using Apache Spark - Object Storage - Data-Platform Connectors
- 3. © 2015 IBM Corporation3 About me… § My name is Thomas…Hi J § Open Source Systems Engineer for 11 years and he has 8 years of experience in Cloud and hybrid environments. § Prior to IBM I worked as a Systems Architect, DevOps / Systems Engineer – Cloud Operations § I like…Spark, Python, Linux, Configuration Management tools, Hadoop, and wrangling cats…
- 4. © 2015 IBM Corporation4 Built-in learning to get started or go the distance with advanced tutorials Learn The best of open source and IBM value-add to create state-of-the-art data products Create Community and social features that provide meaningful collaboration Collaborate Visit: http://datascience.ibm.com Introducing the Data Science Experience
- 5. © 2015 IBM Corporation5 IBM Data Science Experience Community Open Source IBM Added Value Powered by IBM DataWorks Platform in the Cloud • Find tutorials and datasets • Connect with Data Scientists • Ask questions • Read articles and papers • Fork and share projects • Code in Scala/Python/R/SQL • Jupyter and Zeppelin* Notebooks • RStudio IDE and Shiny apps • Apache Spark • Your favorite libraries • Data Shaping/Pipeline UI * • Auto-data preparation and modeling* • Advanced Visualizations* • Model management and deployment* • Documented Model APIs* • Spark as a Service * DSX product roadmap items Core Attributes of the Data Science Experience
- 6. © 2015 IBM Corporation6 Tailored Experiences For Users Collaborating Together Architects how data is organized & ensures operability Gets deep into the data to draw hidden insights for the business Works with data to apply insights to the business strategy Plugs into data and models & writes code to build apps Ingest data Transform: clean Create and build model Evaluate Deliver and deploy model Communicate results Understand problem and domain Explore and understand data Transform: shape OUTPUT ANALYSIS INPUT Data Engineer Data Scientist Business Analyst App Developer DataPlatform Forge Data Science Experience Watson Analytics Bluemix
- 7. © 2015 IBM Corporation7 DSX has RStudio built into the experience…
- 8. © 2015 IBM Corporation8 Modelling Energy Usage in NYC – BlocPower Blog Link: http://ibm.co/29KLbvu "BlocPower operation is diverse from outreach and targeting, origination of investment-grade clean energy projects to financing projects through our crowdfunding marketplace. Data is the underlying tool of our operation and IBM's Data Science Experience will facilitate a closer integration across it and help our business scale up faster. “ — Tooraji Arvajeh, Chief Engineering Officer, BlocPower
- 9. © 2015 IBM Corporation9 Use Shiny apps to share your analysis with business users
- 10. © 2015 IBM Corporation10 Interactively explore the analysis of your data science team
- 11. © 2015 IBM Corporation11 Adjust parameters on-the-fly and visualize model predictions
- 12. © 2015 IBM Corporation12 BigInsights (HDFS) Cloudant (DBaaS) dashDB (Analytics) Swift (Object Storage) SQDB (Managed DB2) Data Sources IBM Cloud Public Cloud Cloud Apps On-Premises Execute SQL Statements Streaming Analytics via Micro-batch M.L. and Statistical Algorithms Distributed Graph Processing Framework § General compute engine § Basic I/O functions § Task dispatching § Scheduling Spark Core Spark SQL Spark Streaming MLlib Machine Learning Graph From a Notebook you can use IBM Analytics for Apache Spark to blend multiple data types, sources, and workloads
- 13. © 2015 IBM Corporation13 The Spark Service uses Bluemix Object Storage as its preferred data store for building performant applications § Object storage provides inexpensive, scalable and self-healing retention of massive amounts of unstructured data § Every object exists at the same level in a flat address space § Bluemix Object Storage has a drag-and-drop upload and Swift API for programmatic access § DataPlatform Connectors enable users to easily move data in and out of Bluemix Object Storage
- 14. © 2015 IBM Corporation14 All of the supported targets are compatible with each source Supported Data Sources for DSX via on- premises and cloud Connectors Cloud Sources On-Premises Sources Cloud Targets On-Premises Targets Amazon Redshift Apache Hive Amazon S3 IBM DB2® LUW Amazon S3 Cloudera Impala Bluemix Object Storage IBM Pure Data for Analytics® Apache Hive IBM DB2® LUW IBM Cloudant™ Teradata Bluemix Object Storage IBM Informix® IBM dashDB IBM BigInsights™ on Cloud * IBM Pure Data for Analytics® IBM BigInsights™ on Cloud * IBM Cloudant™ Microsoft SQL Server IBM DB2® on Cloud IBM dashDB MySQL Enterprise Edition IBM SQL Database IBM DB2® on Cloud Oracle IBM Watson™ Analytics IBM SQL Database Pivotal Greenplum PostgreSQL on Compose Microsoft Azure PostgreSQL SoftLayer Object Storage PostgreSQL on Compose Sybase Salesforce Sybase IQ SoftLayer Object Storage Teradata
- 15. © 2015 IBM Corporation15 IBM DSX KEY FEATURES Sparkling Data Prescriptive Analytics Shiny Data Connections Schedule Jobs §Self Service Data Science platform DSX Notebooks Data Community Data Shaping RStudio Projects Scheduling
- 16. © 2015 IBM Corporation16
- 17. © 2015 IBM Corporation17 Legal Disclaimer • © IBM Corporation 2014. All Rights Reserved. • The information contained in this publication is provided for informational purposes only. While efforts were made to verify the completeness and accuracy of the information contained in this publication, it is provided AS IS without warranty of any kind, express or implied. In addition, this information is based on IBM’s current product plans and strategy, which are subject to change by IBM without notice. IBM shall not be responsible for any damages arising out of the use of, or otherwise related to, this publication or any other materials. Nothing contained in this publication is intended to, nor shall have the effect of, creating any warranties or representations from IBM or its suppliers or licensors, or altering the terms and conditions of the applicable license agreement governing the use of IBM software. • References in this presentation to IBM products, programs, or services do not imply that they will be available in all countries in which IBM operates. Product release dates and/or capabilities referenced in this presentation may change at any time at IBM’s sole discretion based on market opportunities or other factors, and are not intended to be a commitment to future product or feature availability in any way. Nothing contained in these materials is intended to, nor shall have the effect of, stating or implying that any activities undertaken by you will result in any specific sales, revenue growth or other results. • If the text contains performance statistics or references to benchmarks, insert the following language; otherwise delete: Performance is based on measurements and projections using standard IBM benchmarks in a controlled environment. The actual throughput or performance that any user will experience will vary depending upon many factors, including considerations such as the amount of multiprogramming in the user's job stream, the I/O configuration, the storage configuration, and the workload processed. Therefore, no assurance can be given that an individual user will achieve results similar to those stated here. • If the text includes any customer examples, please confirm we have prior written approval from such customer and insert the following language; otherwise delete: All customer examples described are presented as illustrations of how those customers have used IBM products and the results they may have achieved. Actual environmental costs and performance characteristics may vary by customer. • Please review text for proper trademark attribution of IBM products. At first use, each product name must be the full name and include appropriate trademark symbols (e.g., IBM Lotus® Sametime® Unyte™). Subsequent references can drop “IBM” but should include the proper branding (e.g., Lotus Sametime Gateway, or WebSphere Application Server). Please refer to http://www.ibm.com/legal/copytrade.shtml for guidance on which trademarks require the ® or ™ symbol. Do not use abbreviations for IBM product names in your presentation. All product names must be used as adjectives rather than nouns. Please list all of the trademarks that you use in your presentation as follows; delete any not included in your presentation. IBM, the IBM logo, Lotus, Lotus Notes, Notes, Domino, Quickr, Sametime, WebSphere, UC2, PartnerWorld and Lotusphere are trademarks of International Business Machines Corporation in the United States, other countries, or both. Unyte is a trademark of WebDialogs, Inc., in the United States, other countries, or both. • If you reference Adobe® in the text, please mark the first use and include the following; otherwise delete: Adobe, the Adobe logo, PostScript, and the PostScript logo are either registered trademarks or trademarks of Adobe Systems Incorporated in the United States, and/or other countries. • If you reference Java™ in the text, please mark the first use and include the following; otherwise delete: Java and all Java-based trademarks are trademarks of Sun Microsystems, Inc. in the United States, other countries, or both. • If you reference Microsoft® and/or Windows® in the text, please mark the first use and include the following, as applicable; otherwise delete: Microsoft and Windows are trademarks of Microsoft Corporation in the United States, other countries, or both. • If you reference Intel® and/or any of the following Intel products in the text, please mark the first use and include those that you use as follows; otherwise delete: Intel, Intel Centrino, Celeron, Intel Xeon, Intel SpeedStep, Itanium, and Pentium are trademarks or registered trademarks of Intel Corporation or its subsidiaries in the United States and other countries. • If you reference UNIX® in the text, please mark the first use and include the following; otherwise delete: UNIX is a registered trademark of The Open Group in the United States and other countries. • If you reference Linux® in your presentation, please mark the first use and include the following; otherwise delete: Linux is a registered trademark of Linus Torvalds in the United States, other countries, or both. Other company, product, or service names may be trademarks or service marks of others. • If the text/graphics include screenshots, no actual IBM employee names may be used (even your own), if your screenshots include fictitious company names (e.g., Renovations, Zeta Bank, Acme) please update and insert the following; otherwise delete: All references to [insert fictitious company name] refer to a fictitious company and are used for illustration purposes only.