![Step 3: Predict
Apply the trained model to make predicAons on new data
POST prediction/v1.1/query/mybucket%2Fmydata/predict
{ "data":{
"input": { "text" : [
"J'aime X! C'est le meilleur" ]}}}](https://image.slidesharecdn.com/gae-cloudcomputingexpo-2010-101112225452-phpapp01/85/Google-Cloud-Technologies-Overview-58-320.jpg)
![Step 3: Predict
Apply the trained model to make predicAons on new data
POST prediction/v1.1/query/mybucket%2Fmydata/predict
{ "data":{
"input": { "text" : [
"J'aime X! C'est le meilleur" ]}}}
{ data : {
"kind" : "prediction#output",
"outputLabel":"French",
"outputMulti" :[
{"label":"French", "score": x.xx}
{"label":"English", "score": x.xx}
{"label":"Spanish", "score": x.xx}]}}](https://image.slidesharecdn.com/gae-cloudcomputingexpo-2010-101112225452-phpapp01/85/Google-Cloud-Technologies-Overview-59-320.jpg)
![BigQuery via REST
GET /bigquery/v1/tables/{table name}
GET /bigquery/v1/query?q={query}
Sample JSON Reply:
{
"results": {
"fields": { [
{"id":"COUNT(*)","type":"uint64"}, ... ]
},
"rows": [
{"f":[{"v":"2949"}, ...]},
{"f":[{"v":"5387"}, ...]}, ... ]
}
}
Also supports JSON-RPC](https://image.slidesharecdn.com/gae-cloudcomputingexpo-2010-101112225452-phpapp01/85/Google-Cloud-Technologies-Overview-70-320.jpg)
Google Cloud Technologies Overview
- 4. Google Cloud Development Technologies Google BigQuery Google Predic/on API Google Storage Google App Engine Google App Engine for Business (new) ExisAng New!
- 6. 6 IaaS APaaS SaaS Source: Gartner AADI Summit Dec 2009 Cloud Computing as Gartner Sees It
- 8. 8 The PaaS value proposition
- 11. 11 Some App Engine Partners
- 13. 13 >62M Users Social Networking at Scale
- 17. Cloud development in a box 17 • Downloadable SDK • ApplicaAon runAmes • Java, Python • Local development tools • Eclipse plugin, AppEngine Launcher • Specialized applicaAon services • Cloud based dashboard • Ready to scale • Built in fault tolerance, load balancing
- 18. Specialized Services 18 Blobstore Images Mail XMPP Task Queue Memcache Datastore URL Fetch User Service
- 19. Language runtimes 19 Duke, the Java mascot Copyright © Sun Microsystems Inc., all rights reserved.
- 21. Extended Language support through JVM • Java • Scala • JRuby (Ruby) • Groovy • Quercus (PHP) • Rhino (JavaScript) • Jython (Python) 21 Duke, the Java mascot Copyright © Sun Microsystems Inc., all rights reserved.
- 22. Always free to get started • ~5M pageviews/month • 6.5 CPU hrs/day • 1 GB storage • 650K URL Fetch calls/day • 2,000 recipients emailed • 1 GB/day bandwidth • 100,000 tasks enqueued • 650K XMPP messages/day 22
- 25. App Engine Health History 25
- 26. Development Tools for App Engine 26
- 28. SDK Console 28
- 30. Two+ years in review 30 Apr 2008 Python launch May 2008 Memcache, Images API Jul 2008 Logs export Aug 2008 Batch write/delete Oct 2008 HTTPS support Dec 2008 Status dashboard, quota details Feb 2009 Billing, larger files Apr 2009 Java launch, DB import, cron support, SDC May 2009 Key-only queries Jun 2009 Task queues Aug 2009 Kindless queries Sep 2009 XMPP Oct 2009 Incoming email Dec 2009 Blobstore Feb 2010 Datastore cursors, Appstats Mar 2010 Read policies, IPv6 May 2010 App Engine for Business
- 32. Google App Engine for Business Details • Enterprise applicaAon management – Centralized domain console (preview available today) • Enterprise reliability and support – 99.9% Service Level Agreement – Direct support • Hosted SQL – RelaAonal SQL database in the cloud (preview available today) • SSL on your domain • Secure by default – Integrated Single Sign On (SSO) • Pricing that makes sense – Apps cost $8 per user, up to $1000 max per month 32 Google App Engine for Business
- 33. Enterprise App Development with Google 33 Build your own Google App Engine for Business Buy from others Google Apps Marketplace Enterprise Firewall Enterprise Data AuthenAcaAon Enterprise Services User Management Buy from Google Google Apps for Business Enterprise Application Platform
- 34. 34 App Engine for Business Roadmap Enterprise Administration Console Preview (signups available) Direct Support Preview (signups available) Hosted SQL Preview (signups available) Service Level Agreement Available Q4 2010 (Draft published) Enterprise billing Available Q4 2010 Custom Domain SSL Limited Release EOY 2010
- 36. Part II - Google’s new Cloud Technologies Topics covered • Google Storage for Developers • Prediction API (machine learning) • BigQuery
- 37. Google Storage for Developers Store your data in Google's cloud
- 38. What Is Google Storage? • Store your data in Google's cloud o any format, any amount, any Ame • You control access to your data o private, shared, or public • Access via Google APIs or 3rd party tools/libraries
- 41. Google Storage Technical Details • RESTful API o Verbs: GET, PUT, POST, HEAD, DELETE o Resources: identified by URI o Compatible with S3 • Buckets o Flat containers • Objects o Any type o Size: 100 GB / object • Access Control for Google Accounts o For individuals and groups • Two Ways to Authenticate Requests o Sign request using access keys o Web browser login
- 42. Performance and Scalability • Objects of any type and 100 GB / Object • Unlimited numbers of objects, 1000s of buckets • All data replicated to mulAple US data centers • Leveraging Google's worldwide network for data delivery • Only you can use bucket names with your domain names • Read‐your‐writes data consistency • Range Get
- 43. Security and Privacy Features • Key‐based authenAcaAon • AuthenAcated downloads from a web browser • Sharing with individuals • Group sharing via Google Groups • Access control for buckets and objects • Set Read/Write/List permissions
- 44. Demo • Tools: o GS Manager o GSUAl • Upload / Download
- 45. Google Storage usage within Google HaiA Relief Imagery USPTO data Partner ReporAng Google BigQuery Google Predic/on API Partner ReporAng
- 47. Google Storage - Pricing o Storage $0.17/GB/Month o Network Upload - $0.10/GB Download $0.15/GB Americas / EMEA $0.30/GB APAC o Requests PUT, POST, LIST - $0.01 / 1000 Requests GET, HEAD - $0.01 / 10000 Requests
- 48. Google Storage - Availability • Limited preview in US currently o 100GB free storage and network from Google per account o Sign up for waitlist at hp://code.google.com/apis/storage/ • Note: Non US preview available on case‐by‐case basis
- 49. Google Storage Summary • Store any kind of data using Google's cloud infrastructure • Easy to Use APIs • Many available tools and libraries o gsuAl, GS Manager o 3rd party: Boto, CloudBerry, CyberDuck, JetS3t, and more
- 50. Google Prediction API Google's prediction engine in the cloud
- 51. Introducing the Google Prediction API • Google's sophisticated machine learning technology • Available as an on-demand RESTful HTTP web service
- 52. How does it work? "english" The quick brown fox jumped over the lazy dog. "english" To err is human, but to really foul things up you need a computer. "spanish" No hay mal que por bien no venga. "spanish" La tercera es la vencida. ? To be or not to be, that is the quesAon. ? La fe mueve montañas. The Prediction API finds relevant features in the sample data during training. The PredicAon API later searches for those features during predicAon.
- 54. A PredicAon API Example AutomaAcally categorize and respond to emails by language • Customer: ACME Corp, a multinational organization • Goal: Respond to customer emails in their language • Data: Many emails, tagged with their languages • Outcome: Predict language and respond accordingly
- 55. Using the Prediction API 1. Upload 2. Train Upload your training data to Google Storage Build a model from your data Make new predicAons 3. Predict A simple three step process...
- 56. Step 1: Upload Upload your training data to Google Storage • Training data: outputs and input features • Data format: comma separated value format (CSV) "english","To err is human, but to really ..." "spanish","No hay mal que por bien no venga." ... Upload to Google Storage gsutil cp ${data} gs://yourbucket/${data}
- 57. Step 2: Train Create a new model by training on data To train a model: POST prediction/v1.1/training?data=mybucket%2Fmydata Training runs asynchronously. To see if it has finished: GET prediction/v1.1/training/mybucket %2Fmydata {"data":{ "data":"mybucket/mydata", "modelinfo":"estimated accuracy: 0.xx"}}}
- 58. Step 3: Predict Apply the trained model to make predicAons on new data POST prediction/v1.1/query/mybucket%2Fmydata/predict { "data":{ "input": { "text" : [ "J'aime X! C'est le meilleur" ]}}}
- 59. Step 3: Predict Apply the trained model to make predicAons on new data POST prediction/v1.1/query/mybucket%2Fmydata/predict { "data":{ "input": { "text" : [ "J'aime X! C'est le meilleur" ]}}} { data : { "kind" : "prediction#output", "outputLabel":"French", "outputMulti" :[ {"label":"French", "score": x.xx} {"label":"English", "score": x.xx} {"label":"Spanish", "score": x.xx}]}}
- 60. Step 3: Predict Apply the trained model to make predicAons on new data import httplib header = {"Content-Type" : "application/json"} #...put new data in JSON format in params variable conn = httplib.HTTPConnection("www.googleapis.com")conn.request("POST", "/prediction/v1.1/query/mybucket%2Fmydata/predict”, params, header) print conn.getresponse() An example using Python
- 61. PredicAon API CapabiliAes Data • Input Features: numeric or unstructured text • Output: up to hundreds of discrete categories Training • Many machine learning techniques • Automatically selected • Performed asynchronously Access from many platforms: • Web app from Google App Engine • Apps Script (e.g. from Google Spreadsheet) • Desktop app
- 62. PredicAon API v1.1 ‐ new features • Updated Syntax • Multi-category prediction o Tag entry with multiple labels • Continuous Output o Finer grained prediction rankings based on multiple labels • Mixed Inputs o Both numeric and text inputs are now supported Can combine continuous output with mixed inputs
- 63. Google BigQuery Interactive analysis of large datasets in Google's cloud
- 64. Introducing Google BigQuery • Google's large data adhoc analysis technology o Analyze massive amounts of data in seconds • Simple SQL-like query language • Flexible access o REST APIs, JSON-RPC, Google Apps Script
- 67. Key CapabiliAes of BigQuery • Scalable: Billions of rows • Fast: Response in seconds • Simple: Queries in SQL • Web Service o REST o JSON-RPC o Google App Scripts
- 69. WriAng Queries Compact subset of SQL o SELECT ... FROM ... WHERE ... GROUP BY ... ORDER BY ... LIMIT ...; Common functions o Math, String, Time, ... Statistical approximations o TOP o COUNT DISTINCT
- 70. BigQuery via REST GET /bigquery/v1/tables/{table name} GET /bigquery/v1/query?q={query} Sample JSON Reply: { "results": { "fields": { [ {"id":"COUNT(*)","type":"uint64"}, ... ] }, "rows": [ {"f":[{"v":"2949"}, ...]}, {"f":[{"v":"5387"}, ...]}, ... ] } } Also supports JSON-RPC
- 71. Security and Privacy Standard Google Authentication • Client Login • OAuth • AuthSub HTTPS support • protects your credentials • protects your data Relies on Google Storage to manage access
- 73. Using BigQuery Shell Python DB API 2.0 + B. Clapper's sqlcmd http://www.clapper.org/software/python/sqlcmd/
- 76. Recap • Google Storage o High speed data storage on Google Cloud • Prediction API o Google's machine learning technology able to predict outcomes based on sample data • BigQuery o Interactive analysis of very large data sets o Simple SQL query language access
- 77. Further info available at: • Google Storage for Developers o http://code.google.com/apis/storage • Prediction API o http://code.google.com/apis/predict • BigQuery o http://code.google.com/apis/bigquery
- 78. Q&A