![skyalert.orgCyclic workflow graphTriggerCRTS[“Geometry”][“Moon angle”] > 30and SDSS[“Photoprimary”][“g-magnitude”] < 18Actionannotatorfollowup requestdynamically loads modulerun(triggerEvent, portfolio): <business logic>can build event and inject recursivelysend messageAlerts and event cascade18](https://image.slidesharecdn.com/uwescience-091115091553-phpapp01/85/Virtual-Science-in-the-Cloud-19-320.jpg)
Virtual Science in the Cloud
- 1. Virtual Sciencein the CloudRoy WilliamsCalifornia Institute of Technology
- 2. humanscloudssensorsbeginner to expertsharinglogins and accessclick to code to workflowpersonal storagebig data and replicationcompute and scalingsoftware as componentinteroperabiltysurvey and eventcontrol or autonomousThe New Science
- 4. Service Oriented Architecture3. bindservicerequestrequestclientresponseresponse2. findservice contractregistry1. publishPrinciple: Click or Code
- 5. VO Data ServicesCone Searchradius+position list of objects encoded as VOTableSimple Image Access ProtocolSimple Spectrum Access Protocolspectra have subtleties protocol more complicatedAstronomical Data Query LanguageFor database queriesCore SQL functions plus astronomy-specific extensionsSky region, XmatchTable Access ProtocolExposes relational databasesWhat tablesWhat table schemaHere is a query in ADQL
- 6. VO Compute ServicesAsynchronousMay not get immediate answerjust get a place to check backSecurityExpensive resources, big requests, sequestered dataStrong or Weak or NoneScalableGraduated path to powerful computation and big dataCloud storeVOSpaceSharable
- 7. VO Registrypublish -- find -- bindRegistry MetadataDescriptions of data collections data delivery servicesorganizations, etc.Based on Dublin Core with astronomy-specific extensionsRepresented as XML schema; extensibleContents stored in Resource Registries exchange metadata records through the Open Archives Initiative Protocol (OAI-PMH)
- 8. Distributed RegistryAstrogridCfANCSACDSESOSTScI/JHUNOAOCaltechHEASARCJapanVOOngoing harvesting March 07(CfA, ESO, NOAO soon)
- 9. Semantics & SearchIdentifiers ivo://nasa.gsfc.gcn/SWIFT#BAT_GRB_Pos_374875-722Free tags beard Fred pudding Controlled Vocab (UCD) phot.flux;em.irControlled Vocabinterop (SKOS)Ontology Greek isA Man, Socrates isA Greek Socrates isA ManData Models Each sky position will have a circular positional error estimate ...Text markup Outflows from <object>NGC 666</object> are irregular ...Schema Columns are Magnitude, Position, Identifier , ...Metadata (registry) forms Full Registry: true; ManagedAuthorities: authority, nasa.heasarcFormal service description
- 10. Cloud Based Toolscode & presentationdata
- 12. Open SkyQuery.netVO Astronomical Crossmatch Service Query builder
- 13. PresentationExecution Query planning
- 14. Query execution
- 15. WorkflowInternationalAuthorsSubscribersGCN Brokerannotation from archivesAstronomersAmateursStudentsMicrolensingOptical transientsRadio transientsX-ray transientsGamma transientsskyalert.orgEvents and annotation disseminated to subscribers in real time with intelligenceFollow-up SchedulerTelescopeTelescopeTelescope
- 16. SkyalertPush-based workflowCan be cyclicPortfolio aggregation by citationAnnotation as software componentsStream owner builds templateDjango, Python, Jquerynow 4 developers via SVN
- 17. Skyalert Stream Registry... will be VO registry
- 18. Roleshuman or robot1. browsequery, human computing, WWT/Googleskyalert.orghuman or robot2. subscribehuman or robot3. author4. annotatecontrib software componentsarchive, miningpushinjectwebportfolios dbIM/tweet/email/TCPtriggersactions
- 19. skyalert.orgCyclic workflow graphTriggerCRTS[“Geometry”][“Moon angle”] > 30and SDSS[“Photoprimary”][“g-magnitude”] < 18Actionannotatorfollowup requestdynamically loads modulerun(triggerEvent, portfolio): <business logic>can build event and inject recursivelysend messageAlerts and event cascade18
- 20. Skyalert-LSSTskyalert.orgTest run for LSST mobile app
- 21. Data service from CRTS and Skyalert
- 22. gets JSON event list via http
- 23. LSST building skyalert clone
- 24. Pasadena and Tucson both get events by Jabber/XMPP
- 25. “Unknown” is now choice ofCataclysmic Variable, Supernova, Blazar Outburst, Active Galactic Nucleus Variability, UVCeti Variable, Asteroid, Variable, Mira Variable, High Proper Motion Star, Comet, Eclipsing Variable, Gamma Ray Burst Afterglow, Microlensing, Nova, Planetary Microlensing, RRLyrae Variable, Tidal Disruption Flare
- 26. Tier1 and Tier2 Event NodesEvolving in IVOABrokeringRegistry:Tier1 Stream definitions
- 27. Event ServersTier2AuthoringDistributionJabber/XMPPor raw socketTier1: Immediate Forwarding, Reliable?, Topology?Tier2:Subscription, Repository, Query, Portfolio, Registry, Machine Learning, Substreams etc etc
- 28. NSF Teragrid World’s largest open distributed cyberinfrastructure
- 29. 11 Resource Provider sites, >2 Petaflop HPC & >27000 CPUs, >3 Petabyte disk, >60 PB tape
- 30. Fast network, Visualization, experiments (VMs, GPUs, FPGAs)
- 31. For US researchers and their collaborators through national peer-review processTeragrid 2002job submission and queueing(Condor, PBS, ..)login node100s of nodesuserpurged /scratchparallel I/Oparallel file system/homeglobal file systemmetadata nodeUnix, Globus, C++, ssh, files, MPI, PBS, make
- 32. Architectures 2010Science Gateway (no architecture!)Node farm (condor)Parallel computingMessage-passing MPIShared memoryGraphics Processing Units104 independent tiny threadsData IntensiveFlash memory (TG/UCSD)Graywulf (JHU/Pannstarrs)Immediate resources
- 33. Science GatewaysBiology and Biomedicine Science GatewayOpen Life Sciences GatewayThe Telescience ProjectGrid Analysis Environment (GAE)Neutron Science Instrument GatewayTeraGrid Visualization Gateway, ANLBIRNOpen Science Grid (OSG)Special PRiority and Urgent Computing Environment (SPRUCE)National Virtual Observatory (NVO)Arroyo Adaptive OpticsLinked Environments for Atmospheric Discovery (LEAD)Computational Chemistry Grid (GridChem)Computational Science and Engineering Online (CSE-Online)GEON(GEOsciences Network)Network for Earthquake Engineering Simulation (NEES)SCEC Earthworks ProjectNetwork for Computational Nanotechnology and nanoHUBGIScience Gateway (GISolve)Gridblast Bioinformatics GatewayEarth Systems GridAstrophysical Data Repository (Cornell)Slide courtesy of Nancy Wilkins-Diehr
- 34. GPU for molecular modelling
- 35. Pannstarrs PS1computeUser facingSQL/casjobsworkbenchprivacy/sharestored queriesData valetload/validatemergecrawlreplicatelogworkflowworkflowdatahead/slicehot/warm/coldFault tolerance: multiple replication, fault workflowCost and energy carefully consideredFuture: Hadoop/Mapreduce
- 36. Cloud Supercomputing?Teragrid/Globusvs Cloud/Amazon MIBoth ways to get wholesale computingBoth provide IaaS, Infrastructure as a ServiceVirtual Machine more popular than CTSS stackWhat about parallelism? I/O speed? GPUs? etcWatch 3leaf and ScaleMP for these
- 37. Science and Web 2.0 Easy for groups to form and collaborateIntegrates with user workspaceiGoogle and OpenSocialalongside other aspects of their livesUse existing toolsSlideShare, blogs, google gadgets, facebook, Gwave, Flickr, YouTubeSharing workspaceElectronic logProvenanceVirtual Data as “equivalent script”
- 38. Science and Web 2.0Server delivers only codeBrowser makes presentationAjax and Ajaj and Http “long poll”Jquery and Google toolkitsee WWT and GSky in Skyalert“Everything is a wiki”or a wave?Visible/editable by group/s
- 39. Adaptive Optics Gateway Adaptive optics simulations
- 41. Planet finding coronograph
- 42. 4-day run for 4-sec!
- 43. Parallel parameter sweepsproposed upgrade of the Palomar AO system to a 56x56 subaperture system
- 44. Arroyo
- 45. Arroyo Gateway Architecture1. use HTML/JS from webserver to create job definition.wholesale computing2. Daemon is polling & sees new job, makes local space for it.3. Start job on compute resource & update jpb status.daemon7. User fetches results from webserver4. Fetch &update status of running job. Repeat.5. Output to remote space.webserverDjangoMySQLjob definitions and status5. Daemon copies output from remote to local, updates job status.local space for resultsremote space for resultsretailwholesaleRW and J. Bunn
- 47. E. Deelman, G. Berriman, RW, et al
- 49. now 45,000 jobs per month
- 50. Pegasus for load balancing?Asynchronous services: User needs feedback AJAJ (AJAX but with JSON)
- 51. Detailed progress reports during run
- 52. Strong/weak security model with certificates
- 53. Wide-area Mosaicking158 feetGriffith Observatory, Los Angeles
- 54. Citizen Science
- 55. Human VolunteersScience LayerDescribe what you see in imageEach person has level of expertiseHow to use results most effectivelyGalaxyzoo.org, citizensky.org good modelsGame LayerMakes people come backTop 10 ranking etcAnonymous partner a la gwap.com
- 56. Human Volunteer EvidenceDonalek et alarXiv:0810.4945 [astro-ph] 4 of 10 say artifact artifact
- 57. RW and C. Donalek
- 58. Macromolecule Citizen ScienceA. Cunha
- 60. Classic Machine LearningMetric in “Feature Space”Relevance Vector Machine (Tipping)Feature VectorsLearning from Training setPicking relevant lessonsRW and J. Beck
- 61. New Machine Learning:Information FusionData Portfoliosselected from known set of object typesEvidence objectset of class/proband prior assumptionsmay be correlated priorsAnnotator builds evidencefrom portfoliomay include other evidenceInference (= Expert System)Combines evidence with cost-benefitBuilds ImportanceAlchemy
- 66. Influence DiagramsAutomated Decision through Tripod of DataArchivenearby radio source escalates p(blazar)nearby galaxy escalates p(supernova)HumanCrowded field? Artifact present?Can make follow-up observationMachineFuzzy center escalates p(host galaxy)Moving source escalates p(asteroid)Bobotic follow-up observationdecisionhumanarchivemachinelearning
- 67. Lessons Learned
- 68. User Interface (wrong)and now do some science....Finally get some helpAsk for helpTranslate VOTable formatLearn to use VO RegistryRead about web servicesRead about XMLWait for accountRegister
- 69. User interface (right)in Darwinian evolution every small change must give benefitPower userLearn the VO structurehey this is interesting ....Run bigger jobmore science....Registersome science....Web formAnonymousbe careful with complex authentication!
- 70. Steering the ShipShort term Pragmatismuseful tools nowsimple protocols (eg cone search)“just use RA and Dec”vsLong term Architecturemodular suite of interoperable toolssophisticated protocols (egskynode)sophisticated Space-Time coordinates
- 71. Building Information Standards Documents
- 72. Agreements
- 73. Data Models
- 74. Tight Schema
- 75. Loose Schema
- 76. UML
- 77. XSD
- 78. WSDL
- 79. Semantics
- 80. Meaning
- 81. Usefulness
- 82. Applicability
- 83. Code
- 84. Services
- 85. InterfacesA Data Model is a bridge fromcommunity to computers
- 86. What is a Data Center?machinesservicesdoesn’t matter where or howtesting testing testingdo we have enough power and HVAC?
- 87. Complex scienceComplex machinesSeparate science user from complexityMust have domain science contextMaking simple things simple butPower to scale upDrill-down if wantedMachines are not the objectiveScience through data, compute, sharing
- 88. eScience is for People, right?Getting StartedHelp DeskForumDocumentationKnowledge BaseCalendarContact UsSocial MediaBlog/newsfeedCampus ChampionsSummer SchoolsAdvanced Supportfor DevelopersEducation