Philipp Nagele (Wikitude) Wikitude SDK Tutorial
- 2. Agenda for today ● Who is Wikitude ● What has the Wikitude SDK to offer －Architecture and APIs －Plugins API ● Latest Additions to the Wikitude SDK – 3D Tracking
- 3. 3
- 4. v v v Wikitude AR at a glance. First AR app in the world in 2008 10,000 apps powered by wikitude SDK 3D tracking technology v REFERENCES
- 5. Wikitude’s path to success 20142013 2015 Business Model: B2C Focus: Wikitude AR App 2016201220112009 Founder stage in Salzburg garage: First AR Browser launched. 2010 VCs brought in., Management hired: Martin Herdina, Andy Gstoll Established Wikitude Inc. in USA Business Model: B2B Focus: AR technology and tools Achieved 25 million downloads and app installs gloabally Launch of Wikitude SDK for 2D image recognition Launch of Wikitude Studio, the AR CMS Launch of Wikitude Cloud Recognition service Launch of 3D tracking SLAM
- 6. Wikitude offices wikitude GmbH (HQ) in Salzburg Austriawikitude Inc. in San Francisco, USA wikitude Inc. wikitude GmbH.
- 7. AR creator and CMSCloud Recognition Wikitude AR Products & Tools Connect your own CMS
- 8. Native API JavaScript API Easy access to AR mobile development Xamarin PhoneGap Cordova titaniumunity Android iOS
- 9. GEO / sensor based AR Pioneering products and technologies
- 10. Maturing products and technologies 2D Image Recognition & Tracking
- 11. Tailored SDKs for Smart Glasses Partnership since 2014, optimized SDK for BT-200, full documentation →BT-300 in pipeline Partnership since 2016 optimized SDK for R-7 Partnership since 2014, SDK for M100, full documentation →M300 in pipeline
- 12. Wikitude SDK Architecture 2D On-Device Reco & Tracking 2D Cloud Reco 3D Reco & Tracking Unity3D Unity Plugin Plugins API Custom CV plugin Custom camera plugins Your App Your App Cordova Plugin Titanium Module Xamarin Component Your App Cordova Titanium Xamarin Your App Your App JavaScript APINative API Your App 3D Rendering Augmen- tations LBS Plugin ManagerOpenGL ES2 Rendering ARMv7 NEON optimizations ARMv8 optimizations Camera GPU optimizations OpenCL RenderScript Metal API IMU Hardware Abstraction Hardware Optimizations C++ Layer
- 13. Plugins API - Modularization 13 Description The Plugins APIs allows to extend the Wikitude SDK by 3rd party functionality. Plugins for the Wikitude SDK have access to the camera information, recognized images. This is perfect for additional functionality that also requires camera access. Plugins are written in C++ and can communicate both with the JavaScript API and the Native API.
- 14. Plugins API in detail 14 Custom Frame Provider Image Frame Standard Frame Provider Wikitude CV Engine Rendering Custom CV Plugin Plugins API class Plugin { public: … void cameraFrameAvailable(const Frame&; cameraFrame_); void update(const vector<RecognizedTarget> recognizedTargets_); void startRender(); void endRender … }; 1 2 3 4 5 6 7 class RecognizedTarget { public: const string& getIdentifier() const; const Mat4& getModelViewMatrix() const; const Mat4& getProjectionMatrix() const; const float getDistanceToCamera() const; };
- 15. Use-Cases for Plugins API ● Run own computer-vision related code in parallel to Wikitude SDK －OCR Engine －Barcode Reader －Face Recognition ● Own the camera stream and the camera rendering －Custom effects and shaders －Remote camera stream with local processing
- 16. JavaScript Augmented Reality Experience 16 Architect Worlds consist of: • .html file(s) • .js file(s) • .css file(s) • Defines your augmented reality experience • Use the Wikitude JS API provided within the Architect view context <script src=“architect://architect.js"></script> • Loaded by the Wikitude SDKs Architect view
- 17. Architect world 17 Javascript execution triggers actions in the underlying C++ layer
- 18. Architect view composition 18 OpenGL ES 2 view * Renders camera stream * Renders augmentations Web view * Loads main .html file * Transparent background
- 19. GeoObjects & HTML content Menu bar jQuery support Geo-located POIs Multiple Drawables - 3 images - 3 text labels Web View Augmented Reality View
- 20. Support for jQuery and web APIs
- 21. Image Recognition Augment target images using JavaScript API capabilities (Drawables, Sounds, Animations,Videos, 3D Models) var kitchenTracker = new AR.Tracker("Kitchen.wtc"); var overlay = new AR.VideoDrawable("howdens.mp4", 0.65); var trackable2DObject = new AR.Trackable2DObject( kitchenTracker, "Kitchen", { drawables: { cam: overlay }});
- 22. Types of augmentations 22 Augmentations are represented by AR.Drawable subclasses • Images - AR.ImageResource / AR.ImageDrawable • Videos (also with alpha channel) - AR.VideoDrawable • 3D models - AR.Model • Web views - AR.HTMLDrawable • Labels - AR.Label
- 23. ….architecture, real estate….. ….gaming, filming industry….. ….indoor navigation, retail….. New algorithms and concepts 3D Tracking / SLAM Object Reco & Tracking
- 24. Recognition & tracking (core engine) 3D One core engine with many “flavors” Small Scale Scenes / Spaces / Objects - SLAM - Point cloud - Smaller scenes - desktop,small rooms and environments Object Recognition Edge Detection / CAD models - pre-registeredCAD models and displayed silhouettes - Focus: shapes & lines Positional Tracking Indoor Navigation” - Computer vision plusIMU / sensor fusion - Malls, airports, buildings - Untethered indoor gaming in apartment,houseor anywhere Large Scale Scenes Spaces - SLAM based - Point cloud - buildings / spaces - indoors, outdoors
- 25. 3D Tracking Flavors 1. Object Recognition Edge Detection / CAD models Object Recognition Edge Detection / CAD models - pre-registeredCAD models and displayed silhouettes - Focus: shapes & lines
- 26. Wikitude Lab – RnD Sneak Preview
- 27. 3D Tracking Flavors 2. Large Scale Scenes / Spaces Large Scale Scenes Spaces - SLAM based - Point cloud - buildings / spaces - indoors, outdoors Watch here: https://www.youtube.com/watch?v=Zqb6Fo4uRkM
- 28. Positional Tracking – Demo Video
- 29. Philipp Nagele, CTO Wikitude See more.