SIXTH SENSE TECHNOLOGY

WEAR UR WORLD

SUBMITTED BY:
NEHA ROHILLA SUBMITTED TO: Mr. SANDEEP SINGH

TABLE OF CONTENTS

Abstract.3 Introduction...3 Need of SIXTH SENSE TECHNOLOGY....3 Devices(Components used)...4 Working.4 How to run the software component of sixth sense...4 3 kinds of gesture recognized.5 How the device make search over the internet..5 Technologies related to sixth sense devices..5 Recognition of object held in hand.....5 Advantages of technology..5 Applications6 Limitations..6 Upcoming projects......6 Future scope6 Conclusion..6 Acknowledgement..6 References .6

SIXTH SENSE TECHNOLOGY WEAR UR WORLD

Abstract- Information is traditionally confined to paper or

digitally to a screen. In this paper, we introduce WUW, a wearable gestural interface, which attempts to bring information out into the tangible world. By using a tiny projector and a camera mounted on a hat or coupled in a pendant like wearable device, WUW sees what the user sees and visually augments surfaces or physical objects the user is interacting with. WUW projects information onto surfaces, walls, and physical objects around us, and lets the user interact with the projected information through natural hand gestures, arm movements, or interaction with the object itself. Sixth Sense is a wearable gestural interface that augments the physical world around us with digital information and lets us use natural hand gestures to interact with that information.

way you want, in the simplest way possible. The device brings us closer to reality and assists us in making right decisions by providing the relevant information, thereby, making the entire world a computer. The technology is mainly based on hand gesture recognition, image capturing, processing, and manipulation, etc. The software of the technology uses the video stream, which is captured by the camera, and also tracks the location of the tips of the fingers to recognize the gestures. This process is done using some techniques of computer vision. He invented Sixth Sense / WUW ( Wear UR World) which is a wearable gestural , user friendly interface which links the physical world around us with digital information and uses hand gestures to interact with them.

Keywords Gestural Interaction, Augmented Reality, Wearable

Interface, Tangible Computing, Object Augmentation .

I.

INTRODUCTION
Figure 2

Sixth Sense Technology bridges this gap, bringing intangible, digital information out into the tangible world, and allowing us to interact with this information via natural hand gestures. Sixth Sense frees information from its confines by seamlessly integrating it with reality, and thus making the entire world your computer. (Figure 2) WUW was developed by Pranav Mistry, a Ph. D student at Fluid Interfaces Group at the MIT Media Lab. The Sixth Sense prototype implements several applications that demonstrate the usefulness,

II.

NEEDS OF SIXTH SNSE TECHNOLOGY:

(Figure1) When we encounter something, someone or someplace, we use our five natural senses to perceive information about it; that information helps us make decisions and chose the right actions to take. But arguably the most useful information that can help us make the right decision is not naturally perceivable with our five senses, namely the data, information and knowledge that mankind has accumulated about everything and which is increasingly all available online. Although the miniaturization of computing devices allows us to carry computers in our pockets, keeping us continually connected to the digital world, there is no link between our digital devices and our interactions with the physical world.

FIGURE

III.

DEVICES(FIGURE 4)

viability and flexibility of the system acts as the computer and your connection to the Cloud, all the information stored on the web. The key here is that Sixth Sense recognizes the objects around you, displaying information automatically and letting you access it in any

Camera: It captures the image of the object in view and track the users hand gesture. The camera recognizes individuals, images, pictures, gestures that user makes with his hand.

Coloured Marker: There are colour markers placed at the tip of users finger. The movements and arrangement of these markers are interpreted into gestures that act as a interaction instruction for the projected application interfaces. Mobile Component: The Sixth Sense device consists of a web enabled smart phone which process the data send by the camera. The smart phone searches the web and interprets the hand gestures with help of the coloured markers placed at the finger tips. Projector: The projector projects the visual information enabling surfaces and physical objects to be used as interfaces.A tiny LED projector displays the data sent from the smart phone on any surface in viewobject, wall or person. The downward facing projector projects the image on to a mirror. Mirror: The usage of a mirror is important as the projector dangles pointing downwards from the neck. The mirror reflects the image on to a desire surface. Thus finally the digital image is freed from its confines and placed in the physical world.

The downward-facing projector projects the output image on to the mirror Mirror reflects image on to the desired surface Thus, digital information is freed from its confines and placed in the physical world.

The sixth sense technology uses different technologies like gesture recognition, image processing, etc

. Figure 4 Both the projector and the camera are connected to the mobile computing device in the users pocket. Recognition is made using computer vision technique. These markers act as visual tracking fiducials. The software program processes this video stream data and interprets the movements into gestures. The gestures are different from one another and are assigned some commands. All this intelligence comes from the computer vision software and machine learning technology. The camera also sees what the user sees. It not only captures the gestures, but also the scene and objects around it.

IV. WORKING [5]

V. HOW TO RUN THE SOFTWARE COMPONENT OF SIXTHSENSE [7]

The prototype system runs on windows platform and majority of the code is written in C++ and C Instructions for running WUW_v0.1 WUW_v0.1 is the beta version of SixthSense code (WUW stands for Wear Ur World, a former name of the project). This version runs on Windows. It require Visual Studio and Direct X installed. Camera: choose which camera to use and configure the camera settings here. Tokens: This version of code uses colored fingertips as markers. One need 4 different colors in order to run most of the Apps.Recommended to use Red, Yellow, Blue and Green (or colors that are different enough).

Figure 3
The hardware that makes Sixth Sense work is a pendant like mobile wearable interface It has a camera, a mirror and a projector and is connected wirelessly to a Bluetooth or3G or wifi smart phone that can slip comfortably into ones pocket The camera recognizes individuals, images, pictures, gestures one makes with their hands Information is sent to the Smartphone for processing

SOFTWARE RECOGNIZE 3 KINDS OF GESTURES:

VI.

1.Multitouch gestures ,like the ones u see in Microsoft surface or iphone where you touch the screen and make the map move by pinching and dragging. 2.Freehand gestures ,like when you take a picture . 3.Iconic gestures ,drawing a icon in the air .Like if drawn a star ,show me weather . If drawn magnifying glass ,show me the map .

matching and depending on situation category is decided for the further operation. 1.MATCHING:Processed image is matched with all prestored images stored the smartphone with the help of Image Co-relation Matching Techniques.This technique helps to correlate the relation between the selected image as a input and stored information about ahe object held in hand in the smartphone and the object identity is done and appropriate action is done such as review of the book,information about the products, details about the proteins,fats,carbohydrates etc in a given product. In this category the information which is already stored in the smartphone is used instead of gathering the information from internet. Depending on users requirement the internet can also be used for updated information 2.FIDUCIAL:-

VII. HOW DOES THE DEVICE MAKE THE SEARCH OVER THE INTERNET?
The device is connected to the cloud. It uses a lot of search engine application program interfaces (APIs) like Amazon APIs and so on. While it connects you to the Internet world, it still enables to access all the dynamic information/data while you continue being in the physical world.

VIII. TECHNOLOGIES DEVICES

THAT ARE RELATED TO

SIXTH SENSE

Fiducial area of object is selected to optimize for fiducial matching process.In the next step fiducial marker such as barcode detection is performed and if the object is detected then the object identity of the detected fiducial is found out and object details are collected with the help of smartphone by using smartphone.If the object is not detected then it will again go to starting condition and will try to gather information about the object which is held by user in his hand, In this way information about the product is found out and this collected information is projected on the object.

Augmented Reality: The augmented reality is a visualization technology that allows the user to experience the virtual experience added over real world in real time. Augmented reality adds graphics, sounds, hepatic feedback and smell to the natural world as it exists. Gesture Recognition It is a technology which is aimed at interpreting human gestures with the help of mathematical algorithms. Gesture recognition technique basically special type of hand gloves which provide information about hand position orientation and flux of the fingers. Computer Vision: Computer Vision is the technology in which machines are able to interpret necessary information from an image. This technology includes various fields like image processing, image analysis and machine vision. It includes certain aspect of artificial intelligence techniques like pattern recognition. Radio Frequency Identification Radio Frequency Identification systems transmit the identity of an object wirelessly, using radio magnetic waves. The main purpose of this technology is to enable the transfer of a data via a portable device. This technology is widely used in the fields like asset tracking, supply chain management, manufacturing, payment system etc.

X.

ADVANTAGES OF SIXTH SENSE DEVICES

Portable:

1X.

RECOGNITION OF OBJECT

HELD IN HAND

Whenever user holds an object the camera captures the view of the object in users hand and after that video streaming of data takes place .Then whatever image is selected the different operations are performed on that image such as edge detection , cropping and processing of selected image. The processed ima ge is compared for

All the devices are light in weight and the smart phone can easily fit into the users pocket. Support multi touch and multi user interaction: Multi sensing technique allows the user to interact with system with more than one than one finger at a time.. Sixth sense devices also incooperates multi user functionality Cost effective: A typical sixth sense device cost up to $300. Connectedness between real world and digital world: Forming a connection between the real world and the digital world was the main aim of the sixth sense technology. Data access directly from the machines in real time: With help of a sixth sense device the user can easily access data from any machine at real time speed. Mind map the idea anywhere: With the advent of the sixth sense device, requirement of a platform or a screen to analyse and interpret the data has become obsolete.

Open source software: The software that is used to interpret and analysis the data collected by the device is made open source.

XIV.

FUTURE SCOPE

_ To get rid of color markers. _ Applying this technology in various interest like gaming, education systems etc. . _ To have 3D gesture tracking. _ To make sixth sense work as fifth sense for disabled person.

XI. APPLICATIONS
Map Application: With the help of a map application the user can call upon any map of his/her choice and navigate through them by projecting the map on to any surface . Taking Pictures:This camera takes the photo of the location user is looking at by detecting the framing gesture. Drawing Application:The drawing application allows the user you to draw on any surface by tracking the fingertip movements of the users index finger. Making Calls: The sixth sense device is used to project the keypad into your palm and the using that virtual keypad we can make calls to anyone. Interacting with physical objects:It augments physical objects by projecting more information about these objects projected on them. Book Information:By holding and shuffling through the book pages, the sixth sense provides Amazon ratings on that book, other reviews and other relevant things related to the book. Flight Updates:The system will recognize your boarding pass and let you know whether the flight is on time or not. XII

XV. CONCLUSION
The sixth sense technology and devices are definitely the way forward. The sixth sense recognizes the objects around us and displays the information relating to those objects in a real time environment. The sixth sense technology allows the user to interact the information through hand gestures. This is a quiet efficient way compared to the text and graphic based user interface.

ACKNOWLEDGEMENT
It is with great sense of satisfaction that I present my real venture in computer science in form of a report. I wish to express my heartfelt thanks to all those who assisted me during the research.

REFRENCES
[1] [2] [3]

WHAT HOLDING IT BACK NOW?

Improved image recognition technology is a must for more fluid AR experience. Faster processor,sharp cameras,more accurate sensors etc..

[4]

XIII. WAY FORWARD UPCOMING PROJECTS MAKING USE OF THE SIXTH SENSE TECHNOLOGY
Mouse less: Mouse less is a concept which is developed in order to avoid the usage of mouse in computers. It helps to provide a familiarity of the physical mouse without actually requiring a physical mouse. Third Eye: It is a new technique that enables multiple viewers to see different things on a same display screen at the same time. TaPuma: TaPuma is the short form of Tangible Public Map. It is another attempt to bring the digital world nearer to the real world by exploring the techniques of sixth sense technology.

[5]

[6]

[7]

http://www.ted.com/talks/pattie_maes_demos_the_sixth_sense.h tml http://www.pranavmistry.com P. Mistry, P. Maes. Sixth Sense A Wearable Gestural Interface. To be appeared in SIGGRAPH Asia 2009, Emerging Technologies. Yokohama, Japan. 2009 P. Mistry. The thrilling potential of Sixth Sense technology TED India 2009. Mysore, India http://ieeexplore.ieee.org Rao, S.S.; Electron. & Commun. Eng., Anna Univ. of Technol., Coimbatore, India This paper appears in: Communication and Computational Intelligence (INCOCCI),2010InternationalConferenceon IssueDate: 27-29Dec.2010 Onpage(s): 336-339 Location: Erode E-ISBN: 978-81-8371-369-6 ReferencesCited: 11 INSPECAccessionNumber: 11887446 Date of Current Version: 24 March 2011 Agarwal, A., Izadi, S., Chandraker, M. and Blake, A. High precision multi-touch sensing on surfaces using overhead cameras. IEEE International Workshop on Horizontal Interactive Human-Computer System, 2007. http://code.google.com/p/sixthsense/wiki/Software