In Jun 2016, within a week of its release, Pokémon Go became the most downloaded app in Google Playstore and Apple’s App Store. Players were so engrossed in the game that they forgot to eat, go to office and pick up their kids. No, I am not exaggerating. The reason?
People enjoyed the way it used real environments as gaming interface. A throwback to every player’s childhood, it was akin to playing hide and seek in your own neighbourhood. It showed once again what augmented reality (AR) could do to a simple game, and how receptive people are to the idea. Mobile phones are often blamed for breaking the users’ connect with the real world. But AR strives to bridge that gap.
Why Mobile is leading the Way in AR Technologies
Though the term augmented reality was coined in 1992 by Caudell and Mizell at Boeing, the basic concept of ultimate display, which gave rise to first VR (Virtual Reality) and then AR was proposed by Ivan Sutherland in 1965. You can read the essay in the book Augmented Reality: Principles and Practice or the relevant chapter here.
AR systems consist of these three important components:
- Sensor for tracking and registering real objects and environments.
- Processing unit for analysing sensor data to generate augmentations.
- Display for augmented data.
Mobile phones provide all these and then some more features that support AR technologies:
- Additional sensors like gyroscopes, accelerometers, magnetometers, etc. that make tracking and registering more accurate.
- Instant user feedback that helps in correcting errors, if any, in the augmented data being presented.
Challenges for Mobile AR Technologies
However, mobiles have their own challenges in implementing AR technologies, which we will discuss now.
Limited Computational Power
The most important component in AR is recognizing, tracking, registering and rendering real-time environment and its objects. This is achieved through computer vision technologies. Traditionally, computer vision uses tracked or track-less techniques. For mobiles, track-less technique is possible because of availability of mobile cameras. However, algorithms that use track-less techniques need high computation power, which is not available in mobiles.
High Power Consumption
Processing the algorithms used in AR technologies used in mobiles drains the battery life fast. Also, constant use of camera, GPS systems, etc. for real-time tracking of objects also uses mobile battery quite quickly. With latest mobile batteries flaunting 5000 mAh ratings and better architectural support, future looks bright but affordable high battery life is still some steps away.
Time Lag
The foremost requirement of mobile users is immediate display of the augmented reality display. But current AR technologies introduce some time lag between a user capturing his environment with camera and the augmented display on his screen. This is because the algorithm needs to process the real time information, occlude real time objects with virtual objects (or information) and then display the new augmented reality.
Memory and Hardware Limitations
Mobile augmented reality technologies typically function in always on-always augmented mode. The amount of data captured by the mobile phones is not just huge but humongous in this mode. The current memory and hardware architecture provided by mobile phone manufacturers is not sufficient to hold that much data, let alone process it. A possible solution to tackling memory limitations is offloading the gathered data to cloud. However, this requires continuous internet connection, and availability of cheap and reliable cloud storage.
Display Requirements
All the richness of augmented reality will be lost if the mobile display is not equipped to render the augmented images or video for the users without losing any dimension. Mobile companies need to provide affordable and high-quality displays in their mobiles for full impact and utilization of this AR technologies. Currently, smartphones having such displays are exorbitantly priced.
Environmental Restrictions
The quality of data gathered by a mobile camera depends on amount of light, humidity, noise and other environmental conditions over which humans have no control. In case of poor lighting or background noise, the environmental data provided will be poor and hence the final augmented output will also be poor.
Approaches to Tackling Mobile Augmented Reality Challenges
The approach to tackling challenges posed by implementing augmented reality in mobiles has to be two pronged. One, mobile hardware architecture must evolve to provide better computational power and two, AR algorithms and computer vision techniques must be optimized for higher performance in lesser resources. Scientists – both theorists and experimenters – are busy working to make AR more feasible on mobiles. Here we discuss a few of those attempts.
Mobile Architecture
Technology companies involved in manufacturing mobile processors, display units, batteries and other components are engaged in research to support augmented reality in mobiles. For example, Qualcomm’s Snapdragon professes to be the next step to mobile AR while Movidius, an Intel company, is busy rolling out highly optimized vision algorithms, vision processing units, optimized AR function libraries and more.
Study on Natural Feature Tracking for MAR Applications
Natural feature tracking (NFT) is a technique that tracks objects in real time by determining the pose of an object with respect to a known surface in the surrounding environment. This assumes that there are some parts of the environment that are already known to the system. NFT has found many takers in the field of augmented reality because it professes an ability to integrate vision with graphics to produce wide-area augmented realities.
A study done in 2015 evaluated two possible solutions for using Natural Feature Tracking for mobile AR applications. These solutions tried to transfer part of processing to a server and then tested their performance on Android platform.
Augmented Reality Engine Application Project
Augmented Reality Engine Application (AREA) project tries to provide a flexible architecture to develop location-based mobile augmented reality applications. The kernel architecture, over which the individual applications can be built, takes care of efficient utilization of mobile computational abilities. So the application developers need to concentrate solely on developing applications rather than worrying about optimizing AR techniques and algorithms.
Given the advancements in computational technologies and higher demand for location based tracking, a new kernel called AREA version 2 or AREAv2 was developed. The new kernel worked with Points of Interest (POI) in the mobile camera scan region. The algorithms ensured that:
- POIs were displayed correctly even when the camera was held obliquely
- Multiple POIs could be displayed on screen
- POI concept could be integrated with common mobile OS like Android, iOS, Windows, etc.
Some of these algorithms are shown here:
Besides GPS, AREAv2 uses a set these complex co-ordinates:
- ECEF stands for Earth Centred Earth Fixed
- ENU stands for East North Up
Implementing SURF Algorithm in OpenCV
Originally developed by Intel, OpenCV, short for Open Computer Vision, is a library of functions for computer vision applications.
SURF (Speeded Up Robust Features) algorithm extracts a set of keypoints and descriptors for each image, which it stores for future detection of the same image. Working with SURF keypoints and descriptors for an image speeds up calculations in a specific region.
Applying SURF algorithm to computer vision makes object detection independent of scale and rotation, once the keypoints and descriptors are detected. This has great connotations for implementing augmented reality in mobiles because it will reduce processing and memory requirements, thereby further decreasing battery draining and heating.
SURF algorithm was removed from the default installation package of OpenCV 3.0 because the algorithm was patented by its creator. You need to pull down the SURF algorithm from GitHub and run the OpenCV package again before you can use its keypoints and descriptor methods and functions.
The Future
Augmented Reality is logically the next greatest evolution step for smartphones. All the parties invested in this future are putting their best effort in improving current technologies and coming up with newer ones to fulfil this dream.
What has been your experience with augmented reality? Whether you are a mobile user or a neo AR addict tinkering with codes and algorithm, we would love to know your thoughts on this. Do write about them in the comments below (you need to log in to comment).
Log in with Google
Comments
Report comment