Augmented or Mixed Reality libraries and frameworks for Android and iOS

Augmented or Mixed reality (AR), is a live direct or indirect view of a physical, real-world environment whose elements are augmented by computer-generated sensory input such as sound, video, graphics or GPS data.

Augmented reality enhances one’s current perception of reality, whereas in contrast, virtual reality replaces the real world with a simulated one.

Mobile VR is VR that is using a mobile phone as the primary viewing device (display), often with a headset like Google Cardboard or others.

Building AR and VR systems needs a completely different approach and set of tools. For mobile VR Software Development Kit (SDK) one of most supported SDKs is Google VR that works with Android, iOS, Unity, Unreal. There a few smaller SDKs for mobile, or SDKs for 360 videos in VR.

Wikipedia provides a good description of the work for typical AR systems (mostly they are built based on Computer Vision and most of the SDK/APIs or solutions use OpenCV)

Also you can see typical use-cases are there, as well as a list of well known AR SDKs (Vuforia, ARToolKit, Catchoom CraftAR Mobinett AR, Wikitude, Blippar, Layar, Meta and ARLab

Also Quora has a question about popular AR SDK for mobile devices and here is short list of them:

• ARLab – http://www.arlab.com
• ARmsk – http://armsk.org
• ARToolKit – http://sourceforge.net/projects/… (Supports many devices)
• Moodstocks – http://moodstocks.com
• popcode – http://www.popcode.info/demos (Supports Android and iPhone)
• QCAR – Qualcomm Augmented Reality SDK – https://ar.qualcomm.com/qdevnet/sdk (Android & iOS)
• SATCH – http://satch.jp (powered by Total Immersion – http://www.t-immersion.com)
• Kudan – Kudan.eu
• catchoom
• Vuforia – https://www.vuforia.com
• Wikitude – https://www.wikitude.com/download

Overall there more than 50+ SDKs and they are even grouped by different criteria on wiki-lists. Also this table compares the SDKs and can help you choose one depending on the functions you need.

There is a similar comparison for mobile platforms [1] [2]
From among the live projects (many are dead, some have been bought by Apple) the most discussed are Vuforia, Wikitude, ARToolKit, Kudan, and Catchoom.

IMU sensors – accelerometers, gyroscopes, magnetometers.
And here is some introductory information about those libraries that we found.

ARToolKit

ARToolKit is a set of augmented reality software tools that can be used in AR apps. Its main benefit is an open-source code that implies free access to the library.
ARToolKit supports:

• 2D recognition
• mapping additional elements via OpenGL

The library allows tracking known in advance object markers through a mobile device camera and reproducing their location on a device screen. Then a developer can create an augmented reality interface using the received data.

ARToolKit serves the following different platforms: Android, iOS, Windows, Linux, Mac OS X, SGI. Each operating system needs its own development environment. The development environments are free for all the mentioned platforms.

Despite the benefit of free access to this AR library, the downside it that the development documentation is quite limited. It includes the test apps, but not all of them can be easily built. The examples are very poor, and there is no information about any plans to update the framework.

Catchoom

Commercial versions of SDKAndroid, iOSVisualSearch are licensed by Catchoom Technologies and offer tools to connect your branded mobile application with the CraftAR service in the cloud.

Kudan AR

The Kudan functionality consists of:

• image recognition
• mapping additional elements on the basis of user location and recognized images
• markerless tracking (instead of fiducial marks it relies on the use of natural features like the edges, corners, or textures)
• mapping additional elements via separate component over OpenGL

Kudan is faster than other frameworks. This library helps mobile AR apps to map multi-polygonal models in real time and import 3D models from one of the modeling software packages. In addition, the number of recognized images is not limited and it requires less memory to store files in a device.

Developers can use the basic documentation but the framework manual is short and needs additional information. Also, there is a chance that you can suffer from the limited built-in functionality without a direct access to OpenGL.

In general, the augmented reality libraries described above have a wide range of opportunities for app development for the support of different platforms and to use the set of tools for recognizing and tracking objects.

Nevertheless, when it’s time to choose a particular framework, a developer needs to understand what he gets. Some of the tools you can use for free, by simply going to the website or downloading the software program. Other tools require signing a contract and paying a fee, but they provide more features and provide higher quality. When choosing one of the AR tools, think of your project tasks and results you want to achieve, and, after all, match them with the abilities of the AR libraries.

Vuforia

Being a complete SDK for augmented reality app development, Vuforia supports:

• the detection of several kinds of targets (including objects, images, English text)
• target tracking
• 2D and 3D recognition
• scanning real objects for recognition
• virtual buttons
• mapping additional elements via OpenGL
• Smart TerrainTM, a capability to reconstruct a terrain in real time, creating a 3D geometric map of the environment
• Extended Tracking, a capability that delivers a continuous visual experience even when the target is out of view

In particular, using Vuforia for detecting images, mobile apps can access data that are either local on the device or stored in the Cloud.

The main advantages of the framework include the support of virtual reality devices and a test app with comments showing the Vuforia abilities.

However, the absence of a complete framework manual leads to complications for the developers working with Vuforia for the first time. Although, there are many specific instructions and short tips, they are in random order and cannot replace the required documentation.
The use of cloud recognition has limitations in the free version of Vuforia. Also, a watermark appears in this version once a day.

Vuforia SDK Benefits

Multi-target detection, target tracking, virtual buttons, Smart TerrainTM, and Extended Tracking are some of the features of Vuforia SDK. Vuforia supports the detection of several kinds of targets (e.g. objects, images, English text). Vuforia is especially good for image recognition purposes since Vuforia allows apps to use databases that are either local on the device or in the Cloud, The platform is available for Android, iOS and Unity. There is also a version of the SDK specifically for smart glasses (namely Epson Moverio BT-200, Samsung GearVR, and ODG R-6 and R-7) that is currently moving to its beta phase and is open for early access applications from qualified developers.

Wikitude

Wikitude library supports:

• 2D and 3D recognition,
• scanning real objects for recognition,
• 3D model rendering and animation,
• location tracking,
• HTML augmentation.

Using Wikitude, developers can create apps to reconstruct places at the virtual map or in the list, to do a search of events, tweets, Wiki articles, or to get the recommendations from other users. Besides these things a Wikitude-based app also allows users to receive mobile coupons, information about current specials, and also to play in AR games.

Wikitude can be used for both Android and iOS, as a plugin for PhoneGap, a module for Titanium and a component for Xamarin. The Wikitude SDK includes SLAM (markerless tracking) and supports the Unity programming language. This framework is available for smart glasses Google Glass, Epson Moverio, Vuzix M-100 and ODG R-7.

There is a free trial version for developers. When you want to use a complete version, you need to be ready for regular payments. As for the documentation, it is well structured and detailed.

Wikitude AR SDK

Wikitude AR SDK supports image recognition and tracking, 3-D model rendering and animations (supports only the Wikitude 3-D format), video overlays, location-based tracking and image, text, button, video and HTML augmentations. Wikitude AR SDK is available for Android, iOS, Google Glass, Epson Moverio, Vuzix M-100 and Optinvent ORA1, and as a plugin for PhoneGap, a module for Titanium and a component for Xamarin.