If you’ve downloaded Layar v6.2 for Android or iOS and have tried out some of the vision layers, you’ll probably have noticed that tracking was improved tremendously in this version. I’d like to shed a bit of light on what we’ve done technically to get this far.

So what is tracking in the first place? It’s the ability of the device to recognize and follow the object that is being viewed through the camera and determine its position in space. That way, if it’s augmented in a vision layer, the device knows where to draw the augmented content.

Making augments stick

Until v6.1, we were showing the live camera feed in the AR view with the augmented content on top of it. To do that, we’d take the camera frame from the live feed, do lots of calculations to determine the position of the tracked object and then draw the augmented content. That takes some processing time, so that in the end, the drawn content would always be lagging behind the camera images that you saw. 

We’ve changed that in v6.2: We’re now showing the camera frame that belongs to the calculation for the augmented content, so that the content now sticks much better to the object. You can see that by moving the object smoothly: There is a slight delay between your movement and the image you see, but the content really sticks to the object you move. Don’t move too fast, this will blur the image and the app will lose track of it.

Zooming into the augments

There’s another big improvement in 6.2: When a layer developer uploads a reference image (the image of the object that she wants to augment), we create a fingerprint of that image that is used to recognize and track the features of that object. Obviously you need a certain number of features inside the camera view to calculate a good estimate of the position of the object. If the user holds the camera close to the object, zooming into the object, there are less features that can be tracked and at some point we’d lose track of the object.

We’re now adding features dynamically as you zoom in, allowing the app to track an object much longer. This is really nice when the augments on a page are quite small and you need to get closer to read them. 

A cool example is the Eppo comic book, where you can view the original draft drawings on top of many of the pages. Well, you can now also zoom in really close, as shown in the screenshots below. Don’t zoom in too fast though, the app still needs some time to process the images.

Taming the ARM processors

A lot of the code we write for Layar is platform-independent. The entire vision algorithms are written in C++ and integrated into iOS using Objective-C++ and into Android using the JNI. This way we can write once and use the same code on both platforms. This has allowed us since Layar v6.0 to bring simultaneous updates for iOS and Android.

For Layar v6.2 we’ve decided to move one level down and write the processor intensive code directly in assembler using ARM NEON. The cool thing is that up to recently all iOS and Android devices run on ARM processors. Not all of them support NEON (like NVIDIA’s Tegra chips), so not all of them benefit of our new optimized code.

In our reference implementation, we’ve been able to decrease the processing time of certain critical operations by a factor of 8: For example, preparing a frame for display takes 2ms instead of 15ms when using the ARM NEON implementation. Obviously we don’t achieve this same speed enhancement in Layar itself, where there’s a lot more going on, but writing NEON-optimized code is proving a huge improvement.

Thanks to this optimization we were able to add support for streaming video as AR objects without significant loss in performance. We’ve decided to only support dual-core ARM processors, where different tasks like rendering video and analyzing the camera image to calculate the pose estimation can be distributed to different cores.

Now Intel has joined the club with its x86-based Atom processors. That’s a game changer and obviously our NEON-optimized code won’t run there. But we’ve been pleasantly surprised at the performance of the x86 chip. Even without optimization it runs our C++ code at higher speed than similar ARM chips.

We can conclude that with the arrival of Intel in the smartphone market, a new era of speed improvements has started which is great news for processor-hungry apps like Layar.

Dirk Groten, CTO

All credits to Michael Hofmann, Lawrence Lee, Ronald van der Lingen, Anatoliy Samara and Andrey Koval for these achievements.

Recently, we revealed our new Layar Vision extension and announced the Creation Challenge. Now, the Layar Vision beta is available and the Challenge is open for submissions!

“Augmented Reality is at the stage where it can add real value to existing businesses,” said Maarten Lens-FitzGerald, Layar co-founder and General Manager. 

“This is what we are focusing on with our challenge; we’re not looking for just a cool layer using Layar Vision, but for creative, smart business ideas that will provide lasting value for both user and publisher.”

We’re giving away a total of $55,000 in cash to 10 developers who create the most useful, creative or otherwise impressive experiences with Layar Vision; we want to see content that really makes us say “Wow!”

You can now download the Android APK beta and begin building with Layar Vision! Public versions for both Android and iPhone will be available in the coming months.

Please submit your Creation Challenge layer anytime between now and September 15th. After the submission deadline, we will determine the finalist short-list and announce it a week later.

Our internationally-acclaimed jury of AR enthusiasts, including: Layar co-founder and CEO Raimo van der Klein, Wired UK editor-at-large Ben Hammersley, sci-fi author and AR wizard Bruce Sterling, former AdMob evangelist Russell Buckley and author/futurist Gerd Leonhard, will judge and rank the finalists to determine the top 10 ranking.

Important Challenge dates:

• August 15: Submissions open
• September 15th – Submission deadline
• September 22nd – Finalists announced
• October 3rd – Winners announced

Please visit our Layar Creation Challenge page for complete rules and details, peruse our documentation, and direct any questions or comments about the Challenge to Challenge[at]Layar[dot]com.

Good luck!