If you’ve downloaded Layar v6.2 for Android or iOS and have tried out some of the vision layers, you’ll probably have noticed that tracking was improved tremendously in this version. I’d like to shed a bit of light on what we’ve done technically to get this far.
So what is tracking in the first place? It’s the ability of the device to recognize and follow the object that is being viewed through the camera and determine its position in space. That way, if it’s augmented in a vision layer, the device knows where to draw the augmented content.
Making augments stick
Until v6.1, we were showing the live camera feed in the AR view with the augmented content on top of it. To do that, we’d take the camera frame from the live feed, do lots of calculations to determine the position of the tracked object and then draw the augmented content. That takes some processing time, so that in the end, the drawn content would always be lagging behind the camera images that you saw.
We’ve changed that in v6.2: We’re now showing the camera frame that belongs to the calculation for the augmented content, so that the content now sticks much better to the object. You can see that by moving the object smoothly: There is a slight delay between your movement and the image you see, but the content really sticks to the object you move. Don’t move too fast, this will blur the image and the app will lose track of it.
Zooming into the augments
There’s another big improvement in 6.2: When a layer developer uploads a reference image (the image of the object that she wants to augment), we create a fingerprint of that image that is used to recognize and track the features of that object. Obviously you need a certain number of features inside the camera view to calculate a good estimate of the position of the object. If the user holds the camera close to the object, zooming into the object, there are less features that can be tracked and at some point we’d lose track of the object.
We’re now adding features dynamically as you zoom in, allowing the app to track an object much longer. This is really nice when the augments on a page are quite small and you need to get closer to read them.
A cool example is the Eppo comic book, where you can view the original draft drawings on top of many of the pages. Well, you can now also zoom in really close, as shown in the screenshots below. Don’t zoom in too fast though, the app still needs some time to process the images.
Taming the ARM processors
A lot of the code we write for Layar is platform-independent. The entire vision algorithms are written in C++ and integrated into iOS using Objective-C++ and into Android using the JNI. This way we can write once and use the same code on both platforms. This has allowed us since Layar v6.0 to bring simultaneous updates for iOS and Android.
For Layar v6.2 we’ve decided to move one level down and write the processor intensive code directly in assembler using ARM NEON. The cool thing is that up to recently all iOS and Android devices run on ARM processors. Not all of them support NEON (like NVIDIA’s Tegra chips), so not all of them benefit of our new optimized code.
In our reference implementation, we’ve been able to decrease the processing time of certain critical operations by a factor of 8: For example, preparing a frame for display takes 2ms instead of 15ms when using the ARM NEON implementation. Obviously we don’t achieve this same speed enhancement in Layar itself, where there’s a lot more going on, but writing NEON-optimized code is proving a huge improvement.
Thanks to this optimization we were able to add support for streaming video as AR objects without significant loss in performance. We’ve decided to only support dual-core ARM processors, where different tasks like rendering video and analyzing the camera image to calculate the pose estimation can be distributed to different cores.
Now Intel has joined the club with its x86-based Atom processors. That’s a game changer and obviously our NEON-optimized code won’t run there. But we’ve been pleasantly surprised at the performance of the x86 chip. Even without optimization it runs our C++ code at higher speed than similar ARM chips.
We can conclude that with the arrival of Intel in the smartphone market, a new era of speed improvements has started which is great news for processor-hungry apps like Layar.
Dirk Groten, CTO
All credits to Michael Hofmann, Lawrence Lee, Ronald van der Lingen, Anatoliy Samara and Andrey Koval for these achievements.