April 21, 2010

Augmented Reality on iPhone: Marker Tracking Demo

I've been working on an augmented reality game app for iPhone recently. The player will be able to interact with the game by placing and moving specially designed marker cards in front of the iPhone's camera. The app analyzes video frames in real time and updates the scene accordingly. Here is demo of what I've got so far - marker tracking on iPhone 3GS.

The marker tracking algorithm is done by a slightedly modified version of NyARtoolkitCPP. Analyzing each frame takes < 55ms in average. The OpenGL ES rendering can achieve 30 frames per second. The current algorithm relies on thresholding so it is very sensitive to changing lighting condition. Also, template matching based identification requires matching against the whole pattern library for each detected marker. I'm seeking for better algorithms that use edge detecting and id recognition. StbTracker, the successor of ARToolkitPlus, looks ideal for devices such as iPhone but unfortunately it's not open source.

The most discussed topic by iPhone AR application developers today is real time video frame grabbing. There isn't an elegant way yet. For now, I'm using private APIs to repeatedly fetch the camera preview view. The good news is that we are going to have full access to still and video camera data in iPhone OS 4. A peek at the related APIs in the iPhone SDK 4 beta 2 further confirms that.

The OpenGL ES overlay content is displayed in a second UIWindow above the one containing the camera preview. This only point of dosing so is to get a clean preview image without the overlaying content. These ugly hacks will be all gone once we have iPhone OS 4.


Anonymous said...

Hi Arrix,

Your work looks amazing. I'm compiling NyARToolKit for iPhone 3.0 and I have a couple of errors left.

1. Duplicate symbol _spriteVertices (referring to the 3DGraphicsView & NyARToolKitCrossCompileAppDelegate).

2. glView.delegate = self; (error in NyARToolKitCrossCompileAppDelegate.m)

Any help would be appreciated.

shen said...


Could you email me hengshenwui@gmail.com

I would like to find out if you are available for a freelance project with regards to AR+iphone.


Arrix said...

@Anonymous I guess you are using EAGLView.m from Apple's sample code in which spriteVertices and spriteTexcoords are defined. Try removing those unused definitions. You also need to modify your 3DGraphicsView class implementation. Define a protocol

@protocol _DGraphicsViewDelegate


// Draw with OpenGL ES



Add a delegate property to 3DGraphicsView and update drawView: and setupView: to hand the work to the delegate.

Santhosh S said...

I tried to find v and was not able to ding. it says _DGraphicsView symbol is not available. inrder to create DGraphicsViewDelegate protocol we need _DGraphicsView . also 3DGraphicsView.h class is not available. please let me know how to proceed.

Santhosh S said...

i tried to find 3DGraphicsView.h class which is not available. also compiler gives unable to find specifier for _DGraphicsView. Hence i am not able create _DGraphicsViewDelegate protocol.please suggest how to proceeeed further

Experts from India said...

What was that really interesting nice video....which is that device and which was the app you used....as a iPhone application developer i want to learn that