Project Natal – body motion control and much more

In a couple of our previous articles we wrote about a few new ways for information interaction, and this article will describe a new motion sensing equipment devised for  gaming platforms. Since Nintendo launched their Wii controller it was obvious that the gamers are ready for changes. At this year E3 (Electronic Entertainment Expo), Microsoft presented a system they work on and it’s called Project Natal.

The upcoming motion sensing system will work with every Xbox360 when it launches and will be available on all the new systems. It has full body motion control by tracking 3D movement. Unlike the competition which tracks only several points integrated into the controllers, this system tracks 48 points. The box uses two sensors for input: a video camera, and an infrared. The infrared data is used to gauge depth and the video camera movement, but it’s the software which is really doing the magic.

The raw output of the system melds the two sources and then breaks them down into a wire-frame of objects, a heat map (for depth) and a point-map (which is akin to one of those hand imprint needle toys). The software merges all of this together to create a picture of movement in the room.

Johnny Chung Lee, a PhD graduate student in Human-Computer Interaction at Carnegie Mellon University, revealed that he is working on this project a few days ago. His previous work spans a wide range of topics including projector-based augmented reality, brain-computer interfaces, kinetic typography, haptics, multi-channel audio, tangible interfaces, filmmaking and his most mentioned work was related to the modification of Nintendo Wii controllers.

”The 3D sensor itself is a pretty incredible piece of equipment providing detailed 3D information about the environment similar to very expensive laser range finding systems but at a tiny fraction of the cost. Depth cameras provide you with a point cloud of the surface of objects that is fairly insensitive to various lighting conditions allowing you to do things that are simply impossible with a normal camera.” said Lee. ”But once you have the 3D information, you then have to interpret that cloud of points as ‘people’. This is where the researcher jaws stay dropped. The human tracking algorithms that the teams have developed are well ahead of the state of the art in computer vision in this domain. The sophistication and performance of the algorithms rival or exceed anything that I’ve seen in academic research, never mind a consumer product.“

It also recognizes voices, and offers more interaction between the players and the game. It has a scanning option so it recognizes faces and has the ability to import real objects or individuals into the virtual world. A good example for those features is the next video where a female interacts with a virtual character Milo. Milo is developed by the makers of games such as “Black and White” and “Fable”, the Lionhead Studios.

So, why did we find it interesting enough to write about it? It offers more natural interaction and physical activity (which should be encouraged). There is no doubt there will be additional controllers which could help the gamers in more complex games. The usage of such a system isn’t limited to games or social games. It could be used as a replacement for the remote controller, a helper during a workout (imagine a virtual physical trainer which could actually give you tips and encourage you) or a videophone.