WonderRoom

It started when we got a departmental e-mail (forwarded from a Microsoft representative) informing us about a prize challenge to create an "Out of the Box" project using NUI (Natural User Interface). The challenge is named Microsoft NUI OOBW (Out-of-the-box week). This was about an hour before a semester exam so me and Omer were chilling outside and starting to run ideas. After a couple of days we set out - we were about to build WonderRoom - the interactive playroom. The funny thing is, about a week before the actual convention we realized "NUI" was just Microsoft's way of saying - "user Kinect"...

So the idea behind the project is to demo a "non-standard" use of the Kinect and it's features - real time depth and color map feeds, skeleton tracking, and segmentation. The application is in the form of a Kinect for Windows device on a tripod looking over a child's playroom with the idea that we can bring the child's own toys to life. This is done by hooking the Kinect to a PC running MATLAB with the MATLAB toolbox for Kinect and running our software which recognizes a person and some simple building blocks and guides the child in building a house - constantly providing feedback on success or failure just by "looking" at the child playing. A picture is worth a 100 words, so a video probably a million:

Some tech stuff:

• As said - we used a very normal laptop running Windows, MATLAB 2013 with the Kinect for Windows Toolbox from MathWorks - this is a wonderful toolbox that gives you everything the Kinect has to offer in real-time, straight to the MATLAB workspace. It's only flaw is that it does not include the registration functions between the color and depth maps - meaning you can not correlate a pixel from the depth channel with the RGB channels - this did give us some trouble.
• The recognition is quite straight forward - we use min and max thresholds on the depth map for background subtraction. We then use hue filters and shape tests to find each playing cube and figure out it's orientation and position. For 3D positioning we also use the depth map but for this we had to do some ugly approximation because of the lack of correlation between the RGB and D channels. The positions of all cubes is fed to a simple state-machine that gives the user feed back like -"now lets put the blue triangle on top" etc.
• You can see the full source code (including a C++ attempt which was dumped) on Google Code:
  https://github.com/syagev/playroom

We put up a pitch which included the following video and presented it to some Microsoft VPs from Israel and Redmond:

They seemed to like it and the demo we did live during the pitch so we won first place (and got to go home with an Xbox 360 and a Kinect for each of us), and got some press coverage on Ynet.

All together this was bunch of fun!