Wiimote DesktopVR - FAQ and Adv Discussion

A lot of very smart people have written in questions and suggestions and I think a lot of them are very clever and worth sharing... and I'd like to add some of my own expansions on what I showed in the video. Like they say, 2 brains are better than 1.... it turns out several thousand brains are even better. This post will be REGULARLY EDITTED to add new ideas as they come in. I know it breaks the RSS model, but I don't want to make a new post for every new idea. Sorry.
-------------------------------------------
Shutter/polarized glasses - if you are going to put something on your head covering your eyes for 3D, it is worth considering making shutter/polarized glasses which allows the computer to deliver a different image to each eye. This results in stereoscopic displays. Combined with head tracking, you get a DARPA grade 3D simulator. When just doing head tracking, the conflicting stereo depth cues weakens the illusion. I would love to do this, but it's quite a bit more hardware, and requires synchronization with the display refresh (or light polarization) which often isn't easy especially given the wide variety of display technologies out there.

Split screen for 2 people - if you split the screen and use two head mounted sensor bars, you can do this for two people... and possibly with one tracking wiimote. One wiimote does 4 dots, so it could (with some smarts) handle two heads. The field of view of a wiimote is only 45 degrees, so it might be cramped with 2 people. But, you could use one wiimote per head and still do split screen.

Wiimote Whiteboard - FAQ and Adv Discussion

A lot of very smart people have written in questions and suggestions and I think a lot of them are very clever and worth sharing... and I'd like to add some of my own expansions on what I showed in the video. Like they say, 2 brains are better than 1.... it turns out several thousand brains are even better. This post will be REGULARLY EDITTED to add new ideas as they come in. I know it breaks the RSS model, but I don't want to make a new post for every new idea. Sorry.
------------------------------------
My Bluetooth adapter and LEDs - Though it is reported to work with many bluetooth adapters and chipsets, I've been using this one sold by SparkFun electronics. The IR LEDs I use are Vishay TSAL6400s running at 100mA available from Mouser.

Tip switch - Instead of pressing the side button to illuminate the LED, a tip swtich could be added to illuminate when pressed on a surface. Though, it's mechanically harder to make and finding the right level of pressure for activation can be hard, especially if your surface varies (ex: whiteboard vs carpet)

Can you do this with a webcam? - The wii remote contains a 1024x768 camera with built-in hardware for IR blob tracking of upto 4 points at 100Hz. This significantly outperforms any $40 webcam I'm aware of. It'll work with a webcam, just not as well and not as easily.

Rear-Projection - using a rear projected surface, works great. The software will automatically work as-is so long as your pen is bright enough to shine through the projection surface. The wiimote gets a unobstructed view of the screen which eliminates occlusion problems and maximizes tracking resolution. Though, actually building a rear-projection wall or table can be a little difficult if you don't already have one.

Pressure sensitivity - wacom tablets give nice pressure sensitive behavior. Doing this with the wiimote whiteboard will be tough, at least through the camera data. The brightness of the LED varies significantly depending on angle, surface reflectivity, and distance. Any pressure data based off birghtness will not be reliable. The data could be transmitted via a high-frequency IR channel (like remote contol data) to an IR reciever mounted on the wiimote. (see "Right click" for more)

Right Click - any right click signal based off brightness is not likely going to be reliable. Additionally, using multiple LEDs (1 - left click, 2- right click) won't work at all distances and orientations as the LEDs get too close for the wiimote to distinguish them. The RIGHT way to do this is to use high-frequency IR data transmission (like remote control data) to an IR reciever. This will also give you a way of potentially getting pen ID - which is VERY powerful and may be a future project. Though, this requires quite a bit more hardware.

Multiple wiimotes for better tracking - if you have a large surface, you could have multiple wiimotes covering different areas to keep tracking reslution high, or have the wiimotes cover the same area to avoid occlusion problems, or do some interpolation of the data for higher accuracy. All possible, just with the smarter software. Doing this myself is probably not going to happen, but the source is freely available.

Wiimote Finger Tracking -FAQ and Adv Discussion

A lot of very smart people have written in questions and suggestions and I think a lot of them are very clever and worth sharing... and I'd like to add some of my own expansions on what I showed in the video. Like they say, 2 brains are better than 1.... it turns out several thousand brains are even better. This post will be REGULARLY EDITTED to add new ideas as they come in. I know it breaks the RSS model, but I don't want to make a new post for every new idea. Sorry.
------------------------
LED arrays from security cameras - online stores sell IR LED arrays for night vision security cameras. These should work perfectly for the wiimote.

Gloves - gloves exactly like minority report are definitely do-able and probably a good idea for repeated use.

LEDs on fingers - instead of tape you can use LED lights and do this without the LED array. But, you effectively create IR pens similar to the WiimoteWhiteboard demo which means batteries on the glove too.

Pinching with thumb and forefinger - the wiimote does upto 4 points. So both left and right thumb and finger could be tagged with reflective tape or LEDs. The software needs to be smart about maintaining the correct association and recognizing pinches, but very do-able.

Tagging a wiimote and knunchuck - getting the spatial location of another wiimote and knunchuck using a wiimote and reflective tape is an idea VERY much worth exploring. Now you get bi-manual (two handed) positional input in addition to all the accelerometer data and buttons.

Tracking fingers on surfaces (instead of the air) - unless it's a rear projection surface, the tape will be facing the wrong way... though you might be able to put the tape on the back of the fingers and figure out some method of triggering input. That said, it'll probably work well on a rear projection surface. You may need to offset the view angle of the wiimote so the support glass/acrylic won't retroreflect the light back to the wiimote. Alternatively, use active LED points.

Tagging objects on tables and projection- combining this method of tracking with projection is VERY powerful. This is essentially a field of research called Augmented Reality and will likely the be subject of a future video.

"Unable to launch the DesktopVR program"

blarg.... a number of people have been reporting a "Wiimote FishtankVR has encountered a problem" error. but whom are able to run my other programs. Unfortunately, I can't replicate the error on any of my computers and I'm leaving for the holidays. So, fixes may be slow for the next 2 weeks. If you can shed some light on this, please add a comment to this post and I'll upload a fixed version once this problem is hashed out.

Deleting the "config.dat" file and re-running the program may help.
Installing the most recent version of DirectX may help.

(update 12/30/07) I also got some suggestions that this might be due to a "x86/x64 linkage problem" related to 64-bit version of Windows. The program needs to be recompiled for this. I'll try to post a new version sometime in the next few weeks.

Head Tracking for Desktop VR Displays using the Wiimote

Using the infrared camera in the Wii remote and a head mounted sensor bar (two IR LEDs), you can accurately track the location of your head and render view dependent images on the screen. This effectively transforms your display into a portal to a virtual environment. The display properly reacts to head and body movement as if it were a real window creating a realistic illusion of depth and space.

Link to the Wiimote Projects Page

Source Forge: Wiimote Whiteboard

I'm swamped with requests for compatibility fixes, requests for Mac and Linux versions. ... and as much as I would love to be able to get this working in the hands of as many people as possible, I don't have the skills or time to do all this alone. So, here is my call for help. I have created a SourceForge repository for all WiimoteWhiteboard development.

If you are a Mac, PC, or Linux developer there are thousands of people (especially educators and entire schools) just waiting to use the software you write. Help out, become a hero.

If you tried the WiimoteWhiteboard software and had trouble with your setup, you can post your configuration in the SourceForge Help Forum.

Link to SourceForce: Wiimote Whiteboard

$14 Steadycam

This is an oldy, but a goody. The first online tutorial I ever created was on creating your own low-cost camera stabilizer for only $14 (that's in 2001 dollars). It became a featured article in the premier issue of Make Magazine, posted on countless news/blog sites, and viewed well over a 1 million times. It has become a staple of the low-budget/student film-making community and is currently used in a number of high school and college production classes.

If you can't (or don't want to) make one, you can simply buy one. I currently sell them through littlegreatideas.com for $39.95. It includes some bells and whistles beyond the $14 tutorial version. If you like my projects and want to support me, buying one is currently one of the easiest ways to do that. Then again, I also won't turn down flat out donations. =o)

Link to $14 Steadycam
Link to buying a Poor Man's Steadycam

Wiimote Whiteboard with a Rear-Projected Display

After a few people have written in asking about this, I think it's worth posting about it. The Wiimote Whiteboard technique/software works great with rear-projected surfaces. In fact, I think it's the best configuration because you can put the Wiimote behind the surface near the projector and you don't have to worry about blocking it's view. The camera also gets a prime view of the screen so tracking resolution is high.

I didn't talk about this in the original video because making a custom rear-projection screen is relatively hard... and making a rear projected drafting table, for which stylus work is better suited, is even a little harder. So, this might be a little out of reach for many people. But, it's well worth the effort in my opinion because the result is pretty great.

Link to WiimoteWhiteboard

Squiggly lines and other Bluetooth Drivers

After seeing some videos people have posted of them using the WiimoteWhiteboard software, I have added the following to the troubleshooting section of the help document and mention this on the download page.

-----
"My lines are really squiggly and tracking is bad" - Remember the Wii Remote contains a camera, and the better view the camera has of your screen the better tracking you'll have. So, good placement is the key to good tracking. The more direct view the wiimote has of your screen, the better, The closer it is without losing visibility of your screen, the better. The wiimote camera has a 45 degree field of view. Keep this in mind for placement. Using the "Visible IR dots" diagnostic on in the software, you can test the visibility of the corners of your screen using your IR pen. It may be helpful to turn off cursor control. Activate the LED at each corner location and check if the Wiimote still sees it. Some squiggilyness is unavoidable, but it can definitely be minimized. Smoothing is on the feature list to be added in any future release.
----


Also, I've gotten testimony and links to videos that say this works with other bluetooth drivers besides Blue Soleil. But, I haven't tested them myself.

IR LED pen schematic and IR keychain lights

I've uploaded a simple schematic of the IR LED pen I used. If you use a good LED, you can probably run it safely directly off a 1.5v AAA battery without a resistor. I used Vishay TSAL6100 940nm IR LEDs running at 100mA which I bought from Mouser. But, you can find usable LEDs and everything else you need from RadioShack. A number of people have also written in saying that some places sell IR keychain lights which may work right out of the box. NOTE: If you rough up the end of your LEDs, they will be more omni-directional and more easily seen by the wiimote.

Low-Cost Multi-point Interactive Whiteboards Using the Wiimote

Since the Wiimote can track sources of infrared (IR) light, you can track pens that have an IR led in the tip. By pointing a wiimote at a projection screen or LCD display, you can create very low-cost interactive whiteboards or tablet displays. Since the Wiimote can track upto 4 points, multiple pens can be used.

When updates are made to the software, I'll post them here on this blog.

Link to the Wiimote Projects page

[update] Link to the SourceForge repository to help with development.

Tracking Your Fingers with the Wiimote

Using an LED array and some reflective tape, you can use the infrared camera in the Wii remote to track objects, like your fingers, in 2D space. This lets you interact with your computer simply by waving your hands in the air similar to the interaction seen in the movie "Minority Report". The Wiimote can track upto 4 points simultaneously.

Link to my Wiimote Projects page.

Projector-Based Location Discovery and Tracking

My thesis work at CMU is on Project-Based Location Discovery and Tracking. The basic concept is to discover the locations of light sensors using light patterns from the projector. Once we discover the sensor locations, that information can be used for a wide variety of applications such as automatic keystone correction, multi-projector stitching, tracking input pens, augmented reality, variable resolution displays, and simulating foldable displays.

Here's a link to my thesis work with videos after the jump.

Weclome to my project blog: Procrastineering.com

Hi, everyone! Welcome to my project blog. This is an experiment to see if it is helpful for people get updates to projects that I'm working on and as a documentation tool for things that I'm doing.

My main website can be found at johnnylee.net