Procrastineering - Project blog for Johnny Chung Lee

Computer Controlling a Syma Helicopter

2011-11-30T02:57:00.000-08:00

Recently, I've been playing with these inexpensive Syma Remote Control Helicopters. At the time they were only $20 (but seemed have been price adjusted for the holidays). They're quite robust to crashes and pretty easy to fly. For $20, they're a blast. The other interesting thing about these copters is that the controller transmits commands using simple infrared LEDs rather than a proper radio. This simplicity makes it tauntingly appealing to try reverse engineering. So tonight, I decided to do a little procrastineering and see if I could get my helicopter to become computer controlled.

For hardware, I've been liking these Teensy USB boards because they are cheap, small, versatile, and have a push-button boot loader that makes iteration very quick. They can be easily configured to appear as a USB serial port and respond to commands. For the IR protocol, I started with this web page which got the helicopter responding. But, the behavior I was getting was very stuttery and would not be sufficient for reliable autonomous control. So, I decided to take a closer look with an oscilloscope to get accurate timing from the stock remote control. Some of my measured numbers were fairly different for the web tutorial I found. But, now the control is fairly solid. So, here is the nitty gritty:

IR Protocol:

- IR signal is modulated at 38KHz.
- Packet header is 2ms on then 2ms off
- Packet payload is 4 bytes in big-endian order:
1. yaw (0-127) default 63
2. pitch (0-127) default 63
3. throttle (0-127 for channel A, 128-255 for channel B) default 0
4. yaw correction (0-127) default 63
- Packet ends with a stop '1' bit

Format of a '1' is 320us on then 680us off (1000us total)
Format of a '0' is 320us on then 280us off (600us total)

Packets are sent from the stock controller every 120ms. You can try to send commands faster, but the helicopter may start to stutter as it misses messages.

Download Teensy AVR Code (updated 11/30/2011)

The code is available at the above link. It's expecting 5 byte packets over the serial port at 9600 baud. The first byte of each packet must be 255, followed by yaw, pitch, throttle, and yaw correction (each ranging from 0-127). It will return a 'k' if 5 bytes are properly read. If it doesn't receive any serial data for 300ms, it will stop transmitting the IR signal.

Unfortunately, I can't help you write a program to communicate over serial since that will depend on your OS (Windows, Mac, Linux) and varies by language as well. But, it is fairly easy with lots of web tutorials. The harder challenge will be figuring out how to update the 3 analog values to keep it from crashing. =o) The most likely candidate is to use a camera (probably with IR markers) to monitor the position of the helicopter. But, getting that to work well is definitely a project unto itself.

Good Luck!

Shredder Challenge - Puzzle 2 done! Onto Puzzle 3

2011-11-22T15:45:00.000-08:00

Puzzle 2 is now done! Puzzle 3 is a drawing (not text).

As we get to more complicated puzzles, it's clear that loading, rendering, and UI limitations will become a bigger and bigger issue. My colleague, Dan Maynes-Aminzade ("monzy" for short) is doing his best to figure out ways to handle that. There are a lot of not-ideal solutions.

It's also clear that more computer aided matching will be necessary to maintain progress. Here are zip files for the pieces of problem 4 and problem 5, if you want to try your hand at analyzing them directly.

Puzzle 4 pieces
Puzzle 5 pieces

If you come up with good ideas that work, post them in the comments.

Puzzle 1 done overnight!

2011-11-22T07:48:00.000-08:00

Puzzle 1 was completed overnight! Very cool. They definitely get harder. But, the crowd helped UCSD complete puzzles 2 and 3 within a couple of days. So, we could easily catch up.

On to puzzle 2

DARPA Shredder Challenge - you could win $50,000

2011-11-21T17:23:00.000-08:00

On October 27th, DARPA announced their Shredder Challenge. Try to unshred 5 documents for $50,000. There has been a few notable efforts such as UCSD's web based approach. But, that page has recently been compromised due to malicious users.

With the help of a few colleagues, we had also created a web-based version very early on. But, it was missing some of the login engineering and UI of the UCSD effort. So, we never made it public. But rather than attempt to build a full on competitor, we've decided to open up the tool we built for anyone to try to win the contest themselves!

http://www.unshred.me/1

You can create a private branch of each puzzle if you want to try to give it a go alone, or you can contribute to the main public copy of the puzzle. If our image analysis tinkering goes well, we may add some tools to help make finding matches easier.

NOTE: The deadline for submitting answers to DARPA is December 4, 2011 (11:59PM EST)! Only 13 days left.

GO!

Technology as a story

2011-11-15T08:36:00.000-08:00

Generally, I consider myself a technologist. I work in technology, I choose environments that have people who are excellent at it. New technologies make the world move forward. If it is shared broadly enough, it is impossible to "un-invent" a technology and thus the world has been irrevocably changed, even if just by a little bit.

However, what saddens me is when I encounter technologists with the brilliance to create new and wonderful things, but lack a sense of what is beautiful to people. Technology is most often known for being ugly and unpleasant to use, because technologists most often build technology for other technologists.

But to touch millions of people, you have to tell a story - a story that they can believe in, a story that can inspire them. Technology is a tool by which new stories can be crafted. They are not the end product unto themselves. All too often, I find engineers and researchers who are eager to build the technology without understanding the story that goes around about why people should care, why what they built can be inspiring rather than just enabling.

It is not a skill you learn at school. I have encountered people who understand this, and others who don't. I can't say that I have mastered this ability. However, I can at least respect how powerful it can be and strive to be better at it.

As an engineer, as a technologist, as a researcher, or inventor... I encourage you understand the power of stories. A story isn't merely the sequence of events in a book or film. It can be a story about you, and how your life or the lives of the people around you could be a bit different... or how the world could be different than it is today.

It is inspiring to see what a talented artist can do with the very simplest of tools. I recently came across the following video which I feel exemplifies this idea.

When you build something or design something, take a moment to imagine the stories than can be told around what you create and to share that story with others.

The Kinect Effect

2011-11-01T02:14:00.000-07:00

This is hands down, the best Microsoft commercial I have ever seen. It has soul. It has spirit. It has open optimism about what a company and creative enthusiasts can do together. They are even showcasing kinect projects on the official website Kinect.com

Bravo.

My hats off the the tens of thousands of creative developers who have explored the wide ranging uses of Kinect and, of course, to Microsoft & Xbox for seeing that it is a very positive thing to embrace. Yes, Kinect is a product that ultimately must make money through games and applications. But, it can also have a remarkably positive impact on our culture.

Best $3000 I ever spent.

Giving Computers a Human-Scale Understanding of Space

2011-08-12T12:06:00.000-07:00

Computer vision and Human-Computer Interaction are just about to hit their stride. Within the past 4 years, the real-time/robotics computer vision research community has made leaps and bounds - much of it out of the Active Vision Group at Oxford and the Robot Vision Group at the Imperial College of London. One of the first pieces of work that really started to impress was PTAM (Parallel Tracking and Mapping) by Georg Klein:

Full up markerless augmented reality had been a long time dream of many. But, few people knew actually how to do it. PTAM was the first system that showed promise that it could handle the rough conditions of real-time motion of a handheld camera.

Also from Oxford, Gabe Sibley and Christopher Mei started demonstrating RSLAM (relative simultaneous localization and mapping) which provides fairly robust real-time tracking over large spaces. The following video uses a head-mounted stereo camera rig:

Just in the past couple weeks, some new projects done with the help of Richard Newcomb show what happens when you combine this tracking ability with either a depth camera like Kinect, or try to do traditional reconstruction from the RGB. These projects are called KinectFusion (a Microsoft Research Cambridge project) and DTAM (Dense tracking and mapping) respectively.

The following video uses a normal RGB camera (not a Kinect camera):

It's important to remember that no additional external tracking system is used, only the information coming from the camera. Also, it's worth pointing out that the 6DOF position of the camera is recovered precisely. So, what you can do with this data reaches well beyond AR games. It gives computers a human-scale understanding of space.

This is pretty exciting stuff. It'll take a little while before these algorithms become robust enough to graduate from a lab demo to a major commercial product. I usually like to say that "people will beat the crap out of whatever you make, and quickly gravitate to the failure cases". But as this work evolves and people begin build useful applications/software on top, it'll be an exciting next few years.

UIST 2011 Student Innovation Contest

2011-08-09T14:03:00.000-07:00

UIST 2011 is just a couple months away, and Microsoft Hardware is generously providing the toys again this year. This time it's a touch mouse that provides a full capacitive touch image (which is fairly unique). If you are a student, try to enter, win some prizes and get to meet a bunch of other people interested in interface technology.

Official Contest Page

Myth of the Dying Mouse

2011-07-28T03:06:00.000-07:00

It's definitely not the most polished delivery I've made (ignite talks don't let you control your slides, which is very unsettling for me). But, here's a 5 minute ignite talk I recently gave entitled "The mouse and keyboard are NOT going away, and there's NO SUCH THING as convergence".

Kinect Projects - The First 5 Months

2011-05-06T13:17:00.000-07:00

Since it's release in November of 2010, there have been thousands of projects use the Kinect camera from independent developers, artists, and researchers. This is just a short montage of a few that I have enjoyed seeing.

It Gets Better

2011-05-03T22:42:00.000-07:00

Why your arms don't suck.

2011-04-14T22:40:00.000-07:00

Oooooooo..... me likey:

It's rare I see a product demo video and say, "Man, I wish my life would be longer so I can see the amazing future we will have." While I am fairly certain the yet un-purchasable robot above will be the cost of a small house, it is hard to contain my techno-lust. Having worked with a 6DOF robot before, they can be deceptively hard to program and without running a manufacturing line - the immediate utility of owning such a device is debatable. However, if these do end up being adopted by some manufactures, it does potentially reduce both the time to design/produce and the cost to manufacture consumer products. While this means the already blinding rate which new products are released will continue to accelerate, it also means that the bar for producing mass manufactured devices will also come down. As companies adopt re-programmable manufacturing/assembly tools, creating a new product may eventually be a matter of loading new files into all the machines on the floor. I think that's an exciting future and perhaps one day the "Print" button on your computer may take on a much more powerful meaning.

A small educational comment about the arms of this robot. They appear to be 7 degree-of-freedom arms... which is actually the same number of degrees of freedom that your arms have. If you grab a pole, or place your hand on the wall... without moving your shoulder (or your hand), you still have some freedom over the position of your elbow. But why do we need 7 when objects in the world only have 6 degrees of freedom (x,y,z, yaw, pitch, and roll)? The 1 extra degree of freedom is what allows us to reach around obstacles. If we only had 6 degrees of freedom, there would be only 1 way to reach out to pick up an object. So any obstruction along that path would prevent us from getting our food or some tool we needed to survive. Arms that contain 7 degrees-of-freedom have a dramatically larger operating range increasing their utility in uncooperative environments like the real world. For some reason, I find it quite satisfying that there is a mathematical basis for the evolution of our arms.

The re-emergence of DIY vs Big Organizations

2011-03-17T04:56:00.000-07:00

Wow! Limor (of Adafruit Industries) is on the cover of Wired. Wicked. Congrats, I am filled with envy. =o) Even if she does look a little bit Photoshop'd. =o/ But, nice homage!

It's great to see someone from the DIY community get such recognition. I haven't read the issue yet, but extrapolating from some of the phrases on the cover, I imagine it is a celebration of the re-emergence of independent tinkering.

I have actually put quite a bit of thought toward this topic having recently jumped back and forth between the DIY hobby culture, serious academic research, and massively funded commercial product development. I've had the fortune to observe people trying to make new and interesting things at extremely different scales...from $100 budgets to $100,000,000 budgets.

One thing that I find very consistent: good ideas come from anywhere. The biggest factor in predicting where good work will come from is "how much does this person actually care about what they are working on?" In fact, big budgets and a sense of entitlement can actually hinder the emergence of interesting ideas. Having the *expectation* to do really great work can lead people or organizations to develop tunnel vision on "big" ideas, and miss out on smaller ideas that end up having a lot of impact or dismiss seemingly silly approaches that actually end up working.

There's a really great TEDx talk by Simon Sinek that touches on this. He actually brings up a number of great points in his talk, but the one I want to highlight here is his anecdote about Samuel Pierpont Langley vs. the Wright Brothers in pursuit of powered flight. Langley represented the exceptionally well funded professional research organization, and the Wright Brothers were the scrappy passionate pair of DIY'ers. Today, we now know the Wright brothers as the ones who created the first airplane and most have never heard of Langley. Big investment is not a very strong predictor of valuable output. But, an individual's willingness to continue working on the same problem with very little to no pay... is a good predictor.

The great thing about the hacker community is that, generally, most of them fall into the later category. Independent developers and hobbists care enough to spend their own money to work on the projects they believe in. As a result, I'm finding that the delta in the quality of ideas from a well funded research group, and the independent community (in aggregate) is getting smaller and smaller by the month. Increasingly, the best "hobby projects" surpass the quality level of "true research" work in the same area. This startling lack of contrast (or sometimes inversion) becomes laughably evident when I am reviewing academic/scientific work submitted for publication on a project that uses Kinect, and then the newest Kinect hack pops up on Engadget that simply beats it hands down.

Now, I could simply make a kurmudgenly claim that the quality of professional/research/academic work has gone down. But, I actually don't think that's true. In my opinion, what is happening is that the quality of independent projects are getting better.... fast. Which, I think resonates with this observation of a "DIY Revolution".

But, why is this re-emergence happening now? Wasn't is just a few years ago people were lamenting about how "black boxed" consumer products had gotten, and that the good old days where you could open up a product and futz with the innards in a meaningful way were gone? What's changed to cause this apparent re-birth?

I have a theory.

My Theory about the Re-Emergence of the DIY community:
In the 90's and early 2000's, Moore's law was absolute king. The primary deciding factor in purchasing an electronic product was simply how fast it was. This meant an intense focus on tighter and tighter integration of components and all the functionality was disappearing into tiny little black chips that could not be accessed nor modified by mere mortals. But now, people barely talk about raw "megahertz" or "megabytes" anymore. General purpose computers have gotten "fast enough". We now want specialized kinds of computers: one that fits in our pocket, plays games in 3D, one shaped like a tablet, one that goes in our car, one that can go under water, or get strapped your snowboard and not break. We have reached a surplus in computing power that makes it affordable to build (and buy) devices for smaller and smaller needs. Our imagination for what to do with computing has simply not kept up with Moore's Law. So, we find more uses for more modest amounts of computing power. But, what does this have to do with the DIY community?

A byproduct of having such an immense surplus in computing, is that the tools you can buy within a hobbist budget have also gotten exponentially better in just the past 3-4 years, while the improvement in professional tools have been more modest. The difference in capability between the electronics workbench of a professional engineer and a hobby engineer is getting really really small. Kinect is an overwhelming example of this. The cost of a high quality depth camera dropped nearly 2 orders of magnitude overnight. As a result, hobbists are out pacing many professionals in the same domain simply due to sheer parallelism. Perhaps not as dramatically, but this is happening with nearly all genres of electronic and scientific equipment. One day, maybe we'll see backyard DIY electron beam drilling for nano-machining.

When it is no longer about who has the most resources, it's about who has the best ideas. Then, it becomes a pure numbers game:

Take 10,000 professional engineers vs. 1 million hobbists with roughly equivalent tools. Which group will make progress faster? Now, consider that you have to pay the 10,000 engineers $100K/year to motivate them work, and the 1 million hobbists are working for the love of it. Does that change your answer? Even if it doesn't, you have to concede that there does exist a ratio which will make the output of these two groups equal. It's merely a matter of time.

If you follow me through this argument, which I won't claim to be bullet proof but it explains the trends we are observing quite nicely, then this has an interesting implication on organizations that are currently funding big research groups. When it's simply a matter of who has the best ideas, it's tough to try to employ enough people to get good coverage. You could try to spend a lot of energy on trying to find the "best" people, but that's about as challenging as predicting the stock market. Some inventors simply go "dry" of good ideas and end up not providing a good lifetime return on investment (I fully expect this to happen to me someday. I just hope it happens later rather than sooner.)

So to me, this suggest 3 options for big exploratory organizations:

1. Start tackling more resource intensive problems - things that fundamentally cannot be done today for a few thousand dollars, but at some basic level requires materials, tools, energy, computation, space, manpower that is impossible to obtain at a hobby level. The LHC and space programs are good examples of this. Even if the end goal may be of debatable near term economic value, there is a high probability that unexpected derivative technologies/projects will bring commercial/educational benefits elsewhere.

2. Empower everyone within your organization to do exploratory work. The tools are cheap and "research groups" have no monopoly on good ideas. It's hard to know where lightning will strike, so make sure you encourage it anywhere and hope you haven't missed a spot.

3. Partner with the outside developer community. There is plenty of precedence where using the resources you have to channel the creative power of the masses through the platforms you control can bring a tremendous amount of value if done in an organized manner. It is the rocket fuel that powers companies like Facebook, Twitter, and Groupon to go from non-existence to dominating entities in less than 3 years. The same can absolutely happen with traditional physical electronics and other consumer goods. It simply requires treating your customers as potential partners, rather than assuming they are all potential predators.

Display and Interaction work at Microsoft Applied Sciences

2011-02-26T01:26:00.000-08:00

Yay, it's great to see my old group Microsoft Applied Sciences in the press! Lots of great display and interaction work being done there. Fantastic mix of software, hardware, and optics people. Congrats!

As it becomes increasingly cost effective to manufacture a more diverse set of computing form factors, exploring new ways of providing input to a computer and sending output to a user will become an essential part of developing new genres of computing products. The time when raw computing horsepower was the key differentiator passed us several years ago, and the rate of device specialization has shot up dramatically. Less computing power is fine, if it is where you need it when you need it in the form you need it. As a result, you likely see the speed at which wild interface technology research moves into to product also accelerate.

Consequently, if you are a young engineering student. There will be a steady stream of good jobs for people who like to write software for new kinds of input/output devices. =o)

Windows Drivers for Kinect, Finally!

2011-02-21T15:06:00.000-08:00

Yay! This makes me happy. Microsoft officially announces support for Windows Drivers for the Kinect Camera as a free download in the Spring.

This was something I was pushing really hard on in the last few months before my departure, and I am glad to see the efforts of colleagues in the research wing of Microsoft (MSR) and the XBox engineering team carry this to fruition. It's unfortunate this couldn't have happened closer to launch day. But, perhaps it took all the enthusiasm of the independent developer community to convince the division to do this. It certainly would have been nice if all this neat work was done on Microsoft software platforms.

I actually have a secret to share on this topic. When my internal efforts for a driver stalled, I decided to approach AdaFruit to put on the Open Kinect contest. For obvious reasons, I couldn't run the contest myself. Besides, Phil and Limor did a phenomenal job, much better than I could have done. Without a doubt, the contest had a significant impact in raising awareness about the potential for Kinect beyond Xbox gaming both inside and outside the company. Best $3000 I ever spent.

In my opinion, all the press coverage around the independent projects brought a lot of additional positive attention to the product launch. That unto itself became the topic of international news.

But to take this even further, it would be awesome if Microsoft went so far as to hold a small conference to actually showcase people doing interesting projects with Kinect. It is a really great device, and such an outreach program would give Microsoft an opportunity to engage with very enthusiastic partners to potentially build new applications around it both inside and outside of gaming. At the very least, it would be a cheap way to recruit potential hires.

There are lots of smart people outside of Microsoft that would like to build interesting stuff with it. Most of it probably won't be a "Microsoft-scale" business initially, but worth enabling and incubating in aggregate. Though, a large portion of the expert community is already using the Kinect camera in their own projects on just about every OS and every develoment tool in existence. So, Microsoft will need to give researchers and independent developers a reason to go back to thier platform - be it opportunities to engage with people at Microsoft/MSR, other Kinect developers, or opportunities to share thier work though larger distrubtion channels such as XNA, app stores, or XBox downloadable games. We have just seen the beginning of what can be done with low-cost depth cameras.

Low-Cost Video Chat Robot

2011-02-09T01:43:00.000-08:00

Since I relocated down to Mountain View, I wanted a good way to keep in touch with my fiance who is still back in Seattle. So, I decided to mount an old netbook I had on top of an iRobot Create to create a video chat robot that I could use to drive around the house remotely. Since it was a good procrastineering project, I decided to document it here.

There are two major components to the project: the iRobot Create which costs around $250 (incl. battery, charger, and USB serial cable) and the netbook which I got for around $250 as well. At $500, this is a pretty good deal considering many commerical ones go for several thousand dollars. The software was written in C# with Visual Studio Express 2010 and only tested on Windows 7 with the "Works on my machine" certifcation. =o) I'm sure there are TONs of problems with it, but the source is provided. So, feel free to try to improve it.

   Download Software:
     VideoChatRobot v0.1 (posted 2/9/2011)
     VideoChatRobot v0.2 (posted 2/11/2011)

Included are the executable, C# source, and two PDFs: one describing installation and usage of the control software, the other more information about modifying the charging station.

The software does a few nice things like try to setup UPnP router port forwarding automatically, queries the external IP needed to make a connection over the open internet, maintains a network heartbeat which stops the robot if the connection is lost, a control password, auto-connect on launch options, and even mediates the maximum acceleration/deceleration of the motors so it doesn't jerk so much or fall over.

The UPnP port forwarding is far from perfect is not well tested at all. If it works for you, consider yourself lucky. Otherwise, ask a friend how to set up port forwarding to enable remote control over the internet.

Once you have all the parts: the netbook, the robot, the serial cable, the software. You can probably be up an running within 5 minutes. Assembly is merely plugging cables together. Mounting the netbook can be done with velcro or tape. Building the rise stand is more challenging, but entirely optional. I happen to have access to a laser cutter to make my clear plastic stand, but you can probably make something adequate out of wood.

Optional: Modifying the Charging Station
Probably one of the more interesting parts of this project from a procrastineering standpoint is the modifcation to the docking station so that it would charge something else in addition to the robot base.

What I did is admittedly pretty crude and arguably rather unsafe. So, this is HIGHLY NOT RECOMMENDED unless you are very comfortable working with high voltage electricity and accept all the personal risks of doing so and potential risks to others. This is provided for informational purposes only and I am not responsible for any damages or harm resulting from the use of this material. Working with household power lines can be very dangerous posing both potential electrocution and fire risk. This is also unquestionably a warranty voiding activity. DO NOT ATTEMPT THIS without appropriate supervision or expertise.

Now that I've hopefully scared you away from doing this... what exactly did I do? A high level picture is shown here:

The PDF document in the download describes changes in more detail. But, I had a lot of trouble trying to tap the existing iRobot Create charging voltage to charge something else. Primarily, because the charging voltage dips down to 0V periodically and holds for several milliseconds. That would require making some kind of DC uninterruptable power supply and made the project much more complex. The easiest way to support a wide range of devices that could ride on the robot was to somehow get 120V AC to the cargo bay... for those of you with some familiarity with electronics, you probably can see the variety of safety hazards this poses. So, again this is HIGHLY NOT RECOMMENDED and is meant to just be a reference for trying to come up with something better.

I actually do wish iRobot would modify the charging station for the Create to officially provide a similar type of charging capability. It is such a nice robot base and it is an obvious desire to have other devices piggy back on the robot that might not be able to run off the Create's battery supply. I personally believe it would make it a dramatically more useful and appealing robot platform.

Usage Notes
At the time of this post, I've been using it remotely for about a month on a regular basis between Mountain View and Seattle. My nephews in Washington DC were also able to use it chase my cat around my house in Seattle quite effectively. Thus far, it has worked without any real major problems. The only real interventions on the remote side have been when I ran it too long (>4 hours) and the netbook battery dies or having the optional 4th wheel on the iRobot Create pop-off which can be solved with some super glue. Otherwise, the control software and the charging station have been surprisingly reliable. Using remote desktop software like TeamViewer, I can push software changes to the netbook remotely, restart the computer, put Skype into full screen (which it frustratingly doesn't have as a default option for auto-answered video calls), and otherwise check in on the heath of the netbook.

"Hi, Google. My name is Johnny."

2011-01-18T01:56:00.000-08:00

I have some big news to announce on a personal front: Very recently, I have left Microsoft to join a special projects team at Google. After more than 2 and a half years working as a core contributor to the human tracking algorithms for Kinect, it was an extremely difficult decision and I leave behind many great colleagues in Redmond.

It was a wild ride, helping Kinect along through the very early days of incubation (even before it was called "Project Natal") all the way to shipping 8 million units in the first 60 days. It's not often you work on a project that gets a lavish product announcement by Cirque du Soleil and a big Times Square Launch party. The success of Kinect is a result of fantastic work by a lot of people. I'm also very happy that so many other people share my excitement about the technology.

It was great to be a part of such a unique project. I look forward to seeing all the creative and unexpected ways that game developers will use the data from the camera to create fun experiences. The Xbox is exceptionally well positioned to do great things in the entertainment space. It's a great console, and a great platform, with a lot of potential. I genuinely look forward to seeing how it will evolve over the next few years and I absolutely wish the Xbox team the best of luck.

Khan Academy: The closest thing to downloading knowledge into your brain + a $5000 challenge to you

2011-01-02T01:37:00.000-08:00

A few months ago, I rediscovered the Khan Academy after stumbling across a presentation by Salman Khan. If you aren't familiar with this, I recommend making it an absolute priority take a quick scan of some of the videos. Here's a summary video from the website:

Sal has an astonishingly approachable and understandable method of explaining topics in his videos, and also has an incredibly deep understanding of the material he talks about hinting at lower levels of complexity that he might be skimming over, but sometimes revisits in future videos. The Khan Academy videos are, in my opinion, perhaps one of the most interesting things to happen to education in a very very long time. If I may, "disruptive". Anyone with an Internet connection can go from basic fundamentals all the way up to a college level education in many topics in a clear, organized, understandable manner... all for free. His teaching style is more effective along many dimensions than any I have personally experienced in any classroom.

Recently, I've realized that I need to learn more linear algebra. Over the years, I have picked up little bits here and there doing computer graphics, basic data analysis, but I never had a proper understanding of it enough to understand why it really works or more importantly... apply it to solve completely new problems that might be somewhat non-standard. I managed to never take a proper linear algebra course in college or grad school.

So, why do I suddenly care about learning linear algebra now? and consequently should you care? Well if you want to understand how the Wiimote Whiteboard program works, you need a little bit of linear algebra. If you want to understand how video games are rendered on the screen you need linear algebra. If want to understand how Google works, how parts of Kinect work, or how the $1 million dollar Netflix Prize was won, financial modeling, or in general analyze the relationship between two large data sets in the world... you need linear algebra. I'm discovering more and more that any modern sophisticated engineering, modeling, prediction, analysis, fitting, optimization problem now usually involves computers crunching on linear algebra equations. Unfortunately, this fact was never properly explained to me in college so I never prioritized taking a class.

I understood the basics enough to do computer graphics, rendering stuff of the screen (3D to 2D). But much of modern computer vision, is about reversing those equations, going from 2D back to 3D, which involves solving a lot of linear algebra equations to recover unknown data. And I've found that computer vision papers seem to be the worst places to look for a clear explanation of the math being performed. The almost appears to be a desire to see how obtusely one can describe their work.

Fortunately, Khan Academy has over 130 videos on Linear Algebra. Since I knew I would be traveling this holiday, I decided to load up all of the videos on my phone to watch during down time. Watching videos here and there while sitting on the plane, trains, buses, or waiting in lines, I was able to watch all 130+ videos, which cover a 1st year college Linear Algebra course, in about 3 weeks. Pure awesome.

However, the Linear Algebra lectures stopped just as I though it was getting to the interesting part. I was hoping it would get to covering topics such as Singular Value Decomposition, numerical analysis, perspective projections...reversing them, sparse matrices, bundle adjustment, and then real-world application examples. I'm going to order some books on these topics, but I really really love the video lecture format Sal uses in the Khan Academy and wish they continued.

The $5000 challenge:
As an attempt to continue expanding this lecture series in the Khan Academy I want to encourage people who feel like they can give clearly understandable lectures on these topics to pick up where Sal left off. Apparently, there is an informal method of adding your own videos to the academy. I've already donated some money to the Khan Academy (a not-for-profit 501(c)(3)). But, as a call the community, to incentivize people who are able to produce good video lectures on advanced Linear Algebra - for each video posted (and passes the "clearly understandable", Khan academy style, 10 minute video lecture) that continues the Linear Algebra series I will donate $100 to Khan Academy up to $5000. So, not only would you be educating thousands (possibly millions of people), you will be ensuring that your material stays free.

If you do take me up on this offer and do post a video, let me know at johnny@johnnylee.net, and I will review the video. If it passes the bar, I will donate the money and then send you the receipt.

Simulated PS2 Controller for Autonomously playing Guitar Hero

2010-12-16T04:44:00.000-08:00

It's been a long while since I've posted a personal procrastineering project. The past two and half years have been pretty heads down with developing Kinect (congrats to all those involved). But, since that has now successfully launched. I've had a some time to spend on little side projects.

As an exercise to teach myself a little bit more real-time computer vision/robotics, I wanted to see if I could get a computer to autonomously play certain console video games. Video games are nice because they can provide a relatively decent simulation of a 3D environment, emphasize demand for real-time vision processing, I don't have to go out in the field to run a test, and there is no penalty for screwing up. This could also be done with PC games, but rich games are more often console based and it kind of black boxes the activity so I am really forced to depend solely on the data contained in the video stream.

However, a pre-requisite to this activity is programmatically sending controller commands to a console. This is not something that appears to be very common on the intertubes, so I thought I would detail my efforts here to hopefully fill that gap. There used to be a product that would allow you to do this with an Xbox 360 called the XIM2. It was primarily targeting people who wanted to use a mouse and keyboard to play first person shooters on a console. Unfortunately, the XIM2 is now discontinued and the nice looking soon-to-be-available XIM3, I'm told, will not have PC-to-Console control functionality, bummer. So, off to build my own.

The approach I ended up taking is actually very similar to the origins of the XIM. I used a microcontroller to simulate a PS2 controller. The nice thing about the PS2 controller is that it is a relatively simple communication protocol, and due to its vast popularity you can find low-cost adapters to use it with a PS3 or Xbox 360. These are each the order of $5-$10. So by creating a computer controlled PS2 controller, you can support 3 consoles. So, you can programmatically control current generation games as well.

Simulating a controller
Most of my starting points were nice websites such as this and this, which neatly describes the Playstation controller protocol. Most of the material online is geared toward using a PS2 controller with your hobby/robot projects rather than trying to simulate a PS2 controller to command the console. In theory, you simply would reverse the instructions and it would magically work. Unfortunately, that wasn't quite the case. The PS2 console is very picky (sometimes erratic) about timing, goes through a reasonably sophisticated handshaking and configuration process when you plug it in (which is game dependent) before it will accept data from the controller as input. Thus, my controller simulator had to survive that entire start up process looking like a valid controller.

The microcontroller platform I chose was the Teensy 2.0 USB development board which has an ATMEGA32U4 and is programmable via the built-in USB port (so, no external programmer needed). It can be configured to become a variety of USB devices and is only $18. Since it can become a USB serial interface and has hardware for SPI (for the PS2 controller protocol), it can do this entire project without any additional communication hardware.

The PS2 controllers use the SPI protocol for communicating with the console, typically using a transmission clock of 250KHz. But, SOME games, will cause it to shift to 500KHz unexpectedly between messages. So, hardware SPI is a must in the microcontroller. BTW, if you are selecting a different micro, you need to make sure you can simultaneously clock data out as new data is clocked in. Lastly, you need to manually pull the ACK pin low to tell the console something is listening on the line which isn't part of the standard SPI spec.

The PS2 controller initially boots up as a simple Playstation One controller which only provides digital button output (mode 0x41). The game console will make a variety of queries and configuration messages to enable the joysticks, analog button pressure, motors, etc. (mode 0x79) It's unclear what all of the configuration message mean. There are a lot of them, and they vary quite a bit depending on what game is running. Not very simple. But, as long as you mimic the behavior of a valid controller under several games, it appears to work. There was a minor issue that the USB interrupts were causing de-synchronization between the console and micro. So, the USB interrupt handling is disabled while communicating with the console (which is roughly 100us every 15ms). This did not cause an issue with my computers, but may be one possible source of USB errors on other systems just as an FYI.

Firmware Files

Teensy PS2 Controller Sim Firmware v1
Teensy PS2 Controller Sim Firmware v2 (updated 5/10/11)
- fixed ACK timing bug to make it work with PS3 and Xbox360 adapters.

I won't dive into the details, but above are the firmware files for the Teensy development board. These are (of course) provided at your own risk with no implication of support. There is a precompiled .hex file which you can point your Teensy programmer toward that should get you up and running quickly. The C source is in there. For those of you who are industrious, also provided are logs of the SPI messages sent between a PS2 and a valid controller for a few different games.

It's worth noting that I probably didn't have to make such a thorough simulation of a valid controller to work with the PS3 or Xbox 360 adapter since they are likely much simpler and less varied in their setup handshaking. But, I wanted to use my PS2 since it is just laying around and a better simulation would increase the likelihood some random PS3 or Xbox 360 adapter will work.

Wiring
Wiring the Teensy is pretty easy. You need a PS2 cable (from an old controller or extension cable) that you are willing to cut and solder. This only requires 6 wires, detailed in the picture below. I read online that if you bought a cheap knock-off cable, the color coding may not match. So, refer to this pinout diagram if you are unsure.

Program your Teensy with the .hex file in the archive provided, and the LED should begin to blink in "search mode". When you plug in the PS2 connector to the console (while the USB side is connected to your PC since it is USB powered), the LED should go solid but slightly dim. This means it is steadily chatting with the console. If it is flickering without your involvement, that means it is struggling to pass the setup handshake and you should feel free to debug that problem on your own. =o) It seems to work on my PS2 and my PS3(via adapter). I haven't gotten a Xbox 360 adapter yet, but I'm optimistic this will work assuming the adapter works at all.

Sending commands
As I mentioned, the Teensy 2.0 can be configured to provide a USB serial interface. If you are using Windows, you need to download these drivers. Mac and Linux don't need drivers. Now, a serial port should appear in your device list. Any programming environment that can access a serial port can now send button presses to the game console. When you do send a message, the LED on the Teensy will blink bright.

The Teensy is expecting 7 byte messages of the format:

0x5A (buttons1) (buttons2) (joyRX) (joyRY) (joyLX) (joyLY)

Here is a Windows C++ header file that might be helpful in formalizing the following:

Joystick neutral is 0x80, up and left are 0x00.
These are the bit masks for the two button bytes, which should be set to 0 when pressed, set 1 otherwise:

Buttons1:
PS2Button_Select = 0x01,
PS2Button_L3 = 0x02,
PS2Button_R3 = 0x04,
PS2Button_Start = 0x08,
PS2Button_Up = 0x10,
PS2Button_Right = 0x20,
PS2Button_Down = 0x40,
PS2Button_Left = 0x80,

Buttons2:
PS2Button_L2 = 0x01,
PS2Button_R2 = 0x02,
PS2Button_L1 = 0x04,
PS2Button_R1 = 0x08,
PS2Button_Triangle = 0x10,
PS2Button_Circle = 0x20,
PS2Button_Cross = 0x40,
PS2Button_Square = 0x80,

The starting 0x5A byte is used as a start byte to help segment message packets. If you send data without prepending with this byte, you will get 'x' characters sent back to you on the serial port which is the microcontroller complaining your messages aren't formatted correctly.

Autonomously Playing Guitar Hero
As I mentioned at the beginning of the post, the whole purpose of this project was to see if I could analyze the video stream from the game console to autonomously play the game. Arguably one of the easiest games to automate is Guitar Hero, which basically turned millions of willing people into bad midi file readers. So, I'm about to use a lot more processing power and hardware to also make a bad midi file reader. =o) In case it wasn't clear from my description above, this is a flow chart of what is going on here:

The Playstation spits out the video image, the PC analyzes that image, and then sends controller events back to the Playstation. Now, this may seem absolutely absurd to a normal human being. But, I'm an engineer at heart and this makes total sense to me. The nice thing is that assuming I can make the "analyze that image" software infinitely smart, I can hopefully play all sorts of games autonomously - perhaps even driving games, first person shooters, or platformers. Doing all that will be tough, but I'll learn a lot in the process. Some of that may even be useful for analyzing things in the real world as well. But, for now, lets just start with Guitar Hero on Expert mode.

For video capture, I'm just using a $30 USB capture device. Since real-time processing was priority for me, 640x480 images @ 30Hz is plenty of resolution to start with. In fact, for this starter project, I'm just using 320x240. For a computer vision library I am using OpenCV. Although it has some eccentrcities, it is a very powerful toolkit for getting up and running quickly. I'm not going to go into too much detail about OpenCV. But roughly, what I do for playing Guitar Hero is:

1. Unwarp the fret board so it is square (using a homography)
2. Use a template match to find notes along each column.
3. Track notes over time to ensure they are not spurrious.
4. Once notes hit a trigger line, queue the button press.

There are a bunch more subtle details to make it work just right. It seems to work pretty well hitting 95% of the notes on expert mode. It doesn't really handle star notes, sustains, star power, or other weird special effects. But, it can finish all the songs on Expert Mode. Thanks to the simple UI design of "just hit the X button to advance", it can play the entire Career mode from start to finish without me touching it. Here's a video of it finishing the last rift of the last expert song.

For those extremely interested, here is a copy of the OpenCV project that does this. This is DEFINITELY provided without any support and requires getting OpenCV installed and running.

Want to get your hands on an experimental research prototype from Microsoft Hardware?

2010-08-12T09:53:00.000-07:00

It's that time of year again, and we are coming up to UIST 2010 - a research conference on new interface hardware and software technology. Once again, Microsoft Applied Sciences is sponsoring the student innovation contest. Last year, it was the pressure sensitive keyboard. This year is the: Adaptive Keyboard.

From the Applied Sciences page, "[The adaptive keyboard] is a research prototype developed by Microsoft Hardware to explore how combining display and input capabilities in a keyboard can allow users to be more productive. The keyboard incorporates a large, touch-sensitive display strip at the top. In addition, the display continues underneath the keys, allowing the legends to be modified in real time. This lets you do things like change the character set to a different language or display command icons."

If you are interested in participating, visit the UIST student contest page for rules and important dates. The entry deadline is August 17th!

Playing Nintendo with your eyes

2010-08-06T10:00:00.000-07:00

Haha, now this is how you make a science project/research video! Great work guys. Next time I'm in the area, I'd be happy to buy you guys lunch. It looks ridiculous, but great at the same time. 150K views and counting. These Waterloo Labs guys really have something going with their "Engineer Awesome" mantra. My hats off to them. You can check out their other videos on their YouTube channel.

"Congrats"

2010-06-22T13:31:00.001-07:00

I just wanted to say, "congrats". To everyone involved at Xbox, everyone at the game studios. For a really great showing at E3 last week. For all the blood, sweat, and tears that got us to this point. And for helping to bring to market the most sophisticated consumer device ever created.

"Congrats"

Laptops + Pico projectors = Bonfire

2010-04-01T10:23:00.000-07:00

Some colleagues of mine at Intel Research/University of Washington recently got permission to post a video of thier research project from UIST 2009. It's a really nice idea that I wish I had done myself and, in my opinon, is one of the better applications of cheap pico projectors that I've seen.

Using pico-projectors and cameras mounted on a laptop, you can take advantage of nearby table space around the laptop to effectively expand your workspace at will. The video is a little stiff but, like many good ideas, you get the jist of it pretty quick.

Guitar Hero without a guitar

2009-10-26T14:26:00.000-07:00

This year, I'm helping the User Interface Software and Technology (UIST) conference put their proceedings videos online so that more people can access them. So far, I've gotten most of the videos from this year uploaded to the conference YouTube account. If you've closely followed the tech media coverage, you might recognize projects like the pressure sensitive keyboard and Mouse 2.0.

However, one of my favorite projects this year was a muscle sensing system that (among other things) allows you to play Guitar Hero without a guitar. It directly senses the electrical signals in your arms and maps those to the appropriate button presses. This was done by Scott Saponas, a Phd student at the University of Washington exploring a variety of biometric sensing techniques for input.

I also really like this project which combines a large touch table with other physical input devices such as multiple mice and keyboards all working together nicely. This prototype was done by Bjoern Hartman who has recently joined UC Berkeley's faculty.

I like systems that combine many modes of input so that you can dynamically choose the right device for the job and can gracefully scale to multiple people simultaneously. We are pretty far past having a 1:1 ratio between between people and computers. Yet, most systems today are still designed with 1 device and 1 user in mind.

Pressure Sensitive Keyboard

2009-08-08T14:18:00.000-07:00

Some of my colleagues in the Applied Sciences group in Microsoft Hardware have recently gotten some media exposure for one of their recent projects: the pressure sensitive keyboard

Congrats! It's a very nice prototype, and I look forward to seeing what the students at UIST cook up in the innovation contest.

If you aren't familiar with UIST (User Interface Software and Technology), it is a conference dedicated to new interface research. It is one of my favorite conferences to attend. It's where I demonstrated my past work like Automatic Projector Calibration, Foldable Interactive Displays, where Chris Harrison (recently known for Physically Changing Displays) showed off Scratch Input, Andy Wilson showed early pre-Surface touch tables, Hrvoje Benko showed Spherical Surface, MSR-Cambridge presented Second Light, and where Jeff Han first demonstrated FTIR. ...and that's just a small sample from the last 2-3 years.

So if you are interested in new interface technology (or you are part of the tech media) I would encourage you to attend UIST in October this year. You'll get to see what the latest interface researchers are doing all over the world and get to talk to the people behind the ones you have read about.

Rhonda - 3D drawing

2009-07-23T10:01:00.001-07:00

It's always good to give people reminders of what is possible when you don't stick with just a mouse and keyboard. This is a very nice piece of interface work for 3D drawing. The system is called Rhonda. The drawing is a bit on the abstract art side, but it's easy to see the level of control he has.

The great thing about 3d drawing is that the current tools are awful, so new ways of doing it are always interesting. Unfortunately, the bad thing about 3d drawing is that there are a relatively small number of people on the planet that really want to do it. So, it's unlikely these interfaces become widespread outside the domain.

Project Natal

2009-06-01T20:58:00.000-07:00

If you've been wondering why my project blog has been pretty quiet, I can finally say it is because I have been helping Xbox with Project Natal. If you haven't seen the vision video, it is definitely worth checking out:

Now, I should preface by saying I don't deserve credit for anything that you saw at E3. A large team of very smart, very hard working people were involved in building the demos you saw on stage. The part I am working on has much more to do with making sure this can transition from the E3 stage to your living room - for which there is an even larger team of very smart, very hard working people involved. The other thing I should say is that I can't really reveal any details that haven't already been made public. Unfortunately.

Speaking as someone who has been working in interface and sensing technology for nearly 10 years, this is an astonishing combination of hardware and software. The few times I’ve been able to show researchers the underlying components, their jaws drop with amazement... and with good reason.

The 3D sensor itself is a pretty incredible piece of equipment providing detailed 3D information about the environment similar to very expensive laser range finding systems but at a tiny fraction of the cost. Depth cameras provide you with a point cloud of the surface of objects that is fairly insensitive to various lighting conditions allowing you to do things that are simply impossible with a normal camera.

But once you have the 3D information, you then have to interpret that cloud of points as "people". This is where the researcher jaws stay dropped. The human tracking algorithms that the teams have developed are well ahead of the state of the art in computer vision in this domain. The sophistication and performance of the algorithms rival or exceed anything that I've seen in academic research, never mind a consumer product. At times, working on this project has felt like a miniature “Manhattan project” with developers and researchers from around the world coming together to make this happen.

We would all love to one day have our own personal holodeck. This is a pretty measurable step in that direction.

Xbox and Microsoft deserve an enormous amount of credit for taking on such an ambitious project. It’s one thing to say “Wouldn’t it be cool if…”, but it’s another thing entirely to say, “let’s dedicate the resources to really make it happen inventing whatever needs to be invented along the way.” I have to say it's pretty neat building the future.

Inspiring robots set to nice music

2009-04-20T10:59:00.000-07:00

Since, I'm on a bit of a mini-robot kick at the moment. I thought I would share some of the videos I've seen lately that at least inspired it. The elegance of some of these movements and the music remind me of the child-like imagination that we perhaps once had before becoming more jaded with age. At least, the dreams of a little engineer.

This last one is just fun.

Adventures with Bioloid

2009-04-20T00:45:00.000-07:00

A couple of weeks ago, I was participating in the Siggraph Jury review process looking at some of the projects submitted this year. There were a couple of submissions using humanoid servo motor robots. Since I have always had an itch to play with robots that I've never had a chance to scratch, I decided to look into buying one. One of the best selections of these robots I found online was at Trossen Robotics. After a lot of reading and video watching, the current highest rated robots appear to be the Robonova, Kondo KHR-2HV, Futaba RBT-1, and the edutainment Robotis Bioloid. These are all very impressive robots that all (with the exception of the Bioloid) are used in the RoboCup Soccer competitions. Combined with the $900-$1500 price tags, these are definitely not your typical kids toy.

After much deliberation, I ended up going with the Bioloid. It's one of the more well documented robots with a healthy developer community, and it's highly reconfigurable. It comes with an "errector set"-like kit which allows you to build a variety of robots, not just humanoids. However, this modularity comes at the cost of extra weight. So, while the power-to-weight ratio of the servo motors maybe comparable to the higher-end robots, the overall performance of the robot is noticeably slower and clunkier. It also happens to be one of the cheaper robots at$900. I really liked the re-configurability (for future robot projects), the number of degrees of freedom (particularly in the hip), and the size of the community support/English documentation.

When I first got it, I was a little intimidated by the number of pieces in the box. Being an educational robot, I was hoping it was going to be a quick and simple setup. While the instructions are fairly easy to follow, it did take me about 5 hours from opening the box to a completed robot. Assembly requires handing many similar looking parts and lots of tiny screws. However, it is very satisfying to see the robot slowly take form as you assemble the components.

Once it is done, you do get an urge to say out loud "IT'S ALIVE!" with a grin on your face.

The included CD does have software to program and contol the robot, but as I expected, it is somewhat limited to keyframe pose playback or simplfied visual programming. My original intent was to run the robot using my own C/C++ or C# program. So, I didn't spend much time with the included software other than to verify the robot worked and to get an understanding of the control flow. The C development tools described by the documentation are for writing programs that run on the Atmega128 chip inside the robot. What I wanted was to run the control logic on my PC. However, getting my own software to control the robot ended up being quite a bit more challenging than I had expected.

The first major hurdle I had was the physical connection. The kit comes with a serial cable for communication with the robot, but it uses a DB-9 connector that is only found on desktops these days and my main machine is a laptop. The Bioloid has an expansion slot on its control board, the CM-5, for a wireless Zigbee connection. There are a few resources online explaining how to use a Bluetooh Module instead of a Zigbee module. So, I had ordered a BlueSMIRF module (WRL-08332) from Sparkfun in anticipation of doing this.

The Bioloid controller requires 57600 baud serial communication, but the Bluetooth modules typically come set to 9600 baud. To my frustration, the information on Sparkfun's website on exactly how to re-configure the baud rate is a little obtuse. They have different chipset versions with different command sets. Something I burned about 2 hours learning was that newer modules, with the BGB203 chip, CANNOT be configured wirelessly over the Bluetooth connection. They have to be configured via the wired TTL TX/RX connections. Moreover, to change the baud rate and save it to memory requires a TTL connection that can dynamically change its baud rate to issue the "save to memory" command at the new baud rate. My short lived attempt at trying to using a second Bluetooth module was a failure because while it could issue the "change baud rate" command it could not issue the "save to memory" command. =oP Anyway, once I got my hands on a USB TTL-232 cable, things went smoothly. One other important thing to check is the Bluetooth passkey of the module (using the configuration commands). In Vista, to make the Bluetooth serial port binding behave nicely I had to configure the bluetooth connection to use the passkey. It happened to be set to "0000" on my module despite the documentation from Sparkfun indicating it would be "default".

The second problem I ran into was that once I connected the Bluetooth module to the Zigbee communication pins, I discovered that it is NOT A REPLACEMENT for to the PC LINK serial programming cable port at the top of the CM-5. The data from the Zigbee unit is only meant to provide command bytes triggering behaviors in a program running natively on the CM-5. What I wanted was raw access to the servos so I could run control logic on the PC. This can only done via the PC LINK. The data from the Zigbee module never makes it to the servo motor bus. So after some digging, I found a schematic for the CM-5 and found where to piggy back data onto the main PC link. The image below shows where I connected my wires. The TX from the Bluetooth module is attached to the logic level side of the RS232 level converter. The other wires are connected to the Zigbee pins as decribed by the reference above.

This defintiely at your own risk and may behave badly if you try to connect the wired PC link cable at the same time. But since I intend to only use the Bluetooth serial connection, this was not a concern for me.

Now, I can run the included software such as Motion Builder using the Bluetooth connection as if I had the wired PC Link cable attached. Great! The CM-5 provides some commands such that if you open up a ternimal window to the serial port, you can get/set the data for each servo manually. However, the human readable commands use A LOT of bandwidth overhead. Given that the 57600 baud connection is already runnning much slower than the 1000000 native baud rate of the Dynamixel AX-12 servo motors, trying to control the robot via these commands was unbearably slow even if executed programmatically and I kept running into buffer limits on more complex commands.

A not-very-well documented mode of the CM-5 is "Toss Mode" which appears to be a pass through mode to the servo motor bus. Put the CM-5 in Manage Mode, and hit the Start button. In your PC's terminal window, type "t" then hit enter. It should respond with "Toss Mode". At this point, any bytes sent via the serial connection is pushed directly onto the servo motor bus and vice versa. Finally! Exactly what I wanted. After slowly making my way through the Dynamixel AX-12 User's guide, I now have a small C# library that provides direct control/communication with the servos via the serial port. It's still pretty rough but once I clean it up a bit more, I'll probably make it available for download. But, it is a farily straight forward implementation of key commands from the Dynamixel users manual. The hard part was getting the hardware into the right configuration to allow direct communication.

The next step to do is write my own control and logic software to see if I can make it do more interesting things than simply recall preset poses. There's also a mild annoyance in that the 57600 baud serial link is about 17x slower than the 1000000 baud servo bus speed. If this becomes an issue, I might explore making an alternative controller board that would provide 1000000 baud pass through, or even put each limb on a separate bus to parallelize I/O making it even faster. This could result in a 70x speed bump in servo communication which would be helpful with real-time control logic.

(update 4-22-09) It looks like Scott Ferguson has C# libraries for controlling a dynamixel directly via a serial port. He was using a USB2Dynamixel adapter. The bad this is that it doesn't provide power to the servo, only control. So, using the CM-5 a wirelss control/power brick is still fairly attractive.

Magnetic Ink

2009-03-21T14:18:00.001-07:00

In a bit of procrastineering research, I started looking into making my own ferrofluid. Apparently the best stuff to use these days is Magnetic Ink Character Recognition (MICR) Toner. But, it's a little hard to find in bulk. The most amazing work I've seen done with ferro-fluid is by Sachiko Kodama:

It's difficult and messy stuff to work with. Not to mention you need to know how to generate custom magnetic fields to move it. So, it's always been a little low on my project list.

Though, in my brief search for materials, I came across this wonderful artwork by flight404. This is done with an application called Processing, which is a programming environment designed for computational art that grew out of the MIT Media Lab. It's definitely evolved quite a bit since the last time I looked at it if you can create these kinds of visuals. It's beautiful and all free (as in open source). =o) Maybe I'll try my hand at it again (in what little free time I have). Simply trying to recreate something that approximates this visual style would be a satifying exercise.

What would you do with a thousand sheep?

2009-03-19T13:31:00.000-07:00

This is absolutely astonishing! What are YOU doing with your sheep? eh? Whatever it is, it's probably not as good as this:

"Birth of a new art form..."

2009-03-15T13:09:00.001-07:00

It's not often you hear that phrase. I don't think I've ever used it myself, but a few people have been tossing that around when talking about Kutiman's work at Thru-You.com who does video remixes of musicians around the world to create amazing new musical/video pieces. The one that is getting the most blog exposure is called The Mother of All Funk Chords. However, the one I think more clearly demonstrates the subtlety and intricacy of this artistic contribution is below entitled "I am new":

There have certainly been many video remixes before, but this steps it up a few notches in several directions - in no small part facilitated by abundance and wealth of YouTube performances. One of the earlier examples of musical/video editing that I really enjoyed is work by Lasse Gjertsen. He started with "human beat boxing", but really stepped it up in the following video. It's worth remembering that he doesn't know how to play these instruments:

On the topic of the great examples of creating musical mixes with video, this is another wonderful example of what one individual with a camera, video editor, a few instruments, and some determination can create:

Trying your hand at making something like this would be an excellent procrastineering project.

Wonderful slow-motion stabilized video montage of New York

2009-02-16T13:57:00.001-08:00

I'm not sure who Vincente Sachuc is, but he's certainly got a career in cinematography if he wants one. This makes me want to get back into creative film work rather than technical videos. He mentions this is captured with a Casio EX-F1 at 300 fps and edited at 24 fps. A skateboard and a Steadicam Merlin help with the smooth traveling shots. Of course, you could buy one of my Poor Man's Steadycams at 5% of the cost of a Merlin =o). Colorization done in Premiere and Photoshop.

New York 2008 from Vicente Sahuc on Vimeo.

Impressive and Frightening

2009-01-26T13:33:00.000-08:00

This is one of the most impressive (and frightening on several levels) pieces of engineering I've ever seen. When there exists a device that can turn a living tree into logs in under 15 seconds, it is no surprise that deforestation can be a problem. Fortunately, this appears to be a tree farm... for IKEA?

The unbridled and unapologetic efficiency by which this machine performs its function leaves a visceral sensation of both awe and horror. It is distrubingly animal-like. The fact the tree is mostly debarked by the time it hits the ground makes my jaw drop.

Sensitive Object - make any surface touch sensitive

2009-01-09T12:01:00.001-08:00

This is not extremely new technology, but not a lot of people know about it and it has evolved quite nicely in recent years. Sensitive Object is a French company that sepecializes in the use of microphones to detect touches anywhere on an arbitrary objects. With multiple mics they can determine the location of touches or even dragging (as well as multiple dragging touches - not shown in the video below).

I've had a chance to play with it in person, and it's pretty impressive stuff. It works a lot better than I would expect. It doesn't detect touch locations using a triangulation technique (i.e. see how long it takes for the sound to reach each microphone) because that would vary greatly depending on the material the object was made of (plastic, metal, wood, etc) and depend on the shape of the object. Sensitive Object can do any shape like a vase, or statue.

They accomplish this using a pattern matching technique. Each touch sound gets compared to a known table of sound-to-location mappings. Which means you have to enter this mapping during a calibration step. (i.e. give the system a couple examples of touching each location that you want to recognize, touching here sounds like this.... touching there sounds like that). This upfront calibration step is somewhat heavy, but when you are done it's quite powerful. You could turn a cardboard box, a basketball, your car, or even your friend's head into a touch sensitive surface (if it's hard enough). Though, larger objects may only be "bang sensitive" surfaces.

"He was like..."

2008-12-17T10:35:00.000-08:00

While certainly not the technology focused topic I usually post, I definitely wasted a few minutes trying to sing the following chorus in the true spirit of procrastination. Try to sing along (if you can). If you are linguistically incapable, just reading along is amusing enough.

"he was like
she was all
he was all
they were like
we were all,

like oh my god
like totally

we were like
that was all
they were all
he was like
she was like

all totally
like oh my god"

If this was not educational enough for you, the following "anthropological introduction to YouTube" has a boring title, but is an incredibly fascinating and entertaining discussion of the cultural and social phenomena within the depths of YouTube... and relevant to the video above. Like totally. (warning: 1 hour talk, but definitely one of the better uses of 1 hour in my life).

Nice Pen-based Input Research

2008-12-15T17:06:00.000-08:00

One of the things I enjoy to using this blog for is to share cool projects from Human-Computer Interaction (HCI) research. This post highlights projects by Gonzalo Ramos (or "Gonzo" for short) and his co-authors. He has worked on several projects demonstrating how much better pen input software could be. These are just a few I like.

1. The Zlider - A pressure sensitive slider widget that adds additional navigation and control capability to standard slider interactions. Academic research video below. Quick demo montage at beginning, but the demo meat is at 3:07

2. Using a Pen to Effortlessly Bridge Displays. Using a stylus, you can simply drag documents between computer screens or mobile devices. The pen motion also implicity defines the orientation of the displays relative to one another. Academic video below. Demos at the beginning and more mobile screen scenarios at around 2:43

3. Rolling the Pen as Input Using an external tracker and a Wacom tablet, rotating the pen in your fingers can be used to control another parameter without moving the stylus. Academic video below, demo meat at 2:19

You can check more of his projects on his website.

Some great Wiimote IR tracking projects

2008-11-20T12:28:00.000-08:00

I've decided to collect some of my favorite projects I've seen people do with my Wiimote projects, derivatives of them, or distantly inspired (through the creator's own admissions). It's a surprise, and flattering to see how many people seem happy to credit me. Thanks all! The list gets more "unusual" the further you go down.

Two Wiimote Whiteboards to make a competative relay race:

Great IR wands for the Wiimote whiteboard. I've been meaning to make these, but I haven't gotten to it yet.

Some nice two handed, two finger pinching systems:

Wiimote Wheelchair art. Unfortunately, no video but more information at this link.

Head tracking prototypes with Anime assets. The effect of the girl coming out of the screen (about half way through the video) is very nicely done with the "haze" layer. His other videos are also worth checking out. I don't know what he does for a living, but he's good at it.

Wii Theremin gallantly created/performed by Ken Moore:

Finally, a video on "chicken head tracking". It doesn't use the Wii remote, but was posted as a response to my video and I love it!

Scratch Input and Low-Cost Multi-spectral material sensor

2008-11-13T19:28:00.000-08:00

Chris Harrison, a PhD Student at my old program at CMU, presented a couple projects of his at UIST 2008 that I really really like. The first is his "Scratch Input" device. The basic idea is that if you place a senstive microphone on the bottom of a mobile device. Any large, hard surface you put it down on can now be used as an input gesture surface. A variety of gestures can be distinctly and reliably detected with some simple machine learning. Video (academic) below include a nice demo where he turns his entire wall into an MP3 player controller:

The other project he presented was a simple, cheap multi-spectral sensor for recognizing various materials. It includes an IR LED, UV LED, RGB LED, a photoresistor, and a TSL230 TOAS optical sensor. With these, he read the reflectively under different illuminations to recognize 27 different materials with 86.9% accuracy, be this your jeans, your backpack, your desk at home, your desk at work. This means coarse location awareness of mobile devices for cheap, some opportunities for more intelligent power management, and implicit security behaviors when placed on familiar or unfamiliar surfaces. Very nice work.

SurfaceWare - sensing glasses for Surface

2008-11-07T03:21:00.001-08:00

My colleague, Paul Dietz, in the Applied Sciences group released a video of one of his first projects he did when he joined Microsoft. These glasses use the transparent material of the glass as prisms that sense the amount of liquid in them by watching the amount of internally reflected IR light. Check out the video:

If you aren't familiar with how Surface works, it is a rear projected table that also has a bright IR emitter inside that illuminates objects placed on the surface which are then visible to an IR camera. The video does a good job explaining how the glasses work.

This is actually a revisit of an older project of Paul's called iGlassware. That one used passively powered RFID sensor tags in the base of the glass to capacitively measure the liquid level. The table had a big RFID antenna in it. Paul was also a key developer of Mitsubishi Electric Research Lab's Diamond Touch table being skillfully demonstrated by Ed Tse below.

Ed is currently at Smart Technologies, who helped push out their new touch table:

Andy Wilson

2008-10-09T17:17:00.000-07:00

I was re-watching some videos of work done by one my colleagues Andy Wilson, and I don't think his work gets as much attention as it deserves given how amazing it is. If you think my stuff is cool, you should bow down to his greatness... or at least watch these videos.

Working with the PixArt camera directly

2008-09-04T17:51:00.000-07:00

This has been a pretty whirlwind past few months. Lots of things have happened, almost none of which procrastineering related which is why I haven't posted anything here. But, one of the things that I have poked at in the past few weeks was creating a PixArt to USB-HID device which allows the camera from the Wiimote to appear as a relatively easy to access USB device. This addresses several problems with using the Wiimote such as running off batteries for extended periods and flakey platform specific Bluetooth drivers. It's also possible to read from the Pixart cam at over 100Hz if you read directly via I2C as well as track visible dots once you remove the IR filter. Of course, none of this was discovered by me. All credit belongs to the numerous individuals who have contributed thier knowledge to the various Wiimote hacking websites. Normally, this project wouldn't be worth a post, but all the information on how to do this is pretty scattered and difficult to follow. So, I figured I would contribute by trying to making this all a bit clearer.

This project is fairly advanced. You must be comfortable with working with microcontrollers. Several simpler devices such as the Arduino or the Basic Stamp may work, but I used the 18F4550 PIC Microcontroller which provides built-in full-speed USB capabilites. But first, let talk about the PixArt camera:

Here's the pinout thanks to kako and a PCB picture. The Reset pin is active low, so use a pullup resistor to Vcc. The Wiimote runs the camera with a 25Mhz clock, but it also works with a 20Mhz clock so you might get away with fudging this a bit. The I2C communication is fast 400Khz and the slave device address is 0xB0. Most microcontroller development platforms should include I2C communication capabilities. If yours doesn't, get a better dev kit =o). Desoldering the camera can be hard with so many pins. But, careful use of a hot air gun will do the trick. The first part is to initialize the camera over I2C. Here's the pseudo code for initializing to maximum sensitivity (actual CCS C code in comments):

write(hex): B0 30 01
wait 100ms
write(hex): B0 00 00 00 00 00 00 00 90 //sensitivity part 1
wait 100ms
write (hex): B0 07 00 41 //sensitivity part 2
wait 100ms
write(hex): B0 1A 40 00 //sensitivity part 3
wait 100ms
write(hex): B0 33 03 //sets the mode
wait 100ms
write(hex): B0 30 08
wait 100ms

It's still somewhat mysterious to me what all these mean, but in this mess is the sensitivity and mode settings described at Wiibrew. The above code uses the sensitivity setting suggested by inio "00 00 00 00 00 00 90 00 41, 40 00" experssed in the 2nd, 3rd, and 4th message. The wait times are conservatively long. After you initialize, you can now read samples from it:

write(hex): B0 37 //prepare for reading
wait 25us
write(hex): B1 //read request
read 8 bytes
wait 380us
write(hex): B1 //read request
read 4 bytes

This yeilds one sample from the camera containing 12 bytes, 3 for each of the 4 potential points. The format of the data will be the Extended Mode (X,Y, Y 2-msb, X 2-msb, Size 4-bits). The wait timings approximate what the Wiimote does. I've called this routine 1000 times per second without ill effect. Though, I doubt this is actually scanning the sensor and instead is just reporting the contents of an interal buffer. But, people claim 200Hz updates are possible. So, you can use that as a suggestion.

Hooking this up to your microcontroller is pretty straight forward. Give the camera 3.3v power using a voltage regulator, ground, a 20-25Mhz clock, and connect the SDA and SCL lines (don't forget your pull up resistors), and pull up the reset pin.

The CCS C Compiler for the PIC18F4550 includes USB-HID sample code. It's simply a matter of stuffing the data you got from the PixArt camera into the input report buffers for the USB. With this, you could actually create a USB mouse profile and make it control the cursor without any software or drivers at all. If set it up as a full speed device, it's possible to get 1ms reports providing extremely low latency updates. CCS provides relatively affordable PIC programmers as well. Explaining how to set all this up is not within the scope of this post, but it should be plenty to get you started. If you want to make a PCB, you can try ExpressPCB which can get you boards in-hand for as low as $60.

Update 9/6/08: Just a note about the clock. Since my PIC was using a 20Mhz resonator, I just piggy backed the Pixart clock pin off the OSC2/CLKO pin of the PIC which seemed to work fine. Also, Kako has more details (in Japanese) on doing this with an Arduino

More Wiimote Projects - A Brain Dump

2008-06-23T00:01:00.000-07:00

It’s been a while since I’ve posted anything. That’s largely because I’ve been traveling a lot, giving talks, and most recently relocating to a new city. It became clear to me a while ago that I wasn’t going to get around to making more videos anytime soon. So, I figured I would make a post about the projects that I would probably make videos of if I had more free time. The content of this post has been in the talks that I’ve been giving, but I’m just sitting down to write it out now for my trusty blog readers.

1. Throwable Displays using the Wii remote

This I actually built and demoed in my lab at CMU. But, it only existed for about two days before I had to break it down to move and I didn’t get a chance to document it. Several months ago, a patent filed by Philips made some of the tech new sites about throwable displays in games. But it was a concept patent pretty far from a working demo. However, it turns out it’s pretty easy to implement using a projector, a wiimote, an IR emitter, and some of our trusty retro-reflective tape. It essentially combines the techniques from the finger tracking and the wiimote whiteboard projects. You put a little bit of reflective tape on each corner of a square piece of foam core, turn on the IR emitter so the Wiimote can see the four corners, align the camera tracking data with a projector using the 4-point calibration, and then the projector can display images perfectly aligned to the edges of a moving piece of foam core. The process of using a projector to augment the appearance of objects is called “Spatially Augmented Reality”.

Research colleagues of mine made a really fun demo where they tracked an air hockey puck from above and projected down on the air hockey table to display all sorts of visual effects that responded to the location/motion of the puck. They were demonstrating a fancy new type of high-speed tracking system. But, the Wiimote works quite well at 100Hz. I wish I had documented the throwable display on video, because it worked quite well. You really could pick it up and throw it around and the video image stays fairly locked onto the surface. There's a small latency primarily due to the 60Hz refresh of the projector. I even made a rough demo of the air hockey table, but it was VERY rough - just drew a line tail behind the puck. Again, a little patch of reflective tape on the puck and IR ring illuminated Wiimote above. However, the throwable display concept is actually a simpler implementation of a project I did earlier on “Foldable Displays” (tracked using a Wii remote) which I did make a video of, but not in tutorial format like my other Wii videos:

2. 3D tracking using two (or more) Wii remotes

Since the tracking in the Wiimote is done with a camera, if you have two cameras you can do a simple stereo vision triangulation to do full 3D motion capture for about $100. This was actually already done by some people at the University of Cambridge:

This is text book computer vision algorithm, but I haven’t gotten around to making a C# implementation. Obviously, you can use more than 2 wii remotes to increase tracking stability as well as increase occlusion tolerance. This would be a VERY useful and popular utility if anyone out there wants to make a nice software tool to transform multiple wiimotes into a cheap mocap system.

3. Universal Pointer using the Wii remote

The nice thing about the camera is that it can detect multiple points in different configurations. The four dots could be used to create a set of barcode-like or glyph-like identifiers above each screen in a multi-display environment. This would not only provide pointing functionality on each screen, but also provide screen ID which means you could interact with any cooperating computer simply by pointing at its screen. No fumbling for the mouse and keyboard, just walk around the room, or office building, or campus, and point at a screen. If all the computers were networked, you could carry files with your Wiimote virtually (using the controller ID) letting you copy/paste or otherwise manipulate documents across arbitrary screens regardless of what computer is driving the display or what input device is attached to the computer. You just carry your universal pointer that works on any screen, anywhere automatically. This makes a big infrastructure assumption, but it really alters the way one could interact with computational environments. The computers disappear and it becomes just a bunch of screens and your universal pointer.

Similarly, arbitrary objects could have unique IR identifiers. For example, if each lamp in your house had a uniquely shaped Wii sensor bar on it (and they were computer controlled lamps, of course), you could turn on a specific lamp simply by pointing at it and pressing a button or dim it by rotating the wiimote. If was an RGB led lamp, you could specify brightness, hue, and saturation with a quick gesture..

4. Laser Tag using Wii remotes

If you put IR leds on each of the Wii remotes, they can see each other. So, you can have a laser-tag like interaction just using Wii remotes – no display, except perhaps if you wanted a big score board. You’d have to validate which Wii remote you were shooting at, which you would do using some kind of IR LED blink sequence for confirmation. Just wire up the IR leds to the LEDs built into the Wii remote, so you can computer control their illumination.

5. IR tracking with ID using the Wii remote

This is more technical (and related to the above idea), but it addresses an important issue that I have yet to see done in either commercial or research systems. The problem with IR blob tracking using cameras is that you can’t which blob is which. You could blink the LEDs to broadcast their ID. But, this 1) would be slow because the ID data rate is limited by the frame rate of the camera 2) really hurts your tracking rate/reliability because you don’t know where the dot is when the LED is off. Now, the Wii remote’s camera chip gives 100Hz update, which might be tolerable for a small number of IDs. But, this approach doesn’t really work well when you want fast tracking with lots of unique IDs. One solution is to attach a high speed IR receiver to the side of the Wii remote for data transmission and simply use the camera for location tracking. IR receivers used in your TV probably support data rates of around 4000 bps - much higher than the 50 bps sampling limit you could squeeze out of the Wii remote. So, as the LEDs furiously blink their IDs at 4Kbps, they look like they are constantly on to the camera. This yields good tracking as well as many IDs. Now when you have multiple LEDs transmitting simultaneously, you’ll get packet collisions. So, some type of collision avoidance scheme would be needed of which there are many to choose from. It will also be necessary to re-associate the data packet with a visible dot. So, not all the LEDs can be visible all the time. But, you only have to sacrifice a small number of camera frames to support a large number of IDs. You can also probably boost performance if you are willing to accept short term probabilistic ID association.

WiimoteWhiteboard v0.2 - slightely updated/fixed

2008-03-27T15:11:00.001-07:00

Hi, all. I got a few moments yesterday to make a couple of small improvements to the Wiimote Whiteboard software. Most notably, I improved the mouse emulation code. There were problems where it wouldn't work with some programs like PowerPoint, Alias Sketchbook, etc. So, those work fine now.

I also added a "Tracking Utilization" feedback to the GUI which tells you how much of the camera you are utilizing for tracking. This gets updated after calibration. This provides a way to evaluate how good your Wiimote placement is which directly impacts tracking quality. Getting this number to 100% is virtually impossible in any usable configuration, but dropping below 50% is a sign of not-so-great Wiimote placement.

I've also updated the code to use Brian Peeks's WiimoteLib v1.2. I think this may help a little with greater Bluetooth/Vista compatibility. On a technical note, the WiimoteLib in this download is slightly modified to increase the Wiimote camera sensitivity.

You can use this link to download the new version: WiimoteWhiteboard v0.2

Inspiring Students

2008-03-22T10:02:00.000-07:00

One of the great, unexpected, and perhaps most influential aspects of creating these videos has been how many people they have inspired and sparked an innovative spirit in. I've gotten hundreds of emails from young students that express this enthusiasm. But, perhaps one of the best testimonials is this news article about kids in the Clara Byrd Baker Elementary School's Lego Club in Williamsburg, VA. The students there, lead by Kofi Merritt, are getting excited about innovating in technology by creating their own electronic white boards.

Tracking multiple laser pointers @ 200Hz using the Wii remote

2008-03-13T20:08:00.001-07:00

A couple of videos (vid1 and vid2) from sha433 demonstrate that if you are willing to crack open your Wii remote (you'll need a tri-wing screw driver) and take out the IR filter, the camera will track visible points of light, including laser pointers. That's pretty neat. Additionally, it has also been recently discovered by a few people that if you read the data directly off the camera chip using I2C, you can get 200 Hz tracking (vs 100Hz that you get over Bluetooth). It looks like there might be a latency hit, though, depending on how you get the data into your computer.

So, if you are willing to get a little down and dirty with the hardware, you can pull out quite a bit more capability! Good job Sha!

YouTube Awards: Nominated for Best Instructional Video of 2007

2008-03-13T15:55:00.001-07:00

Hey all, (shameless plug) it looks like my video on head-tracking was nominated for the Youtube Award - Best Instructional Video of 2007! Very neat. =o) Please, go to the site and cast your vote! Thanks!

EA's Boom Blox to include Wiimote Headtracking

2008-02-22T01:44:00.000-08:00

Louis Castle announced yesterday at GDC2008 that EA's upcoming title "Boom Blox" will ship with an easter egg that allows head-tracking using the Wii remote! Very cool. Here's the Digg article someone created for it - which you should all digg to help drum up some attention and demand for this feature. Reward the developers who decided this was worth including and send a signal to EA and the greater game development community that this is a desired step forward in the evolution of game play technology. The expected release will be in May.

I'm proud. If this pans out, it'll be only 5 months between the initial research prototype to integration into a major product release. Sweet! Happy to see my stuff being used. Humorously, there were 3 other demos on the GDC expo floor showing variations of my head tracking demo. =o)

Just in case you are wondering: No, I don't get any royalties or benefits for the use of this technique in games. Personally, I'm much happier impacting the state of technology on such a large scale in such a short period of time rather than struggling to transform it into personal financial gain. In terms of my original intent behind creating the head-tracking demo, it has already been a wild success beyond my highest expectations.

Joystiq article

Digg article

Official Wiimote Project Forum!

2008-01-09T13:53:00.000-08:00

The number of comments on my blog posts have gotten rather unweildy making them difficult read and thus, not extremely useful. It has become clear that a full up discussion forum is needed to manage the threads of conversation. I know there is already a lot of chatter scattered all around the web about these Wiimote projects, so hopefully this isn't too late to centralize some of that discussion.

JD has honorably stepped up the challenge. So,... announcing the official discussion forum for wii remote projects!

http://www.wiimoteproject.com/

Everyone give JD a pat on the back, and try to use the forum for discussing wii remote projects. I'll try to add links to this forum on the main project page.

Link to the WiimoteProject Forum

Wiimote DesktopVR - FAQ and Adv Discussion

2007-12-21T22:59:00.000-08:00

Wiimote Whiteboard - FAQ and Adv Discussion

2007-12-21T22:39:00.001-08:00

A lot of very smart people have written in questions and suggestions and I think a lot of them are very clever and worth sharing... and I'd like to add some of my own expansions on what I showed in the video. Like they say, 2 brains are better than 1.... it turns out several thousand brains are even better. This post will be REGULARLY EDITTED to add new ideas as they come in. I know it breaks the RSS model, but I don't want to make a new post for every new idea. Sorry.
------------------------------------
My Bluetooth adapter and LEDs - Though it is reported to work with many bluetooth adapters and chipsets, I've been using this one sold by SparkFun electronics. The IR LEDs I use are Vishay TSAL6400s running at 100mA available from Mouser.

Tip switch - Instead of pressing the side button to illuminate the LED, a tip swtich could be added to illuminate when pressed on a surface. Though, it's mechanically harder to make and finding the right level of pressure for activation can be hard, especially if your surface varies (ex: whiteboard vs carpet)

Can you do this with a webcam? - The wii remote contains a 1024x768 camera with built-in hardware for IR blob tracking of upto 4 points at 100Hz. This significantly outperforms any $40 webcam I'm aware of. It'll work with a webcam, just not as well and not as easily.

Rear-Projection - using a rear projected surface, works great. The software will automatically work as-is so long as your pen is bright enough to shine through the projection surface. The wiimote gets a unobstructed view of the screen which eliminates occlusion problems and maximizes tracking resolution. Though, actually building a rear-projection wall or table can be a little difficult if you don't already have one.

Pressure sensitivity - wacom tablets give nice pressure sensitive behavior. Doing this with the wiimote whiteboard will be tough, at least through the camera data. The brightness of the LED varies significantly depending on angle, surface reflectivity, and distance. Any pressure data based off birghtness will not be reliable. The data could be transmitted via a high-frequency IR channel (like remote contol data) to an IR reciever mounted on the wiimote. (see "Right click" for more)

Right Click - any right click signal based off brightness is not likely going to be reliable. Additionally, using multiple LEDs (1 - left click, 2- right click) won't work at all distances and orientations as the LEDs get too close for the wiimote to distinguish them. The RIGHT way to do this is to use high-frequency IR data transmission (like remote control data) to an IR reciever. This will also give you a way of potentially getting pen ID - which is VERY powerful and may be a future project. Though, this requires quite a bit more hardware.

Multiple wiimotes for better tracking - if you have a large surface, you could have multiple wiimotes covering different areas to keep tracking reslution high, or have the wiimotes cover the same area to avoid occlusion problems, or do some interpolation of the data for higher accuracy. All possible, just with the smarter software. Doing this myself is probably not going to happen, but the source is freely available.

Wiimote Finger Tracking -FAQ and Adv Discussion

2007-12-21T22:18:00.000-08:00

"Unable to launch the DesktopVR program"

2007-12-21T18:00:00.000-08:00

blarg.... a number of people have been reporting a "Wiimote FishtankVR has encountered a problem" error. but whom are able to run my other programs. Unfortunately, I can't replicate the error on any of my computers and I'm leaving for the holidays. So, fixes may be slow for the next 2 weeks. If you can shed some light on this, please add a comment to this post and I'll upload a fixed version once this problem is hashed out.

Deleting the "config.dat" file and re-running the program may help.
Installing the most recent version of DirectX may help.

(update 12/30/07) I also got some suggestions that this might be due to a "x86/x64 linkage problem" related to 64-bit version of Windows. The program needs to be recompiled for this. I'll try to post a new version sometime in the next few weeks.

Head Tracking for Desktop VR Displays using the Wiimote

2007-12-21T07:18:00.000-08:00

Using the infrared camera in the Wii remote and a head mounted sensor bar (two IR LEDs), you can accurately track the location of your head and render view dependent images on the screen. This effectively transforms your display into a portal to a virtual environment. The display properly reacts to head and body movement as if it were a real window creating a realistic illusion of depth and space.

Link to the Wiimote Projects Page

Source Forge: Wiimote Whiteboard

2007-12-18T15:48:00.000-08:00

I'm swamped with requests for compatibility fixes, requests for Mac and Linux versions. ... and as much as I would love to be able to get this working in the hands of as many people as possible, I don't have the skills or time to do all this alone. So, here is my call for help. I have created a SourceForge repository for all WiimoteWhiteboard development.

If you are a Mac, PC, or Linux developer there are thousands of people (especially educators and entire schools) just waiting to use the software you write. Help out, become a hero.

If you tried the WiimoteWhiteboard software and had trouble with your setup, you can post your configuration in the SourceForge Help Forum.

Link to SourceForce: Wiimote Whiteboard

$14 Steadycam

2007-12-17T19:47:00.001-08:00

This is an oldy, but a goody. The first online tutorial I ever created was on creating your own low-cost camera stabilizer for only $14 (that's in 2001 dollars). It became a featured article in the premier issue of Make Magazine, posted on countless news/blog sites, and viewed well over a 1 million times. It has become a staple of the low-budget/student film-making community and is currently used in a number of high school and college production classes.

If you can't (or don't want to) make one, you can simply buy one. I currently sell them through littlegreatideas.com for $39.95. It includes some bells and whistles beyond the $14 tutorial version. If you like my projects and want to support me, buying one is currently one of the easiest ways to do that. Then again, I also won't turn down flat out donations. =o)

Link to $14 Steadycam
Link to buying a Poor Man's Steadycam

Wiimote Whiteboard with a Rear-Projected Display

2007-12-17T15:20:00.000-08:00

After a few people have written in asking about this, I think it's worth posting about it. The Wiimote Whiteboard technique/software works great with rear-projected surfaces. In fact, I think it's the best configuration because you can put the Wiimote behind the surface near the projector and you don't have to worry about blocking it's view. The camera also gets a prime view of the screen so tracking resolution is high.

I didn't talk about this in the original video because making a custom rear-projection screen is relatively hard... and making a rear projected drafting table, for which stylus work is better suited, is even a little harder. So, this might be a little out of reach for many people. But, it's well worth the effort in my opinion because the result is pretty great.

Link to WiimoteWhiteboard

Squiggly lines and other Bluetooth Drivers

2007-12-14T12:19:00.000-08:00

After seeing some videos people have posted of them using the WiimoteWhiteboard software, I have added the following to the troubleshooting section of the help document and mention this on the download page.

-----
"My lines are really squiggly and tracking is bad" - Remember the Wii Remote contains a camera, and the better view the camera has of your screen the better tracking you'll have. So, good placement is the key to good tracking. The more direct view the wiimote has of your screen, the better, The closer it is without losing visibility of your screen, the better. The wiimote camera has a 45 degree field of view. Keep this in mind for placement. Using the "Visible IR dots" diagnostic on in the software, you can test the visibility of the corners of your screen using your IR pen. It may be helpful to turn off cursor control. Activate the LED at each corner location and check if the Wiimote still sees it. Some squiggilyness is unavoidable, but it can definitely be minimized. Smoothing is on the feature list to be added in any future release.
----

Also, I've gotten testimony and links to videos that say this works with other bluetooth drivers besides Blue Soleil. But, I haven't tested them myself.

IR LED pen schematic and IR keychain lights

2007-12-12T15:46:00.001-08:00

I've uploaded a simple schematic of the IR LED pen I used. If you use a good LED, you can probably run it safely directly off a 1.5v AAA battery without a resistor. I used Vishay TSAL6100 940nm IR LEDs running at 100mA which I bought from Mouser. But, you can find usable LEDs and everything else you need from RadioShack. A number of people have also written in saying that some places sell IR keychain lights which may work right out of the box. NOTE: If you rough up the end of your LEDs, they will be more omni-directional and more easily seen by the wiimote.

Low-Cost Multi-point Interactive Whiteboards Using the Wiimote

2007-12-12T15:43:00.000-08:00

Since the Wiimote can track sources of infrared (IR) light, you can track pens that have an IR led in the tip. By pointing a wiimote at a projection screen or LCD display, you can create very low-cost interactive whiteboards or tablet displays. Since the Wiimote can track upto 4 points, multiple pens can be used.

When updates are made to the software, I'll post them here on this blog.

Link to the Wiimote Projects page

[update] Link to the SourceForge repository to help with development.

Tracking Your Fingers with the Wiimote

2007-12-12T15:36:00.000-08:00

Using an LED array and some reflective tape, you can use the infrared camera in the Wii remote to track objects, like your fingers, in 2D space. This lets you interact with your computer simply by waving your hands in the air similar to the interaction seen in the movie "Minority Report". The Wiimote can track upto 4 points simultaneously.

Link to my Wiimote Projects page.

Projector-Based Location Discovery and Tracking

2007-12-12T14:39:00.001-08:00

My thesis work at CMU is on Project-Based Location Discovery and Tracking. The basic concept is to discover the locations of light sensors using light patterns from the projector. Once we discover the sensor locations, that information can be used for a wide variety of applications such as automatic keystone correction, multi-projector stitching, tracking input pens, augmented reality, variable resolution displays, and simulating foldable displays.

Here's a link to my thesis work with videos after the jump.

Weclome to my project blog: Procrastineering.com

2007-12-12T14:29:00.000-08:00

Hi, everyone! Welcome to my project blog. This is an experiment to see if it is helpful for people get updates to projects that I'm working on and as a documentation tool for things that I'm doing.

My main website can be found at johnnylee.net