Friday, December 21, 2007

Head Tracking for Desktop VR Displays using the Wiimote

Using the infrared camera in the Wii remote and a head mounted sensor bar (two IR LEDs), you can accurately track the location of your head and render view dependent images on the screen. This effectively transforms your display into a portal to a virtual environment. The display properly reacts to head and body movement as if it were a real window creating a realistic illusion of depth and space.

Link to the Wiimote Projects Page

96 comments:

  1. I second that! Great work, amazing.

    ReplyDelete
  2. Bravo!
    Keep up, your time will come soon I think.

    ReplyDelete
  3. why does it windows come up with

    Microsoft Windows

    Wiimote FishtankVR has stopped working

    Windows is checking for a solution to the problem...

    straight away, really want this working :(

    ReplyDelete
  4. just great...maybe you could modify the code to do this with a ir-webcam and two ir leds.This would be great...

    ReplyDelete
  5. Please tell me you are working with game developers for this!

    ReplyDelete
  6. This is brilliant and has so many applications outside of gaming. You have inspired me to pull out some of my art installation concept from college for which the technology didn't exist at the time. Thank you.

    ReplyDelete
  7. Hmm this isn't working for me. When I try to run the application it says "Wiimote FishtankVR has encountered a problem and needs to close. We are sorry for the inconvenience."

    The WiimoteWhiteboard app was working fine for me, but not this for some reason. I do make sure that the Wiimote is connected before I run the application.

    ReplyDelete
  8. Here's the error I found using the Visual Studio debugger:

    "An unhandled exception of type 'System.IO.FileNotFoundException' occurred in Unknown Module.

    Additional information: Could not load file or assembly 'Microsoft.DirectX, Version=1.0.2902.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35' or one of its dependencies. The system cannot find the file specified."


    You are catching the file not found exception for config.dat, so it seems like it's for some other file, but I can't find any other files being read in your code.

    ReplyDelete
  9. Now here's the real trick: Get a large square room with white walls, several projectors (calibrated and stitched to create a seamless and continuous field of view, even along the floors), Wimote with the VR glasses...

    I think the actual implementation would be somewhat trivial considering what you've accomplished so far.

    ReplyDelete
  10. absolutely awesome...

    this has an unbelievable no. of applications in the real world...........

    ReplyDelete
  11. Really amazing!

    I wonder Nintendo ever though there would be so much extra uses for a wiimote? ;)

    Subscribed to your blog. :)

    ReplyDelete
  12. JL,

    You rule!

    Just the 3 Wii videos should have given you your PhD. They are not only bright ideas, but they are also highly subversive kind of open-hardware ideas. Implement a multi-thousand dollars device with a 50$ found-everywhere gimmick is a industry threatning and market shaking proposal.

    I am just thinking on asking Eng. students in my University to copycat your ideas and try to make other device as graduation projects. We have a lot of projectors, but we can´t afford the interactive whiteboard. Your idea can affect the 3rd world!

    Cheers,

    Jerry

    ReplyDelete
  13. I'm not sure what wrong with the program. I can't replicate the error on my machines. I've add a blog post about this. So, the discussions concerning this error can be centralized.

    ReplyDelete
  14. Brilliant! Now if we can just get this to control "bob and weave" head and body movement in an FPS engine, we're set. Half-life or Bioshock with this ability would be wicked!

    This is your finest wiihack yet.

    ReplyDelete
  15. Why don't you have an array of LED lights behind the WII light up your face and track it that way?

    ReplyDelete
  16. Is the software also running on a mac?

    ReplyDelete
  17. Wow! man, you are truly a genius, congratulations! I took the liberty of translating some of your explanations in Spanish and put them on my blog, Im a Costa Rican software developer and I believe the world must know about these projects, as this is really interesting. Also I did post your vids on my blog and a link to your project website, so everyone can see that If thats ok with you. This is way too cool man.

    If you want to take a look on my blog, go visit, http://elchinchorro.ticoblogger.com

    Needless to say, if you have a problem with me posting your vids on my blog, tell me and Ill remove them...

    ReplyDelete
  18. Wow you are just amazing. Too bad bluetooth drivers are weird on Vista x64 but once I get access to an XP computer I'll have to check this out. Nintendo should hire you and make some amazing VR games.

    ReplyDelete
  19. ehm.. ehm.. sorry im lost for words.. THIS IS THE NEXT GENERATION GAMING!!! would you please please make this code for linux too!! i would appreciate it, also im gonna be learning C++ and maybe i'll be able to make a mod or sumit to make this works as the mouse in a game :D or in xbox 360/ps3 as the right analogue stick to make really virtual reality (not sure with x360 coz its a closed platform)!! but first step is making it work with the PC as mouse!!

    THANK YOU VERY MUCH FOR FINDING OUT THIS !! also your other 2 wiimote project are awesome.. i wonder why nintendo didn't use that ??

    THANK YOU VERY MUCH ONCE AGAIN!!


    greeetzz

    ReplyDelete
  20. I've managed to get this to work on my own laptop, but I'm having all the targets at the same size. The whole 3d effect doesn't work because of this. How can I fix this? Nevertheless, great job making the program.

    ReplyDelete
  21. You're my new hero. Would it be easy to patch your software to lua to control movement in WOW? People already have headsets for communication, so adding LEDs would not be at all noticable.

    ReplyDelete
  22. You've got yourself a fanboy here. This is the greatest thing I've seen since augmented reality.

    ReplyDelete
  23. Chapeau! I bought a wii remote straight after I saw your first wiihack to try it at home. But this application is simply amazing and I can't wait to try it! Keep up the good work ;)

    ReplyDelete
  24. That is amazing. Now the rest of the world needs to catch up.

    ReplyDelete
  25. Brilliant! Have you thought about working with the Freetrack guys ( http://www.free-track.net ) to get the wiimote working with their software? That would be great for PC head tracking.

    Thanks for your work!

    ReplyDelete
  26. Hi
    Beautiful idea !
    I would be happy to help, as a specialist of 3D video. Have a look at http://www.popims.com/fr/hexa.htm
    to see what we could do together.
    greetings !

    ReplyDelete
  27. Why don't you have an array of LED lights behind the WII light up your face and track it that way?

    More specifically, would your eyes reflect enough IR to track? That could be used for a whole lot of things, such as some eyepopping (no pun intended) storefront displays etc etc.

    ReplyDelete
  28. While the illusion is impressive, the practical implementation of it is extremely limited. The body motions demonstrated in the video were unrealistically large and the illusion only exists during these motions.

    I've played with the portal idea in an open-source head tracking program called FreeTrack. I found that if you're going to go to the effort of head tracking, you might as well use it for hands free head-look instead of a simplistic portal illusion which only works with a fixed view.

    ReplyDelete
  29. You are definitely one brilliant man! Did you ever consider using your finger tracking-setup for the head-tracking demo? The human face contains two fixed-distance reflective markers named eyes:) Though your approach with the safety glasses is by far the most ergonomically advanced approach, it would be even more elegant to work without a device at all. Maybe blinking is fast enough not to cause a disruption in detection. Otherwise there would be a software-based solution...
    Again: you are inspiring! Thank you

    ReplyDelete
  30. You are definitely one brilliant man! Did you ever consider using your finger tracking-setup for the head-tracking demo? The human face contains two fixed-distance reflective markers named eyes:) Though your approach with the safety glasses is by far the most ergonomically advanced approach, it would be even more elegant to work without a device at all. Maybe blinking is fast enough not to cause a disruption in detection. Otherwise there would be a software-based solution...
    Again: you are inspiring! Thank you

    ReplyDelete
  31. That's very cool Johnny, well done!

    I guess turning your head gives the wiimote camera the false impression that you're moving closer and further away? - but i guess during game-play one would be expecting the player to looking at the screen and not gazing around the room.

    To avoid the faff of having LEDs on your head, how about an IR floodlight (IR LED array) at the screen and then reflective balls mounted on your glasses??

    ...and thinking about it now, why not have one reflective ball on your head, and measure the intensity of the reflection to calculate distance??

    Just my 2pnth - but I'm sure you've thought of this stuff already.

    Wishing you every success with this and other ideas you have...

    God bless

    Marcel

    ReplyDelete
  32. Just what I was looking for. Greetings from Brazil. U´re a fuck%&¨%$$# genius!

    ReplyDelete
  33. I don't know what is happenning but I can´t sense Z on my set, neither "axis rotation", just x and Y position. :(

    ReplyDelete
  34. Absolutly great! You made my day ... Greetings from germany

    ReplyDelete
  35. To avoid the faff of having LEDs on your head, how about an IR floodlight (IR LED array) at the screen and then reflective balls mounted on your glasses??


    ^^^ This is a good idea I think, means that you only need one setup for both tracking methods (fingers and head). Also probably cheaper as you won't keep needing batteries for the leds on your head, and shiny stickers will probably be lighter that leds or glasses.

    p.s. like everyone says, you're the man!!!

    ReplyDelete
  36. I am just amazed by the way the Wii-mote is being used. Tried the application on my 40-inch TV and it's awesome.

    Like everybody says, you definitely are the Man 8-)

    ReplyDelete
  37. from one JL to another - like a lot of people posting comments here and on youtube, you are seriously my hero. Such and awesome guy, such brilliant work, so down to earth.

    ReplyDelete
  38. Absolutely Brilliant!

    I just checked your Utube video and I'm exited about the possibilities of this. I hope you have contacted Nintendo to help them further develop their next leap inot the future of gaming!

    I will try your VR Fishtank and see what I can do to help fine tune your project.

    YOU ARE THE MAN! Keep up the Great work!

    ReplyDelete
  39. Quick question - where did you source the glasses with the LEDs on them?

    ReplyDelete
  40. Bob: "While the illusion is impressive, the practical implementation of it is extremely limited. The body motions demonstrated in the video were unrealistically large and the illusion only exists during these motions. "

    I thought this would be a problem for a while, but eventually concluded the following...

    If you revert back to the 2d screen/image, a 3d image only looks obviously 2d WHEN you move your head. Your brain is expecting the perspective to change and when it doesnt the illusion crashes.

    In fact this may work in the game developers favour, as with a little research they could identify the maximimum amount of movement a gamer will make during a session and would be able to reduce the amount of programming required respectively.

    I wish I had the programming skills to have a bash at this. The first idea that sprung to my mind was a 1st person tennis game. The court, stadium, net and ball would look amazing, even with very simplistic graphics.

    Besides, In my oppinion I believe the most unrealistic aspect of playing games is the fact that 'we' are in a motionless state. This could all change with this technology. The wii has already taken the first step in this direction.

    ReplyDelete
  41. Imagine this:

    At CES this week, they were demo'ing PDA sized projectors. What if you mounted one of these on your head with the VR tracking. Adding more IR points would allow for more/better head tracking. And if you can track colors, too, you should be able to get full 3D immersion.

    Layne

    ReplyDelete
  42. Exactly right Charzy11, you don't need to move much to get the effect. The illusion fails like you say as soon as you move an inch, so you only need to move an inch for the illusion to work. If you want to move more then you get an even better effect.

    ReplyDelete
  43. Hello,

    Have you considered using the red eye effect to avoid the need for the glasses? If you shine enough infra red into the room from a single point (the TV) then when the person looks a t it the Wiimote will see two white or pitch black dots. You can probably see this if you put your face very close to the finger tracking demo setup. Do you see this as an option to avoid the glasses?

    ReplyDelete
  44. Hi there

    just a short question:
    have you ever hear about trackIR?
    it is from naturalpoint and works a similar way.
    would it be possible to make the wiimote work the same way in games?

    ReplyDelete
  45. Freakin' awesome!! You are a genius! I was instantly amazed, now I am trying to replicate your equipment, but I can't find IR LEDs to replace the ones in my safety googles. The googles I have (look just like the ones in your video) require LEDs that accept 6v and all I can find at radio shaq (sic) and on line are 1.2-1.8v ANY HELP??

    ReplyDelete
  46. I admire your sharing of programming for the advancement of your idea, without any sort of fees. You are great and the science/technology field could use more of your type!!

    ReplyDelete
  47. This is an idea (virtual window into a 3D space based on head position relative to the frame) I've been kicking around for a couple of years. Unfortunately I'm not even close to smart enough to figure out a way to actually implement it.

    Well done sir. I'm in awe.

    ReplyDelete
  48. incredible I have made a post about this on my blog http://trakatra.com/2008/01/27/videojuegos-en-3d-con-wiimote/

    I hope you the best with this and all your projects.

    ReplyDelete
  49. You just made a mention in new scientist. Well done..:D

    ReplyDelete
  50. WHat if you used a projector against a white wall, instead of a screen? So you are not shooting through a window?

    ReplyDelete
  51. This is too cool!

    DUDE!

    You rock seriously awesome.

    ReplyDelete
  52. There are ed hardy shirts
    ,pretty ed hardy shirt for men,

    ed hardy womens in the ed hardy online store

    designed by ed hardy ,
    many cheap ed hardy shirt ,glasses,caps,trouers ed hardy shirts on sale ,

    You can go to edhardyshirts.com to have a look ,you may find one of ed hardy clothing fit for you
    Top qualitymen's jacket,
    These cheap jacket are on sale now,you can find
    north face jackets inmage on our web
    Ralph Lauren Polo Shirtsbuberry polo shirts

    Do you wannaghd hair straighteners for you own , we have many
    cheap ghd hair straightenersin style and great,you can choose one from these
    hair straighteners
    Authentic chaussure puma
    chaussure sport
    And chaussure nike shoes
    Come here to have a look of our Wholesale Jeans
    Many fashionMens Jeans ,eye-catching
    Womens Jeans ,and special out standing
    Blue Jeans ,you can spend less money on our
    Discount Jeans but gain really fine jeans, absolutely a great bargain.
    http://www.weddingdressseason.com
     

    http://CLOTHES-WHOLESALE.US
    de

    ReplyDelete
  53. Truly a nice blog and excellent anylasis.I like it! Wish you a nice day! I like orange nike shox tl1 shoes ,I like to buy the cheap nike shoes by the internet shop . the puma shoes and the adidas shoes (or addidas shoes) are more on internet shop .i can buy the cheap nike shoes and cheap puma shoes online. It’s really convenient.
    Many persons more like Puma sneaker shoes than nike air rift shoes . the Puma Cat shoes is a kind of Cheap Puma Shoes .
    If you want to buy the Cheap Nike Air shoes ,you can buy them online. They are same as the Nike Air shoes authorized shop. Very high-caliber Air shoes and puma cat shoes . the cheap puma shoes as same as other. edhardyis the world famouse clothing brand famouse for the mens edhardy shirt , mens edhardy jacket , mens edhardy shirt , mens edhardy jeans,womens edhardy boots , women long sleeve shirt, ed hardy accessories , edhardy clothing, edhardy sunglasses, ed hardy jeans, ed hardy, ed hardy clothing, ed hardy clothes, ed-hardy.co.uk, ed hardy shirts, ed hardy mensare quite popular among the youth who like to pursue fashion and show their personalities.
    Wholesale and retail are both acceptable to us. Welcome to our site and free to look! Thank you and wish you a nice day. Good Luck!

    ReplyDelete
  54. NewStreetFashion
    Ed Hardy
    stylish design
    Ed Hardy Wholesale
    fashion excellent quality
    wholesale Ed Hardy
    ED Hardy clothing bring you a super surprise!
    ed hardy wholesale clothing
    The quality is so good
    christian audigier

    ReplyDelete
  55. Each hermes bag takes an average of 18 hours for production, with the entire construction tackled by a single craftsman. The Hermes Lindy turns out 5 Birkin each weekly. All of the herme are made from very high quality genuine leather such as Togo, Epsom and exotic leather like ostrich and alligator.
    daidaihua are the "Hottest" slimming product in the weight loss circles. lida slimming popularity seems to growing with each passing day. And this is largely because slimming capsule is a great way to reduce your body weight without fearing any side effects. slimming capsules use the power of all herbal and natural ingredients to speed your metabolism and reduce appetite at the same time. Fucus vesiculosus is a sea plant extract and one of the main reasons lida is being used in weight control is that it contains iodine. Iodine is used by lida daidaihua to produce thyroid hormone which is responsible for speeding your metabolism to ensure faster fat burning.
    Over the last nearly 30 years Tiffany has designed inspiring and evocative silver jewelry for silver earrings Production of Tiffany necklaces , one of the best gauges for popularity of a collection, has lately been increasingly focused on Picasso's inspiring pieces. Collections such as the Loving Heart, Hearts of Picasso and Sugar stacks are currently the top selling designs on leading silver necklaces online shops.
    Everyone wants to copy celebrities who have all been seen with the replica Chanel handbags version. But please do not have a kind of fetishism for Chanel handbag or stars. The first point you should study to be a fashionista is to dress Chanel to impress yourself, yet not to imitate celebrities or chanel watches appeal to a possible partner.

    ReplyDelete
  56. Each hermes bag takes an average of 18 hours for production, with the entire construction tackled by a single craftsman. The Hermes Lindy turns out 5 Birkin each weekly. All of the herme are made from very high quality genuine leather such as Togo, Epsom and exotic leather like ostrich and alligator.
    daidaihua are the "Hottest" slimming product in the weight loss circles. lida slimming popularity seems to growing with each passing day. And this is largely because slimming capsule is a great way to reduce your body weight without fearing any side effects. slimming capsules use the power of all herbal and natural ingredients to speed your metabolism and reduce appetite at the same time. Fucus vesiculosus is a sea plant extract and one of the main reasons lida is being used in weight control is that it contains iodine. Iodine is used by lida daidaihua to produce thyroid hormone which is responsible for speeding your metabolism to ensure faster fat burning.
    Over the last nearly 30 years Tiffany has designed inspiring and evocative silver jewelry for silver earrings Production of Tiffany necklaces , one of the best gauges for popularity of a collection, has lately been increasingly focused on Picasso's inspiring pieces. Collections such as the Loving Heart, Hearts of Picasso and Sugar stacks are currently the top selling designs on leading silver necklaces online shops.
    Everyone wants to copy celebrities who have all been seen with the replica Chanel handbags version. But please do not have a kind of fetishism for Chanel handbag or stars. The first point you should study to be a fashionista is to dress Chanel to impress yourself, yet not to imitate celebrities or chanel watches appeal to a possible partner.

    ReplyDelete
  57. Nice in addition to educational position! It's really what i needed. The other fellows writing was therefore sufficient on the contrary i wish to adjoin some criticism as well. What about a bit dissimilar like health. My site is very differently refreshing! phentermine 37.5.

    ReplyDelete
  58. sinema filmleri izle film izle
    sinema filmleri izle dizi izle
    lig tv maç izlettir lig tv izle
    maç izle lig tv izle lig tv izle

    canlı maç dinle izle canlı maç izle
    Canlı maç izle canlı maç izle

    sikiş izle sikiş izle

    sikiş izle sikiş izle
    sikiş sikiş
    sikiş izleme sikiş
    sex izle sex izle
    sikiş yeni sikiş filmleri
    redtube sikiş redtube sikiş

    sikiş sikiş
    sikiş yeni sikiş filmleri
    sikiş amatör sikiş
    sikiş gizli sikiş
    sikiş kızlık bozma

    sikiş sikiş
    porno porno izle
    porno porno izle
    sikişsikiş
    sikiş izle sikiş izle
    sikiş izle sikiş izle
    sikiş amcık

    ReplyDelete