A lot of very smart people have written in questions and suggestions and I think a lot of them are very clever and worth sharing... and I'd like to add some of my own expansions on what I showed in the video. Like they say, 2 brains are better than 1.... it turns out several thousand brains are even better. This post will be REGULARLY EDITTED to add new ideas as they come in. I know it breaks the RSS model, but I don't want to make a new post for every new idea. Sorry.
-------------------------------------------
Shutter/polarized glasses - if you are going to put something on your head covering your eyes for 3D, it is worth considering making shutter/polarized glasses which allows the computer to deliver a different image to each eye. This results in stereoscopic displays. Combined with head tracking, you get a DARPA grade 3D simulator. When just doing head tracking, the conflicting stereo depth cues weakens the illusion. I would love to do this, but it's quite a bit more hardware, and requires synchronization with the display refresh (or light polarization) which often isn't easy especially given the wide variety of display technologies out there.
Split screen for 2 people - if you split the screen and use two head mounted sensor bars, you can do this for two people... and possibly with one tracking wiimote. One wiimote does 4 dots, so it could (with some smarts) handle two heads. The field of view of a wiimote is only 45 degrees, so it might be cramped with 2 people. But, you could use one wiimote per head and still do split screen.
I can just imagine one of the Nintendo engineers, halfway through a coffee, horribly burning themselves over this.
ReplyDeleteThis is pure genius. And love the glasses...
If anyone at Nintendo reads this, I have to say that I want a version of Metroid Prime with this technology now!!! :-)
ReplyDeleteMan, the 3D effect is amazing, you can really feel like the objects are coming out of the screen. Awesome.
You're single handedly expanding the user experience galaxy by a light year or 1000000000 !!!
ReplyDeleteAll hail the engineers at Nintendo for providing us with the hardware and all bow down ( deeply ) for Johnny's amazing work and sharing it with the world !!!
If only you would adopt me or accept me as your slave, Johnny !
I hope you got a patent...
ReplyDeleteThat's really great.. I'm just putting together a headset of my own following your concept, to try out, and was thinking on trying to program your split-screen idea. Any objections?
ReplyDeleteSteve
http://www.i-vue.net/?gclid=CJ3g9IewvJACFRmDkwods3bdPQ
ReplyDeletePossibly combining the infrared LED's with this headset, or something like it, to avoid the "window frame" sensation you seem to get when using the head tracking with a fixed panel, could also sidestep the refresh rate problem of trying to create shutter style 3d, as the glasses usually use a separate lcd screen for each eye.
This comment has been removed by the author.
ReplyDeleteHe can't patent it since it already exists I think from another company in 2002.
ReplyDeleteHere's an idea for the program: Have it output a resolution that is higher than what your screen is producing, then have the orientation of the screen depend on your head placement.
ReplyDelete(essentially, treat your desktop environment--and more importantly, any game you are playing--like the image in the video.
I immediately got an idea for the single-player problem after watching your video, albeit at a pretty hard framerate hit. You can render every other frame for the second player, and using the 3D lcd-goggle-flicker trick segregate the appropriate images for each player (obviously this would still only be a monocular view of the screen). This gives you full use of screen real estate for each player, no need to do split-screen, though I don't see it working well for more than two players without pushing out more frames per second, I'm not sure the hardware or video standards could support that.
ReplyDeleteActually shutter glasses are quite "cheap": 100$.
ReplyDeleteThen any nVidia card can output stereoscopic content with nVidia consumer stereo drivers.
So this is a really great and cheap VR solution ;)
For more VR news, check out my VR blog!
Hi, thanks a lot for all this info, i hope that you could also add a way to track your posts.
ReplyDeleteGood day!!
Johnny, I just want to say great job! Keep up your great work. You are definitely developing a large fan base!
ReplyDeleteAlso you may want to know these facts:
I found your site from a video link headline on the geek section of fark.com.
I've started a discussion thread of your Wii remote VR tracking project at the folowing locations:
OpenGL.org advanced programming forum:
http://www.opengl.org/discussion_boards/ubbthreads.php?ubb=showflat&Number=231979#Post231979
and
The Future Pinball Simulator forums:
http://www.futurepinball.com/forum/viewtopic.php?p=27421
You can do 3D without "Shutter" glasses (ugh), not only I am sensitive to the low refresh rate, the contrast is not good and the glasses can become un-synced, leading to instant reversal of my eye convergance, OUCH.
ReplyDeleteThis should work, notice that normally the mirror is placed vertically, and that the effect is completely visible from all angles that the mirror reflection is visible. The Polarization works at any angle remotely horizontal (+/- 15-20°), and your setup is not six-axis, so this shouldn't be a problem.
http://forums.nvidia.com/index.php?showtopic=32547&st=0&p=180665&#
I love your setup, but could a simple (and much cheaper) webcam be used? This way more can use it and complex setups need not be over-expensive? Or am I missing something in the hardware that is necessary (besides the obvious need for an IR filter? I would think you could "borrow" the one on a VCR for test purposes.)
@ Sebastion, my second monitor cost less than $100 on ebay, and I get a second monitor when I am not using the 3D. Since the setup is passive and the paper glasses are $0.50, my setup scales nearly infinitely for very little money. Not to mention it is entirely flicker free and has much higher contrast then "shutters".
Dual polarized projection works as well, but requires a silver screen to preserve the polarization.
I'm not going to try act like an expert on this type of stuff, but the first thing I thought of after I watched the video was how cool it would to be to create a 3d environment like that. Like for example shopping out parts of different pictures and creating a scene. Since it's relatively cheap to test it out, you'd be able to create some pretty cool stuff. Hell, you'd probably even be able to do something like notpron with that kind of stuff too.
ReplyDeleteHello, I am amazed at your nifty ways of utilizing the wiimote to crazy potential. This last article about desktop VR is particularly interesting. Consider the possibilities from a GUI standpoint. Bump Top has explored one way of using a 3d interface with a computer:
ReplyDeletehttp://bumptop.com/
But with a desktop VR system, you could engage the user much further. Also, is this interface limited to a wiimote tracking system? I do not know if any such software exists, but consider the eyetoy from sony. Many games have been designed around tracking of visible light, namely recognizing motion and moving bodyparts. What about laptops with built in cameras? the macbook, for example, would be a good place to start. If there were a way to manipulate a VR desktop on a macbook using the isight camera, you could really get things going on the PC/GUI usage of desktop VR. (assuming the iSight is IR sensitive) A clip on attachment of IR LEDS could be used like in the first video of the "minority report" style of touchless interface, except reflectors could be worn on a pair of glasses or as stickers on one's eyebrows or something of the like. This would give you the same interface as using the safety goggles for the wiimote tracking.
However, why use LEDs? You could implement a facial recognition software to track the user's eyes/face to achieve the same effect as using IR LEDS with the wiimote. Having a built in camera, a macbook or other notebook could use this software to create the same sort of desktop VR interface but without any accessories or headgear. This could not only be used for GUI purposes, but would make for a really neat interactive slideshow, or could even be used in PC based games down the road. That, and with the use of face tracking via webcam, any PC could use the software. Webcams are becoming a standard device nowadays.
Not to mention the incredibly large percentage of cellphones that have video cameras on them, perhaps there would be a way to use its camera and bluetooth link (or wired) to a computer without a camera. Or have the software running standalone on a cell phone to use as an interface altogether (for cellphones with cameras on the same side of the screen).
Thoughts?
To all the people saying "can i do this with a cheap webcam" isnt the whole point of this project that fact that johnny is using an off the shelf product (US$39.99 at toys r us) which has HARDWARE tracking of the IR points therefore taking out one of the biggest obstacles?
ReplyDeleteI cannot run the program. :(
ReplyDeleteI installed Visual Studio 2008 Express Edition to try to debug it, and this is the exception I get when trying to run it:
System.BadImageFormatException was unhandled
Message=" is not a valid Win32 application. (Exception from HRESULT: 0x800700C1)"
Source="mscorlib"
StackTrace:
at System.Reflection.Assembly._nGetEntryPoint()
at System.Reflection.Assembly.get_EntryPoint()
at Microsoft.VisualStudio.HostingProcess.HostProc.GetEntryPointAptThreadState(String assemblyFile)
at Microsoft.VisualStudio.HostingProcess.HostProc.RunParkingWindowThread()
at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
at System.Threading.ThreadHelper.ThreadStart()
InnerException:
Just a quick question:
ReplyDeleteDo you track the camera (basically the "head" of the person wearing the sensor bar) by averaging the positions of the two IR points which are given to you by the Wiimote?
can the wiimote track different wavelengths? (if they are far enough apart on the spectrum)
ReplyDeleteFor Instance: a "green," "red," and "blue" source.
This would allow for multiple players and inputs. (right and left click, scroll)
If not, then it should be feasible to track different intensities, correct?
These projects have gone from "really freaking cool" to "totally awesome" to "if Nintendo hasn't contacted you for a job offer they are morons".
ReplyDeleteOh, and if that's not enough, check out 1:16 in the background. There's a freaking electric cello!
This project has driven me to work on my own similar project (getting something like this into a game) so I bought a Bluetooth receiver today and got this all up and running.
ReplyDeleteA thought/suggestion:
Allow multiple cameras. My off-the shelf bluetooth receiver supports 7 devices and I already have 2 remotes so why not support as many remotes as the user can hook up?
I am thinking that with two remotes and two dots you could have the software calculate the angle and position of both remotes so the user doesn't have to go through any special setup. All they need to do is make sure there is some overlap between the remotes.
With 4 remotes you could almost double the horzintal and vertical range (or almost quadruple either individually).
Out of curiosity, are there any other IR cameras on the market that perhaps offer a larger angle?
I can't get it to work like it's supposed to be working..
ReplyDeletethe desktop vr works ok
wiimote ok motion sensing ok
but in desktopvr the screen
depth is all wrong, it wont
produce the desired effect of
"out of screen"
I dont know what's wrong help anyone?
-Pete
Did you press the space bar once you had the sesor bar on your head? I didn't do that at first and it didn't perform as expected.
ReplyDeleteAlso, standing further back (TV instead of monitor) helps since the range of view on a Wiimote is fairly limited. With a computer monitor you can't move very far before you are out of range.
I can imagine a way to get rid of the 45 degree problem. I'm not sure of your electronic hacking skills but this is simple, you could probably hook up the wiimote to a servo motor that follows [on an axis]the infrared signals...BRILLIANT!
ReplyDeleteBut I guess that would only work if the image is projected in the "3d glasses" otherwise you would have to be rotating that "30 billion inch" TV too :P
Until the Mac version is out ... anyone interested in shooting some prototype video of the wiiboard and/or head tracking using Edusim ... Download here - http://edusim3d.com -- I would be interested in seeing the results for a k-12 interactive touch/virtual environments projects we are working on (no luck with bluetooth/parallels to date).
ReplyDeleteBest Regards,
Rich
====
I was thinking about shutter glasses combined with your demo the other day.
ReplyDeleteA problem with most 3d systems that use glasses is that your eyes need to sit horizontally relative to the screen.
But since here the Wiimote can read the angle of your head at all times, it could also correct how the left and rigth images are shown on screen, adding even more to the experience. I'd love to see that.
I can't imagine a practical way to do this with polarized glasses, though.
Bonjour,
ReplyDeleteI'm following you work.. very good job !
Bye.
Valere.
Hi,
ReplyDeleteGreat stuff! Nice programs to dive in in the pre-holiday vacation days!
The problem FishTankVR is occuring when no or the wrong DirectX is installed. Installing the DirectX SDK was the solution for me, but I guess the generic DirectX (non-SDK) install should resolve the problem too.
Cheers,
Gab
Btw: Just like pete everything works, but I can't get the depth in either. Also tried space, which did align the camera but still no depth.
ReplyDeleteAlso the approximate distance is changing. Let's have a look in the code, although I expect it do be something else. Any help is more than welcome.
Cheers,
Gab
still having the same problem,
ReplyDeleteI did press the space bar to angling the camera etc..
I have my computer connected
to my 32" LG LDC tv..
and as seen in the video the room does not have to be very big as you are moving closer to the screen the out of screen effect remains.. I fon't know whats wrong with my setup...
-Pete
This comment has been removed by the author.
ReplyDeleteWhen I ran the .exe file in the DEBUG folder of the project it actually worked.
ReplyDeleteThen I delete the .dat file in de DEBUG folder, ran the DEBUG version again and it still worked. But not full screen.
Then I copied the .dat file from the DEBUG to the RELEASE folder and then it worked as expected. Altough I didn't seem to get as much effect as in the YouTube movie... But we'll keep playing
(it doesn't sound like a good solution, but seems to do the job)
(BTW: Sorry for flooding the blog with comments, I don't know how to edit previous ones..., or should I just delete those?)
Is it possible to add this wiimote-tracking functionality into nVidia/ATI display drivers? If so, I can imagine being able to do this kind of stuff with any video game.
ReplyDeleteThe head-tracking video is amazing. I think you may have stumbled upon what the next Wii (and indeed, X-Box and Playstation) will have.
ReplyDeleteI'm no computer scientist, so I could be completely wrong, but could this effect but coupled with 3D glasses (ie: a red and a green lens) to make things jump out of the screen even more? With headtracking, the 3D wouldn't be static either and would look "correct", I imagine. As you've already got the glasses for headtracking, presumably adding coloured lenses would be easy?
Amazing work! I really admire your willingness to release the software to the public as well.
ReplyDeletethis would be amazing with pc games like halo or call of duty to control the view point. it would feel like your actually looking around instead of moving around a mouse or the analog stick on a game controller.
ReplyDeleteThis comment has been removed by the author.
ReplyDeleteIf only this was in Strangelhold, the bullet dogding would be way more fun with this. I may buy a Wiimote to try this out, gonna need extra Wiimotes when I get a Wii anyway.
ReplyDeleteHere from the Slashdot article - http://games.slashdot.org/article.pl?sid=07/12/23/1359259
ReplyDeleteI just wanted to say, that if I was a developer with lots of money (sadly I'm just a poor developer heh), I would want to give it all to you!
The head tracking system looks like ten kinds of awesome!
Maybe the next HL2 mod I make could have something like that built in...
You are very good!!, from Spain... very very goob
ReplyDeleteIt would be very interesting if you could explain the projection maths a bit. For instance, what the angles are and how you calculate them?
ReplyDeleteThis doesn't track head angle does it, just body position? Could you do that with a third infra red light?
ReplyDeleteI was just thinking about how all this worked, and i'm confident what i'm thinking won't work with wii hardware, but i realized how to make fully functional 2 person VR with this technique. it would require two pairs of shutter glasses, and a system capable of pushing over 144fps, simply put, it broadcasts data set A for user 1 and datas et B for user 2, and closes A's shutters when B is on screen, and B's shutters when A's is on screen. at 144+ fps, each user gets 72fps, which is more than enough to make the dual broadcast un-noticeable. Only hard part is the hardware to do it.
ReplyDeleteHey Johnny,
ReplyDeletethis is a very nice piece of work.
One thing you might think about is whether the perspective movement gained by the glasses could be 'overloaded' over the nunchuck interaction/input so that a user moves around to provide that input rather than juggling with 2 hands?
Simon
Any chance of the demo program running under Vista?
ReplyDeleteInstructions to recompile NE1?
My wiimote is correctly connected (working fine with GlovePIE), but the program crashes on startup.
Hi Johnny, this is Metaldave from: http://www.revoeyes.blogspot.com/ and you're work is an inspiration to us all thank you so much for what you've done and continue to do.
ReplyDeleteI was having a conversation with one of my associates about the Wii VR topic and I have a question for you. My associate seems to think that a game with more intense graphics like Resident Evil 4 would not be able to run smoothly on the Wii while having the Wii Remote and Sensor track every movement in 3D space because of the advanced physics that would need to be put to use in order to calculate every movement.
My argument against him was that maybe you don't need all those advanced calculations and a normal game with great graphics could work fine because it's only an optical trick caused by the Remote and sensor and that raw processing power may not be needed to do it.
So, would graphics be an issue when developing a game that uses this VR method or can any game be used to the Wii's full graphic power without a hit to performance???
I tried to email you but I got a failure delivery notice. johnny@cs.cmu.edu does not seem to work for me.
Thank you for your time and please keep up the outstanding work!
-Metaldave
You should really start your own company. Amazing ideas!!! Hope you patent them.
ReplyDeleteI would think, Metaldave, that the overhead would me minimal, practically nonexistent. Once it's calibrated (which would probably be the most complex process), it simply translates movement seen by one wiimote to movement of the screen. It's the exact same process as using the wiimote pointer to aim, except instead of moving an aiming reticule, you're moving the game's camera.
ReplyDeleteTo simplify: movement on an IR camera translates to movement of an in-game element.
Can someone reply to my post, letting me know what type of dongle is compatible with all of this and also blue soleil? I'm pretty much a newb with this stuff but I want to get it working on my home machine and I've heard model numbers can matter when it comes to dongles working with the wii and blue soleil.
ReplyDeleteThanks
I've always wondered if the Wii and the DS are hardware manufacturing stepping stones to full VR under Nintendo's longer term R&D. After all, aside from the head tracking potential for the Wii, the DS has set up an infrastructure for manufacturing cheap wi-fi-enabled hardware with two small, light LCD screens...sounds perfect pieces for a Head-Mounted Display to me... :D
ReplyDeleteMan, your solutions are simply amazing. Elegant, cheap and... simple.
ReplyDeleteThat's art man.
I've just just discovered processing and your incredible virtual technique within a week of each other. Processing looks like it could be a quick and powerful way of generating three dimensional environments, and coupled with this Wiimote genius it could be a way of creating immersive environments relatively quickly... my only problem is that I'm new to them both and only have 'hobby time' to look at it, but I'm already obsessed and trying to get my Mac and Wiimote to talk to each other! Johnny, you're my new hero :)
ReplyDeletejohnny, that is absolutely genius! Fantastic! Can't WAIT to try this out over the holidays!
ReplyDeleteRe the 2 player angle, sure, you COULD try and hook up two via split screen or with flickering glasses, but wouldn't it make more sense just to use two displays?
This is using bluetooth via PC anyway, so the first implementation I can think of would be in PC games, where multiplayer almost universally involves each player having their own machine and screen.
Even from a console point of view, it wouldn't be that hard to set up a splitter box to send the signal to 2 different displays and have each render a different view. Resolution might not be great, but since I the VR effect is evident to all of us even on a low res youtube video, i doubt it would be a problem!
Once again, amazing work, I can't wait to see what you come up with next.
This comment has been removed by the author.
ReplyDeleteJohnny Chung Lee, your a genius. I just saw your head tracking video with the wiimote, brilliant truly brilliant.
ReplyDeleteAs people have said, get a patent, and apply for a job with Nintendo.
While your there tell them to make cheap VR goggles for the Wii successor. DS+Wiimote=cheap VR goggles.
Your a hero!
You are my new hero!
ReplyDeleteI really think you've found something here...
Excellent stuff Mr Lee - some fantastic ideas and some great home-brew stuff going on here. I'm possibly inspired to go out and do some of it myself...
ReplyDeleteThank you for this!
ReplyDeleteGreat job.
This is like wow. Like the post above said. Is there a possibility to get this to work with existing graphics drivers so that we can use it with games now...or would nvidia and ati have to put it in?
ReplyDeleteI must concur, your work here has pushed the GUI to a new level, you have unlocked THE key my freind, your name will go down in the history books.
ReplyDeleteI always get an error saying the Wiimote is not connected. As far as I can tell, it is connected. The WiimoteTest.exe app works fine as well as any of the GlovePie scripts. Any ideas?
ReplyDeleteI like it, but I do wonder if the tradeoff achieved by reversing the wiimote/ir setup is worthwhile.
ReplyDeleteKeeping things as they were, but strapping a wiimote to your head, you would look a bit of an arse, but the benefits over your configuration are quite real:
For a start, you wouldn't have to remain in 45 degrees of your TVs direction, you could go pretty much anywhere in front of it, as long as the TV was in 45 degrees of your head's direction.
The program could also track the heads of an arbitrary number of players (well, as many as bluetooth lets you) with just the one bar up front and a wiimote on each user's head. (In your case, the wiimote up front would max out detecting the ir dots on some fixed number of people.) Of course, this isn't particularly relevant to the head-tracking example.
If two players were to set up with their own displays and wiimote/ir LED systems, the risk of a wiimote "locking on" to the wrong sensor bar would probably be greater with your setup.
You'd also get small gains in having a speaker and some buttons on the side of your head, heh.
Still, doesn't have the appeal of the safety glasses.
"You'd also get small gains in having a speaker and some buttons on the side of your head, heh."
ReplyDeleteHaha, you won me over with that point. ;) The big problem with the wiimote attached to the head is that user adoption rate will be far lower then if you offer them a stylish pair of goggles or clip-on IR LEDs. While attaching the wiimote to your head does offer some distinct advantages, your software would likely not hit of nearly as well.
Realistically, I think you would be better off figuring out some other way to widen the field of view of the camera in order to keep the users attachments to a minimum. One way is to use multiple cameras, though the math involved in figuring out where a person is and where the cameras are (relative to each other and the display) would be difficult. Another option is to go with a more expensive camera option such as the Track IR system, but again you would suffer low adoption rates compared to a wiimote, something that 13 million people already have at least one of in their home.
That all being said, you could design any software to support either mode, attached camera or stationary camera. That way the more dedicated players can afix the camera to the head while the casual players can just wear the stylish safety glasses.
This comment has been removed by the author.
ReplyDeleteStrapping a Wiimote to your head isn't going to work. When you change the angle of your IR glasses (by looking left and right, but not moving) the dots stay in place.
ReplyDeleteWith a Wiimote attached to your head, when you look left (but not move), the angle will completely change and the Wiimote will think you moved. Sorry, kinda hard to explain.
The tracking resolution of the wiimote is really high i've noticed when i tried the whiteboard with a IR-pen. When you use the Wiimote for VR, this high resolution isn't really neccessary, it doesn't matter if it registers you 1 cm out of place. So would it be possible to put some kind of cheap lense in front of the Wiimote to make it`s angle bigger? I think this would work and is much cheaper then building a Wiimote-matrix
If you went with the wiimote on head idea you wouldn't be tracking the absolute position of the two sensor bar dots but rather their position relative to each other.
ReplyDeleteIf you rotate your head, the dots won't move much relative to each other because your angle of view is close to the same. However, when you move from in front of the monitor to the side, the dots will get closer together and their "size" (distance) will start to grow apart from one another.
Tracking their relative positioning would allow you to determine where the user is, relative to the screen which is all we care about.
A curved lense of some kind may be a cheaper solution, but for an end-user it's a very large undertaking to modify a wiimote and something that will likely cause it to stop working with their Wii. A wiimote array on the other hand is something that is likely free for many users (people who already have 2 Wiimotes for their Wii) and doesn't require that they risk damaging or have to deal with modifying any hardware.
A lense might not have to be put into the Wiimote, it might just be possible to put it in front of it, so no hardware modification is required.
ReplyDeleteOf course to know this for sure, someone has to try it. Unfortunately I haven't got one here.
Excellent! Keep up the good work!
ReplyDeleteOK, I do not have a good knowledge about programing and all(wish I had ^^), but I really would like to know if it is possible to, in stead o a software, make it kind of a driver(and this way, being able to use your amazing piece of work with every software, such as games and 3D designing softwares)
ReplyDeletewould it work?
I know there is a company called TrackIR(or something like this) thar sort of use this technology, but the really suck, because you dont get this imersion, u just turn your head to make the camera turn
Compiz already have a plugin to allow stereoscopic rendreing
ReplyDeletehttp://forum.compiz-fusion.org/showthread.php?t=4194
Nice! Would you mind sharing where you got the glasses and how you replace the IR leds... and what kind of IR leds you got?
ReplyDeleteTHANKS!
You're so going to get hired by Nintendo eventually...
ReplyDeleteMan, what a great effect. I do a lot of 3D modeling on 3Ds Max. Do you think there would be a way to utilize this technology in that program? Having a true 3D view over the model you are working with would be infinitely useful.
ReplyDeleteIs there any way to change the resolution without recompiling? My TV automatically pillarboxes if it is set to 1024x768, so I need to set it to 1920x1080. I don't have Visual Studio installed on my laptop, so I can't recompile it.
ReplyDeleteIs there a way to change this in the config? If not, maybe a feature for future versions?
Is there any way to change the resolution without recompiling? My TV automatically pillarboxes if it is set to 1024x768, so I need to set it to 1920x1080. I don't have Visual Studio installed on my laptop, so I can't recompile it.
ReplyDeleteIs there a way to change this in the config? If not, maybe a feature for future versions?
Isn't there any possiblility to let the Wii remote be recognized as a mouse (pointer)?
ReplyDeleteIf you would combine that with your glasses and Headtracking interface, you don't even need new games, since all 3D shooter games already display in 3D.
It is then not possible to aim and shoot with your wii remote, but it would be possible by simply looking at it (mouse = crosshair) and pressing space (or any other key) for shooting.
|f I look at he video, this should deffinately be possible.
Dear sir,
ReplyDeleteIs it possible to do something like this: http://www.naturalpoint.com/trackir/
I think I could then save a few euros using my wii stuff to play Flight Simulator 2004!
Best regards,
Eunito.
You could have this work with all Direct3D (or OpenGL) games but it would require hacking your drivers I think. Basically, you would need to adjust the camera position one last time before the data is sent off to the 3D card to be rendered. Your driver would have to have some understanding of relativity though, which would likely require some configuration per game.
ReplyDeleteFor something like this to really work well I would think that the display driver companies are in the best position to program it, such as nVidia or ATI/AMD as it's a very low level operation.
I might have missed something in the discussion above (didn't manage to read through all of it), but I think it was not noted here.
ReplyDeleteYou really don't need a Wii for most of the Johnny's supertricks. Any webcam (my laptop comes with one built in right over the monitor) with a piece of developed photofilm (you know, the developed film has these black parts in the beginning before the actual frames begin) covering it does well as a simple IR camera.
Taking pictures with this camera is a matter of "cam.getImage()" (in Python at least), and looking for two bright spots on them is a matter of a rather simple nested loop (especially if you know the spots are always separate).
It makes for an even cheaper 10-dollar/50-lines-of-code head tracking system.
And thanks for some great ideas and marvelous procrastination moments, Johnny!
In the video you can see that the targets in front seem to really pop off your screen. However I can't get anywhere near the same effect here.
ReplyDeleteSure it looks 3D, as in the targets in front are rendered as bigger, and the targets in back are darker and harder to see. And they do move with you very naturally. They just don't seem to pop off the screen, like they do in the video.
I tried it on a 22" widescreen monitor and an 80" projector. I had to adjust the screenHieght and sensorBarWidth but left everything else the same.
I wondered if it may just be that the effect looks better on film than in person. Closing one eye does seem to exaggerate the effect slightly.
For those of you who have actually tried it out for themselves, or better yet Johnny Chung Lee himself, can anyone confirm that the effect looks as good in real life as it does on the video?
Does it really look like you you can reach out and touch that closest target? Or does the video make it look a little better than it really feels when you're there?
You guys are easily amused.
ReplyDeleteActually I am too the neat part isn't the head tracking or the tracking of irled's it's the fact that nintendo has all this wrapped up in a small cheap package with bluetooth and tracking 4 blobs thats the neat part.
The stuff that Johnny's doing while really neat is nothing new except hes using the wii remote for it..
NOW if you want to see something really cool hes working on I suggest you look at the project using projectors with combo ir and visible light and dlp to create projected images over 3d surfaces in real time and even make foldable
displays... NOW thats some cool stuff
I don't want to burst the bubble, this is great. But it has been done before. MY friend has this thing that is used for flying games, so that you can move your head slightly to looks arround the cockput, so you can see whats to your side by just moving your head slightly. But the way this guy has used it for the 3d effect is awesome!
ReplyDeleteI too am unable to produce the sensation of depth that I noticed in the video.
ReplyDeleteIt is 3d and tracked appropriately I just dont get the feeling that something is coming out of the screen.
Do you, or anyone else here, have any suggestions?
Nice work. Did you consider combining this with the original project where you were reflecting the IR light? Assuming you weren't wearing anything reflective, I think your eyes would be the brightest points - which would allow tracking the exact location of your eyes without wearing anything on your head at all. Might not work, but worth a thought.
ReplyDeleteAnother idea for getting true stereo vision I just thought of, is having a special projection screen which reflects only in a certain direction, and have two projectors - so each eye sees something different... But this probably wouldn't work if a main part of the game was moving your head constantly.
On second thoughts, maybe your eyes would be the darkest points, I don't know. But reflective patches on the glasses would be lighter to wear than IR leds, probably.
ReplyDeleteSo other than the screen with the targets, is there any other use for this yet? is it being developed?
ReplyDeleteThis is completely amazing.
ReplyDeletecan someone with more programming knowledge than myself maybe compile Johnny's code to work on my mac? I got DarwiinRemote to connect my Wiimote to my mac, but no luck getting the program to run using crossover. I'd really like to try it without getting a Bluetooth card for a PC or installing parallels.
Also the macbook camera is IR sensitive.
I'm very intrigued with this project... could someone please tell me what file format the "3D room with the targets" is in?
ReplyDeleteI have been brainstorming about the possibilities of Johnny's program and I'm wondering if it would be possible to make your own "rooms" (or even games) in a 3D software animation program such as Blender or Maya (and how did Johnny make his?). Could someone further shed some light on this?
Thanks :)
@luke o
ReplyDeleteThe targets are a single png image. He placed them in a "box" with x, y, z coordinates (actually 3 of them are drawn outside the box) and drew the lines of the walls and the lines tracing the targets back into the distance by just giving it more coordinates. The x and y coordinates are generated randomly and the z distances are calculated to place the targets an even distance apart. Then the whole thing is rendered using DirectX.
So there is no real file format, you are however correct that you could make 3d rooms fairly easily. You just have to move the camera position to match where the Wiimote tracks the person at, and re-render the room. This is essentially what he is doing with DirectX.
@johnny chung lee
When you watch the video you made, it looks pretty amazing. The targets really come out of the screen. Do you feel like it looks the same when you wear the glasses and you're really there?
This stuff is amazing! I would love to work with you.
ReplyDeleteDUDE you are Awsome
ReplyDeleteHow about combining this with the finger-tracking to create a 3D remote control? Imagine touching those bulls-eye targets to switch components, having sliders to adjust volume, buttons to switch channels? All floating in front of a TV image...
ReplyDeleteJohnny,
ReplyDeleteAmazing! I ordered 2 Wii Remotes from Amazon after watching your videos.
One thing I also liked about the videos is their clarity and cogency.
I was wondering if you could post something about the process (if their is one at all) to which you go about creating these rather compelling videos.
Thanks.
Per Igor's comment - could you have 2 wiimotes watching your location and in some way filter (polarise?) the light so that you could distinguish the head lights from fingers and hence provide different effects?
ReplyDeleteS
Is there any way to do this but using the IR cameras from 2 wii remotes together? This could allow you to have more area to move around.
ReplyDelete"Man, what a great effect. I do a lot of 3D modeling on 3Ds Max. Do you think there would be a way to utilize this technology in that program? Having a true 3D view over the model you are working with would be infinitely useful."
ReplyDeleteA full 3D view would be hard. But you can easily get the sensation of seeing it at it's side (think of replacing the targets in johnny's program with your 3D model). I am currently trying to implement what Johnny has done in OpenGL (though I'm really lazy) and I would test it out with 3D models (.obj or .3ds files).
Very very interesting demo here. I'm curious what is involved with the process that makes this work? I only skimmed the code, I didn't have enough time to sit down and determine what is really happening..
ReplyDeleteEssentially, this looks like it is changing the camera position based on the movement of your head. If this would be very easy to implement in any game, right now.
Movement might be a tad bit problematic... Moving forward without corresponding head movements might diminish the effect.. And the camera would have to only be loosely attached to movement and position.
Still, I'm wondering if this could be applied to something like HL2, where you could simply add support for the head tracking and use it to control the model's position.
I'll have to look through the code and determine if I can see what is going on. I'm sure most of that code is simply making the 3d world. We just need to separate the tracking code from everything else, and apply the same movements to the camera.
I know Java, and C# is a lot like Java... but HL2 is in C++... which I also know... but the language differences are likely to make this a bit harder. If it was in C++ natively, I could probably cut and paste some of it.
Hopefully its not more complicated than just changing the camera...
If you use the Wii remote as the camera in the first person view, just as you would with regular wii games, you'd essentially have an unlimited scope/area to move around.. until you get too close to the TV or way too far from the TV for it to useful.. why would you get up that close anyways.. Only problem would be that you'd look like a friggen dork with a wiimote mounted on your head...
ReplyDeleteThe reason you would have unlimited scope is that since you're always looking at the TV.. the IR sensors from the Wii Remote would always point towards the TV (IR emitters on top / bottom of the TV).. so you'd have a wider range than 45 degrees you have with the current setup...
By all means.. I wouldn't ever be able to come up with this interface.. amazing work Johnny!!
What Nintendo can do to use this is to have another accelerometer and IR sensor built into a headband so you can use your head to play certain games and not look like a dork.. or just an IR sensor built into a headband which attaches to the Wii remote.. Then from there.. use the headband to control the first person camera and first person games.. Possibilities are unlimited.. =P
The Wii was just hacked for homebrew:
ReplyDeletehttp://nintendo-scene.com/812
Any chance you can port your code to linux libraries so that when an Ubuntu bootable dvd comes out we can hook your code into Metroid Prime for real, not to mention into Compiz-Fusion for a true head-tracking 3D desktop on any pc or wii? It would make the low-resolution on the wii matter far less if you had a head-tracking desktop, as moving your head around multiplies the effective resolution. I bet it'd rock for an Android-based iphone clone too...
This comment has been removed by the author.
ReplyDeleteIts working, but the effect isnt as impressive as it looked in the video ^^
ReplyDeleteCould be that i need to try it with a larger screen, but awesome work nonetheless.
A full 3D view would be hard. But you can easily get the sensation of seeing it at it's side (think of replacing the targets in johnny's program with your 3D model). I am currently trying to implement what Johnny has done in OpenGL (though I'm really lazy) and I would test it out with 3D models (.obj or .3ds files).
ReplyDeleteThis is the effect I am going for. I realized that a full 3D view would be very hard. Just getting a better perspective on the models would be helpful enough. Let me know if you get your openGL version working. 3Ds max uses openGL, so the transition should be doable. Also, if you need some 3D models to test your code with, let me know and I can supply you with some.
Did nintendo shut you down already man?
ReplyDeleteYeah Johnny, your site has been down all day. Any info on that? I want to download the software but can't...
ReplyDeleteCould someone create a quick easy shooting game based on johnny's demo?
ReplyDeleteI was thinking something simple, you could leave the targets the way they are but have them move around and respawn once they are shot. Maybe some enemy fire, like shots coming at you and you could avoid them relative to your (head)position.
Nothing too complex, you wouldnt even have to add any art! Something so simple can be totally fun for at least a half hour.
I was thinking something simular to the hidden area in Rez, but with perspective correction and wiimote aiming. :)
This can be done with 2 wiimotes, custom ir shades and the sensor bar on.
Wiimote 1 would track the ir shades, while the sensor bar would track wiimote 2's on-screen aiming. Viola! Funtimes!
nvm my comment, seems to be working now...
ReplyDeleteI'm gonna have to go with the others who've posted that they can't access your site -- I've been trying for a day and haven't been able to ...
ReplyDeleteCan anyone here who has been able to dl the software mirror it somewhere?
Just so you guys know -- it seems to be the entire Carnegie Mellon CS subdomain that is down ... not just Johnny's page.
I had some problems with the site also but after some retries I downloaded it successfully.
ReplyDeleteAs for anyone who is experiencing bugs, I was also (it didn't start), but after installing the DirectX Web Installer and updating my DirectX libs it works fine!
Great piece of work!
ReplyDeleteAs to the multi-user input problem: Fix every pair of glasses with a wireless transmission system (doesn't need a fancy 100m range). You let the user 'connect' with a button that sends the signal
>goggle id<0001
The computer will hold a list with goggles, and determine a time schedule which is broadcast to the goggles using their unique id:
>goggle 1<
>goggle 2<
The glasses will contain an IC, which puts on the LED's when they receive their signal, and turn it off right after.
Once the PC has identified and processed the lights, the screen draws the view of the goggles (flickering). So it will look like this:
'*' = pc
'**' = goggle1
'***' = goggle2
*program on
*waiting signal
**send goggle id '01'
**keep listening
*receive goggle id
*send signal '01'
**detect '01'
**IR's on
**keep listening
*track leds
*send signal 01
**detect '01'
**IR's off
*draw camera player1
**keep listening
***send signal '10'
***keep listening
*receive signal '10'
*send signal '01'
**detect '01'
**IR's on
**keep listening
*track leds
*send signal 01
**detect '01'
**IR's off
*draw camera player 1
*send signal '10'
***detect '10'
***IR's on
***keep listening
*track leds
*send signal '10'
***detect '10'
***IR's off
*draw camera player 2
etc, etc, etc. Sorry if it's hard to follow (it is), but this might actually work. In any older game, it's easily possible to get 100fps+, so 2 players is easily
possible.
Keep up the good work! Although I think you poked nintendo in the eye by presenting this idea (which is obviously something they where saving for later).
I'm having a problem. im running boot camp on a mac with windows XP. so, I was wondering what drivers you need in order to run WiiDesktopVR??? Could anyone help? Thanks!
ReplyDeleteMark, I got it working under XP on a macbook pro, I used the latest version of bluesoleil (5.something), its only a trial, limited to 5mb/s transfer but thats fine for the wiimote.
ReplyDeleteI also installed the latest leopard drivers (ie bootcamp 2.0 drivers) but honestly I don't think that changed anything, I think it was bluesoleil that made it work.
hi, im having another problem running the WiidesktopVR. it says it cant find a wiimote. since im running boot camp with a mac and it has a bluetooth device in it. so i tried downloading bluesoleil to try to fix the problem. but for some reason bluesoleil wont let me even try to sinc any bluetooth devices. so, does anyone know of any other bluetooth programs I could use to sync my wiimote with???
ReplyDeletethank you so so so much for your incredible work, in a few years we'll might say that johnny lee "revolutionized" video games, a bit like going from 2D to 3D, now we're going from "still" 3D to "immersive" 3D !!! Someone has to get shigeru miyamoto to see your work, if he hasn't seen it yet, 1 million people has been watching your latest video !!! I won't be surprised if Nintendo or even Microsoft or Sony approach you offering you a bit of cash to buy you brain (at least what is inside !!!) Keep working (and experimenting...) 1$ per person looking at your video and your a rich man !!! god bless
ReplyDeleteThanks for your amazing work!
ReplyDeleteI'm not sure if you're site is down or if your bandwidth has broken your site, but I cannot access you source code in the zip. I can get the URL from the Google Cache though.
Anybody able to point me to a mirror or cache of the source?
Cheers!
This comment has been removed by the author.
ReplyDeleteThis link is too long, but glue it together...
ReplyDeletehttp://www.inition.com/inition/product.php?
URL_=product_stereovis_philips&
SubCatID_=1&
gclid=COWa_Jz92ZACFQjalAodPjCNPA
They are very cool full 3D flat panel displays. Not that cheap, but not that expensive for what they are either. I'm sure there's a US supplier too (These guys are UK based).
Also not sure how good they would look with the user moving all over the place as the tech they use requires the viewer to be in one of several 'sweet spots'...
Using 2 Wiimotes and LEDs at the front and the back could this detect when you're turning your head too? I assume by measuring the distances between the LEDs and their relative positions some software would know which way you're turning? Couple this with a cheap Head Mounted Display ($200?) and VR would be close at hand.
ReplyDelete^^^ Probably wouldn't work as leds moving closer together means you're moving further away.
ReplyDeleteWouldn't it be easier to combine the leds (or reflective balls maybe easier??) with a wiimote strapped to your head? ie the original setup would track where you are and the wiimote on your head would track which way you are looking. I know a wiimote strapped to your head isn't ideal but compared to most vr headsets I would say that it was quite light.
The problem with a vr headset is that if you're in a game situation or similar then you're limited to the fact you can only walk about 10 feet forward because you'd walk into the original wiimote.
By the way I've had a head tracking headset for ages that uses gyroscopes to tell which way you are looking. You could just use a wiimote instead of the gyroscopes. It works quite well for turning your head in a game and it's very realistic but of course it has no forward/backward movement sensors.
Even using the wiimote I think you're limited to a too small space to realisticly be able to walk round in a game.
I see what you're saying, but with one behind and one in front it would know whether you were moving closer or further away as one set of LEDs would be getting further apart and one would be getting closer together.
ReplyDeleteGet me?
I see what you're saying.
ReplyDeleteI think it would get confused tho, imagine you were stood in the middle and turned your head to the left. According to both wiimotes you'd be moving away from both of them, which obviously is impossible. And how would it know which way you're turning your head, all it would know is that the dots are getting closer together which would be the same for turning your head either way.
Best solution IMO is to strap a gyroscope to your head, or a wiimote!
There should be a key difference though - one wiimote should be getting accelerometer information indicating movement, the other not - right?
ReplyDelete^^ Not sure exactly what you're referring to? If you mean my example then yes, the wiimote by the tv is using the camera function to tell where you are in the room compared to the tv. The one on your head is using the gyroscope function to tell which way round your head is facing. I think it would work like that, maybe it would have to use the camera function with leds near the tv still?
ReplyDeleteIt looks like you just use the Wii Remote. Can we just buy a Wii Remote and use your software or do we need to purchase the whole Wii system?
ReplyDeleteThanks!
Just buy the wiimote. One of the experiments currently uses the sensor bar that comes with the full system but you can replace it with infra red leds instead. Like on the video where he uses the saftey glasses.
ReplyDeleteGreat work. I am currently looking into bridging chuck, processing and the wii remote as you have shown. I am working under linux based system. I would gladly port the DesktopVR to linux. I am far from an expert but spend way to much time in front of my system and could spend it porting your app.
ReplyDeleteIf you want a sensor bar with out a wii
ReplyDeleteWireless Wii Sensor bar
or like the video shows build your own glasses. The glasses should be easy.
Super bright Leds sells the infrared LED's for 1.29ea. I remember that radio shack sold them a while ago, but i dont know about now.
For folks having problems with the Mac in bootcamp you need a later version of the wiimotelib.
ReplyDeleteI've checked a version of wiimotelib 1.2 modified to work with Johnny's code into the WorldWind SVN repository.
The URL is
http://sourceforge.net/projects/nasa-exp/
The directory you want is wiimorelib.
I sent a copy to Johnny prior to the holidays and I'm working to make NASA Worldwind DesktopVR compatible. If I get enough time maybe this weekend. If not...later. :)
Oh, you also need to modify the Wiimote DesktopVR code. That I haven't checked in anywhere...but the changes are more or less straightforward. If folks want it I can post it somewhere.
ReplyDeleteHi!!! Installed .net 2.0 and XP with service pack 2. But program does not start!!! What to do???
ReplyDeleteI thank you for quick help...
why the wii remote , why not just with a simply webcam ?
ReplyDeletethe idea is just ...waow.. I like it's so much ( I hope you can devellop something like than with just a webcam )
This so cool
ReplyDelete> de8msh <
ReplyDeleteFrom a fresh PC running WinXP SP2 to get this running I installed the .NET FX v2.0 and the DirectX 9 SDK (November 2007).
Using a Billionton Bluetooth Class I with BlueSoleil drivers.
For anyone interested; the DirectX SDK Nov2007 doesn't work on a Win2K machine, so I'm just now installing the DirectX 9 SDK (Summer 2003). Will let you know if this works.
> neo_973 <
The Wiimote has the smarts to analyse the camera images and send just the result. It's possible to do this with a plain el-cheapo webcam but would require a lot more effort to rewrite the algorithms to analyse the raw images. Plus a webcam sees broadband optical where the Wiimote only sees infrared, filtering out all the plain optical image making the analysis task a lot simpler.
DeeDub
FYI
ReplyDeleteNo luck with Win2K, the older DirectX SDK (summer 2003) installs (newer version won't install on Win2K) but still get the fishtankVR fault.
No joy with a Dell laptop either, this time running WinXP, with .Net 2.0 and DirectX 9 SDK (November 2007).
Guess it's the cut down graphic card & dell driver, will stick to the desktop instead works fine there.
A thought:
If you could modify a pair of these I-Vue style dual screen LCD glasses, to switch in time with the 3D Stereo effect mode that you get on most Graphics cards these days (the mode that works with shutter glasses to give you 3D vision, except here we don't need those shutter specs) you could get real 3D VR with all the existing FPS games already out there. Just ensure that the I-Vue only refreshes the left eye LCD when the Graphics card is outputting the left eye view, and vice-versa for the right eye.
I'm thinking the VGA card o/p goes to both left & right eye LCD's but add a small micro monitoring the VGA sync line and alternately enabling left eye LCD then right eye (except by enable I just mean enable refresh, you don't want the screen to go blank between refreshes).
Makes sense?
Deedub
Just to clarify that last comment...
ReplyDeleteThe 3D mode of the graphics card basically shows both left + right eye views on your monitor, and a sync line goes to some shutter glasses so you only see left view with left eye etc..
So I'm basically suggesting that you split this 'multiplexed' view back onto two separate screens, one for left eye, one for right.
Maybe this is already possible with the s/w that comes with these dual-screen LCD glasses?
DeeDub
Any details on the safety glasses and infrared (IR) LED's you replaced the original safety glass ones with. I picked up a pair of AO AOSafety Light Vision LED Safety Glasses (http://www.botachtactical.com/aoliviledsag.html)
ReplyDeleteWhat type of infrared LED's should I pickup? I was looking at these ones:
http://www.rentron.com/remote_control/IRLED.htm
Also how difficult is it to replace the existing LED's (i.e. is there any soldering/desoldering etc.)
Amazing projects!! I don't own a wii but I picked up 2 remotes and a wireless sensor bar so I can try some of these awesome experiments .
Hey Johnny,
ReplyDeleteI gotta say, bravo. Not only are you opening some pretty amazing doors, but you're putting the source out there for everyone too. You just got $50 bucks my friend, and it'd be more if I could afford it. You've made some real leaps in progress from the finger tracking, and I finally got a chance to rig up a hat (even though it looks ridiculous) to try out the VR desktop app. Freaking awesome, I love it, and I can't wait to see what else pops out of that crazy head of yours. Keep up the great work!
Again, I see what you mean. but, not only would the LEDs be moving closer together, they'd also be moving in space in corresponding directions
ReplyDeleteWiiMote VR support now in NASA World Wind. :)
ReplyDeletehttp://www.youtube.com/watch?v=AI1sC_G412E
Pretty kewl. It's been said and it will be again but it needed saying
ReplyDeleteYou could combine the earlier fingertip tracking idea with the IR glasses. By pulsing the IR illumination for the reflective tape on the fingers you could easily separate out the fingers (which flash) from the eyes (which don't). Controlling the IR illumination with the PC makes this easier to code, but even just having it free-running should still be possible
Then you can have a three-d world you can interact with (at least with two forefingers like in the demo). Action could be just to flash a standard IR remote - a third steady point could mean 'fire'
tom
It would be very interesting to see this replacing the protective glass with 3D colored lenses for 3D cinemas , projecting the image on a wall with 2 projectors and getting the 3D view out of that ... as then you will be able to get a real depth sense of the experiment.
ReplyDeleteNice thought though and would be lovely to see it in action.
Beautiful work, Johhny -- thanks!
ReplyDeleteFor those of you interested in trying this, but are looking for materials, here's what worked for me:
RadioShack IR LEDs
LED Lighted Frames
Targus Bluetooth Adapter
Replacing the LEDs is pretty simple. The LED holders can be removed from the frames with a Phillips screwdriver, and the housing opened with the same screwdriver. Then just bend and trim your IR LEDs to match the visible LEDs, and swap them out. Screw everything back together, et voila!
Good luck!
Oops! Here's the correct link to the Targus Bluetooth Adapter.
ReplyDeleteI guess this won't let me post a link to Office Depot...
ReplyDeleteWould anyone like to use my website for discussion about these projects? I am not doing it for ads or site traffic, I just want to help the users here communicate with each other as best they can, and give them a great platform to do it.
ReplyDeleteWiimote Project
I just started the site so feel free to use it if you want.
I would love to see this in action with a source mod.
ReplyDeleteIm going to try this first and then look what I can do.
Hey Dubbadee,
ReplyDeleteThanks for posting links to the items. I had constructed a rather ugly hat to do the VR desktop, was too lazy to find the lighted frames like the ones Johnny was using in his video. I just orderd 2 pair (because I could really use 1 pair for regular use too).
Really brilliant idea using moving the LED's instead of the remote.
ReplyDeleteIt gave me a few ideas, mainly on how to get true depth perception. From the software point of view, all you really need to do is create two windows with slightly offset images, move those two windows into two screens on a multi-monitor setup.
How exactly to set up those two screens in a glasses display is still a problem, an IR connection like in those "shutter" glasses would interfere with the whole setup. Could possibly put something together that sends a low res image on bluetooth, definately not a home brew solution though.
The other idea is to use multiple wiimotes in 4 sides of a room pointing inwards to track your IR LED's no matter what direction your facing. Use your imagination.
Wow...that was awesome. I still can't believe what I saw.
ReplyDeleteThis may have been mentioned before but, since a couple of years phillips has been working on a glasses-free 3D TV (which seems to work, having different filters for each pixels), now i don t remember the FPS rate of that TV but I know a similar project of texas instruments (although with glasses) has a FPS of 120 hertz so i will assume phillips TVs do to.
ReplyDeleteyour project blew me away (i m not a techy btw, so i have little idea of all of this) but i think it's real "applicability" would be more for the headtracking itself then the 3D feel, although the 3D feel would still be available.
Now, taking into concideration both previous statements you could actually solve (without the need of filtering glasses, but you still need something to track your head movement) the "two player" problem.
slowing down the frame rate to 60 FPS having a 1 to 2 ratio of different frames, two persons could play on different "screens" with the 3D feel the wiimote offers.
i don't know if by cutting down the FPS, more ppl ould play with different "screens" or the 3D effect of the TV could be implemented.
Again, i am no techy, just an amateur so if some of this does not make sense let me know.
thoughts ?
btw.. amazing project with the wiimote man !! simply amazing !! nintendo, give this guy a job, now !!
Love it!!!
ReplyDeleteJust one word of caution, if you decide to make your own "head tracking device". The wiimote seems to be a spoiled brat. The original "sensor bar" has 10 leds in it, 5 on each side... lots of light coming from there...
ReplyDeleteI tried single led setups, with normal 20mA leds. Hardly detected. Even rigged two 3-led clip-on lights, replacing the leds with IR ones. That's border line. Not very stable, but detected. I also made a plug-in 12v, 10-led "jumbo" that works reliably.
I wouldn't think of doing this with a single led, unless it's a super-strong type.
HI Johnny,
ReplyDeleteAmazing stuff. I had a quiet chat with the 3d cinema guys here at work to see if our cheaper 3d system (working on colour bandpass filtering) would work with your system. This could have reduced your research costs a bit if it had worked and perhaps I could ahve sent you some stuff.
Unfortunately my company's system cannot work with a TV screen due to the sheer number of pixels required (and a number of layers of visual filters that would have to be built into the glass of the screen to support the bandpass filtering), so to save you heading down a dead end the best (and we believe only way) to do single user stereoscopic 3d for head tracking on a TV panel is to use the shuttered/active glasses method you mentioned above.
However using a projector and colour wheel system, as used in my company's 3D Cinema, you could deliver a 3D effect that could be mapped using head tracking. This method is (sort of) supported by DLP-based rear projection screens and by 3-chip DLP projectors and this could be worthy of some further investigation.
Sorry I can't help more. Your system is incredible, I'm off to try it at home...
This comment has been removed by the author.
ReplyDeleteHi everybody,
ReplyDeletelots of talking about 3d tv sets and further on...
Why don't use simple red/blue slides on the ir led glass (like the red/blue glasses that you can find in tv guides sometimes) and display the object in green and red layers moved some pixels???
Is this a cheap way to watch 3d with headtracking?
Simon
ReplyDeleteIt should work with your 3-D projection system if it takes computer input.
I'd be interested in seeing how it works with World Wind and Johnny's code if you could pop me an email we could work out the details...
@ DeeDub Re: webcam head tracking
ReplyDeletewww.free-track.org
http://en.wikipedia.org/wiki/FreeTrack
Hi! could we replace in your Interactive Whiteboards the IR pen for an IR Laser and just project an IR point on the board ?
ReplyDeleteWiimoteProject.com Is now up and running.
ReplyDeleteI have talked with Johnny and We are going to make it the official support forums for his Wiimote Projects. Please register there and start sharing your wisdom :)
If anyone would like to help with the site please PM dice on the new site. (thats me ;) )
I was blown away when I saw this. Great job Johnny. I hope somebody who does code for design software such as SolidWorks notices this and sees the potential. I do solid modeling and would love to be able to interact with a machine design in 3-d space before actual production. Its around the corner I think.
ReplyDeleteThe greatest thing about this project is that it's so simple, yet effective. There exists no better combination.
ReplyDeleteI think this idea is likely to have been posted already, but there are too many to read, so just in case:
Possible uses for muliple Wii remotes.
For the heading tracking; if you place one wii remote pointing slightly to one side, and the other pointing the other way so that there is a small overlap down from the centre of the screen, you have almost 90 degrees. It's just necassary to define which wii remote is on which side.
For the multitouch screen; it could be possible to have 2 (or more) wii remotes aimed at the screen from different angles, then calibrate as normal, and it serves as two purposes. Firstly, it's easier to keep the pen in the view of a camera. Secondly, if both remotes can see the pen, after it's position has been averaged out by the pens, you effectively end up with a higher resolution.
I'm not sure how your code works exactly, but yeah... food for thought.
@kit (myself)
ReplyDeleteJust re-reading my post, I found it is smeared in grammatical errors. In my defense, it's late, and I'm tired.
First of all, really nice job!
ReplyDeleteI was trying to run it on my machine and it while the WiimoteLib can see my wiimote WiiDesktopVR cannot. I started debugging through the code and it seems that the error I'm getting is because
mReadDone.WaitOne(2500, false)
on line 900 of the Wiimote.cs file is failing. I haven't done any C# before so I was wondering if there was anyone here who could point me in the right direction on what this is actually doing (it looks like some thread synchronization or something), or perhaps point me to a page where a question like this is okay to post. I'm running on Windows XP SP 2 on a MacBook Pro.
Thanks
loos
ReplyDeleteTry checking out the latest code from CVS at sourceforge.
https://sourceforge.net/cvs/?group_id=212610
The code has changed to the 1.2 version of wiimotelib and it works on my macbookpro. Heh, in fact its the only thing I have to test on.
Hi Johnny, great videos, and great ideas all! Actually, on the note of polarized view I saw a video from CES 2008 that used the polarization to use one screen to 'split' the images for each viewer instead of using a 3d application. The idea was to use this for single-screen multiplayer games where player 1 only sees the red car, and player 2 only sees the blue car. Obviously all of this combined with head tracking could allow for multiple viewers on one screen. Another great idea, much like all of yours, that seems so simple in retrospect. Keep up the good work!
ReplyDeleteSo I was thinking, what's to stop you from using an LED array and some reflective tape to track the head with two points and the fingers with the other two points? Are all four points required simply for hand tracking? Assuming this isn't the case, there's a big benefit to not requiring LEDs in the head unit, no batteries. The LED array is easily USB powered. The next step up is some sort of coating on the lenses of the glasses themselves that's reflective only in IR, nothing comes to mind but I'm sure there are plenty of options. Stepping up one more from that, you can have a set of polarized, IR-reflective lenses for stereoscopic display, head tracking, and your simple glove with reflective strips for hand tracking and your user has a cheap and completely unpowered and untethered interface device.
ReplyDeleteI'd love to see you implement head tracking on one of Linux's 3d desktop environments, also :)
Visit www.thinkoptics.com as they have a remote with development software that will allow you to develope your own application even further.
ReplyDeleteI just saw alien vs predator and they use night vision googles in it.
ReplyDeleteNight vision googles are not binoculars, both eyes basically see the same image, so it's like you are watching through a camera, and there is no préservation of the stéréoscopic effect.
Thinking about that I remember when I was young I was playing with periscope.
So basically the idea is to have a google with two mini periscops that would make both your eyes see from the same point of view with same angle.
It would be kind of like you are looking from between your two eyes.
We can also imagine a similar google with two mirrors inclinated at 45 degres (north north-west direction) in front of each eye.
The right eye will be able to see through the glass, while the left eye will have is vision reflected on the both mirrors making the left eye see from the same angle than the right eye.
or maybe you can just hide one eye :D
Hi,
ReplyDeleteI would like to rebuild the effect with Java3D.
Has anyone experiences with that?
I also would like to understand the equations in the ReadMeFirst-File.
Has anyone a drawing that shows me the meanings of the variables used?
Thanks, Michael.
Your whole DESKTOP could be used like this; it could be done in LINUX, and you wouldn't even need the WIIMOTE if you have a laptop with integrated WEBCAM.
ReplyDeleteJust an idea.
You should add these two features:
ReplyDeleteFinger Recognition. (without reflective tape) By programing the size of a typical finger shape the computer would automatically detect the closest object withing a small circle shape and reject any other sizes, for ex; the palm, someones face if they are looking up close. This could also be advanced in the automatic detection of, lets say our ears, so we could use the VR desktop feature without using corny glasses, etc.
Virtual Keyboard. After enabling Finger Recognition i thought you could take the finger pointing to the next level and enable multitouch to be compatible with the windows keyboard. Great Idea! That was we could have a floating keyboard in mid air. How is that possible? you ask. Of course the L.E.D has a range and the detection can be altered to only detect in a certain coordinate in mid-air.
"Right Clicking" To relate to the right click feature in Windows O.S., you should simply program the thumb finger and fore finger "Pinching" together to activate the right click feature. just a simple "Pinch"
Questions Email me at EuroMateo@gmail.com
Id be happy to share more of my ideas.
Just as a note...wouldnt it be easier to just have one of those small portable LCD screens mounted on head somehow, and the sensor bar on top of it? You would still get the boxed effect though..and only to use at home so as to not look too geeky lol
ReplyDeleteHow would the wiimote detect fingers with no tape or ears with no LEDs? The tape is needed to reflect the IR light emitted from the array, and the glasses hold the IR LEDs.
ReplyDeleteI managed to get a glove working using a wireless mouse for the left and right clicks and IR LEDs on my finger, couldn't invert the axis on the software though so left was right :D
I'd absolutely love to try this out, but i'm nervous of hurting my Wii with software. Does anyone have any problems running the console for normal game use after running the software?
ReplyDelete"How would the wiimote detect fingers with no tape or ears with no LEDs? The tape is needed to reflect the IR light emitted from the array, and the glasses hold the IR LEDs."
ReplyDeleteDemonic; The tape only makes the IR light reflection stronger. You can just get stronger LED's to get the same effect. The only difference would be that the camera would not only capture fingers, but palms, arms, facial features. Thats what the program would be for; to differenciate from obvious shapes and sizes as oposed to the universal fingertip size.
Other Post update. Instead of ears; Eyes!
Johnny, there are so many applications for your ideas. I am an IT manager for a Radiology practice that routinely utilizes 3D applications for rendering anatomy for diagnosis. The ability to move in and around those objects in an intuitive manner would be tremendous. Your ideas are beyond user interface devices. They are more of a virtual user extension into the system.
ReplyDeleteI participated in a study at the Visualization Lab @ University of Illinois-Chicago with some very similar technology. They were already integrating 3-D into the design. Instead of looking through a window and seeing depth, I was actually immersed in the environment. The pointer device also had two settings: a "solid" laser line coming from the device and a virtual glove. The future of interface technology is looking pretty rockin' right now.
ReplyDeleteIt has probably already been pointed out (too much comments to read them all), but what this technology would really excel at besides gaming obviously is "extended desktops".
ReplyDeleteMeaning on devices like laptops, or even desktop pcs you could have your work spread out on a much larger area than currently. Of course this is nothing new, but this headtracking would take it to a completely new level...
That's insanely cool. Just imagining what you could do with a sensor bar, two more Wiimotes and a bit of tinkering. If you were to build each Wiimote into a glove, and wired up conductive pads to each finger (and thumb) tip, you could track your hands in 3D, then wire it up so that touching each finger to your thumb is equivalent to touching a button. This could give you at least 4 (individual fingers) and up to 15 (? combinations of 2, 3 and all 4 fingers) actions for each hand.
ReplyDeleteI imagine this would be insanely useful to anyone involved in 3D model creation (or in my particular area of interest, protein structure).
I'm not from Nintendo but I'm still working on a very special PC game. Some kind of MMORPG using the latest ios technologies like touchscreen, head-tracking, VR Glasses and other things like the Wii remote thanks to Johnny Chung Lee's idea. I hope I'll put this project to an end!
ReplyDeleteActually webcam CCDs are sensitive to both IR and visible light. Allot of webcams have an IR filter that can be removed. Some of the cheaper webcams have linux drivers and software has been written to do edge detection of flames and such. This software could easily be modified to track the two IR LEDs on the glasses.
ReplyDeleteOn a side note, Nintendo would want you to use a Wii and a Wiimote, because their customers have them at their disposal, and require no extra knowledge on how to set it up. Simply sell some IR glasses like Johnny has described, and get some game developers to go along with it's a win win for customers and Nintendo.
This comment has been removed by the author.
ReplyDeleteIf someone made a Wii Remote battery cover with a fitting for a camera tripod, then we could use existing camera tripods for mounting the Wii Remotes. This provides a cheap (relatively) solution to the other piece of hardware that would need to be included with a head-tracking game. *Johnny Lee Double Fist Pump* Come on Wii game designers!
ReplyDeletePotentially, there could be a additional application of head tracking. With a pair of standard lcd glasses with headphones, the visuals and the audio could be in 3d.
ReplyDeleteIn the desktop 3d example, each disc could have a particular sound, which would get louder or quiter depending on your position. Tilting the head could also change the 'position' of the sounds in the 3d landscape.
In a game scenario, you could be totally immersed (think call of duty, flight sims etc).
Anyway, I'm sure there are a million more applications. Keep up the good work!
Hello Johnny Chung Lee,
ReplyDeleteI would like to ask if you put a fish eye lens on the wiimote , would you then be able to get a wider field of view for the wiimote camera? It might alleviate that 45 degree issue.Would it diminish the quality of the signal from head mounted or finger mounted ir's?
Thanks in advance. you're the Greatest.
I was thinking about the problem of calibrating the wiimote in relationship to a "perceived depth" of manipulatable images. Consider this: I envision a simple 3-d pong game. Something like a tennis game with no net. Would the following work? Create a light-weight racket shaped extension to the wiimote constructed from clear plastic or plexiglass. It would have no strings woven into it, just an empty racket frame. Perhaps this is oversimplifying a solution to things but, would it be possible to place the open frame of the racket over 3-d target like the one's in the head-tracking video and then shift your position until the target appears to be at the same DEPTH of the interior of the racket frame? With the right formulas the coordinate relationship between the racket and the graphic image might be functional as a "key" for calibration.
ReplyDeleteWould this work? Could you then change your position and swing at moving targets and have an effective point of reference for making "simulated" contact with images at a "perceived depth"
For added effect, it would be neat to have some kind of percussive device in the handle to create the sensation of impact(moreso than rumble would, like a little hammer thumping on an anvil).
Keep up the good work! If I had ANY experience programming for the Mac I wouldn't just be wishing you'd get that 3D Desktop program working for my Compy!
ReplyDeleteJust picked up a WiiMote, don't own a Wii, and got a Wireless Sensor Bar so I can put DarwiinRemote to good use.
Can't wait to see what you have in store!
I'm not an engineer, so I'm not sure about the feasibility of what I'm thinking right now.
ReplyDeleteSupposing you want to have two players playing on the same screen and sharing the same 3D experience.
Can't you combine the polarized glasses idea with the old LCD shutter glasses technology ? This way, the screen would play the images this way :
- Image 1 :
Player 1, left eye (horizontal)
Player 2, right eye (vertical)
- Image 2 :
Player 1, right eye (vertical)
Player 2, left eye (horizontal)
I know it's a bit more on the hardware side, but I'd like to know if you think this could be done.
hey, does anyone know how to make it work with a mac, aside from using bootcamp(which i do have, just dont wanna use it)? also, what would happen if you took your idea, and crossed it with 3d movies (blue/green in one eye, red in another, the picture onscreen is two pictures) Would that make a better or worse 3d image
ReplyDeletehey, does anyone know how to make it work with a mac, aside from using bootcamp(which i do have, just dont wanna use it)? also, what would happen if you took your idea, and crossed it with 3d movies (blue/green in one eye, red in another, the picture onscreen is two pictures) Would that make a better or worse 3d image
ReplyDeleteIf you merge this beautifull idea with
ReplyDeletesomething like that http://www.lumus-optical.com/
Your are able to manage a virtual screen and a virtual keyboard !
Very large screen : the right part above the right shoulder !
If your are able to detect the position of fingers, user can use the keyboard below the screen.
To choice who called or what web site to display !
Hi!!
ReplyDeleteWhat kind of "projection" matrix (if that's the trick) are you using in your program to create the ilusion of real 3D?
"What kind of "projection" matrix (if that's the trick) are you using in your program to create the ilusion of real 3D?"
ReplyDeleteAsymmetric:
http://www.vis-sim.com/vegaprime/images/vp_chanprojections_01.jpg
Thnx micah!!!
ReplyDeleteI'll be searching on the net about this subject.
If you have some more resources related with this, please let me know!!
I think you had a great idea, but I also think there is a big problem about this "invention": it's really uncomfortable to have to costantly look at the PC monitor although being moving the head around to change the point of view.
ReplyDeleteSo, I had another idea.
It's a bit complex, but not impossible.
I think that,m using 2 bluetooth-enabled smartphones, we could easily build something like a cheap head-up display.
You mount the smartphones and the WII sensor on the same structure; you put to lens systems in front of the screens, to allow them to be viewed from short distance (a few inches); you put in the phones a program which proper pre-processed images from PC. You "mount" the "device" on your head. And you get your head-up display!
Please let me know if it can be done.
(sorry, some errors in previous post).
ReplyDeleteI think you had a great idea, but I also think there is a big problem about this "invention": it's really uncomfortable to have to costantly look at the PC monitor although being moving the head around to change the point of view.
So, I had another idea.
It's a bit complex, but not impossible.
I think that, using 2 bluetooth-enabled smartphones, we could easily build something like a cheap head-up display.
You mount the smartphones and the WII sensor on the same structure; you put to lens systems in front of the screens, to allow them to be viewed from short distance (a few inches); you put in the phones a program which RECEIVE proper pre-processed images from PC. You "mount" the "device" on your head. And you get your head-up display!
Please let me know if it can be done.
You could use this to map sound volume in earphones too--to give the impression that while a user is turning their head, the sound remains from a single direction.
ReplyDeleteK Robinson
email: zwokkqxpozgc+oybttre_cebpenfgvarrevat at gmail dot com. yes this address will reach me.
Congrats Johnny, seems you inspired EA, of all companies, to take your idea and run with it. Boom Blox for the wii will utilize head tracking for part of the game. Seems to be during replays at least. Nice job, I was hoping to see games use this, and this is a big step in that direction
ReplyDeleteHey,
ReplyDeleteI'm a scientist at UT and this would be an excellent tool for visualizing 3D molecules. I'm not that computer savy, but it appears that it wouldn't be that difficult to alter the demo program to load 3D molecules. If any programmers out there could do this, I'd send you a file to practice with.
To all the people concerned about the framerate hit using LCD shutter glasses, remember a DLP projector is capable of outputting hundreds or even thousands of frames per second at lower colour depth, so if you don't mind everything being in 64k colours you can have a lot higher framerates. Making off-the-shelf kit do it would be a chore but there's no real technical limitation to it.
ReplyDeleteI'm sorely tempted to go and buy all the kit required just to experiment with (projector + shutter glasses + head tracking) = immersive VR :)
Top work Johnny!