Anticipating for a pair of weeks Minority Report, I asked two students to do a Gesture-Driven mouse, using a quickcam and visual basic. The project started in January and the different modules have been first integrated by July 2003, which casually is coincidental with the release of PlayStation II Eye-Toy. So we are in the wave... I have now asked my students to clean the code to put it in the free public domain. If you want to get some clues on visual basic programming and be able to develop EyeToy-like software in your PC, keep watching this page.
Instead of the nice black leather glove with three blue diodes, we use a greenish kitchen glove. This is actually more difficile than Tom Cruise's gadget, as detecting lights is easier than detecting colors. But in this way we avoid extra soldering.
We found that the capture stops when other window event happens, so while mouse can move smoothly (sort of), we need internally relaunch the capture after every event. Also, dragging does not work because the OS takes control. If such feature is needed, you will work at deeper levels of Windows (HID drivers etc.). Fortunately menus and buttons live in upper layers.
To simulate mouse buttons with a gesture, but it seems that a second momenta calculation to fit the ellipse will be enough. On a pentium 4 the analysis takes about 90 milliseconds and we can do it even better, so it seems it will work! Note the small tags next to the images... we use them to disable drawing, which takes a lot of time -half second per image!- in our current, debug, implementation. Ellipse orientation, eccentricity and size are enough parameters to define four states: NOHAND, LEFTBUTTON, RIGHTBUTTON, NOBUTTON.
If you are intrigued, you can download
Let me be serious here: The code is buggy and will not be functional until version 1.0. Or beyond. Do not download it if you are not a programmer or interested on the internals of programming. It involves pointers, callbacks and a lot of obscure tricks, and my students are very young so their code is still very naive and convolute (In any case, you can ask us for help). If you are only interested in the final version, just bookmark this page; or send me an email [arivero at posta.unizar.es] asking to be notifyed
In any case, the .zip can be compiled in any VisualBasic 6.0, and it contains:
We are moving definitively to first and second moments analysis (center of mass and ellipse fit), to determine both the position of the hand and if it is open or closed. A closed hand is a "mousedown click", so one can grasp a object and drag it across the screen. We are keeping edge detection in the analysis code window just as an example for other alternatives, but we are not using it.
Note that all the provided code is to be considered under Gnu Public License, the owners being Enrique Garcia -the analysis code- and Laura Morentin -all the rest-.
If you are aware of similar projects, please tell me.
The main source for recipes on getting video from Visual Basic is Ray Mercer. Also check E. J. Bantz.
The strange relationship between bitmaps and picture boxes can be studied in some articles from the Microsoft Base. Check numbers 187568 and 186260 Also a simple fading example can be of some help. Or a reading of our own code.
To understand use of pointers in visual basic, you need to look for some article on Safe Arrays, which are the movable arrays of VB. There was some other good articles in PDF format, but I do not remember where.