Dec 15, 2007

The Beginning

This project came from my research during another project. I discovered some of the videos online of Jeff Han and his multi-touch research at NYU. It seems to be the case that Mr. Han has inspired quite a lot of people to make their own interfaces, some of which can be found under the Links tab on the right. For anyone interested in creating their own multi-touch should definitely join the NUI Forum. There is a wealth of information and the other members can help with almost any questions.
Now lets get started with Schproket. This endeavour is sure to be 'finger clicking good '©.
Cheers,
Scott

Step 1: What you will need

A multitouch interface is not terribly difficult to construct but there are many supplies and you should know what your getting into. It is easy to get wrapped up in the idea of what kind of software you could make for multitouch, forgetting that you will need an interface if you want to make that software.

So here is a list of the supplies you will need.
Computer with decent processing speed
Digital projector
Webcam
Surge protector
InfraRed LEDs (depends on the size of your table but think around 50 of them for good measure) I am using some I got from Mouser
Plexiglass (cut to the size of your touching surface)
Silicon Rubber, I am using Near Clear
A material to make the table, I am using a medium grade plywood
Mirror to reflect projection upward
Something to hold the LEDs, I am using some C channel aluminum
A projection screen, I am using vellum paper

Minor supplies:
Roll of film, any 35mm film will work.
Sandpaper (and hopefully a sander)
Tools to make the table
Wiring and a power supply for LEDs
Speakers, this isn't necessary but I am including them

Supplies you might need:
Reflective Tape

So now that we have our supplies, lets get to work.

Step 2: Plan

The most important part of this entire process is to have a plan. Know what you are going to make, do and when you are going to do it. Making a multitouch interface has many steps, some of which take several hours to complete. Budgeting your time according to a plan is necessary.

Here is my plan:
Sketch out interface and determine size.
Building frame
Get acrylic
Make and install Silicon rubber sheet
Make LED frames
Wire and Mount LEDs
Hack camera
Test LEDs
Test FTIR
Adjust for any problems
Get software
Test software
Install Projector
Test finished setup
Adjust for any problems

This is a an overview of the entire process. Obviously some of these things can be started earlier. I have been looking at different software available since the beginning of this process, but was unable to actually test any of it until the LEDs are installed. So make a plan.

Step 3: Determine size

Determining the size of the interface is the most important decision you will make. Many factors will affect this decision.

Available space: How much space do you want this interface to take up? I made a mistake in my planning and made my interface larger than I wanted. My only thought was the size of the touching surface. This is important but I forgot the consequences of this decisions, and some of my math was wrong.

Size of screen/camera: The size of the touchable surface depends on the dimensions of the acrylic and also how much space the camera needs to see the whole of those dimensions.




Projection size: Along with the camera you need to consider the projection which will be thrown onto your touching surface. How much height do you need to make a projection of a particular size. Projector Central has calculator for many projectors, even if your projector is not an option you can get a good idea for the height you need. Alternatively you can simply point your projector at your acrylic, back up until its the size you want then measure with a tape measure how far away you are.



So what does this mean? It means the size of your acrylic touching surface (length x width) determines the height of the interface. This is because your light based materials (camera and projector) need a certain amount of space between the lens and the surface in order to fully cover it. Usually your projector will need more height (distance) than your camera.

So once you have some of these measurements sketch out your interface.
Schproket needs to be at least 35" tall in order to fill most of the acrylic (30"x24") and my camera needs to be slightly elevated to capture the full projection and not the full size of the acrylic. This will prevent inputs falling outside the screen size.


Now we can start building the frame for the interface.

Step 4: Building the frame

So we have our measurements and can start to build the frame for the interface.
Schproket is made out of 2 sheets of medium grade 3/4" plywood and 2 2"x4"s.

I added 3" in depth for where the LEDs will be placed along the long edges (now 30"x27"). Then I added 4" in depth to one side to make room for the projector. This prevents the projector from getting in the way of the projection. So my final dimensions are 30"L x 31"D x 35"H.

Schproket is built like a box, so all the sides are solid. I left one side open for access to the inside of the box, which you will need. I won't get into carpentry here but you will need some power tools and basic carpentry skills to make a box.

Things to remember:
The acrylic has to rest on something. Make a lip for the acrylic to sit on. I cut the support 2x4s 1.25" shorter than the full height, then placed a 3/4" piece of plywood on top of the supports stretching across the back and the front of the box. Then placed the acrylic on top of that so my acrylic will sit flush with the top of the walls of the box. The frame to hold the LEDs will also sit on this lip.

Here is the Schproket box after construction with the acrylic on top. The acrylic still has its protective paper cover on it.




A good thing to add to any larger interface is caster wheels. Especially with plywood these things can get heavy.

Step 5: Acrylic Surface

You should have purchased your acrylic already, but if you haven't do so now.
I purchased my acrylic at Canal Plastic. If you are in the NYC area then you should know about Canal Plastic, its an excellent resource for almost any kind of plastic you want and in all kinds of shapes or custom cut.

The acrylic I used is 1/4" clear plexi with the long edges polished. All the sides that are going to have LEDs should be polished. You can polish this yourself but having a professional do it will give you a much better result. Most establishments that will cut the plexi for you will also have polishing available for a minimal cost. It comes with a protective paper glued to both sides of it, I wouldn't remove this paper until you are ready to test your LEDs.

While you can get your interface working with just the acrylic, if you want it to work well then you need to have a projection surface and a sheet of very thin clear rubber.

The projection surface can be a real projection screen or almost any kind of paper. For Schproket I am using sheet vellum cut to the size of my acrylic. This is simply for the projection to catch on something. If you don't have it the projection will go right through the plexi and you won't see the screen you are touching. The only restriction on material is that your surface can not stick to the silicon sheet, try testing a few materials to see what works best. Remember that this is also probably what people will be touching; so a material that is durable or cost effective replaceable is your best bet.

The rubber needs to be very thin and transparent or semi-transparent. The common consensus is on silicon rubber for this. Many people are using Sorta Clear 40 to produce this sheet of rubber. I am not because I have never had any good experience with SmoothOn products. For Schproket I ordered Near Clear from MicroMark, which I find to make far superior products.

This is kind of tricky because you need to make the sheet of rubber. Why? If you set up your entire interface without the rubber it will work, but you will have to press extremely hard on the acrylic to get a decent amount of light to leave the glass. This is about FTIR (frustrated total internal reflection). I am not the expert on this but essentially the light from your LEDs is bouncing around inside the plexiglas. When you press your finger on the top of the acrylic your finger is interrupting the flow of light within it. That forces the light to leave the acrylic directly away from your finger. With just acrylic this is difficult, adding the clear rubber makes it easier. The light is now bouncing around the glass and the clear rubber. When you press your finger on the rubber, which depresses due to its spongey nature, your finger is making a very clear depression in the flow of light. This makes the light leave directly away from your finger. Basically its easier to push rubber than it is plexiglas.




So...we make a sheet of rubber because I have yet to find a place that makes this.

You should make a molding frame around your sheet of acrylic. This needs to be tight against the sides of the acrylic and 1-2mm taller. You are going to pour the silicon mixture directly onto your acrylic.

Follow the instructions for the silicon rubber you have purchased. Be sure to pay attention to the mixing instructions, you want to avoid bubbles forming. Its ok to have some bubbles, its almost unavoidable to have a few.

Step 6: Make LED frames

Most LEDs have a very narrow focus, meaning they only point light in a narrow beam. Your IR LEDs need to be mounted in a frame so that they point directly into the sides of the acrylic.

I picked up a C channel piece of aluminum at Home Depot for a few bucks. It was long enough to cover boths of the sides where I will be placing my LEDs. I measured and cut it into 2 pieces each 30" long.



Then I marked out where the lights will be with a red Sharpie. I evenly spaced them along the 30" to get the best coverage of light.



After marking where the lights will be I drilled through the aluminum with a drill press. I used a 1/4" drill bit so that I could use these handy little LED mounts I found at Radio Shack. The LED snaps into the mount, then just press the mounted light through the drilled hole.

Step 7: Wire and Mount LEDs

Now we need to wire and mount the InfraRed LEDs. There are several options available, the easiest and fastest is to use a breadboard for your circuit. You should probably use this to figure out your circuit anyway, once you have everything working you can always go back and hardwire everything together on a circuit board.
So for now lets use a breadboard to make our electrical circuit for the LEDs.

You can use these LED calculators to determine your circuit needs.
http://ledcalc.com/
http://led.linear1.org/led.wiz

I am using 30 IR LEDs that I picked up through Mouser. They use 1.7 volts to power each LED. My circuit has 6 sets of LEDs in parallel, each set is a series of 5 LEDs. On my breadboard each series has a 47 ohm resistor leading into it. All of the LEDs are powered by a 12v ac/dc adapter. You could run your LEDs off batteries but the adapter is easy to use and since there is a surge protector in the frame anyway its a simple solution for power.

After determining the circuit I mounted all of the LEDs in the C channel frames, 15 on each frame. Then I wired them together. Wiring 5 LEDs together - to + in a row. Then wire the first + and the last - to the breadboard. After 6 sets of 5, the LEDs were all wired in.

Plug the adapter into the outlet and check out your lights. But wait, you can't see them...

Step 8: Hack Webcam for InfraRed

Now we need to hack that webcam to accept InfraRed light. Most cameras have a built in IR light filter which needs to be removed and we need to put 35mm film in to block visible light. Once this is done your webcam will only see InfraRed light.

Not all webcams are the same so you may want to search around for your specific camera, someone may have detailed images of the insides of it.

First, take the back off the camera by unscrewing it. My webcam had one screw holding it together and one side of the plastic frame snapped into the front of the frame.

Slowly take the backing off the camera. My camera had the circuit board connected to the backing, while the front came right off with nothing attached to it.

Take the lens off the circuit board. My camera was screwed in, so I unscrewed it and carefully removed it.

Again, most cameras have multiple lenses so you will have to find the IR filter lens. I found several people online write that the IR lens is tinted red. This was not the case for me and all of the lenses were in a plastic tube, one of top the next. So I just cut the last lens off with my exacto knife. My intention was to lay them out in order from back to front if I had to remove more. I tested it after removing this lens by putting the tube of lenses back into the circuit board and plugging it in to my computer, then point it at some lights or at your TV remote while holding a button down. I got lucky, the first lens I cut out was the IR lens.

Here is my camera with the IR lens removed. I'm holding a button down on my TV remote.


Now you want to block visible light. This can be done with 35mm film that has been fully exposed and developed. All I did was pull a whole roll of film out of the casing, run a light over it all. Then roll it back into the casing and get it developed, remember to ask for process only not prints.

Cut some of your film into a small circle the size of the lens you just removed. It doesn't have to be perfect, just enough to fit where the lens was. You want two of these circles in order to block all the visible light. This film will not block InfraRed light. So place your two circles of film in the camera where the IR filter was. Carefully reassemble your camera, being sure that the film doesn't slide out from its spot. Plug your camera in and look around your house with it. You will see light coming from most houselights, some computer screens, and tv screens but everything else is just black.

Step 9: Test LEDs in frame

Now that the camera can see IR lights, point it at your LEDs in the frames.
Your lights should look something like this. (Note this picture was taken before I put in the visible light filter.)

This is some of my lights seen through the top of the acrylic.

If your lights are not lighting, go back and check your wire connections. Look for a loose connection. If they all look correct, check your circuit or breadboard to make sure the wires are connecting to the correct locations and that your power source is connected.

Dec 14, 2007

Step 10: Test FTIR

So the LEDs are hooked up and working, the acrylic surface is set up, and the frame is built. Put all this together and you can test the effect of FTIR.

You don't need blob tracking software to test this. All you need is some program that will show the incoming video. For this test I used iChat, it has a live video and can take snapshots.

The visible filter blocks everything but IR light, so most of the video should be black. The IR should show up when you press down on the acrylic. You might want to turn any overhead lights off because they more often than not put out some IR light. This won't interfere much with the FTIR but will give you background light. Eventually a projection screen will help block out this ambient IR light, but for now just turn the lights off.

If all is well then your video from the camera should look similar to this.



At this point you will know if your setup is really working or not. If it is not working then you need to troubleshoot.

Troubleshooting
When troubleshooting if you change something, test it. If that did not fix the problem think through what else might be affecting the result. Change one item at a time and test.

What is the problem concerning?
Lighting, camera, or acrylic? These are the main items that could give you problems.

Lighting - Take your camera out of the frame and look at all your lights again to make sure they are all on. Also check if they are aligned straight into the side of the acrylic.

Camera - Take your camera out of the frame and point it at your lighting frames again. Make sure it is picking up only IR light. I kept one LED hooked up to a battery on hand because I had some problems with my camera lenses shifting around. I believe this was because I removed the IR lens and a small spacer which was between the IR lens and another lens. With most cameras you can also check the settings on your computer, play with the settings to determine optimal lighting.

Acrylic - Look over your acrylic for cracks or imperfections on the edges which may interfere with the light entering. Make sure it is aligned with the LEDs. Look over the silicon sheet for large bubbles, uneven surface and clarity.

Step 11: Install Projector

This is one of the steps that can really be done at any time. I found the projector to just be in the way while working inside Schproket. I put it in, made a shelf to hold it facing down at the mirror, then I took it out while I worked on the lighting and acrylic.

The projector is essentially pointing straight down. Try to make it level but most of your 'corrections' can be made with the mirror.

Once the projector is installed, plug it in and turn it on. Then place the mirror on the floor of the frame. You can move the projection around the projection surface by tilting the mirror. Do this until its centered where you want and place something underneath the mirror to hold it in place. Eventually you want to make a brace to hold the mirror exactly in place, you can do this now if you want. For Schproket this wasn't very important, I wanted to adjust the mirror as needed.

This is looking through the acrylic at the projector (black box) pointing down at the mirror.



Projector hooker up to laptop.


Inside frame while projector is on.


View of Mirror with projector on.

Step 12: Blob Detection Software

The next big step to get your multitouch interface working is software. Again, many options available for blob detection software.

Touchlib is an opensource C library currently for Windows. You can find the files and plenty of people using touchlib at the NUI group. If you are comfortable with compiling programs and with programming Touchlib should be work out well.

LibAVG is an opensource Python/XML library for Macs or Unix. LibAVG documentation is pretty thorough and there is a small community working with it and updating it. You will have to compile this as well, but Python is a lot easier to understand and start working with than C. So check it out if your not too comfortable with programming.

I also found this Processing sketch for blob detection. BDWebcam is opensource Processing. Processing is simple like Python but seems to have better libraries for accessing video, maybe I am just missing something.

Blob Detection can be created in a multitude of languages and programming environments. I have read about others using Max/MSP, vvvv, Java/Processing, and PyOpenGL to create their tracking software. Look around and ask questions from what others have made. The only sure thing is that there is not a one download and your done solution yet. It will take a fair amount of work to get your blob tracking working well. Once it is working, a whole new world opens up.
Good luck with your project.

Step 13

Step 14

Nov 24, 2007

Step 15

step 16