The largest community for DJ and producer techniques, tutorials, and tips. Traktor secrets, controller reviews, a massive MIDI mapping library, and more.

The Space Palette -Tim Thompson’s Kinect 3D Controller


As many of you are aware of, the West Coast Championship Controller Battle went down last weekend and the DJ Techtools staff was present for full coverage of the event. Although Edison deservingly ended up taking the crown in the end, we all found another competitor’s setup to be one of the most interesting and innovative controllers we have seen to date. So without further ado, let’s introduce Tim Thompson (aka Artful Codger) and his Space Palette, which utilizes a wooden frame and the Microsoft Kinect to create electronic music on the fly with the literal wave of a hand.

Tim Thompson, the inventor of the Space Palette is no newcomer to DIY controllers and algorithmic music. With a strong interest in computer music and art, tied with a background in software engineering (having worked at notable companies such Bell Labs and AT&T), Tim has been able to create a variety of unique music and art installations that is not only inspiring but also truly captivating. Check out his extensive (10+ years) “What have I been doing?” section of his personal website (http://nosuch.com/tjt/)

Tim’s most recent project, the Space Palette utilizes the depth tracking of Microsoft’s Kinect device and a wooden frame with rectangular shaped cutouts, which are used to control pitch, velocity, and other parameters of specific electronic/drum sounds. Each note is triggered when motion is detected within one of the rectangular cutouts. They are then modulated based on the position and depth of the hand travel producing a 3D touch surface.

The Microsoft Kinect works as a horizontal sensor that functions as an RGB camera, a depth sensor (which is just an infrared laser projector combined with a monochrome CMOS sensor), and a multi-array microphone which together are designed to be used for 3D motion capture, facial recognition, gesture recognition, and voice recognition purposes.

For all practical purposes, the Kinect has a tracking range of 1.2 meters to 3.5 meters (3.9 ft to 11 ft) and can track up to six people or motion analysis of two people with a feature extraction of 20 joints per person within an angular field of view of 57 degrees horizontally and 43 degrees vertically (with the motorized pivot being capable of tilting the sensor another 27 degrees vertically). Another thing you probably didn’t know about the Kinect is that it currently holds the Guinness Wold Record holder for “fastest selling consumer electronics device” selling 8 million units in the first 60 days or 133,333 units per day!

  • After the controller battle, I followed up with Tim via email to inquire further about the Space Palette and his work :

All of our staff were really impressed with your Space Palette, not to mention wildly entertained by your performances (by the way, we all unanimously thought you should have won your last round)

Thanks, it was certainly a great opportunity to show people that controllers can be more than buttons and knobs.  The Kinect blows the doors wide open for innovation, and Space Palette is just the first of many new controllers (both musical and visual) that will be based on it, there’s no doubt.

What software did you use to create the music aspect of your compositions with the Space Palette?

KeyKit is the environment I use – it’s a programming language I developed several decades ago, specifically for algorithmic and realtime MIDI work.   The sounds were done with a variety of VST synths inside Plogue Bidule – e.g. Omnisphere, FM8, Battery 3, Massive.

What software do you use to communicate with the Kinect?

I wrote a program in C++ which uses the libfreenect library to communicate with the Kinect. I don’t use the RGB camera or skeleton tracking, I only use the depth data. The OpenCV image-processing library is used to do blob detection in the thresholded depth image. That program sends OSC messages to both KeyKit (to do musical things) and Processing (to do graphical things). The OSC messages use the TUIO format, which is a standard for multitouch information.

Is there any new technology or devices out there that interest you for another controller/installation (or adding to an existing)?

The other interesting device I’ve used in the last decade is the Fingerworks iGesture pad. Besides being multitouch, the iGesture pad can detect (with incredible precision) the area of each finger on its surface, a feature which I found to be extremely useful for realtime graphics (think “finger painting”) as well as music. The iGesture pad is no longer available because Apple bought the company in 2005, to help develop the iPhone. Apple doesn’t yet officially allow the use of finger area in apps, though some of their hardware is capable of it. I’m looking forward to the day when either pressure-sensitive or area-sensitive multitouch surfaces are widely available and supported. Roger Linn is one of the people trying to make this happen.

Was creating the software mapping/setup with the Kinect and your other software GUI based or more programming based?

All the software I write is done with text-based programming languages.

Could you please explain more specifically what the musical attributes the x, y, and z (outward depth) components of each rectangular space are controlling?

Within each rectangle, the x position of the hand (or other object) determines the pitch. The pitches of the notes are restricted to a particular scale. Each rectangle can have a different range; most of them are two octaves. The smaller the pitch range, the more accurate control you have over the pitch. Any number of hands (or objects) can be used simultaneously. There are no loops or sequences; notes are triggered individually whenever the movement of a particular hand or object exceeds a threshold.

The narrow rectangular spaces in the four corners of the Palette are essentially “big buttons” for control. One of them changes the key, one of them changes the scale, and one of them lets you change the sounds (each rectangle can be cycled through three different sounds). I sometimes allow one of these control buttons to turn on realtime looping independently for each rectangle – this can be convenient and fun, but can also be very confusing, especially for people new to the instrument, so I normally don’t enable this feature.

The y position within a rectangular space is used to control the timing quantization of the notes. Notes played in the lower third of the space are quantized to quarter notes; the middle third quantizes to eight notes; the top third quantizes to sixteenth notes. I think of this feature as implementing “time frets”.

The z position (the depth) gets converted into MIDI aftertouch, and the effect of that is dependent on the sound patch. Typically it’s used to add vibrato, control a filter, or mix between two sounds. The use of hand depth while playing is very satisfying for the player as well as entertaining for the audience.

Are you planning on ever producing or releasing the hardware/software package that creates the Space Palette to the public (or for sales)?

I’m planning on making the Kinect-specific program (the one that sends OSC/TUIO) available, at some point. The software is capable of dealing with any frame that has holes in it – the holes don’t have to be rectangular. Each hole becomes a separate multitouch surface. After the software is trained, the frame is actually optional, though I still consider it an important device both for the player as well as the audience. Now that there’s a TechShop San Jose, I’ll be able to produce frames of different shapes more easily.

Was there anything specific (e.g., an inventor, composer, piece of equipment, or event) that inspired you to start making your own controllers and installations?

Two events were particularly inspiring for me. The first was Woodstockhausen, a yearly event near Santa Cruz where experimental music makers were given an opportunity to entertain each other. I participated for 3 years in a row, doing increasingly-adventurous performances – the second year I performed using only a QWERTY keyboard as the controller, and the third year I performed by dancing on four playstation dance pads, in complete darkness, wearing EL-wire-outlined pants. The other inspirational event was my first visit to Burning Man in 2002, which triggered the most radical change in my work. In 2003 I did my first playa installation which was a 12-foot-high lyre with lighted strings that you played by dancing on playstation dance pads. Burning Man continues to inspire me every year – this will be my tenth year in a row.

What’s the best fan/audience response you’ve ever seen from any of your events?

The best response I’ve ever gotten was when I released the swinging tennis ball in the Space Palette at the Controller Battle.

Are there any interesting stories about the inspiration, creation, or evolution of the Space Palette that you would like to share?

Can’t think of a particular one. Evolution certainly describes how it was created and will continue to be enhanced.

Where can fans see the Space Palette in the future?

I’ll be bringing it to the “Touch the Gear” event on July 17 which is part of the Outsound New Music Summit

http://www.outsound.org/summit/11/schedule_details11.html

I’ll be bringing it to the Nexus fundraiser on July 29

http://www.themml.com/event.php?event_id=8989

And of course I’ll be bringing it to Burning Man.  I’ve applied to be a theme camp, called the Multi Multi Touch Touch Camp, and I’ll be building a shade structure to allow it to be used in our camp during the day as well as at night.

Are you working on any new yet to be seen projects?

Using the Space Palette to paint graphics (simultaneously with the music) is the next step.

Is there anything else about yourself that you would like our readers to know?

One other concept I’d like to include is that I consider Space Palette to be in the class of a “casual instrument”, analagous to “casual games” that are simple and easy to learn, yet still deep enough to be satisfying for a long time. Using the label “casual instrument” is also an attempt to distinguish this from more conventional instruments that allow more fine-grained control (e.g. the ability to reliably play arbitrary melodies). There’s actually nothing preventing the Kinect and the Space Palette from being used to create a more conventional instrument, it’s just not the focus of my current efforts, which are centered around creating installations for people (both musicians and non-musicians) to enjoy at various events. Actually, the most rewarding situation for me is to see someone who does not consider themselves to be a musician walk up to one of my instruments, create their own music, and (most importantly) to see them realize that they are the ones actually creating it. At Burning Man, one of my most memorable moments was watching someone walk away from Monolith 2.0, saying “I did that!”.

You can find our more about Tim Thompson and his work at his website http://timthompson.com/

  • Pingback: Weekly Article Recap for 7/1 -7/7/2011 | IPR Blog()

  • Djteeoh

    If you minimize the “wall” into a portable sized model, like the size of an ipad….then used the same Kinect technology, you could make the first touchLESS midi controller. Same concept just smaller, portable, and useful. It’d be like an MPC, just without actuall buttons.

  • Djteeoh

    If you minimize the “wall” into a portable sized model, like the size of an ipad….then used the same Kinect technology, you could make the first touchLESS midi controller. Same concept just smaller, portable, and useful. It’d be like an MPC, just without actuall buttons.

  • Djteeoh

    while it’s creative and kind of cool, it’s not practical. Who is going to buy this? Who is going to wave their hands for hours to make music? Cool but far from useful.

  • Djteeoh

    while it’s creative and kind of cool, it’s not practical. Who is going to buy this? Who is going to wave their hands for hours to make music? Cool but far from useful.

  • Djcl_ear

    “Space Palette is just the first of many new controllers (both musical and visual) that will be based on it”.

    This is not correct.
    So far, it looks like Space Pallete is a developer’s prototype first shown last weekend.
    However -a week earlier- Ryan Challinor released Synapse, a working kinect application (for PC and Mac).

    It is free and available for download @ http://synapsekinect.tumblr.com/post/6307790318/synapse-for-kinect

    You just need a Kinect device (no Xbox required) and Ableton’s Live 8 installed for it to function. This one does skeleton tracking and so you may actually do dance moves to control the music :), without the need of a carved wooden panel

    There’s a video http://www.youtube.com/watch?v=teHCHsjxI00

    And a timed post about it at Kinect official developer’s forum http://social.msdn.microsoft.com/Forums/en-US/kinectsdk/thread/f7b77162-32b8-4623-87eb-1870bdd45224

  • SkyTek

    I dont understand the fascination with this virtual Minority Report DJ style.
    Dont get me wrong I love progress, and I use  controller. But part of the fun of DJing is the feel of the physical equipment, I think its cool but “everything that glitters aint gold”.

    • Yeah but I don’t think he was ever claiming to be a real alternative to turntables, mixer etc. I think he just did it because, hell, it’s cool, it’s funky, it’s just fun. We’re in a time at the moment where experimenting with sound and technology is far more easy then ever before, so that’s just what people are doing, experimenting. For fun.

      Also, he said straight up that’s it an algorithmic instrument, not a DJ controller 🙂

    • kaiki

      I would dig on having a controller that could be programmed to have any number of buttons, jog wheels, knobs without needing to change hardware. Just have the camera. In the near future, that would be my instrument of choice 🙂

  • Djcl.ear

    @kidalex:disqus  Tim.
    “Space Palette is just the first of many new controllers (both musical and visual) that will be based on it” is incorrect.
    This is a developer’s prototype shown last weekend (as djtechtools says).
    However Ryan Challinor actually released ynapse, a working app (for PC and Mac) one week earlier. You just need a Kinect device (no Xbox required) and Ableton’s Live installed for it to function. This one DOES skeleton tracking and so you may actually do dance moves to control the music 🙂

    It is free and available for download http://synapsekinect.tumblr.com/post/6307790318/synapse-for-kinect

    And there’s also a video http://www.youtube.com/watch?v=teHCHsjxI00

    And a timed post about it at Kinect official developer’s forum http://social.msdn.microsoft.com/Forums/en-US/kinectsdk/thread/f7b77162-32b8-4623-87eb-1870bdd45224

  • JayMan574

    Wonder how much Grass he smoked before he did that…. Maaaaannnnn. Cheezee

  • DJ transient0ne

    Is that moldover in the background at the beginning of the video? With the puke green long sleeve shirt

  • MrPopinjay

    God that mc was annoying, why was he shouting over that man’s performance? Who gives a shit how much time was left?

  • fotifo

    wonder how it would do with traktor

  • Greatmesk

    Tennisball OMG i love the tennisball 😀 greatest response to a “DJ” set ever

    • Spacecamp

      He never claims to be DJing- this was, after all, a controllerism battle, not a DJ battle. 

  • klintala

    I never understood this..  
    it seems like a major down grade from a basic midi controller..
    maybe I’m missing something but wouldn’t you have more control using your 10 fingers?  ..not 2 arms?
    the idea is cool tho..  just kinda pointless.
    it would be sweet to mix it in your set tho..  not with that huge ass tacky board tho..

    just simple stuff, like a vertical filter..  or an x-y effect controller

    • It’s kind of like asking what the point is of a painting, when you can take a picture. I see this as art, not a practical solution to my DJ needs 🙂

      • MrPopinjay

        Well said!

      • Well said indeed

      • Doughboy

        I agree, controllerism is obviously highly associated with DJING, but his “controller” is a form of musical instrument as he said, not a “DJ controller”. Nor does he claim or appear to be DJING. He is performing art, his art that he has created and loves.

  • Diego

    eannnnnn we wanna see you and the crazy midifights!!!!!

  • Very interesting indeed.
    Could you tell us or link to the part of the controllerism battle video where we can see his performance?

    I just started getting into the official Kinnect SDK for DJ means. I’ll inform you if something interesting comes out of it.

    • Tim Thompson

      If you go here – http://www.justin.tv/lovetechsf/b/288875457 – the first performance of the night using Space Palette starts around 1:39:00.

  • Insane is the only word I can think right now…

  • Finkl_00

    Tennis BALL!!!!!

  • ha ha “big fish, little fish, tennisball!!!” er…??? dosen’t quite have the same ring to “big fish , little fish, cardboard box” lol
    love the idea even though it’s so illegal whach out dude you’ll have microsoft on ya back and that that would be bad times =(

    • illegal? it might be against the licensing terms and conditions (which it almost certainly isn’t) but that doesn’t make it illegal.

      you do know that Microsoft recently released a Kinect SDK for Windows? They’re actively encouraging this kind of work 🙂

  • WOW! just WOW!   I cant wait for what the next couple of years will bring.. More DJs will be stepping out of the DJ booth and onto the stage to perform there Sets!   I reckon that you could perform a traktor set with that..   

  • great article!