Hit It: Augmented Reality Drumming

While absentmindedly tapping out a little drum beat on my desk the other day, I got to thinking: why can’t my desk be a better drum? Why doesn’t its surface wobble up and down when I strike it? Why shouldn’t hitting it produce something a bit more exciting than a dull thump? Why can’t the world be my drum kit? So recently I set about building an augmented reality app that would do just that.

The prototype app tracks your hands in real time to detect when you strike a surface and then uses augmented reality to make it look like the surface you hit is wobbling up and down in response.

The sound produced depends on the color of the surface you strike. For these demos, I have dark colors mapping to the bass drum, light colors mapping to the cymbal, and mids to a snare. The hue modulates the pitch, so that striking a red surface sounds different than hitting a green one. Of course since it’s all just software, the drums could be configured in countless other ways, such as having the shape of your hand control the sound, or perhaps varying the sound based on where you play it in the world.

What’s interesting about this little experiment isn’t that it lets you setup virtual drum kits using your phone and a few colorful sheets of paper, but instead that it lets you experience the world around you a little differently. It turns the world into your instrument.

Even with this prototype, it’s fun walking around striking various surfaces to see what sounds they will produce. I mean it’s fun doing this without any reality augmentations too of course, but the app gets even a non-musician like myself seeing the world a little more musically. A more polished take on this app could be a fun tool for musical experimentation, or maybe even for performing live. Could really let you play the room, so to speak (or even the crowd).

As it stands however, the app is just an experiment. It’s not reliable enough to be more than a fun tech demo. Even when the app works perfectly, it can only detect around two or three hits per second and even then it is difficult to maintain anything more than the simplest of beats. It also has to be said that holding a phone with one hand while attempting to play with the other is just plain awkward. This app is begging to be used on a headset with good realtime hand tracking.

Still, building instruments up from your environment is a neat idea and I’d really like to explore the musical possibilities of augmented reality further. AR could be a fun tool for both creating music and visualizing the music you create. Maybe even for creating Holophonor style multi-sensory experiences. And while building on existing instruments—such as percussion and strings—is a nice starting point, there’s certainly nothing to stop you from creating entirely new classes of instruments that fully embrace the technology and could never exist in physical form.