Theremax released for Apple iPad
Over the last few years, everyone in Gordian has been fascinated by the iPhone and iPad, and we’re certainly not alone. In various conversations and napkin sketches, I’ve been working on a musical instrument for the iPad which takes advantage of the large touchscreen, multitouch, and the horsepower of the device. In this case, the key insight behind the creation of Theremax was “How cool would it be if I could SEE the music around me, and play along with it in a visually intuitive fashion?” Like a lot of interesting questions, this led a number of other questions about how we could make the messy, large data of real sound manageable in an intuitive fashion, how best to visualize it, and how to make playing music on a touchscreen interactive in a way that would help people make music.
Plotting musical notes:
One of my favorite tools (since Dave Elrod recommended them to me in a conversation about stereo MPEG compression years back) has been the Fourier Transform and associated frequency-domain tricks. In our current FFT implementation, we’re discovering the challenge that a lot of folks who’ve written tuners, scopes, and frequency analyzers before us have already encountered: real audio data is messy. The instruments we’re using provide a whole bunch of exciting peaks that seems to psycho-acoustically “average” to the identifiable note we hear. Real world data is complex. You can whistle (a relatively simple note, unless you’re an extremely talented whistler), and see a single peak- at the note you whistled – on the screen. You can tap the location, and it matches your whistle… but the peaks Theremax produce are already much messier. One of our favorite development instruments – a live sample of a Hammond B3 organ in “trememlo” mode was so messy we decided not to include it, even though it was a really cool sound. For the time being, we’re dealing with simple plot of the frequency coefficients, scaled out over a logarithmic frequency space, and stretched/trimmed to fit the zoomed area. Look for improvements in our plotting engine as we try out new technologies and filters.
A variable-precision musical instrument:
The biggest problems I encountered in my first tests were: it’s too small. In addition to doing this testing on an iPhone (we decided there simple wasn’t enough screen real estate on the iPhone to “play” Theremax correctly), there was the problem that the frequency domain data from the FFT was very difficult to interpret when visualizing the whole dynamic range. Trying to actually hit the note that corresponded to a peak was even harder. This led me to the conclusion that we needed a way to “zoom in” on the instrument.
Zoom also made it possible to dynamically add in the “fret” lines I’d wanted to supply, but couldn’t fit on the screen. Note: we all agreed that these are useful in an advisory capacity, but we decided *not* to supply support to tune notes to the nearest fret. If everyone sends us feedback on this, maybe we’ll make it an option in version 2!
Design and Implementation:
Theremax is primarily an audio app. On startup, we start up both a listening unit that pushes incoming sound data into our FFT, as well as an openAL component that handles playback. While Apple provides a powerful GUI framework, we opted for OpenGL ES, since we wanted to provide a maximum of interactivity with a minimum of overhead, and since we had little need for “standard” UI widgets.
Visual Aspects:
Working with my friend Stephanie, we settled on a vintage stereo look. Real Theremins are funky pieces of equipment that have been around a long time (in fact much longer than the look we’ve gone for), but we settled on this look because it gave us a fun metaphor for the sliders and switches we wanted to incorporate. The main interactive portion of the screen is presented as a black surface (to contrast with the “touches” and the plotted audio data), both of which we’ve tried to make reminiscent of analog effects. The instrument scale is a draggable, pinchable lit bar that lets you select the ends of the visible range at any given moment in an intuitive fashion.
Experiences with the AppStore and the Apple release process:
I have to say that while each new release of XCode I’ve downloaded has made the process of writing software for the iPad easier, it’s not exactly an intuitive process. I found actually submitting the application a far more challenging experience than actually writing the code. Apple requires a number of steps to create binaries for submissions (cert and signing steps, adding a profile to your project, version and platform toggles which are not really well documented) that make submission quite difficult. Our initial submission was rejected because of “lack of iPhone screenshot” even though we hadn’t intended to release for iPhone. The actual problem was that the main project and each target in the project have hierarchical platform settings, and we’d left iPad+iPhone specified in one target. I have to guess that a lot of my issues were newbie problems, but the first time through is certainly not a smooth or intuitive process.
Bugs:
The initial Theremax release contained a dumb bug that had been fixed in code, but missed the release artifact due to a mistake in our archive->upload process (newer releases of XCode have made this easier!). After receiving a number of bug reports that the instrument slider wasn’t working in a few orientations, we immediately submitted our fix, and then climbed the wall for a week, waiting for this fix to get our to our users. It was finally approved, but only after making a number of folks unhappy, and making us feel terrible. After all this, we discovered the entry here stating that you can request an expedited review for critical bug fixes.
the 4.2 upgrade:
Even though I’d been testing along with the 4.2 upgrade in the simulator (and things had worked fine), I decided to hold off releasing the application for 4.2 until I’d had a chance to test it on a real iPad, running a final copy of iOS 4.2. The application looked good in simulation, and I’d secretly hoped that no 4.2 compatibility release would be necessary. The release came out about a month after the initial application release. Sadly, after getting 4.2.1 installed (a trial involving spending an hour locked out of my iPad because Apple’s servers were too jammed-up to validate my installation after it was completed), Theremax was experiencing byzantine failures on launch. I immediately began digging in, only to find out that my copy of XCode had some problems with the debugging console and 4.2.1, so getting good information about the problem was hard. Eventually, I discovered an off-by-one error that hadn’t been triggered in 3.2 or in the simulator for 4.2, and uploaded a fix. Another wait to push this (critical) fix ensued.
Next Steps:
Synth Enhancements:
The sound in this first version is dead simple. We’re talking casiotone simple. That’s not necessarily a bad thing, considering how much fun everyone has found their old Casios, but the iPad is capable of so much more. I’m a novice music programmer, but I’ve learned a lot of lessons since taking on this challenge, and I’ve received a lot of helpful advice from Rob, who designed our first three instruments. I can’t wait to put some of these lessons to work in providing a richer, more professional sound engine in the next version.
MIDI:
We’ve wanted to add MIDI support right from the start… and the new iOS SDK has obliged by making that easy. That implementation isn’t ready for prime time yet, but we’ll allow you to control remote MIDI devices (e.g. much better synthesizers than we’ve built into Theremax), and receive MIDI input to paint “target” reticles where you’ll need to tap to play along, or where an external harmony is going on for you to riff off of!
Fret Labels:
Even before a full version 2, the most overwhelming feedback we’ve received is that the fret lines are useful, but they could be a lot more helpful if they told you where you were, not just the relative positions of notes. This should be pretty straightforward, and the subject of a “point” release coming soon!
Enhanced sound plotting:
We’re not finished refining the audio plotting technology in Theremax… watch the Theremax page or follow us on twitter for updates on this as we find new ways to visualize the music!
