I followed a 3 day workshop on NuPlay at WORM recently, by Daniel van Gils. NuPlay is a new, free, open source tool which makes it possible to connect all kinds of external gear (midi, sensors, sound) to the quake 4 (id tech 4) game engine without beeing a scripting guru. You can design your own level inside the quake 4 level editor and trigger all kinds of homebrewn objects or manipulate parameters inside this level. By doing this, you can make real time a/v performances or installations. The cool thing is you can use the powerfull real time render engine, physics, lighting, a.i., 3D sound etc. of id tech 4, within your own projects.
I made this little project in 3 days during the workshop:
How does it work?:
Max sends midi to ableton live, triggering some operator synths and...
Live generates a string of midinotes with velocity 0 to 20 and back to 0. The velocity channel is used to tell the game engine which frame of the loop will be displayed. In this case, a loop of a growing flower is playing from frame 0 to 20 and back. So you can re-animate the flower by changing the velocities. Framerate can be changed by changing bpm in live. This technique also allows you to change in and out points of the loop live. Didn't test this yet.
This string of midinotes is send to the NuPlay midi client, which sends it to the right object trigger inside the game engine.
Now i can trigger my 8 flowers live with the monome, while sound and image stay synchronized. I can still walk around in this level.
You can also use 3d sound inside the game engine (now im just triggering ableton synths). By doing this your position in the level will determine the actual sound mix in surround sound. Didn't test this yet, but think it might be interesting.
This was just my first test in using a game engine for a/v performances. I'll definitely do more research on this because i think there are a lot of possibilities re-using this advanced real time game technologies. (although this engine is kind of old(2005)). I hope the id tech 5 engine with user frendlier level editor will be public soon!