PDF using lua for audiovisual compostion

LUA

The lua~ external incorporates domain specific extensions to the Lua language for digital audio. Surprisingly few such extensions exist (notably Geiger’s research project ALUA and recent high-level bindings for the CSound API) and none that provide the rich degree of control to satisfy the authors’ needs. Following Lua’s philosophy, the audio domain extensions in lua~ were designed to provide meta-mechanisms for digital music composition rather than a variety of preconceived musical structures (as appropriate for a domain so fraught with complexity and ambiguity [4]). Computer music compositions may involve serial and parallel processes and structures in a complex web of relationships, eventually producing samples of digital audio. Crucially, such processes and structures are dynamic and possibly actively determined in time. The lua~ external therefore extends Lua’s excellent data-description and functional programming capabilities for digital audio domain in two principal areas, both evaluated under the control of a sample-accurate scheduler:
• Concurrent functional control (via coroutines)
• Signal processing (via unit generator graphs)
A lua~ object embedded in a Max patch can load and interpret Lua scripts that make use of these extended capabilities in order to receive, transform and produce MSP signals and Max messages accordingly. A code sample of a typical script is given in Figure 1.

Concurrent functional control

Concurrent functional control is based upon an extension of Lua coroutines. A coroutine represents an independent thread of execution for deterministic scheduling (also known as collaborative multi-tasking). In lua~, such coroutines are extended to be aware of the sample-clock, with a small number of additional functions to interact with the scheduler. The scheduling of control flow using coroutines in lua~ is a variation of the continuation-based enactment design pattern [13].
Coroutines are launched with the go() function, which takes a delay time as its first argument, and a function as its second argument1. Effectively a copy of this function is inserted into the scheduler’s list of parallel activities, to be activated after the delay time has elapsed. All statements in a coroutine occur instantaneously with respect to the sample clock, with the exception of wait() and play(). The wait(dur) call will yield execution of the function body for dur seconds, in which time other coroutines may execute or signal processing occur, and the now() call returns the number of seconds since the coroutine was launched. All specifications of time are sample-accurate.

Signal Processing

Because the Max/MSP SDK API does not allow dynamic instantiation of any MSP object, a different set of signal processing unit generators has been provided, based on the efficient C++ library Synz [22]. An SDK to extend the DSP vocabulary is planned as future work.
Signal processing primitives (unit generators) are created by calling library constructor functions, such as Sine(), Env(), Biquad() etc. The constructor functions may themselves take numeric or unit generator inputs as their arguments, such that for example the statement Sine(Sine(0.1) * 400 + 500) will create a basic FM synthesis graph modulating between 100 and 900Hz ten times per second. Note that basic operators (+, *, -, /, %, ^) are overloaded for unit generators to aid legibility.
The play(bus, dur, unit) call adds the unit generator unit as an input to bus for a duration of dur seconds (yielding the coroutine in between). This bus may be the global Out bus, which represents the lua~ outlets in Max/MSP, or another bus created by the programmer using Bus().
Figure 1. Code sample layering multiple pulse-trains with distinct algorithmic control of pulse duration & pulse width in each train.

Avoiding block-rate

The scheduler algorithm at the heart of the lua~ external manages the coroutines and the signal processing graphs, avoiding block-rate control limitations. The scheduler lazily evaluates graph sections only when deterministically necessary, maximizing vector-processing potential where possible. Latency between inputs and outputs is only incurred for graph sections with cycles (feedback), and can be minimized to arbitrary control rates. Lua~ thus permits a sample accurate articulation of the composition that may be dynamically deterministic. State changes that involve interpreted code to generate new signal graphs may occur sub-millisecond rates, ideal for generative microsound.

Dynamic graphs & multiplicity

In the Max visual interface, the audio graph cannot be recompiled without audible discontinuities, limiting dynamic audio processing to static, pre-allocated structures. Similarly, the maximum number of parallel voices must also be pre-allocated (e.g. poly~ arguments). In contrast, lua~ supports generative, dynamic signal graphs without discontinuities.

Optimization for real-time processing

The majority of scheduling and signal processing code is written in C++ for efficiency. To achieve sample accuracy, the lua~ interpreter necessarily runs in the high propriety audio OS thread, but the cost of interpreted code is minimized by only calling into Lua for the scheduled state change actions. The Lua memory allocator and garbage collector is optimized for real-time1, and free-list memory pools are used for audio buffers and coroutines to avoid unbounded memory allocation calls.

JIT.GL.LUA

jit.gl.lua is a 3D graphics specific binding of the Lua scripting language for the Max/Jitter environment2. jit.gl.lua provides a compromise between execution speed and flexibility when developing custom 3D graphics routines that lies between patch objects and Javascript on the one hand and custom C externals on the other. jit.gl.lua is also tightly integrated with the Jitter library and in particular the 3D graphics portion of the library, easing some of the burdens of writing 3D graphics routines.

Integration with Jitter

Objects in Jitter whose name begins with jit.gl by convention all receive notifications from a given graphics context they are attached to. Unlike the js (Javascript) object, jit.gl.lua attaches to a graphics context and provides hooks in the embedded Lua scripting environment for receiving these notifications which can be used to automatically call the script when the graphics context calls it as well as manage context dependent resources such as sets of drawing commands stored in a displaylist. Embedded Lua scripts also have access to the jit.gl.lua object they are embedded in through the global this variable. By setting the attributes of the embedding object, global OpenGL state can be managed for the entire script and selectively overridden with low-level OpenGL commands during script execution.
In addition to integration with Jitter graphics contexts, jit.gl.lua provides bindings to much of the Jitter library C functions normally only accessible when writing custom C externals. The most important ones for manipulating 3D graphics are the vecmath and drawinfo libraries. The vecmath library is a full vector and matrix math library for 3D graphics, handling vectors of length 2, 3, and 4 as well as 3×3 and 4×4 matrices. The drawinfo library contains functions for low-level manipulations of the jit.gl.texture object for binding textures to arbitrary geometry and rendering arbitrary OpenGL commands to texture.
Within jit.gl.lua Jitter objects for matrix processing and higher level OpenGL functionality are made available in similar manner to the js object along with a number of extensions. First, named Jitter objects in a patch can be referenced within a script by utilizing Jitter’s name lookup service made available through the jit.findregistered method. Second, an extended binding of the jit.submatrix object allows for the scripting of in-place submatrix processing routines with any Jitter object that processes matrices.