Implementation Strategy

Background

The demand for tightly integrated modalities based upon immanent elements leads to the identification of two key design sites:
Lower level components.
Generic ‘meta-mechanisms’ rather than numerous features.
Lower level components may be necessary to support tight interaction between het-erogeneous modalities and also to encourage the exploration of computational audiovi-sual potential. For example, in the design of a computer game, spatializable sound file playback may be sufficient, however the elevated role of computation in algorithmic composition suggests a lower level approach in terms of signal processing.
The meta-mechanism terminology is drawn from Lua itself. Lua’s authors avoid a language bloated by numerous specialized features by providing a small set of compo-nents with which a user can build desired features in the way he or she wants them. For example, Lua offers no object-oriented class inheritance system, however many varia-tions of object inheritance can be built upon its metatable construct. Meta-mechanisms avoid preconceived behavior through generality and thus encourages composers to ex-plicitly specify the behavioral features of their works. Generalizing mechanism may also support cross-modality.
On the Choice of a Programming Language The design of programming envi-ronments for audiovisual composition can be broadly divided into two camps: visual Graphical User Interface (GUI) environments and text-based environments utilizing domain specific languages. While generally more approachable for novice users, GUI-based environments such as Max/MSP/Jitter and Pd/GEM (Puckette, 2002) lack many of the formal notions a programming language embodies. Such environments more specifically address task scheduling and message passing rather than the computational generality of our endeavor.2 In contrast, a programming language as primary interface addresses the design sites listed above while making use of notions immanent to com-putation itself.
Regarding composition, Roads (2001) identifies two key benefits of programming language representations: ”First, the compositional logic is made explicit, creating a system with a degree of formal consistency. Second, rather than abdicating decision-making to the computer, composers can use procedures to extend control over many more processes than they could manage with manual techniques.”
Using a programming language as the primary artistic interface leads to a notion of composition-as-script. This notion is not new: a number of contemporary tools em-phasizing the aesthetic role of computation such as SuperCollider (McCartney, 2002) and Fluxus (Griffiths, 2007) exemplify the same concerns for independently musical or spatiotemporal media.
Furthermore we note that many of these tools build upon existing languages rather than invent new ones. This leads to a number of clear benefits:
Documentation easing learning curves
Potentially numerous extension libraries adding additional capability for free Code re-use: repositories and portability
Revision, debugging and profiling minimizing error and improving user experience Facilities formally verified in the community, supporting future scalability
Given these advantages, writing a new language only seems appropriate if no ex-isting language can satisfy the demands of a domain. A preferable solution is to use a flexible language that is clearly designed for extension to specific domains: a domain oriented language (Thomas and Barry, 2003).
Lua The Lua programming language has been designed throughout as an ‘extensible extension language.’ Extensible refers to the ease by which new semantic and functional features can be added within Lua or through underlying C functions, while extension refers to the ease by which Lua can be embedded within a host application as a higher-level configuration and control language (Adobe Lightroom and World of Warcraft are notable examples (Ierusalimschy et al., 2007)). The author of SuperCollider states that an extensible language allows the programmer to “focus only on the problem and less on satisfying the constraints and limitations of the language’s abstractions and the com-puter’s hardware” (McCartney, 2002).
Typical use of Lua as an extension languages combines libraries of performance sensitive code and key data structures modeled in C/C++ with the dynamic aspects of the application such as control flow and interface configuration written in Lua. Together with a number of generic benefits summarized in Table 1, these features make Lua ideal for the development of the domain-specific language in our project.
Data-description language Suitable as a readable and enactive document format High-level programming features Compact descriptions of complex algorithms and relationships, support for both imperative and functional styles
Incremental garbage collector Driven by game-programming needs, suitable for real-time performance
Well-regarded performance Amongst the fastest languages of its class
Ease of development ANSI C, small library, well defined API
Flexible license Compatible with both GPL and commercial use

Contributions

We have extended the Lua programming language into the domains of temporal, spa-tial, and sonic composition. These extensions, and the associated host application, are together called LuaAV. This is the focal point for our explorations in computational audiovisual composition as a medium for compositional, technical, and philosophical inquiry within the audiovisual arts.
The core application of LuaAV is a simple platform upon which users can define custom application environments (such as window arrangement and menu structure) and execute Lua scripts in order to actualize compositions into performances. These scripts make use of the grammar and vocabulary added by our domain-specific exten-sions. The extensions themselves are embodied as dynamically loadable Lua libraries (modules), making them independent of the core application. Consequently, any other application embedding Lua that allows a user to load scripts and modules will be able to execute scripts written using LuaAV. The subsequent sections of this paper detail the development of these modules.
LuaAV is intended to be cross-platform. OS dependent support resources such as windowing and file management are currently implemented on the OSX native Cocoa (Apple Computer – Cocoa, 2008) framework as well as the cross-platform GLUT (Kil-gard, 2008) library, but will be mirrored with native resources for Linux and Windows in the near future. A public website of software resources, documentation and com-munity pages related to LuaAV, and computational audiovisual composition using Lua in general, can be found at http://www.mat.ucsb.edu/lua-av, with a related community mailing list at lua-av@mat.ucsb.edu.

Temporal Structure

Sharing temporal mechanisms between sonic and spatial structures clearly supports generality, thus we begin with temporal support in the form of the Lua module time. This module must provide a low-level basis for the gradual determination of audiovisual output, and present a high-level interface for the gradual articulation of temporal expression, making use of notions immanent to computational audiovisual composition.

Computational Time and Composition Time

The fundamental force of time in computer architecture is the inexorable movement from one discrete instruction to the next. Computational ideas must be encoded into an executing state space via a series of such instructions. Any desired temporal structure in the executing performance process must emerge from this state space. From the stand-point of the developer, the problem can be restated as a low-level, generic mapping of composition time, i.e. the flow of articulation in performance, to computing time, i.e. the determined series of instructions.
We also note that the computational instruction flow is topological, in that the tem-poral distance between events is unstated. In practice individual instructions execute at a rate beyond the threshold of perception. To produce a real-time performance we must slow this process to a desired temporal metric.
In summary, computational articulation of temporal structure involves specification of the ordering and spacing of events, in turn demanding support for scheduled activity and measured durations respectively.
Event Spacing: Measured Durations In order to allow specific temporal intervals to elapse between selected instructions, we need to provide a metric of time to measure against. The computational audiovisual domain suggests at least three sources for this metric: system time (based upon CPU hardware), frame-time (counting image buffer swapping events in the display system), and audio sample-time (based upon the sample-rate clock in the audio driver). These clocks are independent: audio clocks may drift slightly compared to system timers (Bencina, 2003), and frame-rate may vary adap-tively with windowing system resources. In addition we may need user-space metrics within a composition, such as rhythmic tempo, as abstract temporal structures within and against which aesthetic form may occur. A user-space metric may be independent or synchronize with an existing clock.
The time module answers these demands by providing a generalized accumulator as a clock type, making no assumptions as to the semantics of the clock units (they could be nanoseconds, seconds, frames, samples, beats, mouse-clicks etc.) These accumulators can be polled for current value using the now() method, and manually incremented using the advance(n) method. To support synchronous relationships, clocks can be nested such that one becomes the driving parent of another: whenever the parent advances, so does the child. A relative offset and dynamically variable rate characterizes the child-parent relation.
In addition we implement a special clock available through time.system using nanosecond units and offering high-precision access to the CPU clock. This system clock supports two alternatives to manual increment: the update() method polls the sys-tem timer and advances all child clocks accordingly (suitable for event-driven hosts), while the execute() method hands over control of the application to the system timer which makes low-level system sleep calls to control execution in real-time.
Event Ordering: Scheduled Activities Explicit articulation of temporal structure in computation must be built upon operations that divert the implicit, ephemeral causality between instructions. We identify four general classes of such operations, summarized in table 2.
Based upon this analysis, we chose a scheduling model that makes particular use of Lua’s inherent support for concurrency and events. Specifically, we extend Lua corou-tines3 by inserting conditional scheduling within the yield-resume mechanism, based upon either the clocks described above or arbitrary event notifications. The body of a coroutine can encapsulate any valid Lua code and thus any combination of memory and control flow instructions, completing coverage of the operation classes identified.

Implementation: Coroutine Scheduling

A coroutine can be created under control of the time module using the go function, which takes a function and an optional list of arguments. Any active coroutine can yield its execution using the wait function, whose argument is either a numeric duration (relative to a specified or default clock), or an arbitrary Lua value as an event token. In either case, this call suspends the coroutine and places it under the ownership of a condition within the time module scheduler.
An event notification may be triggered using the event function, whose first argu-ment is the Lua value event token in question (see script 1). Event tokens can be tables, functions, strings or any other valid Lua type except numbers. Any coroutines pending
function clickwatcher()
while true do — infinite loop
local result = wait(’click’)
print(‘a click occurred:’, result)
end
end
go(clickwatcher) — launch a coroutine
event(‘click’, ‘left’)
event(‘click’, ‘right’)
Script 1: Using go, wait and event for event-based conditions. The clickwatcher function becomes the body of a coroutine when launched using go, resuming upon a ‘click’ event token notification via event. The additional arguments ‘left’ and ‘right’ are passed as return values to the wait call, and populate the result variable, for event-handling purposes.