Cours computational audiovisual composition using Lua

Introduction

In this paper we document extensions to the Lua programming language to support time-based audiovisual composition. Significantly, these extensions enable the tight real-time integration of computation, time, sound and space, and follow a modus operandi of development going back to immanent properties of the domain.
In general terms, we are interested in enabling and encouraging audiovisual com-position with an elevated aesthetic role of computation beyond the computer-aided or computer-assisted. In particular, we are focusing on a computer programming language.

Implementation Strategy

Background

The demand for tightly integrated modalities based upon immanent elements leads to the identification of two key design sites:
Lower level components.
Generic ‘meta-mechanisms’ rather than numerous features.
Lower level components may be necessary to support tight interaction between het-erogeneous modalities and also to encourage the exploration of computational audiovi-sual potential. For example, in the design of a computer game, spatializable sound file playback may be sufficient, however the elevated role of computation in algorithmic composition suggests a lower level approach in terms of signal processing.

Generality of our endeavor

In contrast, a programming language as primary interface addresses the design sites listed above while making use of notions immanent to com-putation itself.
Regarding composition, Roads (2001) identifies two key benefits of programming language representations: ”First, the compositional logic is made explicit, creating a system with a degree of formal consistency. Second, rather than abdicating decision-making to the computer, composers can use procedures to extend control over many more processes than they could manage with manual techniques.”

Contributions

We have extended the Lua programming language into the domains of temporal, spa-tial, and sonic composition. These extensions, and the associated host application, are together called LuaAV. This is the focal point for our explorations in computational audiovisual composition as a medium for compositional, technical, and philosophical inquiry within the audiovisual arts.
The core application of LuaAV is a simple platform upon which users can define custom application environments (such as window arrangement and menu structure) and execute Lua scripts in order to actualize compositions into performances. These scripts make use of the grammar and vocabulary added by our domain-specific exten-sions. The extensions themselves are embodied as dynamically loadable Lua libraries (modules), making them independent of the core application. Consequently, any other application embedding Lua that allows a user to load scripts and modules will be able to execute scripts written using LuaAV. The subsequent sections of this paper detail the development of these modules.

Temporal Structure

Sharing temporal mechanisms between sonic and spatial structures clearly supports generality, thus we begin with temporal support in the form of the Lua module time…

Computational Time and Composition Time
The fundamental force of time in computer architecture is the inexorable movement from one discrete instruction to the next. Computational ideas must be encoded into an executing state space via a series of such instructions. Any desired temporal structure in the executing performance process must emerge from this state space. From the stand-point of the developer, the problem can be restated as a low-level, generic mapping of composition time, i.e. the flow of articulation in performance, to computing time, i.e. the determined series of instructions.

Implementation: Coroutine Scheduling
A coroutine can be created under control of the time module using the go function, which takes a function and an optional list of arguments. Any active coroutine can yield its execution using the wait function, whose argument is either a numeric duration (relative to a specified or default clock), or an arbitrary Lua value as an event token.
In either case, this call suspends the coroutine and places it under the ownership of a condition within the time module scheduler.

Concurrency: Future Directions
Readers might note the absence of multithreading in the discussion of concurrency above. A Lua interpreter itself cannot be safely used across multiple operating system threads (a design decision of the Lua authors).
In contrast to coroutines, the scheduling of operating system threads is predominantly outside the scope of the executing pro-gram, introducing indeterminism in the timing of instructions between threads that pre-vents micro-temporal interdependencies. Safely sharing data structures between threads demands nontrivial solutions that are not conducive to a transparent programming inter-face (Lee, 2006). The call for tightly interleaved interaction between sonic and visual processes is very difficult to maintain with indeterminately scheduled threading.

…….