MULTIMEDIA COMPOSITION ENVIRONMENTS

Digital media composition

Designing an environment for multimedia composition in the digital domain requires constructing a model of multimedia representation. The ideal specificity of a model depends upon the problem it is trying to solve, however digital media composition is a highly unconstrained problem. A good model should balance design for efficiency with minimization of preconceptions about its potential usage.
Building a multimedia composition where audio, video, 3D graphics, MIDI, OSC, and other hardware and communication protocols need to be pulled together in real-time to realize the composition requires a good amount of technical knowledge as well as a large, modular code base. To handle the complexity of this task and allow rapid sketching of compositional ideas, a number of general purpose multimedia composition environments have been developed of which the Max/PD family [27, 19] is one of the most widely used. With a basic knowledge of the environment, a user can link together sonic and visual processes and interactively construct and audiovisual composition.

Using visual data-flow programming for multimedia composition

The Data Flow Architecture pattern [14, 20] performs sorted operations on data elements (tokens) through a directed graph of modules (nodes) connected via ports (arcs). Modules read tokens on their input ports, process them according to a module-specific internal algorithm, and then write tokens into their output ports. Tokens can flow through the network in regular rates (stream flow) or irregular rates (event flow).
The Max family implements data flow through a visual patching interface in which audio is processed in stream flow, while other data types are processed in event flow. Users connect processing nodes together via ‘patch cords’, analogous to patching together a modular synthesizer. While this type of environment facilitates rapid sketching, the Max visual data-flow interface also carries some inherent limitations (Figure 1).
« Rather than a programming environment, Max is fundamentally a system for scheduling real-time tasks and managing communication among them. Programming as such is better undertaken within the tasks than by building networks of existing tasks. This can be done by writing  »externs » in C, or by importing entire interpreters… »
Extending Max by importing interpreters is a very powerful way to add new functionality to the composition environment. Interpreters bring an entirely new interface to Max with new modes of interaction and new methods of composition. Many extensions (externals) for Max exist which embed interpreted languages, whether generic languages (Javascript in js [28], LISP in maxlisp [4], Python in py/pyext [6]), or domain specific languages (such as csound~ [22] and rtcmix~ [5] for audio DSP and Ruby bindings for PD’s GridFlow graphics [3]). Each of these language bindings address particular issues of multimedia composition within Max but none of them present a unified framework for composition with audio and graphics while remaining sensitive to the performance issues of real-time systems.

Lua used as a Domain Specific Language for Multimedia

For real-time multimedia composition, interpreted languages must be expressive, efficient to work with, have a low computational cost, and be flexible in terms of how they fit into the compositional workflow. The Lua scripting language has become widely adopted by the game developer community for precisely these reasons. Lua’s authors describe Lua as an extension language [9] specifically designed to be embedded within host programs and extended by domain-specific APIs for interaction with the host program. Lua meets the needs of an extension language by providing good data description facilities (associative tables that can also function as arrays), using clear and simple syntax (suitable for non-professional users), offering flexible semantics for extensibility, being small and easy to embed, and without needing to satisfy generic programming constraints such as static type-checking.
Lua has grown from a configuration language to a full-fledged programming language, supporting higher-level features such as first-class functions and coroutines. Higher-level programming structures can be designed for the domain specific API from basic meta-mechanisms provided by Lua (e.g. by using metatables one can support many different types of inheritance). Finally, for an interpreted language, Lua fares very well in terms of speed [1] and memory usage (the virtual machine only has 38 instructions), and incorporates an incremental garbage collector suitable for real-time use.

BENEFITS AND IMPLEMENTATION

In this section we discuss some of the general benefits of embedding Lua with the Max environment. Different domains have different constraints and call for different design philosophies in their programming interfaces. In the following sections we present domain specific implementations of Lua bound Max externals for real-time graphics & audio respectively.

Textual rather than visual data representation

Lua’s extensible data description facilities make it very easy to define complex and expressive hierarchical data structures suitable for formal descriptions of multimedia content. Since the data description is textual and programmatic, it can be generative, support large numbers of objects and – most importantly for multimedia composition – it can be dynamic over time.
Furthermore, hierarchical data structures and function calls in Lua support lexical scoping, such that local and (relatively) global variables can be used (in contrast, Max variable names are almost always global).

Procedural control structures

Control structures natural to textual programming languages (if, while, for etc.) are often difficult to represent and may be inefficient in a visual data-flow patch, but can be highly expressive tools for multimedia composition.
Textual interfaces generally have much less of an overhead because events are compiled by a virtual machine into byte code rather than message-passing graph iterations.