Nonlinear Audiosystems Design Tools

Audio programming, nonlinear audio design
may 18, 2020


As a game audio designer I am hindered during my creative process, due to the discrepancy between the nonlinearity of my work and the linear character of standard audio production software.

Standard linear audio production tools offer little to none nonlinear sequencing, transitioning, parameter adaption and probability functionalities. Furthermore, linear sequencers are not optimised to provide a visual representation of nonlinear systems. Current standard middleware solutions are more focussed on optimising implementation and less on tackling the obstructions in the design stage of a project. These issues cause testing and prototyping to take unnecessary long and obscures communication with collaborators. This obstructs the workflow and discourages innovation.

Nonlinear Audiosystems Design Tools or NADT, is a range of experiments in the form of prototypes for tools to tackle these issues. By prototyping various tools to address issues caused by the nonlinearity of video games in relation to the linearity of audio production software, I hope to improve the process of designing and testing nonlinear audiosystems. The project has been created as part of my thesis on Tools for Designing Nonlinear Game Audio Systems.

Main issues

The current issues that arise are divisible into 3 general subjects: The inability to quickly test and prototype audio systems with their audio, the lack of overview current standard tools provide (which also hinders communication) and the lack of visual and interactive context.

With an approach that favours quick testing and prototyping, I hope to approach the problem from multiple angles. This should provide a more general sense of what works well and what doesn't, before focussing on specific elements. The main disadvantage of this approach however, is that many interesting details will not (yet) be addressed.

Modular 'flexible' tools

To be able to function as a proof of concept on how to tackle the issues, the tools need to be flexible and can't be specific to any one game or phase in game audio.

'.. the game industry is undergoing constant change .. . The only solution to this unpredictable situation is granularity--that is , the goal to build a flexible set of tools that are constantly able to update, evolve and adapt' - Florian Füsslin, Audio Director at Crytek (Game Audio Programming: Principles and Practices)

Current status

This project is still in progress. So far, I have worked on 3 experiments. All of which have reached their research goals, but can be much further explored in the future.

Click bellow to cycle between the different experiments done so far.

Automatic sound loading and parsing

The first experiment is a Python terminal application, that allows the user to quickly test quantised nonlinear audio systems. The project tackles the disturbance caused by the time and effort spent from creating audio to being able to hear and test it within its nonlinear system (and eventually within the context of the game). Export audio directly into specific folders using specific file and folder names and it will automatically sequence and layer the audio based on the data obtained from the file and folder titles.

The project tackles the disturbance caused by the time and effort spent from creating audio to being able to hear and test it within its nonlinear system (and eventually within the context of the game). Directly loading the audio with given data about transitions and layering in the file and folder names, allows for quicker testing. The video shows this experiment demonstrated on an early musical concept for Rookery, a game currently being developed. Click here for more information on the project.

Quantised synchronisation

I decided to start with quantised grid-based bar synchronisation, based on the following quote from Colin Walder:

'For basic synchronization, I start with bars, beats and grid since it is trivial to prepare the interactive music to support these, and because the are they are the most straightforward to understand in terms of mapping to gameplay.' (Game Audio Programming: Principles and Practices)

The system to be tested with this tool, is based on layers containing different variations. Audio is sequenced on a given bpm and loop length. For every layer a variation is chosen and played. The data obtained from a filename indicates to what variations a variation within a layer can transition. This means that every layer transitions independent of each other. Because of this, every variation of every layer needs to be loopable over each other.

File loading

To add a layer, a folder needs to be created in the ‘Bounces’ folder. The title of every folder should end with an underscore followed by the number of the layer. Within every folder, audiofiles containing variation-loops can be put. A variation-loops filename should start with the number of the variation, followed by its title, followed by the numbers of variations it can transition to, all separated by underscores.


Status

Quick testing, by directly bouncing the audio with specific titles, saves time. However, the options are still very limited and file names do need to be named very specifically. User input could be completely avoided by obtaining bpm and the amount of beats from the folder or file titles as well. This system is very simple and can only test a specific type of nonlinear sequencing. However, it has achieved its goal in testing this type of approach.



Visual Parameter Adaption

VPA is a prototype tool that uses a visual overview and interface along with the automatic file loading functionality carried over from the previous experiment, to look into upgrading the process of prototyping nonlinear systems and improving (interdisciplinary) communication. Because audio is a nonvisual medium and therefore always is experienced linearly, it can prove difficult to explain or fully understand nonlinear audiosystems design.

Files are loaded from a given folder location and information on layers and transition-possibilities is obtained from them. Layerboxes can be spawned from the Spawner. A layerbox holds a layer with all its variations. When dragged into the sequencer the layerbox is given a X and Y parameter. These parameters can be set to certain values by moving the X and Y sliders. The transitions between variations within layers aren't visualised (yet), but behave in the same way as the previous experiment where the numbers at the end of a filename indicate to what variations a variation can transition. As the current prototype is bound to time restrictions and made to quickly test the most important features, I have created a mockup showcasing some future possibilities. The mockup shows how components can dynamically be loaded in, based on what is useful for the project. This also makes it easier to add components in the future.

As visual context is an important part of game audio, a component to load in visual context has been added. In the future I'd like the program to style the entire interface based on input visuals. I'd also like to add some simple 'template' games, so that the audio can be tested in its interactive context as well. The 'Layer Containers' now show what variation of the layer is being played. As oneshots are another important context to test with looping audio, a component to play sound effects has also been added. In the future I'd like to add sidechaining and randomisation (pitch, gain, etc) functionalities.

Project Rookery

The video shows the project used to test a musical concept for project Rookery. Rookery is a walking simulator set in the Victorian era in London, currently under development. Click here for more information on the project.

Systems design

The current prototype has 4 main components. The Fileloader loads audiofiles and parses information obtained from their names. The Spawner allows the user to spawn Layers containing horizontal loops with variations, that can then be dragged into the Sequencer. The Sequencer checks if certain parameters are checked to see if and what sound needs to be played. The Controller eventually calls the audio. Below is a general overview of the current system.

Improvements

As an investigation into using a visual approach to upgrade the process of designing nonlinear audio for games and communicating about them, VPA has achieved its goal. However, the project only scratches the surface of nonlinear game audio possibilities and its current version conditions the user quite heavily into using certain techniques. Moreover, the current system isn't easily adaptable enough, mainly due to the Unity based 'objectcomponents approach'. This is also one of the reasons for the next (third) experiment completely stepping away from this approach.



Procedural audiosystems design and implementation

The latest experiment is a prototype for a framework/tool designed to improve procedural music systems prototyping and implementing. Even though procedural sound design has become somewhat standard in game audio, I find procedural music is still relatively underrepresented. It can be difficult, time consuming and expensive to prototype, test and implement. Because it can be difficult to test, it often is a risk to even attempt procedural music. Procedural music won't work well for every game, but because it is the most dynamic form of nonlinear music, I believe it should be easier to try out. Games such as Mini Metro and Monkey Island have successfully showcased how procedural music can exponentially elevate an entire game.

PAS allows for quick testing and implementing procedural music, using a sample-based sequencer. The automatic file loading functionally has again been carried over from the first experiment. This project shies away from a visual approach, to put more focus on quick implementation and building a general modular framework. PAS's more universal approach and lack of visuals makes it an easy library to implement in any game. This does however lead to the programmer also having to be the composer and vice versa.

Procedural Imagination

PAS is made as a general framework/tool, but is also used for and tested on 'Procedural Imagination'. A procedurally generated game world that dynamically responds to the player's actions. Currently being developed by Josien Vos.

Systems design

External gamedata is used to trigger and generate Cycles. These Cycles set audio parameters for Layers. Layers hold audiofiles and data on when and how to play them. The Sequencer receives ticks from the Clock and checks variables within Layers to trigger audiofiles.

×

For a more intricate audiosystem, checking what to play and calling the sounds should be split to reduce the risk of delays. Click here for a more in depth dataflow overview.

Layers

The loaded audio is divided into vertical layers, to be played independently or over each other. A layer can consist of vertical loops or oneshot sound effects. A layer holds data on how, when and what variation to play.

Generating audiodata

Layers contain 'musical values', among which a rhythm and a tonal layer. These rhythms and melodies are used in the sequencer to check when and what file to play. The rhythm calls on what ticks audio needs to be played. The tonal layer decides what notes need to be played, based on the Layertype (melody, countermelody, chord, percussion, etc) and the available samples within the layer.

Cycles

A Cycle holds Parameters, that hold values to adjust, values to adjust to and what layers to adjust. Using a Cycle is the only way to adapt audiodata. Cycles can be triggered from game events using a AdaptionMoment or from the sequencer, using a CycleTimer. A DynamicCycle only sets the on/off value of layers. Using a CycleTimer the DynamicCycle can create a natural sounding buildup. For now this buildup is mostly random. In the future specific dynamic structures could be given and/or influenced by external parameters. This system is build so that any one parameter or game event can easily be linked to multiple aspects of audiodata. This prevents conditioning the user into using certain systems.

Unity & C#

To allow for quick testing and implementing, PAS is written in C# for Unity. However, because PAS should serve as a general framework, PAS moves away from a component based Unity hierarchy and uses little to none Unity-specific elements. Only the ProceduralAudio component needs to be placed in a scene.

Improvements

The most important goal for this experiment is to investigate improving the audiosystems designing process and encouraging experimentation by removing the obstructions in the workflow. The current MVP for this project have been achieved. However, the project could be further developed indefinitely. Because the focus lies on quick prototyping a lot of interesting features have been overlooked, the most important features I'd like to add in the near future are: