Building a node-based sequencer for the web

Thanks to the Web Audio API music-making applications are everywhere on the web now. A blog post about building such an app with diagram-js and p5.sound.

Gitter
Fig. 1: The Gitter application

After hacking bpmn-js and turning it into a node-based sequencer during a hackathon last year I wanted to re-build the application using diagram-js, the diagram library bmpn-js is built upon. If you’re interested in seeing the finished application you can try it out here.

What is a node-based sequencer anyway?

Sequencers are an integral part of lots of music-making hardware and software. They control a sequence of musical events which can be edited and usually also recorded. In a drum machine, for example, it controls which drum sound is played at which time and then loops this sequence infinitely. The most common layout which can be seen in Fig. 2 is a two-dimensional matrix of buttons that represent the individual steps of each sound.

Gitter
Fig. 2: A typical sequencer

Another way of showing the individual sequences is a net of nodes. Here the distance between elements determines the sequence. Such a net can be seen in Fig. 3. It is made up of emitters which represent the start of a sequence and listeners representing a step and holding a sound. The matrix below contains the same information presented in a different way.

Gitter
Gitter
Fig. 3: Node-based sequencer

What makes a node-based sequencer fascinating is that by moving one element you can change many sequences at the same time often leading to unexpected but interesting results.

Building a diagram editor with diagram-js

Building an application for showing and editing diagrams on the web can take a long time when done from scratch. Usually, the way such applications work is very similar. This is where diagram-js comes into play. It takes the greatest common divisor and wraps it in a library that can be used for building diagram editors of many kinds.

What you get out of the box

diagram-js comes with a lot of functionality you can use right away like a scrollable and zoomable canvas, the awareness of shapes that are connected and have parent-child relationships, creating and deleting elements, moving and resizing elements and tools like a lasso tool.

Gitter
Fig. 4: The lasso tool comes with diagram-js

diagram-js uses the powerful concept of dependency injection enabled by didi. When instantiating your editor you can specify which modules you want to use besides the core modules like the event bus and the canvas. You can use both existing modules like the lasso tool and your own modules to create your custom editor. Fig. 5 shows how you can create your own editor and instantiate it with the desired modules.

import Diagram from 'diagram-js';

// diagram-js modules
import lassoToolModule from 'diagram-js/lib/features/lasso-tool';

// my editor modules
import myModule from './features/myFeature';

class MyDiagramEditor extends Diagram {
  constructor() {
    const diagramModules = [
      lassoToolModule,
      // ...
    ];

    const myEditorModules = [
      myModule,
      // ...
    ];

    super({
      modules: [
        ...diagramModules,
        ...gitterModules
      ]
    });
  }
}

const editor = new MyDiagramEditor({
  container: '#canvas'
});

// access my instantiated module
const myModuleInstance = editor.get('myModule');
myModuleInstance.sayHello();
Fig. 5: Instantiating your custom editor

You can see the full implementation of the Gitter editor here and its instantiation here.

One thing that took me quite while to understand is the syntax of specifying your module. A module is an object with properties that the injector will look at in order to figure out what your module contains and how to use it. Fig. 6 shows an example of a module with some functionality.

module.exports = {
  myModule: [ 'type', MyModule ],
  myFactory: [ 'factory', myFactory ],
  myValue: [ 'value', 'foo' ]
};
Fig. 6: A simple module

You can specify three different types of properties. type specifies a constructor function that will be called with new and therefore will return an instance. If you know AngularJS 1.x services, type is equivalent. factory specifies a function that will be called without new and can therefore be used as a factory function for instances. I’ve personally never used this type. Finally value can be used for any value like objects, arrays and primitive types. The value will be returned. In practice you’re going to use mostly type. Fig. 7 shows a simple module with one type that will be instantiated when creating the editor.

class MyModule {
  constructor(eventBus) {
    eventBus.on('diagram.init', () => {
      console.log('Hello from MyModule');
    });
  }
}

MyModule.$inject = [ 'eventBus' ];

module.exports = {
  __init__: [ 'myModule' ],
  myModule: [ 'type', MyModule ]
};
Fig. 7: Creating a simple module with a `type` property

There are two interesting things here. $inject is the static property of your type you’re going to use to specify which properties you want to inject into it as constructor arguments. The other interesting thing is __init__, which is not part of didi but implemented in Diagram. It simply specifies that your type shall be instantiated so you don’t have to do it manually.

When building your custom editor with diagram-js your directory structure would typically look like Fig. 8.

/core
  /MyCustomRenderer
  /MyCustomElementFactory
  /...
/features
  /MyCustomFeatureA
  /MyCustomFeatureB
  /MyCustomFeatureC
  /...
Fig. 8: Typical directory structure of a custom editor

I won’t go into detail about the implementation of Gitter with diagram-js here. If you’re interested in how it was built check out the project on GitHub.

p5.sound for painless web audio

The Web Audio API is pretty powerful but like the WebGL API, it’s not very easy to master. In the initial implementation during the hackathon, the whole application was build using the plain Web Audio API without any libraries. Things like precise scheduling and playing sounds became time-consuming to build and ended up not being very performant. For Gitter, I wanted to use one of the many libraries out there which make working with web audio a lot easier. I first tried out Tone.js but experienced performance problems with even simple things which led me to abandon it and look for something else. The next thing I tried was p5.sound an extension of p5.js, a JavaScript implementation of Processing. The extension comes with features like global transport, loops, and sounds which are easy to use.

Sequences, phrases and parts

p5.sound comes with the concept of musical sequences that can be used as building blocks for a sequencer. A phrase is the smallest building block and represents a ‘pattern of musical events over time’. Think of it as one row of the matrix I’ve shown you above. The sequence is a simple array that contains the information for each step. Fig. 9 shows a simple sequence.

function playSound(time, playbackRate) {
  mySound.rate(playbackRate);
  mySound.play(time);
}

const mySequence = [ 1, 0, 0, 0, 1, 0, 0, 0 ];

myPhrase = new p5.Phrase('myPhrase', playSound, mySequence);
Fig. 9: A simple phrase

A part is a building block that contains one or more phrases. It can be instantiated with the desired number of steps and can be looped. Fig. 10 shows a simple part with two phrases.

const myPart = new p5.Part();

myPart.addPhrase(myPhraseA);
myPart.addPhrase(myPhraseB);
Fig. 10: A simple part with two phrases

The Gitter application has one phrase for each listener that contains the sequence to be played and one part that contains all phrases and is looping infinitely. Maintaining the sequences for each listener when editing the diagram becomes easy with those building blocks provided by p5.sound. If you’re interested in the implementation have a look at Audio, Sequences and SequenceUtil.

What’s next?

Using diagram-js and p5.js made building the application in a considerably short amount of time possible by providing all the basics that I needed for building a node-based sequencer. There are still some open issues on GitHub that I may or may not tackle. In the meantime feel free to try the application and give feedback.

After having spent some time building music-making apps for the web I want to focus on getting up to speed with machine learning and more specifically deep learning in the next couple of months. Eventually I would like to combine building music-related applications with machine learning.

Published by in javascript and webaudio and tagged diagram-js, gitter and p5.js using 1319 words.