Anthony Mattox Interaction Design + Digital Art

I played around a bit with processing.js, an awesome bit of software that runs processing scripts in a browser using the HTML canvas element. I wanted to use it to create an animated full browser background for a web page. I couldn’t find any info on how to do this online, but came up with a solution. It may not be the most elegant, and performance starts taking a big hit at large sizes, but I wanted to share it in case anyone wanted to build on it or already has a better answer. I’ll also note that I haven’t tested this thoroughly.

Simply resizing the canvas element just stretches the rendered image so it needs to be changed within processing.

Since the processing script is essentially converted into javascript it’s possible to communicate between the processing script and other javascripts. So, I added an listener for the window resize event (using jQuery), which calls a function in the processing function to resize the canvas element.

The javascript looks like this:

// javascript (requires jQuery http://jquery.com)

var ProcessingInit = function() {
  function resizeWindow() {
    var pCanvas = Processing.getInstanceById('pCanvas');
    pCanvas.resize($(window).width(),$(window).height());
  }

  $(window).resize(resizeWindow);
  resizeWindow();
}

It’s wrapped in a function so that it can be called from processing when the application is started. ProcessingInit(); in the setup function. This makes sure the canvas element is fully instantiated before it tries to manipulate it.

It’s pretty straightforward on the processing side as well. That function simply calls the size function again. I’m not sure if that’s good practice, but it seems to work fine.

// processing

void setup() {
  size(800,600);
  ProcessingInit();
}

void resize(float X, float  Y) {
  size(X,Y);
}

I’ve been playing around with Pure Data for a while now, along with a few other pieces of audio software. The results of this can be heard in the audio for my recent flash game, Pulsus.

Pure Data is a great application which allows you to control every detail building audio applications from very basic components. Unfortunately, much of the program is very poorly documented, so I’d thought I’d share a few things I’ve managed to figure out.

Here’s a quick walkthrough to create a very basic midi controlled, polyphonic synthesizer in pure data. It doesn’t sound like much, but it’ll get things working.

Generating a Tone

This basic synthesizer will consist of two patches, one patch to generate a tone, and another which will read a midi signal and control a few instances of the tone generator.

This is the oscillator patch which generates a tone from a midi message. The tone is created on the left side and the envelope (the change in volume over the course of a note) on the right.

The tone patch has an inlet at the top, which accepts a message containing a pitch/velocity pair, and an outlet at the bottom which sends out the audio signal generated. The [unpack] object splits the pitch and velocity. The pitch is sent to a series of objects on the left side which generate the tone. The velocity is sent to the right where the envelope will be generated.

The pitch, in the form of a midi note number, is converted to a frequency in Hertz which is then sent to an oscillator, generating an sine wave.

Two messages will be sent for each midi note, the first when the note begins and the second when the note ends. The ‘note off’ message will be the same but the velocity will be 0. The velocity, how hard the note was hit, is usually used to set the volume of the note. The velocity is sent to a [select] object. If it is zero, the message is sent to release the note, otherwise to attack. Each of these send a message to a [vline~] object which sends a signal for the amplitude which either ramps up or down.

Finally, the tone is multiplied by the envelope and sent out the parent patch. When the note is hit, a message is also sent to the oscillator which resets the phase.

MIDI and Polyphony

Now that we have our oscillator we need to create a patch which will control a few instances of it. The patch will receive midi data from a hardware controller or from another piece of software. The midi settings will need to be configured for PD to receive the data.

The PD patch which takes midi input and handles the voice allocation, controlling a number of oscillators.

At the top of our patch we have a [notein] object which reads midi from channel 1. [notein] will output pitch/velocity pairs. The [poly] object handles the allocation of different voices. When pitch/velocity pairs are sent to it it sends on the data with the addition of a voice number.

In order to play multiple notes at once, the messages will be routed to a number of different oscillators, each a different ‘voice’. When [poly] receives a note, it will give the number of the first available voice. When it gets a ‘note off’ message (velocity of 0) it makes that voice available again so that that oscillator can play another note when it needs to.

Now we can route the processed messages to a number of instances of the ‘tone’ patch. We do this by packing the messages into lists and using the [route] object which sorts them based on the first element in the message. Awesome. Finally we just add all the signals from the oscillators together and send them to the audio output. It first also multiplies the signal by a amplitude set by the slider so that we can adjust the volume of the synth.

Ok, So What Now?

Now that we have a simple little synth we should be able to get some sounds. A midi signal can come from a physical midi controller or from software. You can use a program, like Ableton Live, to create a composition and then send out midi. You can also create other pure data patches to generate midi. A patch could create generative melodies, act as a sequencer, or create midi from another input such as the keyboard.

The synth at this point just plays a sine wave for each note. It’s a start, but not thrilling. To create different timbres, the [osc~] object can be replaced with something to play different waveforms and we could add modulators or a more complex envelope generator. On top of that we can run the audio signal through filters either in pure data or in other software. A bit of reverb will make it sound a little less flat.

If you have any suggestions post them in the comments. I’m certainly no expert, and I’d love to hear some other methods.

Recently I performed with a musical group, creating generative imagery as a backdrop. I decided try using quartz composer again, primarily because it renders graphics so quickly and smoothly, but also because it’s fairly easy to put something together. Unfortunately I ran into a significant roadblock. Quartz composer has no way to control elements while they are running. Although you can completely restructure and modify your script while it is running, there are no interface elements to control simple changes in values. Obviously, this makes QC so much less valuable as a performance tool.

Having already set up most of my script in Quartz Composer I decided to find a way to control it some other way. Pure Data, an application similar to Quartz but for the production of sound, allows you to easily build interfaces. The data is then sent out of pure data as midi controller values, sent through a virtual midi channel, and grabbed by Quartz Composer. From there the values can be used for whatever is necessary.

Figuring out the details took a little while and some help from my friend Michael Scoot-Nelson, but the setup is actually quite simple. In Pure Data, just set up as many controller as you need in the form of sliders or other UI elements.

In this case, I have a number of vertical sliders labelled for a particular application. Above each controller I have an object which receives a bang [r b]. In Pure Data the value from each slider is only sent when the slider is changed. Sending a bang to the value ‘b’ pushes all the values down and sends them out to the midi channel. The Sliders should be set to a range from 0-127.

The values from the sliders are plugged into the first inlet of control out objects (ctlout). The ctlout objects also need a controller number and a channel number. These are used to get the value on the other end. The channels can all be set to one, and each slider should have it’s own controller number.

In Quartz composer, a ‘Midi Controller Receiver’ object will grab the values sent from Pure Data. Viewing it’s settings in the inspector, the number of inputs can be changed. Each input is listed with an integer, these are the controller numbers which must be matched to the ones set in pure data. The channels being received can also be changed.

There is one final step to getting everything hooked up. In Pure Data, under Preferences > Midi Settings must be set to an Midi Channel that Quartz Composer is listening for. On a Mac it’s easy to set up this channel with the IAC Driver in Audio Midi Setup.

I also created this little guy here. One element I was controlling in Quartz Composer (a particle generator) could only be set on or off and I needed it to vary smoothly. This creates what is essentially a pulse width modulation simulated analog output. Pressing the ‘on’ and ‘off’ buttons do just that, and pressing the pwm toggle lets you use the slider to set it more or less on.

Metro’s turn the output to on every tenth of a second. It is then turned off (ten times a second) after a short delay (less than a tenth of a second). If the slider is at the bottom it turns back off almost immediately, so for the greater part of a second it is off. If the slider is at the top, it takes nearly a tenth of a second to turn back off, so it is mostly on. In the middle it will be on half the time. The result, at least for my particle generator, was the appearance of it’s rate changing while really it was just turning on and off very quickly. I’m not sure where else this would be useful, but here it is.

It might be a little hard to read, but if it would be useful to anyone I can upload the pd file somewhere.

Until recently I’ve just been saving all of my images out of Processing using a simple saveFrame command and throwing all of my images into one giant folder. Looking into this folder I see a few key problems. I’ve come up with a little system to help me keep better track of renderings of my generative works. Unfortunately it won’t work retroactively, but it’ll be useful for future work, and perhaps it can be helpful to some other Processors.

This very adding this very simple snipped to a Processing sketch will save out an png of the render window whenever the ‘s’ key is pressed. #### will be replaced by the current frame number.

void keyPressed() {
  if (key=='s') {
    saveFrame("name_of_sketch-####.png");
  }
}

This works out pretty nicely, but if you run the sketch twice and just happen to press ‘s’ on the same frame the first image will be overwritten. Another issue, which might not seem to bad until you have a huge library of images built up, is the order of the files. If the sketch is run many times, all the saved images will be mixed together. Ideally images saved from one run of the sketch should be named sequentially so all the images from one run can be seen together.

Adding an arbitrary value to the particular run and sticking it into the file name fixes these two issues. Many of my scripts include a function to reset the program so it can be run a few times without being completely restarted. In this case, incrementing this arbitrary global value maintains the order of a set of runs.

int G;

void setup() {
  G=floor(random(10000,90000));
  //noiseSeed(G);

}

void draw() {
}

void keyPressed() {
  if (key=='s') {
    saveFrame("name_of_sketch-"+G+"-####.png");
  } else
  if (keyCode==BACKSPACE || keyCode==DELETE) {
    // code to reset script
    G++;
  }
}

This system isn’t perfect, but is a vast organizational improvement for just a few extra lines of code

Another useful trick, if you have a script which utilizes a lot of Perlin Noise, is the set the noiseSeed to a random integer (like G in the script above), and include that in the file name. This way, if you want to rerun the script with the same parameters later, you can always set G to the integer in the file name of a particular saved image.

I’ve gotten a few questions recently about how to render Processing scripts for printing. There are a few obstacles, but there are two simple methods which work pretty well. Between the two I’ve been able to get high resolution renderings of different types of scripts without having to do anything terribly complicated. The image will be exported as either a vector (shapes) or a raster (pixels) image. Depending on the sketch one method might be much more useful.

Exporting a Scalable Vector Images

The best way to get a high resolution image from Processing is to export the sketch as a vector image. With respect to graphics and images a Vector is just a fancy word for a shape. Unlike a Raster image which records visual information in an array of pixels, vector images are composed of shapes. For some applications this can much more useful. Typefaces, many logos, and other graphics composed of flat shapes are created with vectors. The shapes in a vector image can the be colored, outlined, and styled in many other ways. Vector images can be much smaller than a rasterized version, and most importantly, they are scalable. Since the image is made of shapes it can be stretched as large as you need it and will lose no quality.

Shapes drawn with processing commands are vector shapes, and Processing includes a library to export this data as a PDF document, which can then be opened and used by any vector editing program.

To capture the vector data import the pdf library, call beginRecord(), draw your shapes, and then endRecord(). A simple code to capture a frame on a mouse press could look like this.

import processing.pdf.*;

boolean capture=false;

void setup() {
}

void draw() {
  background(255);
  if (capture==true){
    beginRecord(PDF,"vector_file.pdf");
    render();
    endRecord();
    capture=false;
  }
}

void mousePressed() {
  capture=true;
}

The script above would have all the drawing functions within the render() function. To be less obtrusive the beginRecord() and endRecord() functions can be within separate conditionals. The code between them will be run all the time and the data will be recorded only when the mouse is pressed. The new draw() function would look like this:

void draw() {
  background(255);
  if (capture==true){
    beginRecord(PDF,"vector_file.pdf");
  }

  //
  // drawing code
  //

  if (capture==true) {
    endRecord();
    capture=false;
  }
}

Notes about exporting PDFs.

• The exported PDF will be in the sketches folder and titled with the second parameter of beginRecord().
• You can open the PDF with a vector program like Adobe Illustrator and edit the shapes.
• Any shapes drawn, even if they are outside the rendering window will be in the PDF. Remove the Clipping mask to reveal all the shapes.
• For fills and strokes to work they must be called after beginRecord().
• The size of the vector file is dependent on the number of vertices in your image.
• For cumulative rendering place beginRecord() in the setup and endRecord() with a key or mouse press before closing the sketch.

Exporting Large Raster Images

Using a raster image can be a better option if you have a program with a lot of complex shapes. Saving pixel images is also easier, faster, and less processor intensive, but you lose the scalability. To save an image use the saveFrame() function.

void keyPressed() {
  if (key=='s') {
    saveFrame("image-####.png");
  }
}

The #### will be replaced with the frame number. TIFF, TARGA, PNG, and JPEG files can be saved (default is TIFF), and again the image will be in the sketch’s folder. To create very large renderings you can simply increase the window size and scale up the sketches parameters as necessary. I’ve rendered sketches much larger than my screen-size with good results, but can’t promise it will work on all platforms. On a mac you can enter exposee and an extra large window will shrink to fill the screen, although you can’t interact with it.

After playing with binary cellular automata I thought I’d expand the script a little to accommodate more than two states of a pixel. Three loops are nested and each is iterated once for each state. If the index of each loop equals the three values above the current point the function returns the current state of a counter which is incremented in the innermost loop. In a binary system this is a little more complex than just testing each of the eight possible combinations, but is almost necessary when dealing with more states. I’ll post my code at the bottom of this post. But first, some pretty pictures.

Read On »

In my last post I explained how to control the brightness of multiple light emitting diodes connected to an Arduino with an interface scripted in Processing. The script which I created was great because it just took a series of values sent via USB and lit the LED’s appropriately. This is convenient because it is not specific to any input which might be needed to control the lights. The script to send a value serially along with an indicator character can be added to any Processing script. Naturally one of the first things I had to do with it was create an audio visualizer. With the Arduino programmed as it was all I had to do was use a sound library to break an audio input into frequency bands and send the values down.

The circuit is pretty straight forward. Six LED’s are connected through resistors to the six pins which support PWM (3, 6, 5, 9, 10, 11) and to a ground pin. I have everything crammed onto a tiny breadboard on my proto-sheild cause it’s cute and self contained. That’s just my style. PWM stands for Pulse Width Modulation and is the a way to control the brightness of LED’s as well as some other components. It is a digital output and produces the effect by switching on and off very quickly. The result can be visualized as a square wave. When you send a higher value to a PWM pin it will spend more time on than off. This blinking is faster than we can see so the LED appears to be changing brightness according to the amount of time it spends in the on position.

Read On »

As I started working with my shiny new Arduino board I quickly learned how to build basic circuits and program the micro controller to respond to various stimuli, but what really interested me in the technology was the ability to send information between the board and a computer. I’ve done a fair amount of programming in Processing and was excited to bring some of my work off screen. There is a nice script on the Arduino site on sending data from the board to a computer to control elements within a processing sketch. Unfortunately, the reverse is not as well documented.

I needed a script to send multiple variables from Processing to an Arduino to control a few components. After a bit of research, and more trial and error, I put together a script which controls the brightness of three LED’s through a virtual interface on a monitor. Here are my scripts and the circuit I came up with. It seems simple enough to me but if anyone has suggestions I’d love to hear them.

Here three color LED’s are connected to pins 9, 10, and 11 which can use Pulse Width Modulation (PWM) to control the brightness of each LED. They are connected to their respective pins through appropriate resistors and also connected to a grounded row in the bread board. In the image I am using the prototyping shield from Adafruit which makes it easy to build small circuits.

Read On »

My processing work in my blog as slowed to a trickle recently. This is due to a couple of large projects which I can’t wait to reveal. One is a fun web project and the other a flash application. I’ve also been playing with my fancy new toy here and am beginning to make some progress. Along with the Arduino board I’m also using a prototyping shield from Adafruit. It creates a nice little workspace with a breadboard and some extra power sources and grounds. It also contains two led’s, one of which I wired to digital pin 0 on the underside of the board. This just keeps the light on (as long as I’m not using the pin for something else) when the board has power, and makes it blink when data is being transmitted. The shield covers up the power and transmission led’s on the Arduino itself.

In the photo above I have an led display I took out of something else and, for the sake of learning, programmed. Turning the potentiometer controls the display. Nothing thrilling, but it’s good practice. I’ve also been trying to understand serial communication so I can build devices which send and receive complex data.

Anyway, there’s a quick update, and here is my Arduino code in case it might be useful to someone. It’s not put together very well. In particular there should be a 2d array containing all the numerals and their corresponding led’s. But nonetheless.

Read On »