Why Colorcoder?

Colorcoder has been programmed to sonically decrypt covert messages composed by Ellsworth Kelly in his 1951 painting ‘Colors for a Large Wall’. The information embedded within the artwork relates directly to Kelly’s service within the ‘Ghost Army’ during WW2 and more specifically to the sonic deception formulas and techniques they composed during this period. It features in the Tate Etc magazine. Issue 36: Spring 2016.

A Short History of Colorcoder

Before the artist Ellsworth Kelly became a pioneer of hard edge painting he was a member of the 23rd Headquarters Special Troops unit, also referred to as the ‘Ghost Army’. Deployed in France during World War II, the Ghost Army was a U.S. military tactical deception unit assembled by actor Douglas Fairbanks Jnr. It consisted of approximately 1,100 artists, actors, musicians, and other creative types sequestered from art schools (Kelly was studying at the Pratt Institute in New York at the time) and advertising agencies to design camouflage and to construct fake military installations, rubber tanks, and to fabricate soundscapes (acoustic techniques that were later used in the first Gulf War, when speaker systems placed behind sand dunes, dictated the movements of the battlefield by transmitting imaginary battle sounds).

The directive of the Ghost Army was to saturate the Nazi’s with duplicitous acoustic signifiers of mass troop and artillery manoeuvres (mounted on armoured vehicles, large speaker systems transmitted soundtracks in covert locations so that the presence of mass numbers of soldiers and equipment was fabricated) and to more generally transmit disinformation about the numbers, plans, and whereabouts of the allied forces. More specifically Kelly was part of the 603rd Camouflage Engineers within the wider unit alongside fashion designer Bill Blass and photographer Art Kane. For Kelly’s part, he was predominantly engaged in the creation of dummy and inflatable vehicles, air-fields, and troop encampments (so that Nazi reconnaissance aircraft would see them) and in the production of camouflage. You can see in his post-war work how he begins to deviate from conventional camouflage, gradually morphing into The Meschers of 1951, then taking the quantum leap in abstraction to produce the 8ft squared Colors for a Large Wall (in the same year) before finally honing in on this style to create 9 Squares.

In the early 50’s Kelly started imbuing his Color Field paintings with coded messages relating to experiments and research carried out during WW2. Covert information that holds clues to the composition of weaponised frequencies that were deployed by Ghost Army operatives (the first ‘battle djs) via 3 turntable set-ups. In response, AUDINT’s 1st wave of Bill Arnett, Hippolyte Morton, and Walter Slepian (who also previously served in the Ghost Army) start to mimic the 64 square formula of Colors for a Large Wall by using inverted record covers and illustrating them with the pictorial and written references that would aid their research into the painting’s meanings. AUDINT’s current sound work is an App called the ‘Colorcoder’, which is programmed to sonically decrypt Kelly’s paintings, extending the research undertaken by Arnett, Morton, and Slepian.

What is Colorcoder?

Colorcoder is a creative application that helps you to ‘sonically decrypt’ Kelly’s Colors For A Large Wall (1951) by ‘hearing’ it as well as ‘viewing’ it.

Each unique colour used within the painting has a palette of associated sounds that span the audible range and can be layered in various ways to create different sonic textures. Moving the AUDINT turntable icon around the painting – using the mouse or other interaction modes – triggers sound that is ”representative” of Kelly’s colour palette.

These various ways of moving around the painting – ‘drawing’ a freehand path from square to square using the mouse; selecting specific squares in a particular order using the grid; bouncing around the painting at different speeds and angles to cut across the squares in an irregular fashion; or following a constant horizontal path back and forth between successive rows or vertical path up and down between successive columns – equate to the different ways our eyes might scan across it, taking it in.

Where do I download it?

Colorcoder is available for download with it’s accompanying ‘README’ PDF via the AUDINT site.

How do I use it?

[expand title=”While we’ve tried to make Colorcoder intuitive to use… we’ve also produced a demo video.” startwrap=”” endwrap=”” trigclass=”highlight”]

Colorcoder – Functionality Demo from Monomatic on Vimeo.

[/expand]

Here’s a brief overview of the interface:

The ‘HSB COLOUR, WAVELENGTH & FREQUENCIES’ section shows how an area of the painting, determined by the size of the AUDINT turntable icon, is sampled and averaged and displayed as the corresponding hue, saturation and brightness values. It also shows the values of the “equivalent” wavelength and frequency of light and the scaled down audible frequency.

Adjusting the ‘ICON SIZE’ dial(s) changes the dimensions of the AUDINT turntable icon.

The ‘INTERACTION MODES’ section allows you to select from five different ways of moving within the painting – through the mouse; an interactive grid; a ‘bouncing ball’ mode (mouse left click-drag-release anywhere within the painting); and preset horizontal and vertical paths – with their respective ‘DRAG’ and ‘SPEED’ controls.

The ‘SONIC TEXTURES’ section allows you to layer up to 8 ‘OCTAVES’ of sound from the lowest audible frequency derived from the hue. Selecting new ‘OCTAVES’ will load the associated audio files if they’re not already active – “loading…” will appear above the ‘AUDIO ON’ toggle and any automatic movement of the AUDINT turntable icon will freeze for a few seconds while it completes.

Different combinations of octaves can be activated through a set of ‘OCTAVE PRESETS’ which load (bottom row) and save (top row) these custom configurations (we’ve filled the first four presets to get you started). When the application is closed any changes you’ve made to these presets are saved for the next time you open it.

This is Creative Software

This is creative software… not commercial software. Colorcoder was developed in the C++ creative toolkit openFrameworks v0.9.0. While we are happy to share it with others to play with, we do so with the proviso that it has not been rigorously tested across a range of hardware/OS versions and may not run on machines lower than the minimum spec. The application is provided “as is” and under the terms of the BSD licence below.

Minimum Spec

Colorcoder is currently only available for OS X.

It was built on a MacBook Pro (Retina, 15-inch, Late 2013), 2.3 GHz Intel Core i7, 16 GB 1600 MHz DDR3 running OS X El Capitan Version 10.11.1 and has been tested on several Apple Mac computers down to the lowest spec of a MacBook Air (11-inch, Late 2010), 1.4 GHz Intel Core 2 Duo, 2 GB 1067 MHz DDR3 running OS X Yosemite Version 10.10.5 (which worked fine bar the screen resolution).

We suggest running Colorcoder on Apple computers with the following minimum spec:

• multicore 64-bit processor;
• min 2 GB RAM (4 GB recommended);
• OS X 10.9 and above.
• min screen resolution of 1280x800px

BSD Licence

Copyright (c) 2015, MONOMATIC

All rights reserved.

Redistribution and use in binary form, with or without modification, are permitted provided that the following conditions are met:

1. Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.

THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS “AS IS” AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

FAQ

[expand title=”1. Running Colorcoder for the first time.” startwrap=”” endwrap=”” trigclass=”highlight”]

We’ll assume you know how to find the ‘Colorcoder_OSX_final’ ZIP file on your computer once you’ve downloaded it. You may want to move it to a more convenient location such as your ‘Desktop’ folder.

Double mouse left-click on the ZIP file and it will expand creating a ‘Colorcoder_OSX_final’ folder with the application and a ‘README’ text file inside. If you then launch it in the usual way via a double mouse left-click on the application icon, OS X is likely to open a dialogue box:

“Colorcoder_OSX is an application downloaded from the Internet. Are you sure you want to open it?”

You do – and you won’t need to confirm this again.

Colorcoder can take up to 30 seconds to load – be patient.

To close the application make sure it’s in focus (mouse left-click on the title bar or anywhere in the app window) and press the ‘esc’ button on your keyboard – or mouse left-click on the red dot at the top left of the title bar.

[/expand]

[expand title=”2. How does what I’m hearing relate to a colour in the painting?” startwrap=”” endwrap=”” trigclass=”highlight”]

Well… the answer to this isn’t simple.

In short, like all mappings of colour to musical tone, Colorcoder is essentially interpretive and driven by aesthetic considerations.

If you want a more detailed answer, read on…

In computational terms we:

• use the HSB colour model to sample and extract the hue, saturation and brightness values of a single pixel – or the average of an adjustable square area of pixels – under the AUDINT turntable icon;
• use a simple function to calculate the wavelength of light with a colour “equivalent” to this hue value;
• convert this wavelength to it’s corresponding frequency;
• pitch this frequency down by a factor of 2⁴⁴ – which effectively translates it to the bottom of the audible frequency range;
• ‘stack’ 8 successive octaves up from this ‘base’ frequency;
• and finally trigger audio files according to the extracted hue and the number of active octaves.

For anyone interested we’ve included some of the code for this in the ‘Can I see the source code?‘ FAQ below.

However, we make no claims to have realised a mechanism that meaningfully maps a colour to a musical tone. We are fully aware that our ‘pseudo-scientific’ approach is simplistic and deeply flawed – and we encourage anyone interested in understanding quite how simplistic and deeply flawed it is to read the following online essay on Pitch and Color Recognition by Kevin Brown.

We do appreciate that the question of whether it is indeed possible to find a meaningful mapping between colour and pitch has occupied the minds of numerous artists, scientists, philosophers and art historians for centuries. It’s a key motivation for the synaesthetic art forms of visual music, early abstract cinema and according to Michael Betancourt (2006), the “synaesthetic abstraction” of early expressionist painting. The numerous attempts to pursue this tantalising relationship found inspiration in the theories of natural philosophers that predate The Age of Enlightenment – such as Isaac Newton and his studies into the nature of light, first published in his 1704 treatise Opticks – and trace a formative lineage to the ‘invention’ in 1734 of the Ocular Harpsichord by Louis Bertrand Castel (Fransenn, 1991) and the many mechanical light-projection inventions, collectively known as “colour organs” and associated forms such as Lumia that followed.

Still, we want to make it clear that we ourselves are not particularly interested in this “synaesthetic tradition” – or in finding a meaningful mapping between colour and pitch. Our reading of the historical discourse around the many attempts to map colour to musical tone – particularly when re-focussed through the transdisciplinary ‘lens’ of modern neuroscience and cognitive psychology and the contemporary sensory-centric philosophy of the likes of Casey O’Callaghan – is that despite a deeply rooted universal meme that suggests musical notes can be visualised as colours, the scientific evidence patently indicates they can’t.

So to reiterate, Colorcoder is a piece of creative software – it’s essentially interpretive and based on aesthetic considerations.

References:

• Betancourt, M. (2006) Visual Music and the Birth of Abstract Art. [Online] [Accessed on 13th April 2013] http://www.michaelbetancourt.com/pdf/Wilson_ Lecture-Abstraction.pdf
• Brown, K (1994-2015) Pitch and Color Recognition. [Online] [Accessed on 15th April 2015] http://www.mathpages.com/home/kmath578/kmath578.htm
• Franssen, M. (1991) The Ocular Harpsichord Of Louis-Bertrand Castel. [Online] [Accessed on 14th May 2014] http://www.tbm.tudelft.nl/fileadmin/Faculteit/TBM/Over_de_Faculteit/Afdelingen/Afdeling_Values_and_Technology/sectie_filosofie/medewerkers/Maarten_Franssen/doc/OcuHarpsCastel.pdf

[/expand]

[expand title=”3. When I make the AUDINT turntable icon large it does’t seem to work properly in ‘MOUSE’ and ‘BOUNCE’ modes?” startwrap=”” endwrap=”” trigclass=”highlight”]

This is because the larger the icon the more pixels are being averaged to extract the hue. You can see this in action in the two squares at the top of the ‘HSB COLOUR, WAVELENGTH & FREQUENCY’ section – the left square displays the ‘subsection’ of the painting being sampled, the right square the averaged colour. If the resulting average colour doesn’t closely match the colour of a particular square in the painting it won’t trigger. This isn’t an issue for the ‘GRID’, ‘HORIZONTAL’ & ‘VERTICAL’ modes where the icon is always fully within a square at some point… but it is for the ‘MOUSE’ and ‘BOUNCE’ modes where it might never wholly be within a single square. Reduce the icon size to very small and it works much better in these modes.

Also see the ‘What are the ‘MIDI Out’, ‘MIDI Panic’ and other toggles in the ‘SONIC TEXTURES’ section for?‘ FAQ below. ‘ICON SIZE’ is certainly more responsive when Colorcoder is used as a MIDI device.

[/expand]

[expand title=”4. What are the ‘MIDI Out’, ‘MIDI Panic’ and other toggles in the ‘SONIC TEXTURES’ section for?” startwrap=”” endwrap=”” trigclass=”highlight”]

These are ’surplus to requirements’ for most users and can effectively be ignored. They have no effect on Colorcoder when it’s being used in ‘AUDIO ON’ mode. But if you really want to know…

MIDI or Musical Instrument Digital Interface is a technical standard that describes a protocol, digital interface and connectors that allows a wide variety of electronic musical instruments, computers and other related devices to connect and communicate with one another.

Earlier iterations of Colorcoder used the same colour to frequency computational process above but rather than trigger audio files it converted these frequencies into MIDI Note and pitchbend data (while also adding “expressive” controller data from the saturation value, volume from the brightness value and pan from the relative position of the mouse) to trigger a sound patch in the Native Instruments Massive software synthesiser.

The audio you hear in this version is the recorded output from Massive using a patch that we considered both behaved and sounded ‘best’ with the output MIDI data.

For those with a knowledge of MIDI we’ve decided to keep this functionality in Colorcoder – it actually works far more effectively as a MIDI device – although we provide no further explanation of how to use it other than set the MIDI input port of your sound source to “Colorcoder MIDI Out”.  We also include a link to a ZIP file of some of our shortlisted Massive sound patches (although you could use any MIDI software synth or hardware sound module).

[/expand]

[expand title=”5. Is Colorcoder going to be developed further?” startwrap=”” endwrap=”” trigclass=”highlight”]

A version for Windows 10 is in development and should be ready shortly.

Over and above this – possibly. Future iterations of Colorcoder may see the app developed into an interactive gallery installation or a live performance tool. Particularly since Colorcoder is essentially a general ‘sonification engine’ – it’s not specific to Ellsworth Kelly’s Colors for a Large Wall. Any image, graphic or photograph (or video for that matter) could be loaded into the app and converted to a sonic texture somewhat representative of the colour within a given area of the screen.
[/expand]

[expand title=”6. Do you offer any support for Colorcoder?” startwrap=”” endwrap=”” trigclass=”highlight”]

No. Colorcoder is a zero-budget project and accordingly we just don’t have the resources to offer any ongoing support. See the ’This is Creative Software…’ section above…

[/expand]

[expand title=”7. Can I get the source code?” startwrap=”” endwrap=”” trigclass=”highlight”]

Colorcoder is not currently open source – but here’s some of the coding we developed to convert a colour to MIDI which may be of interest.

 // HSB to wavelength then frequency conversions //-------------------------------------------------------------- float ofApp::hueToWavelength(float hue) {          /*      wavelength (nm) to hue (degree) and ranges      780-720 : 345-360 - cool reds - 15 range      720-625 : 0-20 - reds - 20 range      625-590 : 20-45 - orange - 25 range      590-565 : 45-75 - yellow - 30 range      565-500 : 75-150 - green - 75 range      500-485 : 150-195 - cyan - 45 range      440-485 : 195-250 - blue - 55 range      380-440 : 250-345 - violet - 95 range      */          wavelength = 0.;          if ((hue > (345./360.)*255.) && (hue < (360./360.)*255.)) {         wavelength = ofMap(hue, (345./360.)*255., (360./360.)*255., 780., 720.);     } else if ((hue > 0.) && (hue <= (20./360.)*255.)) {         wavelength = ofMap(hue, 0., (20./360.)*255., 720., 625.);     } else if ((hue > (20./360.)*255.) && (hue <= (45./360.)*255.)) {         wavelength = ofMap(hue, (20./360.)*255., (45./360.)*255., 625., 590.);     } else if ((hue > (45./360.)*255.) && (hue <= (75./360.)*255.)) {         wavelength = ofMap(hue, (45./360.)*255., (75./360.)*255., 590., 565.);     } else if ((hue > (75./360.)*255.) && (hue <= (150./360.)*255.)) {         wavelength = ofMap(hue, (75./360.)*255., (150./360.)*255., 565., 500.);     } else if ((hue > (150./360.)*255.) && (hue <= (195./360.)*255.)) {         wavelength = ofMap(hue, (150/360.)*255., (195./360.)*255., 500., 485.);     } else if ((hue > (195./360.)*255.) && (hue <= (250./360.)*255.)) {         wavelength = ofMap(hue, (195./360.)*255., (250./360.)*255., 485., 440.);     } else if ((hue > (250./360.)*255.) && (hue <= (345./360.)*255.)) {         wavelength = ofMap(hue, (250./360.)*255., (345./360.)*255., 440., 380.);     }     return wavelength; }  const int speedOfLight = 299792458; // m/s  //-------------------------------------------------------------- float ofApp::wavelengthToFrequency(float wavelength) {     frequency = speedOfLight/(wavelength/1000000000.);     return frequency; }  //-------------------------------------------------------------- float ofApp::frequencyToAudible(float frequency) {     audible = frequency/powf(2.0, 44);     return audible; }  // calculates the midi Note No + pitchbend values //-------------------------------------------------------------- midiNoteNo = round(69.0+12.0*log2f(audible/440.0)); midiNoteNoPitch = 440.0*pow(2, (midiNoteNo-69.0)/12.0); pitchbend = round(4096*12.0*log2f(audible/midiNoteNoPitch));  

[/expand]

[expand title="8. Can I contact you about Colorcoder or related projects?" startwrap="" endwrap="" trigclass="highlight"]

We’re friendly people always happy to hear from folk interested in our work. So if the FAQ or other info on this page hasn’t answered your query; you’d like to discuss other ideas; or you have an opportunity you think may be of interest then drop us a line - info@monomatic.net.

[/expand]