iTones, Interactive Installation, 2014

iTones is an interactive gallery piece that transforms a user, and her smartphone, into a musical instrument. A user is confronted with a minimal set up: black headphones hanging on the wall next to a repurposed iPhone armband. A drawing, done directly on the gallery wall and labeled “iTones,” demonstrates how the piece is worn. 

The user is required to use her own smartphone, which she inserts in the wristband and plugs into the earphones. At the moment of plugging in, the system starts to play the ringtones on the black headphones. Audio filters are mapped – using a hidden Kinect and PD patch – to the movements of her limbs. The systems allows her to use her body as an instrument, varying the pitch and frequency of the ringtones as she moves her arms and legs. The gesture of “plugging in” to the system is an important initiation into the experience, underscoring users’ reliance on their smartphones. Concurrently, it enables them to give the pre-selected sounds of their personal data a new audible life.

 

The Concept

This piece was inspired by an early work by Polish-born artist Krzysztof Wodiczko: Personal Instrument (1969). His interactive, wearable artwork was the first in a series of public interventions that articulated the façade of freedom of speech in a public space dominated by the repressive socialism of Poland in the 1960s. In his piece, the sounds of public space becomes a metaphor for citizens and government to negotiate the “boundaries of freedom and ways of practicing it.” Wodiczko’s goal was to articulate the voicelessness of the citizen who was forced to listen to state directives, but unable to make himself heard. 

By performing in public, Wodiczko fought censorship. As he explained: “I am an artist in the listening, not in the speaking, and though I do not have the right to say what I really want to say… let me at least be allowed to listen to what I want to hear…’’

The Personal Instrument (1969), Krzysztof Wodiczko

Concept sketching to update new piece.

The Update

In the new digital spaces of our Dynamic Media Society, we can find ourselves in a situation of similar duress. iTones updates the Personal Instrument to reflect 21st-century power struggles, not between government and citizen, but between algorithm and user.

The sounds of public space, which represent freedom, are replaced with the pre-programmed cell phone ringtones of a user, to represent the user’s personal data repurposed by the algorithmic systems of dataveillance for ulterior means.

The sounds of public space, which represent freedom, are replaced with the pre-programmed cell phone ringtones of a user, to represent the user’s personal data repurposed by the algorithmic systems of dataveillance for ulterior means. Just as Wodiczko revealed the dominated citizen by empowering him as a conductor of the sounds of public space, iTones reveals the user as a used user, or Reproducer, by allowing her to transform her ringtones through gesture. The goal is to offer users of iTones a means to metaphorically negotiate agency as reproducer in the face of algorithmic control.

With the Personal Instrument, Wodiczko empowered the citizen through the metaphor of transforming the ambient noises of public space into music. Yet he unveiled the futility of the instrument through its public silence (its music was heard only in the privacy of the artist’s earphones). Thus the citizen was rendered voiceless by the repressive regime.

Similarly, iTones empowers its user by giving her the brief ability to repurpose her data, symbolized by the ringtone, and transform it into music. She takes control of the algorithm. But, as with the Personal Instrument, iTones’ outward silence reveals the same impotence: the regime of algorithmic control is always at work.

Behind the Scenes

The moment of plugging in triggers the hidden computer and Kinect to begin mapping the manipulation of the sound to user-gesture. The computer runs open-source code in Pure Data Extended (PD) and Synapse. Synapse sends user-skeleton coordinates (X, Y, and Z) captured by the Kinect to PD, which translates the coordinates of a user’s arm movements into sound filters in order to transform the ringtone. As the Y coordinates of the arms increase, the frequency of the ringtone increases until it plays 1.5x faster than regular speed. Likewise as the Y coordinates decrease, the frequency does as well, until the ringtone is playing 2x slower. As X coordinates move outward from the core, a reverb and delay filter distort the ringtone. It is integral to this system of sound manipulation that the user be able to disrupt the sound of a familiar ringtone. The filters transform the sounds to the point of annihilation, yet leave room — literal points in space — for moments of recognition: when hands fall close to the core of the body at the level of the heart, either at rest or in gesture, the effects turn off and the ringtone plays at normal speed and pitch.

Sketching for sound manipulations.

Sketching for interaction: 1. approach the piece; 2. plug in and put on the headphones; and 3. move, dance, and listen. 

Designing For the Gallery

In order to design for a gallery experience, I needed to not only problem-solve both the specific technical issues for the space, but also create a system where users would understand what was required of them: to approach the piece; to plug in and put on the headphones; and finally to move their arms (or dance) in order to recognize that their movement directly correlates with the sound.

I designed a minimal set-up: black headphones hanging on the wall next to a repurposed iPhone armband. I included an instructional drawing, done directly on the gallery wall and labeled “iTones,” to demonstrate how the piece is worn.  

Gallery instructions.

Plugging in.

Analysis

iTones empowers its user by giving her the brief ability to repurpose her data, symbolized by the ringtone, and transform it into music. She takes control of the algorithm. But, as with the Personal Instrument, iTones’ outward silence reveals the same impotence: the regime of algorithmic control is always at work.

Though each piece offers a glimmer of hope in the form of musical beauty, the goal isn’t to empower or defeat but rather to articulate a power struggle between two agent yet unequal components. 

To learn more, click here.