“It's an egg,” “it’s a bubble or a drop,” “it reminds me of neurons,” “an atom,” “some sort of a sea creature, a bioluminescent jellyfish!!”

The text and associated visuals present the interactive, multisensory, traveling installation titled Synesthesia, designed by the authors of this essay. Responding to Metode’s open call for experimental essays for “explor[ing] creative and risky ways of assembling new knowledge about surfaces through words and works,” this essay is formatted and structured as a “coded narrative.” The formal format becomes an exploration platform to amplify the employed concepts and theoretical underpinnings of the art installation. As a way to conceptualize the interaction between audience and object that occurred during the public display of Synesthesia, the essay explores the notions of enaction, sensoriality, performance, and embodiment as they relate to the discourse on dynamic skins. The aspiration is that through this interfaced presentation in Metode’s platform, a transformation takes place, adding a new component to the existing multilayered, hybrid, and complex installation. 

Animated text fragments come into view in different graphic styles based on the “genre” of the information they represent. These texts appear in relationship to photographic, audio, and video documentation linked to the project Synesthesia.

Listed here are the text genres implemented in the “coded narrative”:

Script of “play” actions

A script of Synesthesia’s internal and interconnected actions. These actions occur within the parametric boundaries of the installation’s networked systems. The script of “play” actions is often determined by infrastructural and contextual factors, e.g., the internet strength of the exhibition space or the actual 4G vs. 5G limitations of the various countries where Synesthesia has been presented. Synesthesia can be “moody” or operating in full swing based on the context’s serendipities.

“Exhibition visitors’ quotes”

Visitors’ documented first-hand oral reactions to the project. A number of participants were recorded as they interacted with the installation in the various exhibition contexts Synesthesia has been exhibited. 

A descriptive narrative of the theoretical and experiential engagement with the artwork presented in four larger themes: Curiosity, Deception, Enacted Cinema, and Agency.

A series of quotes originated from external sources, mostly from theory-related literature within architectural theory.

As part of the coded narrative, the various text genres are presented dynamically in reference to the installation’s performance. The narrative’s arrangement is fluid and constantly reorganizing based on the readers’ choices. In clustering together the same information genre, the readers can choose to “click” one item within a cluster and will be taken to the associated fragment of the overall narrative. The reader, thus, is invited to intuitively select the sequence of how the narrative might come together and eventually carve their own path of discovering the work.

Note: All photos, videos, and drawings are by the Synesthetic Research and Design Lab unless otherwise mentioned.

Synesthesia as surfaces: a conceptual framework 

Surface as a Performative Informational Matter

Our daily interaction with our environments is mediated through a vast universe of electronics, sensors, and control devices. Our experiences are increasingly funneled through constant negotiations with interconnected systems such as video conferencing, biometric data applications, and evolving “smart” technologies. These systems are not an isolated form of material media, but they constitute an interdependent landscape that shapes the material complex surrounding us. The “life” of such systems is oftentimes manifested in surfaces and interfaces—e.g., interactive screens, digital advertising billboards, XR technologies, or our computer screens—that bridge the analog and virtual, the two-dimensional and three-dimensional, the continuous and ruptured, the complex and “novel,” and the traditional and expected. These surfaces’ interest lies in their embeddedness of both material and informational matter. In this integration, the disentangling of the more familiar narratives on cognition and the built environment is found. Furthermore, through these integrations, the links between human perception and the physical surface are challenged, expanded, and enacted.

A number of disciplines explore the interplay of information, senses, and materiality and are invested in the fusion of data and matter, with advanced technologies of the matter including programmable, adaptive, and interactive surfaces. Nevertheless, surfaces as a discourse of the matter are, as Tim Ingold claims¹, an often dismissed scholarly domain. But by looking closely at technologically conditioned surfaces, as we argue in this essay, we can shed light on the ways we embrace new scientific, symbolic, aesthetic, and technological theories and on the ways we produce new materialities and their meanings. In reevaluating the surface as a performative matter, perhaps we can employ the new tools at our disposal to explore more embodied, participatory, and empowered ways of being.

Embodied Cognition, Design, and Cinematic Space Overlaps

During the last four decades, emergent theories of the mind, human experience, and computational design have shed light on the potential of transformable, dynamic environments. Environments manifest the capacity to respond to various sensory and perceptual models, call for more intuitive relations, and address embodied interactions with the world around us, as Larry Busbea discusses in the book The Responsive Environment: Design, Aesthetics, and the Human in the 1970s.² Synesthesia is part of an applied research that traces theories of embodied cognition.³ The research follows the early 1950s’ cognitive sciences to the enactivist theories of the mind of the late 1960s,⁴ and the parallel evolution of the 1960s design methods movement to the 1980s design thinking approaches.⁵ J.J. Gibson’s theory on ecological psychology has played an important role in reframing notions such as agency, interaction, and perception, as Ingold discusses in his Metode keynote essay for Deep Surface and is also explored in this work.

By unfolding the dynamics observed between scientific methods on cognition and design processes, the research reveals a continuous evolution towards softening harsh dichotomies as proclaimed in the beginnings of the cognitive sciences and design methods approaches. Early rational approaches set a base for studying complex systems—from the human body to society—by splitting the body and the mind, the biological brain and the self. In tandem, they created sharp distinctions between reason and subjectivity, sense and sensibility, representation and action, and embodied and abstracted cognition. As psychology, neuroscience, cognitive science, and artificial intelligence evolve, new paths are engendered. In embracing the unknowns of perception and intuition, while including the body, Synesthesia aspires to offer a valuable resource of awareness for the multitude of experiences and interactions within our everyday interfaced surroundings.

In an attempt to post-conceptualize the display of the design installation Synesthesia, that is, to unpack and understand our own work based on observations and descriptions of the interaction between the audience and the displayed installation, our essay also connects the enactivist theories of the mind with enactive cinematic approaches and debates between embodiment and cinema.

The Synesthetic Research and Design Lab

The Synesthetic Research and Design Lab, hereafter SR&DL, serves as a collaborative research and prototyping platform where interactive design, art, digital culture, and emergent health sciences meet, highlighting the recursion between individuals and their environments. The lab, directed by Severino Alfonso and Loukia Tsafoulia—the authors of this essay—is a research-design platform that develops practical and theoretical methodologies that critically frame the interactions between humans, objects, and environments under the scholarly underpinnings of embodied cognition. The lab collaborates with the Jefferson Health Center for Autism and Neurodiversity, the Occupational Therapy and Neurology Departments at Thomas Jefferson University, and the University College Dublin Inclusive Design Research Centre of Ireland in partnership with SMARTlab teams in Dublin and Cahersiveen, Ireland, and Niagara Falls, Canada. It also partners with self-advocacy communities and industry experts to build collective knowledge that addresses all-inclusive ways of perceptually experiencing our spaces. Through international symposia, publications, and applied research, these collaborations stimulate international dialogue amongst artists, planners, designers, therapists, medical field experts, tech companies, educational institutions, advocates, neurodivergent people, and their caregivers.

The lab develops research-driven and question-framing interactive and experiential installations and prototypes as a way to transgress the art, health, and design fields’ boundaries. In alignment with Anthony Vidler’s concept of the “warped spaces,” the lab investigates design systems that provide a layered understanding of embodied spaces—affective and performative—through the experimental meshing of the physical and the digital realms. 

The Synesthesia Installation

The multisensory, interactive, traveling installation Synesthesia is a recent project by the SR&DL and was conceived as part of the lab’s ongoing research agenda. The title Synesthesia is used symbolically to evoke a sense of place by combining sight, sensation, movement, touch, memory, and perception. Rather than being a direct reference to the neurological condition, Synesthesia investigates the combined sensorial and cognitive aspects that make up an experience. 

This single artifice triggers and responds to overlapping layers of sensory input and output. It operates via entangling senses, emotions, patterns, variations, and information in an enacted, topological domain. In that sense, Synesthesia provides a rational and emotional sense of what it means to live among “conversational” machines and raises awareness of the design potential behind responsive environments and dynamic skins. From the human enacting on the predetermined possibilities the space affords, these fields of inquiry explore how the space provides opportunities for meaningful interaction while adapting to inclusive perceptual factors. The transition from a physical space determined by symbolic and formal aspects to an informational environment that operates through real-time data processing yields a meaningful alteration in human perception and the surface. Synesthesia hence activates a discourse on surfaces that understands them as complex environmental systems, embedded in cultural, aesthetic, technological, material, symbolic, and bodily discussions.

The work also points to the need for acknowledging how the aesthetics and politics of algorithms, information, and the digital technologies impact our capacities to assign meanings to surfaces, objects, and our environments. With Synesthesia, we ask: What is our agency as mediators between the instrumentality of algorithmic protocols, human perception, and the surfaces that manifest them? 

Technical Description

Synesthesia’s large, blob-like volume—approximately 12’ x 14’ x 13’—is lifted by six aluminum poles that scissor-cross in pairs and connect at their top and bottom joints. The metal poles hold fifteen flexible fiberglass rods that bend into the blob’s computed form. The rods serve as the skin’s tensile frame—a neutral light-gray, flexible, and slightly translucent membrane. Such a surface wraps the blob and includes twenty-four portals, referred to as “eyes,” where a series of black 3D-printed micro telescopes are positioned. Throughout the surface of the blob, the same number of thin, round, eye-shaped profiles that are part of the micro telescopes’ front sections are revealed. The blob thus exhibits a continuous surface to the exterior, with the only exception of these “eyes.”

Selected “eyes” on the project’s surface include a complex optics mechanism with an embedded sensor, a small LED, and a beamsplitter positioned at 45 degrees that allows a hidden camera to detect the human eye without the visitor knowing [fig. 3]. This is the “computing brain” of the installation—designed as a camera obscura that records the human eye when in proximity. The video recording of the visitor’s human eye is then projected onto the skin from the interior of the blob. The blob also includes a series of light events that respond to the visitors’ presence in various ways. Together with the live video projections of the eyes, these light events complete the visual effects projected on the blob’s surface. The interior of the blob also includes a series of olfactory, sound, and vibrating events defined through scripting protocols. 

Synesthesia exhibition venues to date

Synesthesia has been exhibited at the European Cultural Center, as part of the 17th Venice Architecture Biennale (May–November 2021), at the Municipal Theater of Piraeus in Athens (December 2021–January 2022), and at the great hall of Trajan’s Market Museum of the Imperial Fora in Rome (April–May 2022), and it will be presented at the IE Creativity Center in Segovia, Spain, during February 2023. Before traveling to Europe, the project launched MICRO • GALLERY, an experimental space of the HOT • BED art gallery in Philadelphia (February–March 2021).¹⁰

Synesthesia invites the audience, or in other words, the participants, into a game of curiosity. It is conceived as an atmospheric, soft interface activating different levels of human-machine participation, delusion, and wonder. 

“It’s a blob. It’s a brain. It’s a misshapen tent. It’s an artificial, cybernetic alien monster that lures you toward it, so it can steal parts of your body.”

Peter Crimmins, WHYY, HotBed Gallery in Philadelphia Exhibition

Prior to entering the exhibition space, an ambient sound can be heard from the distance. This sound is not music nor noise, rather something in between. There are moments of calm sound patterns and others of agitation with intense rhythms of metallic variations that squawk at the listener. A soft mechanical rehrr in the background is discernable as well, sounding similar to the noise of an air conditioner constantly vibrating. 

In darkness, here comes the composed sound: electronic compositions in Stefan Schmidt’s Rodenkopf 

Audio 1: The album rodenkopf by Stefan Schmidt

“The first thing that came to my mind when I listened to the sound—of the installation—was Vangelis’ film scores, especially his Blade Runner composition. But then, the sound changed entirely and forced me to rethink my references.”

Participant comment: Municipal Theater of Piraeus Exhibition

The inner projectors initiate: their background noise is superimposed on the composed sound and a beam of light is cast on the cinematic skin from within, waiting for enaction. 

During that moment, prior to fully engaging with the installation, a dynamic light splashes against the surrounding surfaces of the exhibition space. The visitor leaves their previous state of mind behind and adjusts their perceptual center as they enter a new event. When entering a darkened room, the participant needs to accommodate to an event led by artificial lights with modifiable intensity, positioning, and color attributes. 

“Once I entered the dark space I found the form weird, almost disturbing. It made me feel anxious but also curious. In slowly familiarizing myself with the darkness and when I eventually came closer, the overall experience felt soothing.”

Participant comment: European Cultural Center, 17th Venice Architecture Biennale Exhibition

“It's an egg,” “it’s a bubble or a drop,” “it reminds me of neurons,” “an atom,” “some sort of a sea creature, a bioluminescent jellyfish!!”Participants comments from the various spaces the work has been exhibited.

“The contrast of Synesthesia’s form and softness to the theater’s neoclassic architectural features, its geometric ceiling ornamentation, and its hard marble resembling patinas creates a juxtaposition that forces contextual awareness.”

Participant comment: Municipal Theater of Piraeus Exhibition

“The context is striking. Synesthesia with its multimodal sensoriality, effectual character, and lightness, next to the rich ancient material and symbolic layers of the Fora and in dialogue with the weight of the Roman caryatid that faces it. It made me think of wormholes, a transtemporal trip.”

Participant comment (archeologist): Trajan’s Market Museum of the Imperial Fora, Rome Exhibition

“Folds, blobs, nets, skins, diagrams: all words that have been employed to describe theoretical and design procedures over the last decade. […]The new vocabulary has something to do with contemporary interest in the informe; it seems to draw its energies from a rereading of Bataille and a new interest in Deleuze and Guattari.”

“They are words and forms conceived and manipulated in a virtual space, with, nevertheless, an intimate relationship to production techniques and the technology of materials. Such a relationship would be impossible without the digital interface that construes information, theoretical and practical.”¹²

“In a series of writings between 1893 and 1895, [August Schmarsow] developed a psychological characterization of space, based on a concept of ‘intuited form,’ built up out of the sense of sight joined to “the residues of sensory experience to which the muscular sensations of our body, the sensitivity of our skin, and the structure of our body all contribute.”¹³

Conceived as a deformed sphere—a central, internal, node of multiple projections shaped like a blob and based on topological deformations—Synesthesia acts as a three-dimensional cinema, a complex, tensile physical skin that constitutes both a physical and a virtual interface. As a “folded” cinematic experience, it provides a theatrical space owned by the audience as they participate and engage with the cinematic narrative. 

“Perhaps I recognise similarities with my work in the idea of transmission and technological reproducibility. Maybe there is something in the idea of screen culture of simultaneously enabling both mass-customisation and mass reproduction.”

Comment by Nick Walkley, participant in Metode’s workshops and essay contributor

“It reminded me of a large Noguchi lamp. I know, sure! But, at the same time you can’t use his paper technique at such a large scale, can you? How is this skin put together? What is it made of? And most importantly, what’s the intent?”

Participant comment (industrial designer): HotBed Gallery in Philadelphia Exhibition

“Such a rich exploration of stereometric and topological form explorations, from the smaller internal pieces to the larger blob. Interested to know how all these came to be.” 

Participant comment (physicist): Municipal Theater of Piraeus Exhibition

One of the stages in the curation of Synesthesia at the various exhibition venues showcases the structural joinery components. They are designed to facilitate the traveling nature of the installation and make possible the fast assembly and disassembly of the structure into pieces that can be shipped across the globe. The various types of joints act in both tension and compression respectively. The different filaments and tectonics were iterated, tested, and evaluated as to their friction, strength, production time, and material density capabilities.

It also includes a sampling of the iterative process for the “eye” portals creation and the computing brain of the installation. This is an exhaustive analysis of formal and material aspects in search of a camera obscura effect. We examined the use of optical components to split incident light that have the capacity to both be transparent and reflective so the participants’ eyes can be detected by the hidden cameras without the participants detecting these cameras. The form developed as a hybrid referencing, on the one hand, the organicity of human organisms, and on the other, the geometries that accommodate the industrialized parts of small processors, sensors, cameras, and all systems comprising the “eye.”

Automated motors that rotate internal mobiles of dichroic, reflective material are ON.

Synesthesia’s fluctuating light and patterned sound effects attract the viewer to approach and engage in a sensorial “conversation.”

“If no one is near it, it calls out with an inviting chirp to let you know it is waiting for a visitor. Once you get close, it emits a series of grinding sounds.”

Peter Crimmins, WHYY, HotBed Gallery in Philadelphia Exhibition

Internal white lights turn ON, in slowly fading patterns following their networked “conversation.” They cast their light beams on the dichromic mobiles. Colorful reflections are in motion, in a stable pace until a human-machine conversation starts.

The work provides visual stimulation at low levels of intensity with dynamic light and color projections as animated visuals in Lumia formations referencing the work of Thomas Wilfred. 

“The ways the dynamic light effects are occurring is a mystery. I feel as if it is an experience of the cosmos. A very meditative environment that puts me “in the zone.” 

Participant comment: Municipal Theater of Piraeus Exhibition

“Suddenly, and only for a few seconds, the whole blob gets illuminated! The atmospheric multicolored light patterns disappear, and the artwork turns into a light bulb! It is like a theater director’s trick, a wake-up call of sorts, where the surrounding scenography is illuminated and the material and technical details are momentarily revealed. This is a moment that certainly increased my curiosity. I want to stay around longer!”

Participant comment: Municipal Theater of Piraeus Exhibition

Air is circulated within. A breathing mechanism provokes vibration once participants are in proximity.

The work engages in tactile, embodied interactions with its low-frequency vibration that one can feel but cannot hear. Our sensory modalities are interconnected. Whether the visitor actively engages in all levels of sensorial interactions within Synesthesia’s narrative or is unconsciously impacted by them, the environment created by the artwork regulates the audience’s psychophysiological states, emotions, and feelings. By expanding oculocentric narratives to multi-sensory modalities, Synesthesia’s surface acquires new meanings and activates a holistic praxis of embodiment.

“The very nature of our bodies is what allows us to not only experience visually but also experience gravity, force, and pressure. Vision is multimodal; it encompasses the activation of motor, somatosensory, and emotion-related brain networks. More than twenty years of research on mirror neurons have demonstrated the existence of a mechanism directly mapping action, perception, and execution in the human brain.”¹⁵

In what ways does an understanding of surfaces as participating, active, and complex relate to an anthropocentric understanding of surfaces? In what ways does how a surface is experienced, as active or passive, relate to the precondition of human interaction with this surface?

Comment by Jakob Oredsson, participant in Metode’s workshops and essay contributor

“Consciousness, thinking, and communication activities are intrinsic to our bodies, responding to the characteristics of the physical, social, and cultural environments in which we dwell.”¹⁶

“The blinking holes remind me of a brain, it feels like a billion synapses are taking place once I come close. Once I see through one of them, I can also hear an instantaneous sound, like an alien radio transmission. I feel like I am in conversation with my head.”

Participant comment: European Cultural Center, 17th Venice Architecture Biennale Exhibition 

Audio 2: Sound activated when the presence of the participants is detected by the “eye” sensor mechanism. Downloaded by freesound.org by Brotherprovisional. License by the Creative Commons

The presence of the human eye near selected portals triggers its live video recording and its projection back into the cinematic space, while other portals are deceptive. Upon interaction, the participants’ eyes are captured and projected from within and in real-time on Synesthesia’s skin. The participant becomes an active part of the installation unknowingly. Their image is only perceived by observers, further back from the blob. 

Synesthesia is being looked through but also looks outwards. The portals are simultaneously the “eyes” of the installation that look back at the participants. Synesthesia’s inner gaze of the interactions is live-streamed on the installation’s webpage, creating an active exchange between the installation, the participants, and the wider public, thus cutting across local, global, and virtual scales.

“I can see the interior now, I think through the screens in the back. Was that me projected? Broadcasted on the web?”

Participant comment: Municipal Theater of Piraeus Exhibition

“The other side of the interface. Does it matter which?” 

Participant comment: Municipal Theater of Piraeus Exhibition

“I wonder if it is worth exploring that surface and what it is like when it is turned off, before going into the magic that happens on the other side of them.”

Comment by Nick Walkley, participant in Metode’s workshops and essay contributor

In his Deep Surface essay, Ingold defines the term “interface” as having “two sides: whether a top side and an underside, or an outside and an inside.” However, by drawing on Gibson’s ecological approach to perception, Ingold stresses that an interface is also “perforated by holes or keys that allow information, and sometimes materials, to pass from one side to the other.” Synesthesia makes a reference to the holes found in the often opaque enclosures covering construction sites or private spaces in the city. As city dwellers encounter these unattended openings, they are allowed to explore the “other side” and gain visual access to the unknown territory and the unfolding of events that occur within it. On the one hand, the enclosure functions as the thin urban layer separating the public from the private. On the other hand, the opening in this enclosure functions as a temporary fusion of the two urban fields, as a flashing point—an analogy to the installation’s eyes—that turns from black to white and vice versa on the nolli map of the city. Similarly, in Synesthesia, the unconscious, accidental, and curious performance of the participants actuates Synesthesia’s structured “veil.” The structure invites, the body is pulled in, and in blinking patterns, our organs of perception are dynamically projected back onto the surface as abstracted images, only to disappear shortly after. 

(A hidden secret:) “There is an interior to be discovered but it is hidden from me. My friend told me that there is a hidden camera that does something cool. I am intrigued to debunk the secret, to understand the technique. Where is the music coming from? How does it interact with me, with you? There has to be something else to this, but I don’t yet understand what. Maybe the issue is manifested in the interior?”

Participant comment: European Cultural Center, 17th Venice Architecture Biennale Exhibition 

“How is everything connected?”

Participant comment: Municipal Theater of Piraeus Exhibition

The installation is a digital and social microenvironment that operates at various scales. On the internal microscale of Synesthesia’s body, all the computational and sensory units communicate with each other and eventually create their own network, a microsystem held within Synesthesia’s soft wrapper. 

Synesthesia “bends” the traditional planarity of the cinematic surface. Referencing the term enaction in the visual arts, Synesthesia embraces the shift from traditional cinematic spectacles to works probing the frontiers of interactive, performative, and networked media.¹⁷ The project translates cinematic references into unexpected paths of engagement, participation, and immersion. Enactive cinema in that sense became the perfect transgressive subfield to relate to in a process of post-conceptualizing the installation Synesthesia as a subfield of cinema that meshes the performative values found in theater and interactive art in blurring the “fourth wall.”¹⁸ The screen surface, accordingly, is not inert but rather responsive. With that, the folding, enacted surface merges the audience and performer. 

“The surface is not inert but always participates.” Has any surface ever been passive? Is Synesthesia perhaps exemplifying that this has never been the case?

Comment by Jakob Oredsson, participant in Metode’s workshops and essay contributor

“Such an electric atmosphere; it alters the ordinary conditions of sensation.”

Participant comment (curator): Trajan’s Market Museum of the Imperial Fora, Rome Exhibition

Synesthesia’s interactive interfaces expand human agency through sensors, actuators, real-time responses, and a human-machine participatory constitution. Collectively, the installation equates environment to experience and mediates our bodies’ agency via interlacing action and perception.¹⁹ Designed as a traveling experience, Synesthesia engages a variety of communities and places. This inclusive aspect of the work amplifies the narratives and cultural interactions triggered by it, in developing methods for ethnographic impact and observational analysis.

In writing this experimental essay, we hope to have shown how the notion of surface cannot be taken as an isolated domain, since it holds multiple functions: a physical space, a medium, and a responsive discourse. Furthermore, performance and participation are sought not only as an experience, but as a method of research and a way to create knowledge and awareness of our bodies and our technological environment. 

Time to rest. In the darkness, back to silence, what have we experienced, alone, collectively?

“Perhaps invite the reader to turn the screen off, to contemplate the screen itself, as part of the experience of reading the essay.”

Comment by Nick Walkley, participant in Metode’s workshops and essay contributor