<p>Photo by Orlando Saverino-Loeb</p>

Photo by Orlando Saverino-Loeb

Synesthesia: An Experiment on Enacted Surfaces

Last ned artikkelen som PDF her

An Experiment on Enacted Surfaces

“It's an egg,” “it’s a bubble or a drop,” “it reminds me of neurons,” “an atom,” “some sort of a sea creature, a bioluminescent jellyfish!!”

<p>Fig. 1 <a target="_blank" rel="noopener noreferrer nofollow" href="https://www.synestheticdesignlab.com/synesthesia">Synesthesia</a>, design installation by Severino Alfonso and Loukia Tsafoulia (2021–2023)</p>

Fig. 1 Synesthesia, design installation by Severino Alfonso and Loukia Tsafoulia (2021–2023)

The text and associated visuals present the interactive, multisensory, traveling installation titled Synesthesia, designed by the authors of this essay. Responding to Metode’s open call for experimental essays for “explor[ing] creative and risky ways of assembling new knowledge about surfaces through words and works,” this essay is formatted and structured as a “coded narrative.” The formal format becomes an exploration platform to amplify the employed concepts and theoretical underpinnings of the art installation. As a way to conceptualize the interaction between audience and object that occurred during the public display of Synesthesia, the essay explores the notions of enaction, sensoriality, performance, and embodiment as they relate to the discourse on dynamic skins. The aspiration is that through this interfaced presentation in Metode’s platform, a transformation takes place, adding a new component to the existing multilayered, hybrid, and complex installation

Animated text fragments come into view in different graphic styles based on the “genre” of the information they represent. These texts appear in relationship to photographic, audio, and video documentation linked to the project Synesthesia.

Listed here are the text genres implemented in the “coded narrative”:

Script of “play” actions

A script of Synesthesia’s internal and interconnected actions. These actions occur within the parametric boundaries of the installation’s networked systems. The script of “play” actions is often determined by infrastructural and contextual factors, e.g., the internet strength of the exhibition space or the actual 4G vs. 5G limitations of the various countries where Synesthesia has been presented. Synesthesia can be “moody” or operating in full swing based on the context’s serendipities.

“Exhibition visitors’ quotes”

Visitors’ documented first-hand oral reactions to the project. A number of participants were recorded as they interacted with the installation in the various exhibition contexts Synesthesia has been exhibited. 

// Theme X.

A descriptive narrative of the theoretical and experiential engagement with the artwork presented in four larger themes: Curiosity, Deception, Enacted Cinema, and Agency.

Theory-associated quotes.

A series of quotes originated from external sources, mostly from theory-related literature within architectural theory.

As part of the coded narrative, the various text genres are presented dynamically in reference to the installation’s performance. The narrative’s arrangement is fluid and constantly reorganizing based on the readers’ choices. In clustering together the same information genre, the readers can choose to “click” one item within a cluster and will be taken to the associated fragment of the overall narrative. The reader, thus, is invited to intuitively select the sequence of how the narrative might come together and eventually carve their own path of discovering the work.

Note: All photos, videos, and drawings are by the Synesthetic Research and Design Lab unless otherwise mentioned.

<p></p>

<p>Fig. 2 <em>Synesthesia</em>´s surface is actuated by the human presence and interaction with it. Photographs taken during the installation’s press debut at the Ravenhill Chapel in East Falls, Philadelphia, PA. Photos by John Aquino</p>

Fig. 2 Synesthesia´s surface is actuated by the human presence and interaction with it. Photographs taken during the installation’s press debut at the Ravenhill Chapel in East Falls, Philadelphia, PA. Photos by John Aquino

Synesthesia as surfaces: a conceptual framework 


Surface as a Performative Informational Matter

Our daily interaction with our environments is mediated through a vast universe of electronics, sensors, and control devices. Our experiences are increasingly funneled through constant negotiations with interconnected systems such as video conferencing, biometric data applications, and evolving “smart” technologies. These systems are not an isolated form of material media, but they constitute an interdependent landscape that shapes the material complex surrounding us. The “life” of such systems is oftentimes manifested in surfaces and interfaces—e.g., interactive screens, digital advertising billboards, XR technologies, or our computer screens—that bridge the analog and virtual, the two-dimensional and three-dimensional, the continuous and ruptured, the complex and “novel,” and the traditional and expected. These surfaces’ interest lies in their embeddedness of both material and informational matter. In this integration, the disentangling of the more familiar narratives on cognition and the built environment is found. Furthermore, through these integrations, the links between human perception and the physical surface are challenged, expanded, and enacted.

A number of disciplines explore the interplay of information, senses, and materiality and are invested in the fusion of data and matter, with advanced technologies of the matter including programmable, adaptive, and interactive surfaces. Nevertheless, surfaces as a discourse of the matter are, as Tim Ingold claims1, an often dismissed scholarly domain. But by looking closely at technologically conditioned surfaces, as we argue in this essay, we can shed light on the ways we embrace new scientific, symbolic, aesthetic, and technological theories and on the ways we produce new materialities and their meanings. In reevaluating the surface as a performative matter, perhaps we can employ the new tools at our disposal to explore more embodied, participatory, and empowered ways of being.


Embodied Cognition, Design, and Cinematic Space Overlaps

During the last four decades, emergent theories of the mind, human experience, and computational design have shed light on the potential of transformable, dynamic environments. Environments manifest the capacity to respond to various sensory and perceptual models, call for more intuitive relations, and address embodied interactions with the world around us, as Larry Busbea discusses in the book The Responsive Environment: Design, Aesthetics, and the Human in the 1970s.2 Synesthesia is part of an applied research that traces theories of embodied cognition.3 The research follows the early 1950s’ cognitive sciences to the enactivist theories of the mind of the late 1960s,4 and the parallel evolution of the 1960s design methods movement to the 1980s design thinking approaches.5 J.J. Gibson’s theory on ecological psychology has played an important role in reframing notions such as agency, interaction, and perception, as Ingold discusses in his Metode keynote essay for Deep Surface and is also explored in this work.

By unfolding the dynamics observed between scientific methods on cognition and design processes, the research reveals a continuous evolution towards softening harsh dichotomies as proclaimed in the beginnings of the cognitive sciences and design methods approaches. Early rational approaches set a base for studying complex systems—from the human body to society—by splitting the body and the mind, the biological brain and the self. In tandem, they created sharp distinctions between reason and subjectivity, sense and sensibility, representation and action, and embodied and abstracted cognition. As psychology, neuroscience, cognitive science, and artificial intelligence evolve, new paths are engendered. In embracing the unknowns of perception and intuition, while including the body, Synesthesia aspires to offer a valuable resource of awareness for the multitude of experiences and interactions within our everyday interfaced surroundings.

In an attempt to post-conceptualize the display of the design installation Synesthesia, that is, to unpack and understand our own work based on observations and descriptions of the interaction between the audience and the displayed installation, our essay also connects the enactivist theories of the mind with enactive cinematic approaches and debates between embodiment and cinema.

The Synesthetic Research and Design Lab


The Synesthetic Research and Design Lab, hereafter SR&DL, serves as a collaborative research and prototyping platform where interactive design, art, digital culture, and emergent health sciences meet, highlighting the recursion between individuals and their environments. The lab, directed by Severino Alfonso and Loukia Tsafoulia—the authors of this essay—is a research-design platform that develops practical and theoretical methodologies that critically frame the interactions between humans, objects, and environments under the scholarly underpinnings of embodied cognition. The lab collaborates with the Jefferson Health Center for Autism and Neurodiversity, the Occupational Therapy and Neurology Departments at Thomas Jefferson University, and the University College Dublin Inclusive Design Research Centre of Ireland in partnership with SMARTlab teams in Dublin and Cahersiveen, Ireland, and Niagara Falls, Canada. It also partners with self-advocacy communities and industry experts to build collective knowledge that addresses all-inclusive ways of perceptually experiencing our spaces. Through international symposia, publications, and applied research, these collaborations stimulate international dialogue amongst artists, planners, designers, therapists, medical field experts, tech companies, educational institutions, advocates, neurodivergent people, and their caregivers.

The lab develops research-driven and question-framing interactive and experiential installations and prototypes as a way to transgress the art, health, and design fields’ boundaries. In alignment with Anthony Vidler’s concept of the “warped spaces,” the lab investigates design systems that provide a layered understanding of embodied spaces—affective and performative—through the experimental meshing of the physical and the digital realms. 

The Synesthesia Installation

The multisensory, interactive, traveling installation Synesthesia is a recent project by the SR&DL and was conceived as part of the lab’s ongoing research agenda. The title Synesthesia is used symbolically to evoke a sense of place by combining sight, sensation, movement, touch, memory, and perception. Rather than being a direct reference to the neurological condition, Synesthesia investigates the combined sensorial and cognitive aspects that make up an experience. 

This single artifice triggers and responds to overlapping layers of sensory input and output. It operates via entangling senses, emotions, patterns, variations, and information in an enacted, topological domain. In that sense, Synesthesia provides a rational and emotional sense of what it means to live among “conversational” machines and raises awareness of the design potential behind responsive environments and dynamic skins. From the human enacting on the predetermined possibilities the space affords, these fields of inquiry explore how the space provides opportunities for meaningful interaction while adapting to inclusive perceptual factors. The transition from a physical space determined by symbolic and formal aspects to an informational environment that operates through real-time data processing yields a meaningful alteration in human perception and the surface. Synesthesia hence activates a discourse on surfaces that understands them as complex environmental systems, embedded in cultural, aesthetic, technological, material, symbolic, and bodily discussions.

The work also points to the need for acknowledging how the aesthetics and politics of algorithms, information, and the digital technologies impact our capacities to assign meanings to surfaces, objects, and our environments. With Synesthesia, we ask: What is our agency as mediators between the instrumentality of algorithmic protocols, human perception, and the surfaces that manifest them? 

Technical Description

Synesthesia’s large, blob-like volume—approximately 12’ x 14’ x 13’—is lifted by six aluminum poles that scissor-cross in pairs and connect at their top and bottom joints. The metal poles hold fifteen flexible fiberglass rods that bend into the blob’s computed form. The rods serve as the skin’s tensile frame—a neutral light-gray, flexible, and slightly translucent membrane. Such a surface wraps the blob and includes twenty-four portals, referred to as “eyes,” where a series of black 3D-printed micro telescopes are positioned. Throughout the surface of the blob, the same number of thin, round, eye-shaped profiles that are part of the micro telescopes’ front sections are revealed. The blob thus exhibits a continuous surface to the exterior, with the only exception of these “eyes.”

1 Ingold claims that the essential property of the surface is to withdraw and therefore that “scholars have long held surfaces in disdain.” Tim Ingold, Imagining for Real: Essays on Creation, Attention and Correspondence (London: Routledge, 2021), 149.

2 See Larry Busbea, The Responsive Environment: Design, Aesthetics, and the Human in the 1970s (Minneapolis: University of Minnesota Press, 2020).

3 The embodied turn refers to the period when an interest in the body became established among researchers in the cognitive sciences and language and social interaction. Controversies regarding human cognition and perception in sciences, philosophy, and design converged with the overarching mind-body debate. The debate entails aspects of the ‘mental,’ linked to consciousness, intentionality, and the self, and of the ‘physical,’ related to the embodiment. The mind-body debate bears yet another significant conceptual dipole currently under revision and thus fluctuating: between the computationalism and the enactivism theories of the mind. Common to computationally-oriented research is the idea that cognitive activity occurs in the agent’s nervous system and that cognition and consciousness are a form of computation. Since the late 1980s, though, embodied approaches have resurged, most significantly through embodied robotics and the various versions of enactivism. An essential source of inspiration for embodied and embedded cognition, central to the embodied turn, can be found in the phenomenological tradition. Martin Heidegger, Edmund Husserl, and Maurice Merleau-Ponty emphasized the physical embodiment of our conscious, cognitive experiences. Cognition in their works is physically interactive, embedded in physical contexts, and manifested in physical bodies. J.J. Gibson’s novel theory on ecological psychology as an embodied and situated approach to cognition is inspired by American pragmatism and is tied to the phenomenological tradition. Connectionist psychology also sets perception as an extended process involving whole organisms in motion through their environments and hitherto has played an important role in shaping the embodied turn. 

4 The terms “enaction” and “enactive” are attributed to Francisco Varela, Evan Thompson, and Eleanor Rosch in their 1991 The Embodied Mind. Varela et al. develop an enactivist approach for describing and explaining human experience by employing cognitive science and neuroscience methods, philosophical phenomenology, and Buddhist meditation techniques.

5 See Lucy Kimbell, “Rethinking Design Thinking: Part I,” Design and Culture 3, no. 3 (2011); Molly Wright Steenson, Architectural Intelligence: How Designers and Architects Created the Digital Landscape (Cambridge, Mass.: MIT Press, 2017); Arindam Dutta, ed. A Second Modernism: MIT, Architecture, and the “Techno-Social” Moment (Cambridge, Mass.: MIT Press, 2013). Also Ingrid Halland discusses how cybernetics was related to design methods and the developments towards design thinking. See Ingrid Halland, “Error Earth: Displaying Deep Cybernetics in ‘the Univeritas Project’ and ‘Italy: The New Domestic Landscape’, 1972” (PhD, University of Oslo, 2018). 

<p>Fig 3. View from the outside of one of the “eye portals” that act as the “computing brain” of the installation.&nbsp;</p>

Fig 3. View from the outside of one of the “eye portals” that act as the “computing brain” of the installation. 

Selected “eyes” on the project’s surface include a complex optics mechanism with an embedded sensor, a small LED, and a beamsplitter positioned at 45 degrees that allows a hidden camera to detect the human eye without the visitor knowing [fig. 3]. This is the “computing brain” of the installation—designed as a camera obscura that records the human eye when in proximity. The video recording of the visitor’s human eye is then projected onto the skin from the interior of the blob. The blob also includes a series of light events that respond to the visitors’ presence in various ways. Together with the live video projections of the eyes, these light events complete the visual effects projected on the blob’s surface. The interior of the blob also includes a series of olfactory, sound, and vibrating events defined through scripting protocols. 

Synesthesia exhibition venues to date

Synesthesia has been exhibited at the European Cultural Center, as part of the 17th Venice Architecture Biennale (May–November 2021), at the Municipal Theater of Piraeus in Athens (December 2021–January 2022), and at the great hall of Trajan’s Market Museum of the Imperial Fora in Rome (April–May 2022), and it will be presented at the IE Creativity Center in Segovia, Spain, during February 2023. Before traveling to Europe, the project launched MICRO • GALLERY, an experimental space of the HOT • BED art gallery in Philadelphia (February–March 2021).10

6 See the “3rd International Neurodiversity & the Built Environment Symposium: PlaceMaking,” Thomas Jefferson University, accessed December 1, 2022. 

7 Anthony Vidler refers to the intersection of media and genres as a “warping” in his preface to the book Warped Space: Art, Architecture, and Anxiety in Modern Culture: “The second kind of warping is that produced by the forced intersection of different media—film, photography, art, architecture—in a way that breaks the boundaries of genre and the separate arts in response to the need to depict space in new and unparalleled ways. Artists, rather than simply extending their terms of reference to the three-dimensional, take on the questions of architecture as an integral and critical part of their work in installations that seek to criticize the traditional terms of art. Architects, in a parallel way, are exploring the processes and forms of art, often on the terms set out by artists, in order to escape the rigid codes of functionalism and formalism. This intersection has engendered a kind of ‘intermediary art,’ comprised of objects that, while situated ostensibly in one practice, require the interpretive terms of another for their explication.”

8 SR&DL. “SynestheticDesignLab.” Accessed December 3, 2022.

9 As also discussed by Luciana Parisi, Contagious Architecture: Computation, Aesthetics, and Space, Technologies of Lived Abstraction (Cambridge, Mass.: MIT Press, 2013).

10 See news links for more information and press on each exhibited venue:

SynestheticRDL. 2022. “Synesthesia at Mercati Di Traiano, Museo Dei Fori Imperiali in Rome.” SR&DL. April 22, 2022.
SynestheticRDL. 2022. “Synesthesia at the Municipal Theatre of Piraeus, Athens.” SR&DL. January 17, 2022.
SynestheticRDL. 2021. “Synesthesia at the ECC Architecture Venice Biennial 2021, Palazzo Bembo.” SR&DL. May 27, 2021.
Molly Given. 2021. “New Interactive Art Display Hits Philly before Going to Italy—Metro Philadelphia.” Metrophiladelphia.com. February 17, 2021.

<p>Figs. 4 and 5. Inner and exterior views of <em>Synesthesia</em>’s structural elements, its deformed geometry, and the surface wrapper. Photos taken during the artwork installation at the Trajan’s Market Museum of the Imperial Fora in Rome.</p>

Figs. 4 and 5. Inner and exterior views of Synesthesia’s structural elements, its deformed geometry, and the surface wrapper. Photos taken during the artwork installation at the Trajan’s Market Museum of the Imperial Fora in Rome.

// Theme 1. Curiosity

Synesthesia invites the audience, or in other words, the participants, into a game of curiosity. It is conceived as an atmospheric, soft interface activating different levels of human-machine participation, delusion, and wonder. 

“It’s a blob. It’s a brain. It’s a misshapen tent. It’s an artificial, cybernetic alien monster that lures you toward it, so it can steal parts of your body.”

Peter Crimmins, WHYY, HotBed Gallery in Philadelphia Exhibition

Fig. 6. Press titles reveal the various interpretations of the work.

Prior to entering the exhibition space, an ambient sound can be heard from the distance. This sound is not music nor noise, rather something in between. There are moments of calm sound patterns and others of agitation with intense rhythms of metallic variations that squawk at the listener. A soft mechanical rehrr in the background is discernable as well, sounding similar to the noise of an air conditioner constantly vibrating. 

In darkness, here comes the composed sound: electronic compositions in Stefan Schmidt’s Rodenkopf 

Audio 1: The album rodenkopf by Stefan Schmidt

“The first thing that came to my mind when I listened to the sound—of the installation—was Vangelis’ film scores, especially his Blade Runner composition. But then, the sound changed entirely and forced me to rethink my references.”

Participant comment: Municipal Theater of Piraeus Exhibition

<p>Fig. 7. Entry to the installation at the Municipal Theater of Pireaus. This is the first “stage” of the overall experience in which the visitor hears <em>Synesthesia</em> without seeing it. The wall showcases a collection of the installation’s “internal organs”: structural joinery components and a sampling of the iterative process for the creation of the “eyes.” Fabricated using 3D printing technology. <span>Photo by </span><em><span>Thomas Daskalakis, </span></em><span>NDP Photo Agency.</span></p>

Fig. 7. Entry to the installation at the Municipal Theater of Pireaus. This is the first “stage” of the overall experience in which the visitor hears Synesthesia without seeing it. The wall showcases a collection of the installation’s “internal organs”: structural joinery components and a sampling of the iterative process for the creation of the “eyes.” Fabricated using 3D printing technology. Photo by Thomas Daskalakis, NDP Photo Agency.

The inner projectors initiate: their background noise is superimposed on the composed sound and a beam of light is cast on the cinematic skin from within, waiting for enaction. 

<p>Fig. 8. Inner view showing the projectors’ light beams masked in the shape of an eye and cast on <em>Synesthesia</em>’s surface.&nbsp;</p>

Fig. 8. Inner view showing the projectors’ light beams masked in the shape of an eye and cast on Synesthesia’s surface. 

During that moment, prior to fully engaging with the installation, a dynamic light splashes against the surrounding surfaces of the exhibition space. The visitor leaves their previous state of mind behind and adjusts their perceptual center as they enter a new event. When entering a darkened room, the participant needs to accommodate to an event led by artificial lights with modifiable intensity, positioning, and color attributes. 

“Once I entered the dark space I found the form weird, almost disturbing. It made me feel anxious but also curious. In slowly familiarizing myself with the darkness and when I eventually came closer, the overall experience felt soothing.”

Participant comment: European Cultural Center, 17th Venice Architecture Biennale Exhibition

<p>Figs. 9 and 10. Photographs taken during the installation’s press debut at the Ravenhill Chapel in East Falls, Philadelphia, PA.&nbsp;</p>

Figs. 9 and 10. Photographs taken during the installation’s press debut at the Ravenhill Chapel in East Falls, Philadelphia, PA. 

“It's an egg,” “it’s a bubble or a drop,” “it reminds me of neurons,” “an atom,” “some sort of a sea creature, a bioluminescent jellyfish!!”Participants comments from the various spaces the work has been exhibited.

<p>Fig 11. <em>Synesthesia</em> at the neoclassical building of the Municipal Theater of Piraeus in Athens, Greece. Photo by Thomas Daskalakis,<em> </em>NDP Photo Agency.</p>

Fig 11. Synesthesia at the neoclassical building of the Municipal Theater of Piraeus in Athens, Greece. Photo by Thomas Daskalakis, NDP Photo Agency.

“The contrast of Synesthesia’s form and softness to the theater’s neoclassic architectural features, its geometric ceiling ornamentation, and its hard marble resembling patinas creates a juxtaposition that forces contextual awareness.”

Participant comment: Municipal Theater of Piraeus Exhibition

Fig. 12. Installation process of the artwork at the great hall of Trajan’s Market Museum of the Imperial Fora in Rome. 

“The context is striking. Synesthesia with its multimodal sensoriality, effectual character, and lightness, next to the rich ancient material and symbolic layers of the Fora and in dialogue with the weight of the Roman caryatid that faces it. It made me think of wormholes, a transtemporal trip.”

Participant comment (archeologist): Trajan’s Market Museum of the Imperial Fora, Rome Exhibition

<p>Fig. 13. A large black container was constructed to host the three “stages” of the artwork and ensure a dark environment.</p>

Fig. 13. A large black container was constructed to host the three “stages” of the artwork and ensure a dark environment.

“Folds, blobs, nets, skins, diagrams: all words that have been employed to describe theoretical and design procedures over the last decade. […]The new vocabulary has something to do with contemporary interest in the informe; it seems to draw its energies from a rereading of Bataille and a new interest in Deleuze and Guattari.”

“They are words and forms conceived and manipulated in a virtual space, with, nevertheless, an intimate relationship to production techniques and the technology of materials. Such a relationship would be impossible without the digital interface that construes information, theoretical and practical.”12

“In a series of writings between 1893 and 1895, [August Schmarsow] developed a psychological characterization of space, based on a concept of ‘intuited form,’ built up out of the sense of sight joined to “the residues of sensory experience to which the muscular sensations of our body, the sensitivity of our skin, and the structure of our body all contribute.”13

12 Anthony Vidler. Warped Space: Art, Architecture, and Anxiety in Modern Culture (Cambridge, Mass.: MIT Press, 2000).

13  Anthony Vidler. Warped Space: Art, Architecture, and Anxiety in Modern Culture (Cambridge, Mass.: MIT Press, 2000), p. 4. 

Video 1. Fabric Prototyping Process 

Conceived as a deformed sphere—a central, internal, node of multiple projections shaped like a blob and based on topological deformations—Synesthesia acts as a three-dimensional cinema, a complex, tensile physical skin that constitutes both a physical and a virtual interface. As a “folded” cinematic experience, it provides a theatrical space owned by the audience as they participate and engage with the cinematic narrative. 

“Perhaps I recognise similarities with my work in the idea of transmission and technological reproducibility. Maybe there is something in the idea of screen culture of simultaneously enabling both mass-customisation and mass reproduction.”

Comment by Nick Walkley, participant in Metode’s workshops and essay contributor

<p>Figs. 14 and 15. Artwork in the making process, a study of the skin.</p>

Figs. 14 and 15. Artwork in the making process, a study of the skin.

“It reminded me of a large Noguchi lamp. I know, sure! But, at the same time you can’t use his paper technique at such a large scale, can you? How is this skin put together? What is it made of? And most importantly, what’s the intent?”

Participant comment (industrial designer): HotBed Gallery in Philadelphia Exhibition

“Such a rich exploration of stereometric and topological form explorations, from the smaller internal pieces to the larger blob. Interested to know how all these came to be.” 

Participant comment (physicist): Municipal Theater of Piraeus Exhibition

One of the stages in the curation of Synesthesia at the various exhibition venues showcases the structural joinery components. They are designed to facilitate the traveling nature of the installation and make possible the fast assembly and disassembly of the structure into pieces that can be shipped across the globe. The various types of joints act in both tension and compression respectively. The different filaments and tectonics were iterated, tested, and evaluated as to their friction, strength, production time, and material density capabilities.

It also includes a sampling of the iterative process for the “eye” portals creation and the computing brain of the installation. This is an exhaustive analysis of formal and material aspects in search of a camera obscura effect. We examined the use of optical components to split incident light that have the capacity to both be transparent and reflective so the participants’ eyes can be detected by the hidden cameras without the participants detecting these cameras. The form developed as a hybrid referencing, on the one hand, the organicity of human organisms, and on the other, the geometries that accommodate the industrialized parts of small processors, sensors, cameras, and all systems comprising the “eye.”

<p>Fig. 16a. A collection of the installation’s “internal organs”. Photos taken at the Trajan’s Market Museum of the Imperial Fora Exhibition.</p>

Fig. 16a. A collection of the installation’s “internal organs”. Photos taken at the Trajan’s Market Museum of the Imperial Fora Exhibition.

<p>Fig. 16b. Photo at the Municipal Theater exhibition by <em>Thomas Daskalakis, </em>NDP Photo Agency.</p>

Fig. 16b. Photo at the Municipal Theater exhibition by Thomas Daskalakis, NDP Photo Agency.

Automated motors that rotate internal mobiles of dichroic, reflective material are ON.

<p>Fig. 17. Internal mobiles reflecting and refracting dynamic light patterns.</p>

Fig. 17. Internal mobiles reflecting and refracting dynamic light patterns.

Synesthesia’s fluctuating light and patterned sound effects attract the viewer to approach and engage in a sensorial “conversation.”

“If no one is near it, it calls out with an inviting chirp to let you know it is waiting for a visitor. Once you get close, it emits a series of grinding sounds.”

Peter Crimmins, WHYY, HotBed Gallery in Philadelphia Exhibition

Internal white lights turn ON, in slowly fading patterns following their networked “conversation.” They cast their light beams on the dichromic mobiles. Colorful reflections are in motion, in a stable pace until a human-machine conversation starts.

Video 2. Sequence of inner lights fading in and out. 

The work provides visual stimulation at low levels of intensity with dynamic light and color projections as animated visuals in Lumia formations referencing the work of Thomas Wilfred. 

<p>Fig. 18. Animated light formations cast on <em>Synesthesia</em>’s skin.</p>

Fig. 18. Animated light formations cast on Synesthesia’s skin.

“The ways the dynamic light effects are occurring is a mystery. I feel as if it is an experience of the cosmos. A very meditative environment that puts me “in the zone.” 

Participant comment: Municipal Theater of Piraeus Exhibition

a

b

c

d

e

Fig. 19 a–e. Photos of the dynamic lights cast on the artwork Surface. Taken from various exhibition venues.b. and d. photos by Maria Karatsiori, e. photo by Orlando Saverino-Loeb

“Suddenly, and only for a few seconds, the whole blob gets illuminated! The atmospheric multicolored light patterns disappear, and the artwork turns into a light bulb! It is like a theater director’s trick, a wake-up call of sorts, where the surrounding scenography is illuminated and the material and technical details are momentarily revealed. This is a moment that certainly increased my curiosity. I want to stay around longer!”

Participant comment: Municipal Theater of Piraeus Exhibition

Video 3. Variations in light effects

Air is circulated within. A breathing mechanism provokes vibration once participants are in proximity.

The work engages in tactile, embodied interactions with its low-frequency vibration that one can feel but cannot hear. Our sensory modalities are interconnected. Whether the visitor actively engages in all levels of sensorial interactions within Synesthesia’s narrative or is unconsciously impacted by them, the environment created by the artwork regulates the audience’s psychophysiological states, emotions, and feelings. By expanding oculocentric narratives to multi-sensory modalities, Synesthesia’s surface acquires new meanings and activates a holistic praxis of embodiment.

<p>Fig. 20. Vibration patterns on <em>Synesthesia</em>’s skin.</p>

Fig. 20. Vibration patterns on Synesthesia’s skin.

“The very nature of our bodies is what allows us to not only experience visually but also experience gravity, force, and pressure. Vision is multimodal; it encompasses the activation of motor, somatosensory, and emotion-related brain networks. More than twenty years of research on mirror neurons have demonstrated the existence of a mechanism directly mapping action, perception, and execution in the human brain.”15

In what ways does an understanding of surfaces as participating, active, and complex relate to an anthropocentric understanding of surfaces? In what ways does how a surface is experienced, as active or passive, relate to the precondition of human interaction with this surface?

Comment by Jakob Oredsson, participant in Metode’s workshops and essay contributor

“Consciousness, thinking, and communication activities are intrinsic to our bodies, responding to the characteristics of the physical, social, and cultural environments in which we dwell.”16

15 Vittorio Gallese and Alessandro Gattara, “Embodied Simulation, Aesthetics, and Architecture: An Experimental Aesthetic Approach,” in Mind in Architecture: Neuroscience, Embodiment, and the Future of Design, ed. Sarah Robinson and Juhani Pallasmaa (Cambridge, Mass.: MIT Press, 2017), 165. 

16 Henry Francis Mallgrave, “Know thyself: or what designers can learn from the contemporary biological sciences,” in Mind in Architecture: Neuroscience, Embodiment, and the Future of Design, ed. Sarah Robinson and Juhani Pallasmaa (Cambridge, Mass.: MIT Press, 2017), 13.

Video 3. Inner view of the blinking “eye” mechanism

“The blinking holes remind me of a brain, it feels like a billion synapses are taking place once I come close. Once I see through one of them, I can also hear an instantaneous sound, like an alien radio transmission. I feel like I am in conversation with my head.”

Participant comment: European Cultural Center, 17th Venice Architecture Biennale Exhibition 

Audio 2: Sound activated when the presence of the participants is detected by the “eye” sensor mechanism. Downloaded by freesound.org by Brotherprovisional. License by the Creative Commons

<p>Fig. 21 a,b. A series of portals act as peeking holes and, in blinking light patterns and sounds, invite human participants to come closer. <em>Synesthesia</em>’s blinking “eye” mechanism seen from the skin’s exterior and interior.<span>&nbsp;</span></p>

Fig. 21 a,b. A series of portals act as peeking holes and, in blinking light patterns and sounds, invite human participants to come closer. Synesthesia’s blinking “eye” mechanism seen from the skin’s exterior and interior. 

// Theme 2. Curiosity

The presence of the human eye near selected portals triggers its live video recording and its projection back into the cinematic space, while other portals are deceptive. Upon interaction, the participants’ eyes are captured and projected from within and in real-time on Synesthesia’s skin. The participant becomes an active part of the installation unknowingly. Their image is only perceived by observers, further back from the blob. 

Video 4. Activation of the inner projection once human presence in proximity is detected by the “eye” mechanism.

Synesthesia is being looked through but also looks outwards. The portals are simultaneously the “eyes” of the installation that look back at the participants. Synesthesia’s inner gaze of the interactions is live-streamed on the installation’s webpage, creating an active exchange between the installation, the participants, and the wider public, thus cutting across local, global, and virtual scales.

<p>Fig. 22. Participants interacting with <em>Synesthesia</em> physically. Photo by <em>Thomas Daskalakis, </em>NDP Photo Agency.&nbsp;</p>

Fig. 22. Participants interacting with Synesthesia physically. Photo by Thomas Daskalakis, NDP Photo Agency. 

<p>Figs. 23 a,b. Participants interacting with <em>Synesthesia </em>virtually with the live streaming component presented on TV screens.&nbsp;</p>

Figs. 23 a,b. Participants interacting with Synesthesia virtually with the live streaming component presented on TV screens. 

“I can see the interior now, I think through the screens in the back. Was that me projected? Broadcasted on the web?”

Participant comment: Municipal Theater of Piraeus Exhibition

Video 5. Livestream video of the inner gaze of Synesthesia presented in a separate room prior to Synesthesia’s physical blob at HotBed Gallery, Philadelphia, PA. 

“The other side of the interface. Does it matter which?” 

Participant comment: Municipal Theater of Piraeus Exhibition

“I wonder if it is worth exploring that surface and what it is like when it is turned off, before going into the magic that happens on the other side of them.”

Comment by Nick Walkley, participant in Metode’s workshops and essay contributor

In his Deep Surface essay, Ingold defines the term “interface” as having “two sides: whether a top side and an underside, or an outside and an inside.” However, by drawing on Gibson’s ecological approach to perception, Ingold stresses that an interface is also “perforated by holes or keys that allow information, and sometimes materials, to pass from one side to the other.” Synesthesia makes a reference to the holes found in the often opaque enclosures covering construction sites or private spaces in the city. As city dwellers encounter these unattended openings, they are allowed to explore the “other side” and gain visual access to the unknown territory and the unfolding of events that occur within it. On the one hand, the enclosure functions as the thin urban layer separating the public from the private. On the other hand, the opening in this enclosure functions as a temporary fusion of the two urban fields, as a flashing point—an analogy to the installation’s eyes—that turns from black to white and vice versa on the nolli map of the city. Similarly, in Synesthesia, the unconscious, accidental, and curious performance of the participants actuates Synesthesia’s structured “veil.” The structure invites, the body is pulled in, and in blinking patterns, our organs of perception are dynamically projected back onto the surface as abstracted images, only to disappear shortly after. 

Video 6. Inside_Outside explorations.

(A hidden secret:) “There is an interior to be discovered but it is hidden from me. My friend told me that there is a hidden camera that does something cool. I am intrigued to debunk the secret, to understand the technique. Where is the music coming from? How does it interact with me, with you? There has to be something else to this, but I don’t yet understand what. Maybe the issue is manifested in the interior?”

Participant comment: European Cultural Center, 17th Venice Architecture Biennale Exhibition 

<p>Fig. 24. <em>Synesthesia</em>’s view from the outside, enacted surface. Photo by Francesco Di Majo (@fdimajo)</p>

Fig. 24. Synesthesia’s view from the outside, enacted surface. Photo by Francesco Di Majo (@fdimajo)

<p>Fig. 25. <em>Synesthesia</em>’s view from within capturing the participants’ eyes while interacting with it, layered with the internal structure and the light patterns of the piece.</p>

Fig. 25. Synesthesia’s view from within capturing the participants’ eyes while interacting with it, layered with the internal structure and the light patterns of the piece.

“How is everything connected?”

Participant comment: Municipal Theater of Piraeus Exhibition

<p>Fig. 26. Interactive systems diagram.&nbsp;</p>

Fig. 26. Interactive systems diagram. 

The installation is a digital and social microenvironment that operates at various scales. On the internal microscale of Synesthesia’s body, all the computational and sensory units communicate with each other and eventually create their own network, a microsystem held within Synesthesia’s soft wrapper. 

// Theme 3. Enacted Cinema

<p>Fig. 27. The installation’s surface as enacted by the participant/performer.</p>

Fig. 27. The installation’s surface as enacted by the participant/performer.

Synesthesia “bends” the traditional planarity of the cinematic surface. Referencing the term enaction in the visual arts, Synesthesia embraces the shift from traditional cinematic spectacles to works probing the frontiers of interactive, performative, and networked media.17 The project translates cinematic references into unexpected paths of engagement, participation, and immersion. Enactive cinema in that sense became the perfect transgressive subfield to relate to in a process of post-conceptualizing the installation Synesthesia as a subfield of cinema that meshes the performative values found in theater and interactive art in blurring the “fourth wall.”18 The screen surface, accordingly, is not inert but rather responsive. With that, the folding, enacted surface merges the audience and performer. 

17 Gene Youngblood described “expanded cinema” in his 1970 book Expanded Cinema as an explosion of the frame outward towards immersive, interactive, and interconnected forms of culture. Interestingly, Youngblood also wrote extensively about consciousness, cybernetic cinema, computer films, and synesthesia.

18 In conceptualizing and analyzing the theories on embodiment, cinema became an art field too important to overlook. It is perhaps a crucial medium that best absorbs the translation and manifestation of our theoretical framework onto our praxis. The enactive cinema questions the static relationship that traditionally has existed between the viewer and the cinematic screen and aims to make that very relationship the motive and concept of the film being shown. It thus generates a performative dialog that loops between the space of the cinema, the viewers involved, and the narrative presented. Traditionally, even though the visual and hearing brain functions are activated and are dynamically driven in the cinema space as it stands, the level of bodily driven performance remains a static, introverted, and individualized experience. In response, a fundamental question explored by the enactive cinema assigns a larger role to body performances as it relates to embodied cognition. How the narrative unfolds and how rhythm and soundscape emerge both depend on how the spectator experiences the emotional dynamics between the characters. A fundamental aspect of such cinema resides in the creation of sensory triggers as explained via the enactive theories of the mind, where inputs and outputs promote real-time variations of an environment that, in turn, promotes further performative tasks from the participants themselves, in a real scale cybernetic-like spatial experiment that serves as a cognitive exalting mechanism. 

“The surface is not inert but always participates.” Has any surface ever been passive? Is Synesthesia perhaps exemplifying that this has never been the case?

Comment by Jakob Oredsson, participant in Metode’s workshops and essay contributor

Video 7. 360-degree cinematic view

// Theme 4. Agency

“Such an electric atmosphere; it alters the ordinary conditions of sensation.”

Participant comment (curator): Trajan’s Market Museum of the Imperial Fora, Rome Exhibition

Synesthesia’s interactive interfaces expand human agency through sensors, actuators, real-time responses, and a human-machine participatory constitution. Collectively, the installation equates environment to experience and mediates our bodies’ agency via interlacing action and perception.19 Designed as a traveling experience, Synesthesia engages a variety of communities and places. This inclusive aspect of the work amplifies the narratives and cultural interactions triggered by it, in developing methods for ethnographic impact and observational analysis.

In writing this experimental essay, we hope to have shown how the notion of surface cannot be taken as an isolated domain, since it holds multiple functions: a physical space, a medium, and a responsive discourse. Furthermore, performance and participation are sought not only as an experience, but as a method of research and a way to create knowledge and awareness of our bodies and our technological environment. 

Video 8. Overlapping participants’ organs, superimposed on top of each other. 

Time to rest. In the darkness, back to silence, what have we experienced, alone, collectively?

“Perhaps invite the reader to turn the screen off, to contemplate the screen itself, as part of the experience of reading the essay.”

Comment by Nick Walkley, participant in Metode’s workshops and essay contributor

<p>Fig. 28. Views from one of the portals on the large-scale structure created to host <em>Synesthesia</em> in darkness while exhibited at the Trajan’s Market Museum of the Imperial Fora, Rome. A view of multiple scales and dimensions, across fluid spaces and time.&nbsp;</p>

Fig. 28. Views from one of the portals on the large-scale structure created to host Synesthesia in darkness while exhibited at the Trajan’s Market Museum of the Imperial Fora, Rome. A view of multiple scales and dimensions, across fluid spaces and time. 

Video 9. Instantaneous “finale” … until the next participants join.

References

Alfonso, Severino, and Loukia Tsafoulia. “A Vision of Complexity. From Meaning and Form to Pattern and Code.” In Narrating the City: Mediated Representations of Architecture, Urban Forms and Social Life, edited by A.A. Kavakoğlu, T.N.Hacıömeroğlu, and L. Landrum, 74-92, UK: Intellect Books, 2020.

Alfonso, Severino, and Loukia Tsafoulia. “Performance as Action: The Embodied Mind.” In proceedings of the 2021 International Architectural Research Centers Consortium Performative Environments Conference, 329-336, Tucson: University of Arizona, 2021.

Busbea, Larry. The Responsive Environment: Design, Aesthetics, and the Human in the 1970s. Minneapolis: University of Minnesota Press, 2020.

Dutta, Arindam, ed. A Second Modernism: MIT, Architecture, and the “Techno-Social” Moment. Cambridge, Mass.: MIT Press, 2013.

Gibson, J. J. The Senses Considered as Perceptual Systems. Westport: Greenwood, 1966.

Gibson, J. J. “The Theory of Affordances.” Hilldale, USA 1, no. 2 (1977): 67-82. 

Halland, Ingrid. “Error Earth: Displaying Deep Cybernetics in ‘the Univeritas Project’ and ‘Italy: The New Domestic Landscape’, 1972.” PhD, University of Oslo, 2018.

Harvey, Matthew, Rasmus Gahrn-Andersen, and Sune Vork Steffensen. “Interactivity and Enaction in Human Cognition. Constructivist Foundations 11, no. 2 (2016): 234-264. 

Ingold, Tim. Imagining for Real: Essays on Creation, Attention and Correspondence. London: Routledge, 2021.

Kimbell, Lucy. “Rethinking Design Thinking: Part I.” Design and Culture 3, no. 3 (2011): 285-306.

Parisi, Luciana. Contagious Architecture: Computation, Aesthetics, and Space. Technologies of Lived Abstraction. Cambridge, Mass.: MIT Press, 2013.

Robinson, Sarah, and Juhani Pallasmaa. Mind in Architecture: Neuroscience, Embodiment, and the Future of Design. Cambridge, Mass.: MIT Press, 2017.

Steenson, Molly Wright. Architectural Intelligence: How Designers and Architects Created the Digital Landscape. Cambridge, Mass.: MIT Press, 2017.

Varela, Francisco J., Evan Thompson, and Eleanor Rosch. The Embodied Mind: Cognitive Science and Human Experience. Cambridge, Mass.: MIT Press, 1991. 

Vidler, Anthony. Warped Space: Art, Architecture, and Anxiety in Modern Culture. Cambridge, Mass.: MIT Press, 2000. 

Youngblood, Gene. Expanded Cinema: Fiftieth Anniversary Edition. NY.: Fordham University Press, 2020.

Synesthesia Installation Team

Creators: Severino Alfonso, Loukia Tsafoulia
Synesthetic Research and Design Lab, College of Architecture and the Built Environment, Thomas Jefferson University

Music: Stefan Schmidt, Rodenkopf, www.musicforoverexposedcelluloid.com

Lead Interactive Engineer: David Azar

Live Feed and Real-Time Display: Matthew Ross

Lights Automation Technical Consultants: Julia Lu, Adnan Aga

Project Assistants: Olivia Birritteri, Abigail Kern

Fabric Prototype Collaborators: Anne Hand, Kim Rosner

Fabric Construction: RoseBrand Inc.

For more information about the project and collaborators, please visit:

https://www.synestheticdesignlab.com/synesthesia

Severino Alfonso and Loukia Tsafoulia, “Synesthesia: An Experiment on Enacted Surfaces,” Metode (2023), vol. 1 ‘Deep Surface’