Time Based Drawing/Painting in Immersive Virtual Reality Environments
Authors: Gabriella Kardos and Michael Casey
E-mail: gabriellakardos@gabriellakardos.homechoice.co.uk
m.casey@gold.ac.uk
OVERVIEW
The installation will consist of a physical space containing virtual containers of paints of different colours, a set of tangible objects of different shapes (such as brushes, sticks, balls) and an immersive display system (i.e. VR theatre or SAS3). This space becomes the ‘studio’ of the artist. The artist/spectator/ participant enters the studio space and begins to experiment with virtual painting/drawing by picking up some of the ’brushes’ or other objects that have been designed to function as active transmitters of information. Electric field-sensing input devices will enable the artist to draw using brushes or hand/body gestures.
The research involves the design of a system implemented via artist/scientist collaboration for creating dynamic time-based ‘immersive painting’. The system will have a Play Back option that will enable the viewer to ‘rewind’ the entire creative process and view it again.
The system will also offer the possibility to record sound (words spoken by the performer while she paints, sounds of her body in movement), and manipulate sounds in real time from sources such as nature sounds – water, wind, rain, a heart beat, etc. At the time when the work is revisited these will then be re-played simultaneously with the unfolding of the painting process, thus helping to reconstruct the mood or the creative ambiance present at the time of creation and further help us understand the relationship between environment, the emotional/creative state of the artist and the final output.
ASPECTS OF THE PROPOSED WORK
Using the Network
The artist’s brushstrokes, gestures and sounds can be transmitted over a network to display systems in other locations. To do this the information will need to be interpreted depending on the capability of each client device, such as displays of different sizes and with differing features. Network protocols will be developed for communicating gestures, three-dimensional imaging and audio, that signal the requirements for displaying each type of data. Depending on location, therefore, the experience will be framed in a different way thus adding variation due to cultural and technical factors.
Play Back Option
The artist’s performance will be recorded on a timeline, so that the viewer could rewind the entire creative process by the push of a button. This is a dramatic shift in experiencing a painting as a finished product. One can sit in a chair and watch the brushstrokes emerge one by one, until a complete painting has unfolded, an experience close to that of listening to music where every stroke is like a note, every colour a new instrument. This ‘painting in action’ offers a new way of experiencing the artwork: the viewer is taken on a journey of creation. The element of surprise, the use of chance, the build-up of a crescendo, all these and other factors which are not at first apparent in a static work will be made visible, thus highlighting the ‘process’ rather than the final outcome. The viewer gets the privilege of witnessing this very private activity otherwise buried within the layers of paint. By the same token, the artist could go back in time and make changes to the ‘history palette’.
The Element of Space-Time
- Bridging the Gap between Performance and Drawing/Painting
The intent of a Dance performance is not the drawing, or the tracing of a drawing through space, but the performance itself – the movement of the body through space and time. The act of movement is the performance. Drawing, on the other hand explores the trace left by the moving object upon a surface. Time-based virtual drawing/painting bridges the gap between these separate, but similar disciplines by sharing the element of space-time. 20th Century artists such as the Italian Futurists Giacomo Balla and Umberto Boccioni and others, such as Duchamp with his Nude Descending the Staircase have explored this idea by trying to capture the imaginary trace of an object moving through space, but without being able to ‘reveal’ the unfolding of the movement through time.
Creative and Viewing Experience
Time-based virtual drawing/painting is to be experienced either in private (where the performer/artist is alone with her tools in the immersive environment – CAVE/SAS3) or in public (where the immersive environment is open to spectators, i.e. VR theatre.)
Thus, the medium offers a multi-usage. It can be used:
1. By dancers/artists in public performances
2. As a private medium
3. For educational purposes, to draw complex, spatial diagrams or to explore and compare creative processes of different artists
Jackson Pollock felt like a ‘monkey’ when he was filmed by a reporter painting on glass so that his creative process would be recorded for posterity. Imagine if he could have worked with no one else watching, finish the canvas, leave the room and then return to it the next morning to wipe it clean by the push of a button which would rewind the process. It would mean travelling back in time. Then, by pressing another button he would become the spectator of his own painting performance as the painting would play itself back – ‘paint itself’ once again.
Interactivity
The viewer becomes an active participant either by ‘painting’ in the immersive environment or simply by watching what other artists/members of the public have created before, a feature implemented by the Play-back button. The Play-back option will enable the observer to study carefully every detail of the painting performance by stopping and playing the performance at any stage of its development. Eventually a whole library of painting performances will be accumulated which will be available to the public to experience.
In this installation members of the public are given the choice: to view the work of others or to do their own work. They can choose the role they want to play and the degree of engagement with the work.
Drawing/Painting in ‘Space’
Drawing/Painting in an immersive environment revolutionizes the very nature of the medium by getting away from the two dimensionality of the canvas/wall, liberating it from its ‘support’ – the flat surface.
Revival of the Medium
Since the 70s we’ve been told that “painting is dead”, only to be revived for a short while during the 80s and then marginalized again in the mass of conceptual art and the emergence of new media and the net. With time-based painting in a virtual world the medium loses the physicality which forms its very essence, but gains in ethereality, a change not for the better or the worse, but one which marks a move into the new reality paradigm of the age we’re living in.
Experimentation with VR
This artist studio will help advance experimentation with virtual reality by offering an alternative mode of engagement with VR space: the participant’s role goes beyond that of navigating through a previously constructed replica of physical reality, her role now being to ‘create’ a new world in a very intuitive manner. The world resulting from this process would carry the participant’s point of view, thus giving new meanings to VR space. VR environment becomes a new communication space.
ABOUT THE AUTHORS
Gabriella Kardos
Kardos’s own experimentation has taken her from a traditional form of artistic expression (oil on canvas) to using digital media. Presently she spends most of her time working with digital photography exploring its non-objective, instantaneous results, which paradoxically arise out of a so-called objective medium. The photographs, as well as other related outputs, such as video sequences and sounds are being stored in a library of she is creating in preparation for their use in performance space in real, virtual or mixed reality worlds. The direction in which her research is going is that of time-based transformation of the artwork in which the viewer is present in an immersive space and constructs his/her own narratives.
Michael Casey
Dr Michael Casey is head of the Centre for Creative Computing at Goldsmiths College, a researcher specialising in the areas of machine learning, machine creativity, sound recognition and multimedia information systems. He has made significant and lasting contributions in both scientific and artistic aspects of music and media technology. His recent output has included automatic music generation from Bayesean statistical models of recorded music, Independent Subspace Analysis for blind separation of single-channel mixed signals, factorial hidden Markov models (FHMMs) for separation of single-channel mixtures with known signal characteristics, and minimum entropy Bayesian methods for general sound recognition and structure discovery in music and sound recordings, [1,2,3]. He is a member of the Moving Pictures Experts Group (MPEG) committee of the International Standards Organisation (ISO) and an editor for the MPEG-7 International Standard for Multimedia Content Description for which he has contributed several of the standardised audio descriptors and description schemes.
“... AND THE SLITHY TOVES”
AUTHOR: MARK PALMER
Homepage: http://www.
E-mail: Mark.Palmer@uwe.ac.uk
Abstract
A limited set of concepts has defined the evolution of user interfaces. From the development of the computer as the universal machine to the desire to reproduce actions we take for granted in the physical world instrumentality has been privileged.
This is in contrast to the discoveries made through computers. The discovery of emergent properties has led to a questioning of the hierarchies that have been assumed to dominate reality. In conjunction with the processing power of computers Boolean logic has challenged the belief that there is a set of laws imposed upon an otherwise inert matter and revealed that it may be the relationships inherent within things themselves that generates order.
But beyond the truisms of butterflies beating their wings in South America Boolean logic has other consequences; consequences that were creatively explored long ago, perhaps most famously, by the mathematician Charles Dodgeson. Better known as Lewis Carroll, we discover this exploration through the dialogues and events of the Alice stories and texts such as The Hunting of the Snark 1.
This aim of this proposal is the sensuous exploration of this logic not as an alternative reality but as a means to generate new ways of thinking about, understanding and experiencing what we otherwise dismiss as the everyday.
Concept of Work & Project Description
In some ways Carroll’s stories are as equally identifiable by Tenniel’s illustrations as they are by the texts themselves. If nothing else Tenniel’s wood engravings are the reference point by which we consider the iconography of Carroll’s worlds. What’s particularly appealing about Tenniel’s work is the way in which the engravings possess lines that seem to articulate the forms that he wishes to portray. In this way his illustrations seem poised on the brink of action.
It is this quality of surface and the structure of events that will form the focus of the work. Rather than using static texture wraps as a means to simulate ‘reality’ the intention is to create dynamic texture maps that move across the surface of a scene’s geometry. Borrowing from the style of wood engraving, these simple black and white lines might develop in any number of ways, moving slowly and then quickly, in this direction and that, forming vortices, eddies and flows that might develop in a fractal nature. What’s more different objects in a scene might be played against each other raising questions of relativity. Much like the way in which Gilles Deleuze provides an analysis of this in The Logic of Sense we can think our train is leaving a station because it is in fact the one that’s next to us that’s moving these textures can be used to startle users out of their preconceptions. Apart from a number of pre scripted elements (for instance falling down the rabbit hole to enter the environment suggests itself as one) the system can then be linked into an IA 2 engine (interaction of actors) to generate events depending on the behavior of the system’s users.
Borrowing from the illustrative conventions of books the rendering of this environment can take place across a number of platforms. The problems usually associated with the problem of having to choose one viewpoint to render the scene for a CAVE falls away. Where once other viewers would see the discontinuity between screens as a rupture of the illusion this can now be played on as these surfaces switch between a sense of immersion and the pages of a book.
Indeed this might provide an opportunity to play extending the conventions of a CAVE by using augmented reality within it’s walls so characters enter the space of the CAVE as if they had come off of it’s ‘pages’. The use of ‘pages’ would also allow new forms of navigation to be developed. Scenes could be fractured (in much the same way that multiple windows are used for multiple players within games) or stretched. Rather than one user controlling the experience for all optical tracking would allow relationships between user and screens to develop in such a way that their body becomes the interface. But equally important these conventions would not disadvantage online participation. Indeed the fact the some participants would not be present provides the opportunity to play with the shifts in size so often employed by Carroll.
Other opportunities also offer themselves. If it were chosen to use avatars for online participants it may be the case that we wished to mark their entry into a scene. If the conventions of cel-shading were used to render the scene, new entrants might enter as stylised ‘Disney’ characters before becoming integrated into the world. Conventions such as being ‘fragged’ and ‘respawned’ might be played with in terms of the rendering style of one’s character.
Technical Description
The project would make use of desktop and CAVE environments. The most significant elements of the project would be the development of ‘texture’ and ‘emergent event’ engines. Tracking technologies could be adapted from work already carried out. Perhaps most importantly the proposal can be broken down into a number of achievable goals that can be built upon depending on research budgets and staff time, and could indeed be built into further funding proposals.
Drawings:
Online participation provides opportunities to deal with shifts in scale. The movement and attention of participants would affect events.
About author:
Mark Palmer is the first Watershed Senior Research Fellow in Digital Media at the University of the West of England. His work emerges out of a history of collaborative practice that included the award of a New Technology Arts Fellowship hosted by the University of Cambridge and The Junction and an AHRB fellowship in the Creative and Performing Arts. Having been a sculptor and installation artist who began working with immersive virtual reality his research interest lies in creative genuine multiuser environments where all participants are equal. A part of this project he has also had work published on the philosophy of Deleuze and Spinoza and the implications of the new sciences of chaos and complexity for our understanding of experience and the potential of new media. He has worked on the CADE committee, is one of the editors of Digital Creativity and in 2003 he was elected as CADE’s chair.
WONDERLAND: “SOMETHING BETWEEN TEUTONS AND ALICE´S ADVENTURES”
AUTHOR: LUCIE SVOBODOVÁ
E-mail: luciesvobodova@volny.cz
Concept: The basic concept of this virtual reality installation comes out from a Teutonic spiritual tradition, which says that the Earth is a cavity sphere, with sun situated in the centre of its inner space and we are placed on the inner surface of its shell.
There is colourful, fanciful landscape inside the sphere, which is based on distorted notion of reality known from Alice´s Adventures In Wonderland. The surface of the landscape is divided into irregular chess board fields, separated from each other by glass walls. These walls are coming out of the ground and head upwards, in a way similar to the laylines net of the Earth. There are sixty four of these fields as on the chess board, but they are irregular and their size is varying.
The Earth
Each field of the landscape has a different character in the term of visual aspects and system of rules. Visitor can pass through the glass wall to another field under certain defined conditions. Looking from the outside, the glass walls resemble labyrinth, the maze made out of glass. Being inside, we can see just a small part of the wholes.
Individual landscapes resemble various sceneries known from the Earth, but they are not modelled as realistic copies of real scenes. Everything is presented in changed, fanciful form: deserts, mountains, volcanoes, seas etc., are stylized in various formations, with modified colours, strange plants, mushrooms etc. From the perspective of visitor, the whole space seems to be without limits, because it is placed on surface of the sphere. So there is no beginning, no end. There are surprises hidden at various places, the landscape it built enough spacious to move through for hours
Every field represents the specific ideology, religion or philosophy, which is demonstrated by the relevant behaviour of inhabitants and objects situated inside. Visitor meets a being or beings in every room, occupying solely the space surrounded by glass walls. They have a concrete relation to world-view characterising the specific field, e.g. they could be, by their nature and behaviour, exponents of particular school of thought. They can also illustrate it by direct actions, for example by changing applicable physical laws, through a mutation of objects, characters or forces, regarding to the tradition and premises of their ideology.
Landscape with irregular fields
As in Alice In Wonderland visitor wanders through the landscape, trying to get through to the next spaces. He can enter the space at any location and travel as long as he wishes and it will be possible to move through this space in two ways – by quick jumps in according to a map, or slowly, inspecting one room by the other in details. Visitor can realise, via interaction with inhabitants and objects, which ideology is ruling over that area and he can develop communication in according to that particular school of thought. The communication is nonverbal, visitors and inhabitants use various other ways of understanding to each other – gestures, touch, visualising of thought with the help of signs. Also a large number of relevant clues will be placed in rooms within a specific field, but everything will work visually, with no text being used. The inhabitants will be able to transfer their qualities of character onto the participant, if he is willing so, but that will finish previous ones. The whole project is based on comprehensibility, on crossing the language barriers, with the help of visual information. The soundtrack is just music and sound effects. No words.
Scenario:
There will be various special effects at the landscape and weather will be changing, all depending on outer forces (parameters controlled through the network). Overall character of environment including visual atmosphere depends on concrete participant’s mood. Specific changes of the mood are measured with thumb sensor (electrical activity on the skin surface), it is some way of “reading emotions”. The movement will be controlled by spaceball or joystick. The whole environment will be presented in cave like immersive display system with the 3D glasses, so there will be the real notion of space. The rooms will be also accessible from the internet, if the visitors meet certain conditions. It will be possible through basic browsers and newly created interface. The visitors will have the possibility depict themselves as the 3D avatars, looking the way the visitors wish. They can choose from 64 various ideologies (individual faiths, atheism, shamanism, animism, hermetism – roughly in accordance with the amount of people, who follow that ideology).
About author:
From the very beginning she inclined to use electronic media. She has accomplished five short animated films using combination of classical painting and computer animation. One of the films is based on infinity fractal space, principally like geometric flower. She collaborated with Czech artist, Michal Gabriel, on virtual sculpture, realized interactive projections, 3D graphics and touch pictures. Two years she worked on complex virtual reality installation controlled by EEG sensors. She works with movement, space, interactivity. She used different kinds of electronics media – laser, 3D space, animations, new software and hardware, Internet, virtual reality, artificial live, digital picture and sound. She experiments with different kind of possibilities – sound, word, static picture or movement, biofeedback, mental activity, psychotropic essences and new technologies.
Lucie Svobodova is, from the beginning of her art activity, transforming one basic, but a very extensive theme. She is looking for new forms and, in the same line, for ancient traditions and archetypes. She creates new space and reality, she reveals new relations among usual objects and principles, and also creates new ones. She tries to show, with the help of new technologies, very old mystery of our lives and more discover it.
Education:
1977-81 high school SUPSˇ, Prague, Art Painting 1982-88 University of Applied Arts Prague, Film and TV graphics
Dividing Lines: Re-Constructing the Berlin Wall
Author: Tamiko Thiel
Homepage: http://mission.base.com/tamiko/
E-mail: tamiko@alum.mit.edu
Introduction:
For almost 30 years the Berlin Wall was the most visible incarnation of the Cold War, a brutal swath cutting through the very fabric of a metropolis and the lives of its inhabitants. Now the Wall has almost completely vanished and the few remaining pieces fail to convey what it was like to have a death strip running through the very heart of a metropolis.
Working with Berlin architect Teresa Reuter I will rebuild selected parts of the Wall where it cut especially deeply into the social and urban fabric of the city. We will incorporate archival material of human events that happened at the Wall with a virtual kinesthetic experience of the Wall's impact on the topography of the neighborhoods it destroyed. Past projects have shown me that when concepts of metaphor and surrealism are used to incorporate 2D archival material into an interactive 3D space, users can experience the virtual space as a powerful - and personal - encounter with history.
We would be interested in implementing this as a site-specific "augmented reality" project wherein users could stand in selected locations in Berlin and experience the overlapping realities of the current built environment and the way it used to be.
Use of ALTERNE platform:
I would be interested in collaborating with ALTERNE consortium members in the areas of virtual storytelling, AI technology and new visualization techniques for immersive CAVE-like environments, integrating these technologies into the interactive narrative artworks that I show internationally in venues such as SIGGRAPH, the ICA Media Centre/London and the Tokyo Metropolitan Museum of Photography.
Some of the questions I wish to address in my work:
• Narrative devices: How can we use objects or characters within the narrative to gently entice users to participate in the dramatic structure of the piece? These devices must be readable even to relatively naive users. They must lead users on, give them clues of alternative paths, engage their interest and participation. These devices can include doors, pathways, odd or remarkable objects, characters (human and otherwise) who function as 'lures' and circumstances (environmental or characters) that compel the user to move in certain directions or take certain actions. (The book Alice in Wonderland is a case study in such devices.)
• Tactile interactions with the virtual environment: How can the interface between user and virtual environment be constructed such that both the physical interface itself and the way that the user interacts with it is an integral part of the environment and its narrative? Ideally the interface device should both bring the content of the virtual world into the real world and make users feel as if through the interface device they are directly affecting the virtual world. My favorite example is the camera in Maurice Benayoun's World Skin, where the act of taking a "photograph" of the virtual space eradicates everything that was visible in the viewfinder from the world (http://www.benayoun.com/) An important theoretical text for this topic is Mihaly Csikszentmihalyi's book, "The Meaning of Things," which discusses how objects (and images) become "objects of power," i.e. acquire meaning for an individual.
• Intelligent agents: Character design and programming is extremely time intensive, and artists unfortunately lack the resources of game companies. Characters are however sometimes needed to give a sense of life to a scene (animals, humans who go about their own lives in the background) and/or to interact with the user in specific ways that lead the narrative forward (supply information, lead the user "physically" forward, provoke specific emotional states and reactions in the user.) What compromises can be struck between 2D and 3D, what sort of interaction with what sort of characters is really needed in the context of a specific piece? Are any libraries of behaviors being developed that artists can tap into, or does the wheel have to be reinvented every time since all code is proprietary?
Context:
I have been working on the development of dramatic structure for time-based media since 1991, first with abstract, non-narrative video and since 1994 with interactive 3D (virtual reality.) The uniqueness of my method is that in order to deal with the "Gesamtkunstwerk" qualities of interactive 3D it unites insights from the fields of music, theater, architecture, urban planning, installation art and computer interactivity. It does this by focusing not on the specific type and content of interactions themselves, but on the dramatic structure of the overall emotional experience provoked in users by their interactions with a virtual environment. Besides applying this method in my own work I have taught courses and workshops at UC/San Diego, at the USC School of Cinema, at the Bauhaus-University/Weimar, and in numerous lectures at other universities.
Until now I have focused on developing my methodology of dramatic structure in interactive narrative. I am currently looking for institutional support for my work in order to incorporate technologies such as synthetic agents and tactile computing into my artworks as well. I believe that the complex artistic content and depth of my projects, combined with their international exposure once finished, offer a good test bed for ALTERNE consortium members working in emerging technologies and can provide them with extensive exposure, feedback and user testing.
As an example, my virtual reality installation Beyond Manzanar was to be shown for 6 months in the exhibit "Highlights of the Permanent Collection" at Silicon Valley's San Jose Museum of Art. It has proved so popular that they have extended it to run the entire 2 years of the exhibit. Despite its grave subject matter - internment camps in America - its game-like quality fascinates children; despite its high-tech execution the interface is simple enough and the subject matter compelling enough to attract elderly visitors to interact with the piece - including many former internees of the Manzanar Internment Camp, who use the piece to explain their own experiences there to their grandchildren.
(http://mission.base.com/manzanar/)
Focus:
My method for creating dramatic structure in interactive 3D is independent of the actual hardware technology in use or the specific content. The important commonalities are user interaction in a responsive 3D spatial environment, with the goal of providing a dramatic experience for the user.
I am specifically NOT talking about games – i.e. competitive interactive environments with rules, goals and win/lose criteria. Although much effort is going into game development right now, only a small part of the total population consists of gamers. My experience exhibiting interactive artworks has shown me that a much larger segment of the population is interested in culturally rich exploratory spaces that are engaging but do not have an extensive learning curve or require competitive drive and hours of time in order to reach a satisfying conclusion.
Similarly I do not call my work "interactive theater," because of the implication that there are specifically defined characters or roles that the user is expected to play while interacting with the piece. I focus on a more abstract and general form of dramatic structure that will provide a compelling experience for the user without requiring role-playing.
To build dramatic tension classical drama theory tends to focus on tensions between characters in a narrative as seen from the viewpoint of an observing audience. In interactive 3D however the user IS the audience and IS the main character in the narrative, and the dramatic tension must be built between the user and the environment itself, which may or may not include characters. (I see the case of a user manipulating his or her avatar in real time as only a special case of this more general theory: the focus is still on the emotional experience users have as a result of their manipulations of their avatars.)
I therefore replace character-driven plots with a dramatic framework for the series of emotional states that the environment should evoke in the user. Constraints (on what users can do and where they can go) are used to guide users through the piece and at the same time engage their sense of responsibility for what happens to them in the course of the piece.
Past VR Projects:
Here is a brief synopses of past projects that relate technically and artistically to issues I will be addressing during the fellowship:
Geometries of Power (2002): http://mission.base.com/geometries/
This piece was developed in a 2-week workshop with Bauhaus-University students in October 2002. The goal was to create a multi-user online virtual world whose spatial and interactive structure creates an emotionally and visually engaging experience for the participants. In this piece, users can drag and drop images of politicians and terrorist
