MULTIPLE VIEWPOINT TELEPRESENCE
WSA Team:
Mathias Fuchs, artistic director
Grzesiek Sedek, network technical consultant
Don Foresta, project coordinator
The Multiple Viewpoint Telepresence project is the ALTERNE project which deals directly with the high bandwidth network. The technical objective was to combine the network with the virtual space of the Unreal Tournament gaming engine to create a platform for online presentations of speakers and materials supporting the speaker, to be used over the academic network of Internet 2.
The project evolved from a series of pilots for a virtual faculty in art and science called Global Threads with talks by Benoit Mandelbrot, Jean-Claude Risset and Miller Puckette on line for members of the MARCEL network. The pilots experimented to some degree with the multi-screen potential of Access Grid to begin working with the idea of several sources of information forming an online presentation. Mathias Fuchs was the artistic director of those pilots and his experience with UT in the past made him the logical choice for furthering this form of development through the ALTERNE project.
Telepresence thus brought together the artistic experience of Mathias Fuchs with gaming engines and virtual space, the MARCEL network developed by Don Foresta at the Wimbledon School of Art where it was brought to fruition and the virtual faculty project, Global Threads, a specialized project of the MARCEL network.
Artist’s Abstract:
Game engines for multiuser games (as the Unreal engine we have been working with since 1999) are extremely popular amongst gamers to share gaming experience via the Internet. Little research has been undertaken to use these engines for artistic purposes in the field of telematics. This is so much more astonishing as the iconography and generic form of computer games begin to influence a whole range of other media, from film, TV, books and music to toys and fashion.
The project consists of an exploration of:
- connectivity and
- social play of telematic setups in an artistic context. We intend to explore
- sign making and
- telematic performance,
- multiple signification and
- the aesthetics of Mixed Reality.
The transformed look and feel of the users/ gamers will be communicated via streaming video and streaming audio to a telematic audience and at the same time it will be superimposed with player pawns and virtual environments of the game.
Intention:
To Create an enhanced environment for multi-user conferences via multiuser game engines.
Problem:
Currently most of the available videoconferencing systems available carry the lame look and feel of business-oriented videoconferencing or the crude and sometimes childish appearance of games for kids. (Microsoft's netmeeting, CUSeeMe, iVisit and others)
The potential possibilities of videoconferencing set-ups and multiuser games could be developed in a direction which makes the best use of both of these technologies.
Currently those systems are developed by different development groups for different target audiences: the commercial sector and the leisure industry. We are interested in what videoconferencing would look like if it employed the tools of advanced IT and the aesthetics, design and concepts of artists. How would it feel to be videoconferencing in such an environment?
Research strategy:
We intend to use our artistic experience to cross-breed videoconferencing and multi-user games to create playful environments for international technically advanced communication.
Our system of telematic knowledge spaces using game engines attempts to revive the ancient method of Mnemosyne, used by Greek “singers� (Simonides of Cheos) and philosophers as well as Renaissance scholars. This form of mnemotechnique, called loci or place method, was widely used by orators to memorize complete speeches. The orator picked a building and learned every nook and cranny very intensely until he was able to move about the building in his memory. As a preparation for the speech a plethora of items of different complexity and amount of detail could be placed in the memorized rooms, e.g. a scale for justice etc. While delivering the speech the orator wandered from room to room and collected the hints while the speech unfolded.
The “Virtual Jean-Claude� level contains images of visual illusions which referred to the acoustical paradoxes he was interested in. Not only were we able to make Risset talk by walking through the sequence of rooms we set up, we also were able to make him start and stop musical examples, present pictures and texts. We are currently working on setting up a rich body of Unreal levels which contain the knowledge of artists and scientists and make them available as worlds, the user can freely navigate in. The scholar or user would be able to play Benoit Mandelbrot, Jean-Claude Risset, and Andy Warhol (and be an artist/scientist for 5 minutes).
Technical Results:
The tools used, Access Grid for multicasting over high bandwidth networks and Unreal Tournament for creating the virtual space of presentation were both adequate for the work done and the objectives of the project. Both were a point of departure rather than final technical propositions and the need for improvement was obvious by the end of the experiment. The project allowed the Wimbledon team working with Mathias to identify the weak points of each and to begin to find solutions, which means further development in the same directions.
The key to making Telepresence work as desired is keying a live video conference into the virtual space created with UT. A real-time presentation – in this case, of Jean-Claude Risset - can be seen in one of the virtual rooms covering a full wall in that room. Jean-Claude Risset’s avatar leads viewers from one space to the other to see and hear the additional material on line. Additionally, by having the live transmission within the virtual space we were able to imagine bringing video effects to the live image in real-time. The final technological mix is therefore, UT, the multicasting platform Access Grid and video effects using Pure Data.
On 11th November, 2004 at SCGlobal in Pittsburgh we presented a system of an Unreal Level which was based on a lecture by Jean-Claude Risset. We used his voice, the images he presented during his lecture, and the sounds he played, to set up a level, which contained all the material. We also built an avatar which impersonated Jean-Claude Risset and allowed us to navigate through the level containing his knowledge. Different to a traditional remote lecture we were able to let the virtual “Jean-Claude� control the timing and the structure of his lecture. We could speed up his talk, slow it down, create pauses or repeat certain sections.
In addition to the virtual space created, a toolbox of special effects using Pure Data was developed and experimented at Wimbledon. These tools allow the possibility of adding real-time video effects to transmitted images to enhance the visual effect of the exchange. Students in the Theatre Design department explored the uses of the effects for possible online theatre applications and allowed the WSA team to observe how they could be used without slowing down the transmission of the presentation. Those tools are available over the ALTERNE web site for anyone interested in pursuing the same line of experimentation.
Testing was done over the network using Access Grid between Wimbledon and Manchester to try out each stage of the work done and to observe in real-time how it would be received.
Access Grid is the lingua franca of multicasting today, but it still has latency problems and rather weak audio, which is sufficient for speech but not for music. The use of music challenged the audio side and pre-recorded music, such as in the Jean-Claude Risset experiment, worked reasonably well. Real-time music exchange, subject to latency problems, is another problem and solving it is the objective of at least one of the future projects described below.
The AG interface is not user friendly and more development is necessary in the area as well. The Access Grid community seems to be moving in a direction of more complexity rather than less. It may well be that we find an alternative solution, probably still based on the software VIC and RAT, standard multicasting software programmes now in use. Discussions have begun with the CERN about their multicasting platform VRVS as another possibility.
“It became necessary�, as Fuchs states, “to do considerable time-scaling of Jean-Claude Risset’s lecture as we found out that subjectively perceived time in a game differs a lot from perceived time in a lecture.� This was an important design consideration that was a constant in the work done.
The avatar was somewhat limited and the better solution found through working with Haptek Risset Avatar. is not yet compatible with UT. Work was done with Chris Shaw and People Putty (Haptek) to add to the virtual space a more performing avatar of the subject character, Jean-Claude Risset, which can be programmed to imitate emotion and to give the character a semblance of personality. That company has also been collaborating with the ALTERNE project to find a better method of producing avatars quickly with a programme that any computer literate student can use. On-going research is taking place at Wimbledon in that direction. The avatar of Jean-Claude Risset will be included in the web site.
Another important aspect of our research was being able to record and rebroadcast work done over Access Grid. A tool was built to allow this to take place. We are still not completely satisfied with the sound quality and further work has to be done to improve it. Still this is key to building an archive of online events and work will continue in this area.
The advantages of the tools were simple - everything worked. The technologies used functioned in real-time presentation situations proving the concept, offering a convincing demonstration and indicating future desirable development. The online presentation made during SC Global in Pittsburgh in November 2004 resulted in an invitation to SIGGRAPH. The roundtable discussion made at the College Arts Association in Atlanta in February of this year resulted in a further invitation to that conference next year to present an up-dated version of the project.
Future Research:
Mathias Fuchs felt that the principal practical result was the construction of a “knowledge space around the concept of musical and visual structures which could be used by artists, researchers or students.� He felt that he had built a prototype which “breaks down the conventions and stereotypes of video-conferencing aesthetics and develops a new look and feel for videoconferencing and interactive learning environments.� He also added that he fulfilled his artistic objective without the need for additional tools and that he would continue in the same direction.
The work done is a concrete beginning and not a finished work. It is the point of departure for an idea tested for almost three years and found needing further development. The virtual faculty pilot projects indicated an initial need for a multiple point of view to online presentations. The work done under ALTERNE demonstrated the feasibility of that approach. The final product was proof of the potential and the need to continue working in that direction. The technical choices were, for the most part the correct choices with the reservations expressed above.
The intention is to create an experience, which is immersive and intensive and has a strong link of movement and experience. The German word for experience, “Erfahrung�, hints that there is an etymological as well as an anthropological link of movement (“fahren�) and experience. The current stage of the project is a pilot implementation with still a few problems to be solved with the player pawn model and the level to be extended in content.
We also discovered that a new Action class will have to be written which allows for fade outs of music playing or the lecturer’s voice being played.
The next large step will be to make the level ready for multi-user operation and to experiment with chroma-keying possibilities on Unreal video output.
The future development has already been initiated. A project developed with the CERN to put presentations from its scientists on line will use Telepresence as its platform. A proposal has been written with the CERN, which includes ALTERNE partners, Pavel Smetana and CIANT, Alok Nandi and the Free University of Brussels and MARCEL, to develop the platform further for the best presentation of scientists and scientific materials over the high bandwidth network to the educational and cultural communities. The project called NUCLEUS was positively reviewed by the European commission but not funded. A second request is presently being made
The next pilot for the Global Threads project will also use Telepresence. That event planned for late 2005 will feature Peter Weibel, Director of the ZKM on the artistic side and Aton Zeilinger, nuclear physicist from the University of Vienna on the science side, talking about the role of the observer in art and in science. Both men, when shown the results, decided to use the Telepresence platform for those presentations. The talks will be produced by our team, stocking the support material for those talks in the virtual space and building avatars of both men. The development done by WSA will permit the recording of the online events for later online presentation and availability to a larger audience. They will be available through the Art and Industry category of the MARCEL web site.
A third, and more elaborate project, AGORA 4D, is being finalized now with some of the same players over a longer term, to improve the platform for multicasting, integrating the Telepresence project while adding new dimensions to it and improving its overall performance and the operation of the high bandwidth network multicasting possibilities, including archiving. The consortium presented the first version of the project, designed by MARCEL and IRCAM in Paris to the European Commission last Fall and also received a positive evaluation . That project will include IRCAM, Centre Georges-Pompidou, MARCEL, CIANT, IRISA in Rennes, France, The Public in West Bromwich, UK, the Universities of Genoa and Turin and the Poznan Center for Supercomputing and Networking.
PLAYGROUND
Experimenting with the possible uses of the combined virtual reality and multicasting spaces:
One part of the experimentation at Wimbledon included building real-time tools for adding video effects to a transmission between two or more sites. The tools were developed by Grzesiek Sedek using Pure Data to be able to work the video images while broadcasting to other sites.
The Theatre Design section of WSA played with some of the tools developed for ALTERNE and the Telepresence project and their results can be seen here:
These PD tools are available and can be downloaded from the Tools section of this web site.
