© Copyright 1995 IEEE. Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.
Hunter G. Hoffman, Keith C. Hullfish, and Stacey J. Houston
Human Interface Technology Laboratory, FJ-15
University of Washington
Seattle, WA., 98195
hunter@hitl.washington.edu
We investigated whether subjects could separate memories of events experienced in virtual reality from real and imagined events: a decision process we term virtual-reality monitoring. Participants studied 8 separate spatial configurations of real geometric objects arranged on a life-sized chessboard, 8 configurations in virtual reality (an immersive, computer-simulated world), and imagined objects in 8 other configurations. On a later source identification memory test, subjects were generally able to correctly identify the sources of the events. A Memory Characteristics Questionnaire was administered to assess differences in qualitative characteristics of memories for virtual, real and imagined events. Differences were found that could potentially serve as cues to help people decide where their memories originated. Results are interpreted within the Johnson-Raye [7] theoretical framework.
In virtual reality, life-like changes in visual imagery occur in response to the participant's own actions. Such realistic feedback often leads subjects to report that they feel "in a place" when navigating the computer-simulated world. For instance, subjects experiencing a computer-simulated Sharkworld may have the feeling they are in the ocean, moving around a shark-infested ship wreck. This subjective experience of "presence" in the virtual environment is thought to be the essence of virtual reality.
There are speculations in the virtual reality literature that presence improves sensori-motor or cognitive performance within virtual reality, and improves the efficiency of training and planning (15). It may also improve transfer of training to the real world and enhance learning. There are likely degrees of presence. Despite their immersion, at some level subjects presumably remain aware that they are only standing in the laboratory wearing a helmet. The more subjects focus their attention on the simulated environment, the more present they are likely to feel in the virtual world. On the other hand, if they were allowed to hear people whispering or telephones ringing in the laboratory, these distractions are likely to draw their attention away from the virtual world, reducing their sense of presence. We need a good measure of presence to explore speculations about the virtues of virtual reality, and to optimize the quality of the human-computer interface. Reality monitoring, a successful paradigm from Cognitive psychology, may prove helpful.
REALITY MONITORING
Johnson and Raye [7] have developed a paradigm in which to study reality monitoring (RM), the decision process by which memories of real and imagined events are distinguished (see [6] for a review). The decision process by which people distinguish memories of real, virtual and imagined events is a related phenomenon we term virtual-reality monitoring. The present experiment is the first in a series of studies aimed at developing memory source identification confusions (virtual-RM errors) into an objective measure of presence.
Johnson and Raye propose that differences between real and imagined events as originally experienced are preserved in memory and can later serve as cues to where the memory originated. That is, memory source is inferred by the subject at the time of retrieval, based on cues associated with the target memory. RM decisions take advantage of differences in qualitative characteristics of memories from different sources. For example, compared with memories for imagined events, memories of real events tend to include more perceptual, spatial and temporal, semantic and affective (emotional) information and less information about mental effort. Consequently, a memory with a great deal of visual and spatial detail and very little evidence of mental effort would be judged to have been real. In contrast, a target memory with few perceptual cues, but abundant evidence of mental effort is likely to be identified as imagined.
In one experiment, Johnson and Raye showed subjects a list of word pairs. For some of these stimuli, the subject's task was to self-generate (imagine) the second word in the pair. For example, given the word HOT they had to think of the opposite word COLD and fill it in. For other items, the second word was already filled in and all subjects had to do was read it. Later, they took a source identification (ID) test. The test list consisted of imagined and read (perceived) words from the study list as well as new distracter items. When subjects were asked to identify the source of each item by responding "perceived" "imagined", or "new", they were quite accurate at doing so. In a second experiment, the amount of effort required to generate the imagined word was manipulated. For some subjects, the task was made easier by giving subjects the first letter of the second word in the pair. This was intended to decrease the amount of mental effort required to self-generate the second word in the pair. As predicted by Johnson and Raye, this manipulation increased confusions between memories of perceived and imagined events. This finding that confusion increases with decreases in the amount of mental effort associated with imagined events has now been demonstrated in a number of studies [2][3[4][8].
Increasing the perceptual similarity between memories of real and imagined events has also been shown to increase RM confusions [5][9]. For example, if an artist with a vivid imagination generates an image of a fruit bowl that contains many characteristics typically associated with real events (clarity of detail, vividness, rich colors etc.), these cues could later mislead the artist when she attempts to remember whether she actually saw a fruit bowl or only imagined one. More generally, memories with qualities atypical of their class are likely to result in RM errors.
According to a number of investigators [1], researchers presently lack a theoretical framework within which to study VR, and lack an objective measure of "presence", the sensation of being in a place while experiencing a VE. The present study explores the use of the RM paradigm to fill these needs. We began by investigating whether subjects can discriminate memories of events experienced in VR from real and imagined events. Subjects studied the spatial locations of geometric objects located on a chessboard. The objects were either real, imagined, or virtual. Subjects later took a source ID memory test. After the test, we explored qualitative similarities/differences between the 3 types of memories by asking subjects to rate each class of memories on an adaptation of the Memory Characteristics Questionnaire (the MCQ) developed by [10]. Our questionnaire consisted of a number of scales designed to assess a wide range of characteristics of memories (e.g., visual detail, spatial and temporal information, emotional intensity, and the subjective experience of presence). Following the logic used by Johnson and colleagues, we predict that some of the differences in the qualities of the experiences as originally experienced will get stored in association with the target memories, and will allow subjects to infer the sources of their memories on a later memory test. More specifically, we predict that cues associated with memories of virtual events will generally allow the virtual source of these memories to be identified. That is, subjects will be able to discriminate memories of virtual events from memories of real and imagined events. Our version of the MCQ was administered to investigate the virtual-RM decision process. Ratings of the phenomenological qualities associated with memories of real, virtual and imagined events may point out differences in qualities which subjects could be using in their virtual-reality decisions, helping us to understand how these decisions are made.
THE EXPERIMENT
Method
Subjects.
Twenty four college educated subjects, as well as graduate and undergraduate students from the University of Washington participated in the 2 hour experiment.
Materials and equipment. A configuration consisted of four identical 3-dimensional objects (either triangles, cubes, half-cylinders, or t-squares), placed in one of 32 possible relations on a life-sized chessboard. In the real and imagined world conditions the 8 x 8 square chessboard was 12' x 12'. The objects seen on the chessboard in the real condition, that subjects were asked to imagine on the chessboard in the imagined condition were approximately 14" x 14" x 14". For the imagined condition, subjects were given written directions to determine which objects should be imagined to exist on which squares of the chessboard. In the virtual world, the checkerboard and objects were scaled to approximately the same dimensions as real world stimuli. Configurations were rotated through conditions such that each configuration occurred in each of the three worlds (real, virtual, and imagined) and as new items equally often in the experiment.
The VR system consisted of a Virtual Research stereoscopic head mounted display and a hand held joystick. Both devices were equipped with Polhemus 6 DOF electromagnetic trackers, which allowed the computer to track head movements and to track the position of the joystick. Subjects moved in the worlds with the joystick.
Design and Procedure.
Stimuli were counterbalanced such that each of 8 possible paths, and each of the 4 object shapes, appeared in each world type (Real, Imagined, Virtual, and New) equally often for each subject.
Each subject experienced 24 of the 32 configurations in the study phase (8 real, 8 virtual, and 8 imagined). The remaining 8 configurations served as distracters (new items) in a memory test to be described shortly. The order in which subjects encountered each world type was counterbalanced. Subjects were randomly assigned to one of 3 orders such that each subject was equally likely to encounter each world first, second or third. Once in a world, subjects studied 8 consecutive configurations before changing worlds.
Each subject practiced navigating through "shark world", a VR game, for approx. 5 min. The experiment was described as a study of cognitive abilities in real, virtual and imagined worlds. In each world they would follow a path around 4 objects (or imagined objects) placed in various positions on a life-sized chessboard (see Figures 1, 2, and 3).
They were to study where the objects were placed in relation to each other and to the chessboard in preparation for the upcoming memory test (the exact nature of which was unspecified; see Figure 4). Prior to beginning a new world, subjects were given 2 practice trials (Subjects who received the "imagined" condition first were shown what the real objects looked like). After the study phase, subjects played "Sharkworld" again for approx. 5 min. as a distracter task.
All study items, and 8 new distracter items were presented for the source I.D. test. The order of the test trials was randomized. Subjects were asked a) to identify the source of origin of each test item, and b) to indicate how confident they were in the accuracy of their responses as a percentage of either 0%, 20%, 40%, 60%, 80%, or 100%. The test was self-paced.
After all memories had been identified, subjects completed a subset of the MCQ. These questions assessed a wide range of memory characteristics (e.g., visual detail, spatial and temporal information, and feelings) on a 7-point scale (e.g., the relative spatial arrangement of objects in my memory for the event is: vague 1 2 3 4 5 6 7 clear/distinct ). A questionnaire introduced by [16], supplemented by two new presence questions was then administered to get subjective self-assessments of subjects' psychological experience of "presence" in each of the environments. Each question was asked 3 times in a row, once for each world. The order in which the questions were asked were always real, imagined, then virtual.
RESULTS
Collapsing across conditions, subjects' source ID accuracy was significantly higher than chance (.51 vs. .25 respectively), t(23) = 7.53, p < .001. No significant difference was found in the proportions of correct responses, nor of confidence ratings for items correctly attributed to real, imagined, and virtual sources. No significant differences were found for false recognitions, misses, or source monitoring errors. And there was no correlation between source monitoring errors, and subjective ratings of presence.
Memory ratings.
The mean memory ratings were calculated for each item adapted from the MCQ (see appendix). For each rating, the following planned comparisons were conducted: real vs. virtual events, virtual vs. imagined events, and imagined vs. real events. The outcomes of these comparisons are summarized in the next two sections. Consistent with our predictions, significant differences were found for 5 of the memory characteristics for the real vs. virtual ratings, for 6 of the memory characteristics for the virtual vs. imagined events, and for 11 of the memory characteristics for the real vs. imagined comparison. Although not conclusive, this pattern suggests that there may be more cues to distinguishing real from imagined events than for the other types of decisions.
Real vs. Virtual.
Real events were given significantly higher ratings than virtual events on the following characteristics: Clarity, spatial location, doubts/certainty. Virtual events were rated as having more color, and felt more intense. No difference in ratings of presence were found.
Virtual vs. Imagined.
Virtual events were rated higher than imagined events on the following characteristics: color, vividness, visual detail, event detail, feeling of intensity, and presence.
Real vs. Imagined.
Real events were rated significantly higher than imagined events on the following characteristics: clarity, color, visual detail, vividness, event detail, spatial location, event duration, +/- of feelings, overall memory, doubts/certainty, and presence questions.
CONCLUSION
The results of the source ID test showed that subjects can distinguish memories of virtual events from real and imagined events. Combined with the MCQ ratings, these results support the Johnson-Raye theoretical framework; cues associated with memories of virtual events allow subjects to distinguish memories for these events from memories of real events and imagined events. Apparently, differences in the qualitative characteristics associated with the memories enabled subjects to infer sources at time of retrieval.
Previous research on RM has primarily used words, phrases/sentences, geometric stimuli, pictures of objects, and simple actions [6], [10]. The present results generalize these findings to memories for spatial configurations on life-sized chessboards and to memories for virtual events.
Subjects' ratings on a wide range of characteristics suggested some phenomenalogical cues that could potentially be useful in these decisions. Of particular interest was the finding that subjective ratings of "presence" for real events and virtual events were dramatically higher than ratings of "presence" for imagined events. Future research could investigate the extent to which RM and VR monitoring decisions are based on differential feelings of presence. Presumably, presence is most indicative of an event experienced in the real world. Manipulations that increase the phenomenological experience of presence associated with memories of virtual and imagined events may make them more difficult to distinguish from memories of real events on source ID tasks, and may have predictable effects on characteristics associated with memories for these events, as evident from ratings on the MCQ.
There are presently a number of limitations of VR technology (e.g., field of view, update rate, resolution) that likely reduce the realness of the experience. Future research should explore whether these qualities of the virtual experience are stored in memory, and later help subjects identify the source of these memories as "virtual". For example, one could manipulate field of view to determine whether doing so improved performance on the source ID test and increased differences in ratings (between groups) on the MCQ. It may also be interesting to add some questions regarding these limitations to the MCQ. Findings that these artifacts of the present limitations of VR technology help subjects accurately identify the source of their virtual memories would further support the logic of using VR monitoring as a metric of presence discussed below. As the technology improves and the artifacts are reduced, the cues to source will also be reduced.
One application of VR technology is for training (e.g., training astronauts how to fix the Hubble, [12]). The more closely the virtual worlds simulate the real worlds, the more likely it is that training will transfer from one world to the other. For training purposes, the ideal (but perhaps unattainable) goal is for the virtual world to be made indistinguishable from the real world. We are presently attempting to develop virtual-RM into a metric for assessing how closely virtual events simulate real events, a sort of Turing test for quantifying the fidelity of virtual technologies. The more convincing the virtual world, the more "present" subjects will feel, and the more likely they will be to misattribute their memories for this event to a "real" source.
The Johnson-Raye theoretical framework also suggests ways to minimize transfer effects in situations where transfer is undesirable. For example, there are (admittedly controversial) concerns that violent entertainment contributes to violent behavior in the real world [11]. The advent of violent games to be played in virtual realities will likely heighten such concerns. Undesirable transfer of training could perhaps be minimized by making the phenomenal experience in the virtual world distinctively unreal (e.g., vertical flight enablement, permeable walls etc.). Doing so may help players compartmentalize these entertainment experiences separate from their pool of knowledge about the real world [6],[13],[14].
The data from this exploratory experiment were encouraging in that they showed it is reasonable to study the psychological experience of virtual reality using the Johnson-Raye framework. Unfortunately, these results contained too much variance and performance was too low to assess whether virtual-reality monitoring might provide a good objective measure of presence (e.g., no correlations between virtual-reality monitoring confusions and subjective measures of presence were found). In future research, meaningful stimulus objects (and a larger number of them) will be employed in an attempt to stabilize variance and raise accuracy and confidence.
An objective measure of presence would be a valuable tool for VR researchers, allowing us to assess the value of presence, and the conditions under which it occurs most intensely. In the process, we hope to gain a better understanding of reality monitoring, a central process in cognition.
ACKNOWLEDGMENTS
This research was supported by AFOSR grant #F49620-93-1-0339. The authors would like to thank Drs. Woodrow Barfield, John Dunlosky, Tom Furness, Sheree Kwong See, Elizabeth Loftus and William Winn for their precious input. Special thanks to Dr Max Wells. Finally, we extend our thanks to the U.W. students and members of the community who took part in this study with such complete sincerity.
REFERENCES
1. Barfield, W., Sheridan, T., Zeltzer, D., & Slater, M.
(1995). Presence within virtual environments. In W.
Barfield & T. Furness III (Eds.) Virtual Environments
and Advanced Interface Design. London: Oxford
University Press.
2. Durso, F.T., & Johnson, M.K. (1980). The effects of
orienting tasks on recognition, recall, and modality
confusion of pictures and words. Journal of Verbal
Learning and Verbal Behavior, 19, 416-429.
3. Finke, R.A., Johnson, M.K., & Shyi, G.C.-W.
(1988). Memory confusions for real and imagined
completions of symmetrical visual patterns. Memory
& Cognition, 16, 133-137.
4. Foley, M.A., Durso, F.T., Wilder, A., & Freidman,
R. (1991). Developmental comparisons of explicit
versus implicit imagery and reality monitoring.
Journal of Experimental Child Psychology, 51, 1-13.
5. Johnson, M.K., Foley, M.A., & Leach, K. (1988).
The consequences for memory of imagining in another
person's voice. Memory & Cognition, 16, 337-342.
6. Johnson, M.K., Hastroudi, S., & Lindsay, D.S.
(1993). Source monitoring. Psychological Bulletin,
114, 3-28.
7. Johnson, M.K., and Raye, C.L. (1981). Reality
monitoring. Psychological Review, 88, 67-85.
8. Johnson, M.K., Raye, C.L., Foley, H.J., & Foley,
M.A. (1981). Cognitive operations and decision bias
in reality monitoring. American Journal of
Psychology, 94, 37-64.
9. Johnson, M.K., Raye, C.L., Wang, A.Y., & Taylor,
T.H. (1979). Fact and Fantasy: The roles of accuracy
and variability in confusing imaginations with
perceptual experiences. Journal of Experimental
Psychology: Human Learning and Memory, 5, 229-
240.
10. Johnson, M.K., Suengas, A.G., Foley, M.A., &
Raye, C.L. (1988). Phenomenal characteristics of
memories for perceived and imagined autobiographical
events. Journal of Experimental Psychology: General,
117(4), 371-376.
11. Lazar, B.A. (1994). Why Social work should care:
Television violence and children. Child Adolescent
Social Work Journal, 11(1), 3-19.
12. Loftin, R.B. & Kenney, P.J. et al. (1994). Virtual
Environments in Training: NASA's Hubble Space
Telescope Mission. Proceedings of the 16th
Interservice/Industry Training Systems & Education
Conference, Orlando Florida.
13. Potts, G.R., & Peterson, S.B. (1985). Incorporation
versus compartmentalization in memory for discourse.
Journal of Memory and Language, 24, 107-118.
14. Potts, G.R., St. John, M.F., & Kirson, D. (1989).
Incorporating new information into existing world
knowledge. Cognitive Psychology, 21, 303-333.
15. Sheridan, T. (1992). Musings on telepresence and
virtual presence. Presence: Teleoperators and Virtual
Environments, 1 120-126.
16. Slater, M., Usoh, M., & Steed, A. (in press). Depth
of presence in virtual environments, Presence:
Teleoperators and Virtual Environments.
17. Zeltzer, D. (1992). Autonomy, interaction, and
presence. Presence: Teleoperators and Virtual
Realities, 1, 127-132.
APPENDIX
-each question was rated on a scale from
1 to 7.
1. My memories for the real objects are: dim...sharp/clear.
2. My memories for real objects are: black & white...entirely color.
3. My memories for real objects involve visual detail: little or none...a lot.
4. Overall vividness for real objects is: vague...very vivid.
5. My memory for the real objects are: sketchy...very detailed.
6. Relative spatial arrangement of the real objects in my memories are:
vague...clear/distinct.
7. The real events seem: short...long.
8. In the real events I was: a spectator...a participant.
9. At the time, the real events seemed like they would have serious implications: not at all...definitely
10. Feelings at the time (for real events) were: negative...positive
11. Feelings at the time (for real events) were: not intense...intense
12. I remember what I thought at the time (real event): not at all...clearly
13. Overall, I remember the real events: hardly...very well
14. Do you have any doubts about the accuracy of your memories for the real events: yes, a great of doubt...no doubt whatsoever
15. How strong was your sense of presence in the real chess world?
weak presence...strong presence.
16. To what extent could you "feel" the presence of the geometric objects (excluding the chessboard). in the real world: weak...strong presence
17 Please rate your sense of "being there" in the real chess world: not at all...very much so.
18 To what extent were there times during the real chess world experience when the chess world became the "reality" for you, and you almost forgot about the real world outside: at no time...almost all the time.
19 The real chess world seems to me to be more like: something that I saw...somewhere that I visited.
20. How strong was your sense of presence in the real chess world? weak presence...strong presence.
21. To what extent could you "feel" the presence of the geometric objects (excluding the chessboard) in the real world? weak presence...strong