Building Castle's in VR with an AI companian
Castletopia
5 Minute Read | Detailed Case Study Available
MSc Individual Thesis Project, University of Nottingham, 2022
Guide: Prof. Colin Johnson
Context
My fascination with Computational Creativity
My fascination with computational creativity led me to explore the concept of co-creativity. This is where a human and an artificial system, such as an AI, collaborate in a creative endeavour. As an HCI researcher, I was curious about how users perceived a co-creative AI.
The literature review exposed me to research around co-creativity within the arts and music, but I wanted to explore co-creativity in something much more fundamental, which many people would be able to relate to. Play is a very co-creative activity that is present across all cultures and a fundamental human behaviour.
One of the most universally enjoyed play activities is building structures with blocks, a favourite among children in many cultures. I decided to create a VR environment to simulate this activity, making it accessible to many users and allowing the experience to remain embodied.
I developed an experimental project for my MSc: to understand users' mental models of an AI while engaging in open-ended play within a VR environment.
Music
Previous Literature
Human-AI
co-creativity
Research Gap
Most fundamental activity in human culture
Can AI enhance creativity?
The Research Questions
There are two fundamental questions that I was trying to answer through this research:
-
What is the mental model of the AI companion with which the user co-creates the castle?
-
Does the AI enhance the creativity of the player?
Hand Gestures and Interactions in VR
Storyboarding
I worked on designing the storyboard for the hand interactions and gestures that would allow the user to grab objects and build a castle.
Selecting an object
Grab Object
Drop Object
Rotate Object
Rotate Object
Rotate Base
Pick Object
Delete Object
The Turn Taking Process
A thumbs-up gesture was decided upon as a favourable gesture to signal the AI for its turn. This hand-gesture has been metaphorical popular in English-speaking countries as an approval towards something. Given the context of the study and the recruitment of English-speaking participants, the gesture would have a positive connotation, though in different cultures its meaning can be perceived differently (Strazny, 2004)
Strazny, P. (2004, November 1). Encyclopedia Of Linguistics. Fitzroy Dearborn Pub.
When the player wants to signal the computer to take a turn they lift their left hand and give a thumbs-up
The AI recognizes the signal and decides where to move the block
Once the AI has moved it waits for the human's turn. The human can manipulate the AI's moves, including deleting its placed blocks.
Rule based AI
System Design
I designed a rule-based AI which places blocks towards a castle-building form.
The image shows combinations, based on rules that define placement and orientation of a block placement depending on what the lower block is.
A sample structure created using the rules by which the AI plays. The human player is not limited to these rules and can place any blocks over the drop-zones as long as they are empty and available.
Oculus Quest 2
The Prototype
I scripted and designed the VR experience within Unity and exported a testable prototype to a Quest 2 which would be used to run the experiment on participants.
The following prototype demonstrates me building a castle alone and then with the AI companion.
Listen to the audio by clicking the music button on the right hand corner of the video
A Within Subject Study
Experiment Design
The procedure was for 15 participants to play the castle building activity in VR across two sessions; build a castle while playing alone (session A) and build a castle while playing with the computer (session B). After each session the participants were given questionnaires to fill.
The questionnaire was designed to collect data regarding player's intentions towards an output, expectations, experience of creativity, experience of joy, player's mental model of the AI and their likelihood to want to experience a similar system again.
Session A:
Play Alone
( 8 participants)
Survey A
Session B:
Play with AI
Survey B
Session B:
Play with AI
( 7 participants)
Survey B
Session B:
Play with AI
Survey B
Qualitative and Quantitative
Results
Data analysis using a Wilcoxon signed rank test for within subjects was carried out. The results suggest no significant difference (all p-values are greater than 0.05) in user's perception of creativity while playing alone and when playing with the AI.
Thematic coding was done of answers to open-ended questions to identify categories of themes and their frequency.
Mixed Results
Conclusion
This experiment showed that even a rule-based AI system was effective in making players feel creative in an open-ended play experience. It was not possible to cannot claim that this was significantly more than that of player's experience playing alone. The AI influenced creativity in the following manner:
-
Players discovered new possibilities through the AI’s inputs
-
Players may not have thought of a castle aesthetic as it was inbuilt in the AI
-
Players transferred control to the AI when they had no idea how to proceed
Fourteen out of the fifteen participants did not get close to guessing the rules by which the AI was playing the game. The player’s mental model of the AI was shaped based on their expectations from the AI and the level of control they wanted to give or take from the AI.
Examples
Samples of Castles built by Participants
Play Alone
Play with AI
Play Alone
Play with AI
***