Surreal VR Pong: LLM approach to Game Design

The increase in complexity from 2D to 3D game design makes it fascinating to study from a computational creativity perspective. Generating images given text descriptions using models like DALL-E has recently increased in popularity. However, these models are limited to generating 2-dimensional outputs. While outputs of these models can be used to stylize 3d objects with variable textures, they cannot produce mesh-level interactions. We introduce Codex VR Pong as a demonstration of controlled non-deterministic game mechanics leveraging generative models. We are proposing that prompt-based creation can become part of gameplay rather than just part of game development.