On February 15th, 2024, OpenAI unveiled Sora – an impressive new deep learning model that can generate videos and images from text prompts. Sora can generate videos up to a minute long, at various resolutions and aspect ratios. While the model is not currently available for testing, cherry-picked results from OpenAI suggest it vastly improves on the previous state of the art. OpenAI claims – somewhat pompously – that Sora is a “world simulator.” What’s a world simulator, you ask? Good question! Here is OpenAI’s stated motivation for training Sora:
We’re teaching AI to understand and simulate the physical world in motion, with the goal of training models that help people solve problems that require real-world interaction.
The technical report elaborates on OpenAI's understanding of the theoretical significance of Sora:
Our results suggest that scaling video generation models is a promising path towards building general purpose simulators of the physical world.
These bold statements are taken from blog posts – including the terse “technical report” – that can only be characterized as marketing documents rather than academic articles (let alone peer-reviewed research). It's not the first time we've seen such language from companies working on video generation. Take for example this aspirational statement from Runway, one of the leading startups working on video generation:
A world model is an AI system that builds an internal representation of an environment, and uses it to simulate future events within that environment. [...] You can think of video generative systems such as Gen-2 as very early and limited forms of general world models.
Beyond PR posturing, these claims touch on genuinely interesting questions about the internal structure of state-of-the-art video generation models. Unfortunately, buzzwords like “world simulators” and “general world models” are not particularly helpful to formulate these questions clearly. What follows is an attempt at articulating what, exactly, it would mean for these neural networks to have an internal model of the physical world, and whether we have any evidence that systems like Sora do.
To read the rest of this blog post, head over to https://artificialcognition.net.