David Li & Jonathan Li · Aug 12, 2025

Data-efficient computer world models

We built Photon-1, the world’s first large-scale computer world model.

Photon-1 is a sparse 106B-A5B MoE transformer trained on just 18 years of computer use demonstrations to predict what will happen next on a computer screen. Instead of directly predicting actions, it imagines what should happen next in latent space.

We show that our computer world model can be turned into a computer use model with a small amount of labeled computer use data (<10K trajectories). Online reinforcement learning on computer use environments further improves task execution, despite having no explicit instruction-following computer use data in pretraining. Our results suggest that computer world modeling is a strong objective that imbues planning and reasoning capability directly from watching computer use demonstrations.

Thinking in imaginative space

Humans can learn to use a computer quickly by watching other people use a computer. What we learn from is often not the raw actions a person takes, but what is happening on the screen.

Similarly, our model imagines the future in a latent space. Through a carefully designed encoder, this learned latent space semantically encodes information about how the computer state is changing over time, including differences in text, layout, interactive elements, and cursor position on the screen. Our model does not generate images.

Computer world model pretraining — Figure 1: our encoder converts each screenshot into a set of latent tokens which capture its semantic meaning. The world model predicts the next latent token autoregressively. After predicting enough tokens, the model will have predicted the latent for the next computer state.

Previously, we built models to predict the next action to take on a computer directly without imagination. We found these models to be significantly data-constrained, requiring more computer use data than exists on the internet to reach generally competent levels. Fundamentally, we believe training on action data alone is too sparse a learning signal to instill general understanding of a computer desktop and the work done on such devices. Photon-1 overcomes this problem by predicting the future directly, which provides a denser learning signal.

After training Photon-1, we finetuned an open-source single-stream DiT (Ideogram 4) as an auxiliary image generation model to covert Photon-1’s latent space to a screenshot image for visualization purposes.¹1.This visualization process is not expected to be perfect, as the latent encoder was not trained for an image reconstruction objective. All depictions of Photon-1’s imagination are visualized with this image generation model.

Training Process

We designed Photon-1 to predicts future frames autoregressively using a next-token-prediction objective. Unlike in natural images, screen states feature dense textual information, layout, and iconography, making flow matching or diffusion-based modeling intractable.

To facilitate autoregressive frame prediction, our vision encoder uses finite scalar quantization (FSQ) to encode each screen state into 960 discrete tokens with a vocabulary size of 5⁸. This encoding effectively compresses the image from an average of 59.8K bytes in WebP image to around 2.2K bytes²2. $\log_2(5^8)\cdot\frac{960}{8}$ , a 27x compression.⁶6.Optimized video codecs which also encode across the temporal dimension gave ~11.4kb per frame, meaning our encoder provides ~5.1x better compression. To achieve this rate of compression, Photon-1 uses a differential latent encoder, which encodes video frames as pairs to allow the latents to describe differences between frames.

Latent encoder — Figure 2: our differential image encoder results in a 27x compression compared to the optimized WebP format by encoding the differences between frames rather than the contents of the frames themselves.

We supervise the encoder through semantic objectives, including OCR, screenshot understanding, and screen recording understanding objectives, using internal datasets. We freeze the encoder during the training of the computer world model.³3.Future work includes unfreezing the vision encoder during training, creating an autoregressive BYOL-style embedding.

Our pretraining corpus originates from approximately 2 million publicly available computer screen recordings, filtered down from an internal index of 2 billion videos. We performed extensive video and frame-level filtering to remove non-computer use content, and built an internal keyframe detection model to remove redundant frames. We pretrained Photon-1 from scratch on one epoch of the resulting 575 million frame dataset, equivalent to 552 billion tokens or about 18 years of video sampled at 1 fps.

Throughout pretraining, we tracked several internal benchmarks including screenshot understanding and frame completion correctness, which increased consistently throughout the pretraining process.

Computer world models as computer use agents

To test whether computer world modeling provides useful priors for computer use, we finetune the pretrained model on small amounts of computer-use data (<10k trajectories) to teach the correct action format and instruction input format. Under our computer use setup, the computer world model imagines what should happen first then generates the action to get to that state, allowing use of the computer world modeling priors. The resulting model is able to complete computer use tasks.

Computer world model finetuning — Figure 3: finetuning our computer world model for computer use. This process teaches the model to follow instructions and use the correct action output format. We add special computer use tokens to allow the computer world model to emit actions for computer use. At inference, the model predicts the state of the next frame first, then outputs the action to get from the current state to the predicted next state.

Our online reinforcement learning engine performs realtime rollouts on MacOS and Linux virtual machines at scale, and programmatically verifies the outcomes on the machines to provide a reward signal. On Linux environments, we augment OS functionality with different Linux desktop enviornments (LXQt, Xfce, Mate, Gnome, and Plasma). Each environment is signed in with a unique Google account to register for and use login-restricted web applications⁴4.Sometimes we reuse accounts between trajectories so there are existing web accounts. and contains an internal clone of ChatGPT without rate-limits⁵5.We implemented features making our ChatGPT clone more useful for computer use models. We gave gpt-5.4-mini access to a screenshot tool returning the last 5 actions taken. It can also use a code interpreter, use web search, and generate downloadable artifact files..

We find that online reinforcement learning for computer world models can bring capability to levels comparable with modern LLMs on our internal computer use benchmarks, despite being trained on less data and with less compute.

Reinforcement learning pipeline — Figure 4: Left — Photon-1 computer use performance on an internal computer-use benchmark over time during RL training. Right — overview of online RL infrastructure.

We observe that the predicted latents after RL become difficult to visualize and understand, suggesting that the computer world model is honing its use of the imagination space to improve planning and reasoning, without any text priors.

Imitating human behaviour

Photon-1 learns human computer use priors present in our pretraining data, such as using AI tools to complete tedious tasks. For example, after reinforcement learning, our model learns to use our internal ChatGPT clone to generate documents and answer knowledge-related questions.

Pretraining on human interactions with AI tools teaches our models how to use LLMs as tools, and the ability to guide and correct them.

Towards curious intelligence

We’re building curious intelligence. Curious actors require an environment to play, explore, and learn in, and we believe computer environments are an ideal space to develop them. Models like Photon-1, which understand through observation and learn to act by thinking in imagination, will be a key ingredient in building curious superintelligence.

We’re a team of two based in San Francisco. We’re always looking for exceptional thinkers. If our work sounds interesting to you, leave us a note at team@inductionlabs.com.