Unreal Agent

Unreal Agent (UnrealGPT) is an in‑editor AI copilot for Unreal Engine 5 that doesn’t just suggest ideas—it builds actual content and systems inside your project.

Running as a dockable tab in UE5.6, it connects to OpenAI’s GPT‑class models and drives the editor through Python, scene queries, reflection, screenshots, and external generators. In testing, the agent has successfully created complete PCG assets from scratch (nodes, links, parameter wiring) and authored advanced, layered materials—all from natural‑language prompts.

Describe what you want (“Create a PCG graph that scatters rocks on slopes and grass in the valleys”, “Build a master material with triplanar mapping, distance‑based blending, and a ready‑to‑use instance”), and Unreal Agent plans, executes, and verifies the changes directly in your level and content browser.

What the Agent Can Do

• Design and wire up systems

• Build PCG assets end‑to‑end: create the asset, add nodes, connect pins, and set parameters via Python.

• Author complex materials and material instances, including layered setups, masks, and parameter collections.

• Script editor workflows: batch‑rename assets, set up folders, spawn and configure actors, and more.

• See and understand your world

• scene_query – Search the level by class, name, label, or component types and operate on the matching actors.

• viewport_screenshot – Capture the current viewport; the agent uses the image to reason about layout and results.

• GetSelectedActorsSummary – Focus the agent on exactly what you’ve selected for surgical edits.

• Drive Unreal through powerful tools

• python_execute – The editor superpower

• Runs Python scripts inside Unreal Editor to create and modify actors, Blueprints, PCG graphs, materials, and assets.

• Uses a structured JSON result (status, message, details) so the agent can reason about what actually happened.

• scene_query – X‑ray vision for your level

• Returns compact JSON describing matching actors (name, label, class, location).

• Lets the agent target exactly the right objects for follow‑up actions.

• viewport_screenshot – Visual ground truth

• Captures the current viewport as a PNG.

• Unreal Agent can “look” at the shot to improve layout, spacing, and verification.

• reflection_query – Live API explorer

• Inspects any UClass and returns a schema of its properties and functions.

• The model can discover new APIs on the fly and then immediately use them in Python.

• file_search – UE5.6 Python brain extension

• Searches a UE 5.6 Python API vector store for relevant docs and examples.

• The agent reads real docs before writing code, increasing correctness on complex editor APIs.

• web_search – Scoped internet assistant

• Calls OpenAI’s web search tool for broader context, tutorials, and edge‑case details.

• Great for “how do people usually…”‑type questions around Unreal workflows.

• replicate_generate (optional) – Content on demand

• Talks to Replicate’s HTTP API to generate images, 3D, SFX, music, speech, or video.

• Returns local file paths; Python helpers (unrealgpt_mcp_import) turn those into Texture2D, StaticMesh, and SoundWave assets, ready to use in your materials, levels, or PCG setups.

• Voice & images – Rich control surface

• Voice input via Whisper (/v1/audio/transcriptions): speak your intent, let the agent handle the rest.

• Attach local reference images as additional context for layout, look‑dev, and content generation.

• Closed‑loop, agentic behavior

• After python_execute, the agent is instructed to verify its work with scene_query and viewport_screenshot instead of blindly chaining more code.

• Tool‑loop detection, execution timeouts, and result‑size limits keep runs safe and predictable.

If you want an AI that can own tasks like setting up PCG systems, crafting advanced materials, and automating editor workflows—rather than a passive chat window—Unreal Agent gives you a deeply integrated, open source foundation built specifically to drive Unreal Engine 5.