Problem
Exploring how a language model can control an agent inside a game engine naturally, without menus or fixed scripts.
The video loads when you press play, not beforeBrain AIBrain AI
LLM-driven NPC · Unreal Engine · voice to action
Built an AI-powered interactive agent in Unreal Engine integrated with the OpenAI API. Users control the agent through voice commands, triggering real-time actions like navigation and movement, with a swappable response voice via ElevenLabs.
- Unreal Engine
- C++
- OpenAI
- OpenAI
- Integrated language model
- Voice
- Spoken command control
- UE
- Real-time 3D agent
- ElevenLabs
- Swappable synthesized voice
The story
Solution
I integrated the OpenAI API into Unreal Engine in a modular way: the user's voice commands trigger agent actions (move, jump, and more) in real time, with a swappable response voice via ElevenLabs.
Result
An interactive Unreal Engine agent that executes real-time actions from voice commands processed by OpenAI, with a swappable response voice via ElevenLabs.
The systems I built in the engine
Unreal Engine Developer
Modular OpenAI Integration
Built a modular system to integrate the OpenAI service inside Unreal Engine, isolating the AI logic from the rest of the project so it can be reused or swapped without touching the agent.
Unreal Engine Developer
Intent-to-Action Mapping
Translated the voice commands interpreted by the model into concrete agent actions —moving, jumping, and more— inside the engine, closing the real-time voice-to-action loop.
Unreal Engine Developer
Swappable Voice Layer
Built the swappable voice-synthesis layer via ElevenLabs, so the agent's spoken response can be changed without touching the interaction logic.
Gallery
Next project
GhostlineGhostline
Unity WebGL + Cloudflare Workers