Playing Skyrim VR With ChatGPT Powering NPC Conversations

An AI mod for Skyrim VR lets you have natural language conversations with characters using your voice.

Skyrim VR has long been a favorite playground for modders, and the Mantella AI NPC voice mod takes the immersive possibilities of this VR classic to a whole new level.

The mod lets you have real-time conversations with every NPC in the game. Mantella delivers this by integrating speech-to-text, OpenAI’s ChatGPT large language model (LLM), and text-to-speech. The mod transforms the way you interact with the many NPCs you meet along your long journey in Skyrim, and while there is noticeable latency, you can still see how the concept redefines what it means to inhabit a virtual world.

I tested out the mod on a handful of the thousands of NPCs throughout the game, each with their own distinct backstory and individual personalities. The NPCs are capable of remembering past conversations, have awareness of in-game context such as the time of day, and can even notice some of the items you’re carrying. If you’ve ever dreamed of building a deeper connection with the diverse characters of Tamriel, this mod makes that a reality.

An experimental feature even allows you to prompt NPCs to follow or trade items. There are also options that allow NPCs to become hostile, depending on what you say, adding a new level of skill to conversing with them. These somewhat lifeless automatons are now transformed into more fully realized virtual beings, similar to what you might expect to find on a very basic Star Trek holodeck program.

The Experience 

Mantella allows users to experience its AI agents for free using an Openrouter.ai account, though you probably want to pay for a better experience.

Mantella unlocks the ability for the mod to interact with several different services, some free and some paid. You can select from a pre-populated list that gives an estimation of the average cost associated with the paid accounts when using various AI models. After switching to my paid OpenAI ChatGPT API key with Openrouter set to use the GPT-4o mini model, the wait for a response became much shorter and the conversations seemed to flow more naturally as seen in this example: 

For the initial testing however, I first used the free account which did work but came with significant lag time in the AI’s spoken responses. You can see an example of that here:

Regardless of the model used, participating in conversations felt engaging, with interactions tailored to my choices in ways that made me feel as if I could foster distinct relationships with the world’s many characters. The Mantella mod also enhances the role-playing aspect of Skyrim for those who might be into experiencing the game that way, permitting players to express their imaginations in ways previously unfathomable in a VR fantasy.

However, it’s important to note that AI voice mods like Mantella are still in the early stages. While the potential a mod like this brings to interactive VR gaming is awe-inspiring, bugs and lag issues still persist, particularly during demanding scenes. Conversations can also occasionally get stuck, or the NPCs will just fail to respond altogether, especially when talking to multiple AI enhanced characters at the same time in a crowd.

It is the little inconsistencies like these that remind me that technology like this is still new and evolving. However, for a sci-fi enthusiast such as myself, every moment spent engaged in these new dynamic dialogues felt like a step closer to the dream of conversing with fictional characters in Star Trek’s holodeck. 

Mantella also stands out as an early glimpse into the future of interactive immersive gaming, opening up endless possibilities for storytelling, creativity, and personal connections inside one of VR’s most cherished virtual landscapes. We recently reported on the efforts at Astrobeam, where one of the creators of Job Simulator is looking at this same area.

If you’re ready to take Skyrim VR to the next level and make it an even more immersive experience, give Mantella a try.

Be sure to join us Friday for our weekly VR Gamestream where Beck and I will be playing Skyrim VR using this mod and taking requests from the chat as to what we should ask the NPCs: