An audio reactive rebuild of our sidekick game in Usagi Engine

Idea

Our website minigame started as a small experiment and it didnt take long before we asked "what if it was a desktop app?". That led us into Tauri, which worked well enough to prove the idea had legs, buttons, and a mild desire to become more complicated than planned.

For this version, we rebuilt the minigame in an actual game engine without overlaying it onto the desktop. We chose Usagi Engine because it is a Lua based 2D engine built for rapid prototyping, which sounded ideal for a small playful project that still needed real structure underneath it.

We also added a new challenge: making the minigame react to whatever audio was playing on the player's PC. Music, videos, system noise, suspiciously loud notification sounds - all fair game. The aim was to move beyond a self-contained toy and make the game feel a little more alive on the desktop. This version targetted Windows instead of Mac.

Useful for small teams, solo developers, and curious business owners who want to see how a playful web idea can grow into a more technical prototype without turning into a six-month expedition.

Status

The current build is running well. The core loop is in place, the visuals respond to audio input, and the Windows-first approach kept the scope tighter than trying to support every platform at once.

The main missing piece is narrative. At the moment, the minigame works as a reactive never ending experience, but it needs a clearer reason to exist inside the wider Holeaf world. As a game it can be technically successful and still feel like it has wandered into the room without introducing itself.

We may continue with this version before testing other engines. Usagi Engine has been a good fit for fast iteration, but the audio work did expose one limit: it could not handle the loopback capture we wanted by itself. To solve that, we built a small C# assistant app to capture Windows audio and pass the useful signal back into the game.

That extra helper was not a failure of the engine. It was the expected trade-off from our research: use a fast Lua engine for the game layer, then add a small native companion where the operating system work gets more specific.

Problems Faced

The main challenge was audio capture. Making a game respond to its own sounds is one thing; making it respond to whatever the player already has running on their PC is a different job with its own issues.

Usagi Engine gave us the right shape for fast 2D prototyping, but it was not designed to handle Windows loopback audio capture directly. The C# assistant became a small bridge between the operating system and the Lua game code.

Once that architecture was clear, the remaining work was mostly about getting the signal clean enough to use. Raw audio data is not automatically interesting. It needs to be sampled, reduced, and turned into values the game can respond to without looking like it has had too much coffee.

Codex had this task licked though. GPT-5.5 on medium thinking mode was useful for creating the C# capture approach, checking assumptions, and tightening the loop between automated checks and manual testing. Codex was great at iterating quickly, testing, correcting, and trying again.

After around 20 minutes of back and forth, we had the assistant app capturing audio, the game receiving usable values, and enough manual testing to trust the behaviour. Not production-perfect, but very much prototype-useful.

Techniques That Helped

Preparation made the biggest difference. We had already looked at Usagi Engine before starting, so we were not asking the agent to learn the engine, design the game, generate assets, solve audio capture, and read our minds at the same time. AI is fast, but it still performs better when you stop handing it a fog machine.

Before the build session, we used ChatGPT to help with three areas:

• Sprite direction and generation, with plenty of human editing and rejection along the way.
• Research into Usagi Engine's limits, especially where Lua ended and native platform work might begin.
• Initial prompts for the coding agent, rewritten heavily so they were specific, testable, and grounded in what we actually wanted.

We split the useful knowledge into two working files: agents.md for how the agent should behave, and handoff.md for the project state, decisions, and next steps. That gave the coding session a shared memory without needing a large planning system.

We also chose not to use our usual wtp tasking tool for this one. Since Usagi Engine is designed for rapid prototyping, we wanted to see if the whole thing could be built in a single focused session with pre-planned prompts. We forced the work to stay small, testable, and honest.

The best technical decision was separating concerns early:

• Usagi Engine handled the game loop, visuals, and player experience.
• The C# helper handled Windows loopback audio capture.
• Manual testing checked whether the reactive behaviour actually felt right.
• Automated checks caught the boring mistakes before they became dramatic ones.

That mix gave us a working prototype without pretending every part belonged in one language or one tool.

Lessons Learned

Experience still matters. Twenty-plus years of development work helped us spot the risky parts early, question the first solution, and keep the build moving when the engine could not do everything directly.

The more useful lesson was about knowing our own gaps. We're not deeply familiar with every edge of Usagi Engine or Windows audio capture. Instead of bluffing through that, we used AI to get up to speed, then applied human judgement to decide what was worth trusting.

That made the prompts better. We could give the agent clearer context, better assets, a simple task list, and firm checkpoints for manual testing. The agent did not have to guess the project shape. It could focus on the next useful step. This is the simple idea behind Spec Driven Development - decide as many things up front as you can & you won't be wasting an afternoon unpicking a solution the AI has learned from scraping years of Stack Overflow.

For small business and solo projects, that is often the real advantage of AI-assisted development. It does not remove the need for technical judgement. It makes the learning loop shorter, so long as someone is still steering.

The prototype also reminded us that choosing a tool is not about finding the one thing that does everything. Usagi Engine was a good choice because it made the game layer quick to build. C# was a good choice because Windows audio capture needed native help. The project worked and came together quickly because each part had a sensible job.

If you are exploring a web app, desktop prototype, small game, or AI-assisted build and want a clearer route through the awkward middle, get in touch with Holeaf. We are happy to help turn the rough idea into something you can actually test and trust.