Can a Penpot Design Assistant Run Through Codex Instead of an API?

Idea

This quick experiment started with a local Penpot instance and an existing assistant plugin (AVA) that already talks to OpenAI compatible APIs. The question was simple: what happens if that handoff goes to the Codex CLI instead of the APIs it was designed for?

The practical goal was not to build a polished design copilot in an hour. It was to prove the swap was possible, see where it broke first, and work out whether the extra power is worth the extra wiring. It usually is, right up until it is your job to explain the wiring.

Useful for small teams, solo builders, and developers exploring AI APIs vs command line tools.

Status

Working-ish is the honest version. It does what we set out to do - but we never intended to use it long term.

The assistant can route requests through the Codex CLI, and that alone makes the experiment worth doing. It proves the integration point is not locked to the usual API pattern and opens the door to more capable local workflows.

What is still rough is everything around that core handoff. The assistant itself is not especially good at structuring design work, its skills are thin, and the output still needs a human with judgement to stop it arranging things like a caffeinated intern five minutes before lunch.

So the current state is closer to a promising prototype than a dependable feature. This is a short post for a short project!

Problems Faced

Technically, this was quick. Usually, quick projects have a habit of hiding their awkward bits until they are on stage.

The main problem was not getting text in and text out. That part was fairly direct. The real limitation showed up in the assistant's decision-making once it had more freedom. A design assistant that can access a stronger runtime is only useful if it also knows how to break work into sensible steps, keep context under control, and avoid producing a layout while being confident for all the wrong reasons.

There is also a product question sitting underneath the code question. An API driven assistant tends to be narrow by default: prompt in, response out. A CLI driven assistant can do much more, which means you need clearer guard rails. More power is great right up until your prototype develops opinions about filesystem it's hosted on.

So while this did fit into roughly an hour, the hard part was less about integration and more about seeing where the next layer of discipline needs to go.

Techniques That Helped

The biggest help was simple preparation. Codex was already installed, authenticated, and ready to run, which removed a lot of friction.

A local Penpot instance already being set up and in use also made the feedback loop much faster. Being able to change the plugin behaviour, trigger it again, and see the result immediately is far better than spending half the session wondering whether the bug is in your code, your config, or a service three network hops away.

Keeping the goal narrow helped as well. This was not a full assistant redesign. It was a targeted swap at the integration point: replace the OpenAI compatible API call with a Codex CLI handoff, confirm the round trip works, and only then think about richer behaviour.

That approach helped a lot. If you try to prove architecture, product value, and UX quality all in one pass, you usually end up proving that tired people make strange choices.

Lessons Learned

The useful lesson here is that a CLI based assistant is a different category of tool from an API only assistant. It is not just a new transport layer. It changes what the assistant can see, how it can act, and how much real work it might be able to do on your behalf.

That also means the bar gets higher. Once an assistant can incorporate skills and MCP, prompt quality alone is not enough. You need tighter task framing, better constraints, and a clearer idea of what the assistant should and should not be trusted to handle.

For Penpot and similar design workflows, it felt promising. A local assistant backed by Codex could do more than answer questions. It could be run from the project folder and help inspect structure, prepare assets, suggest implementation changes, or support design to code workflows in a way that is much closer to how real projects actually move.

Small experiments like this are useful because they show where the opportunity is before everything gets polished into a sales pitch. If you are testing similar ideas in your own tools and want a second pair of eyes, get in touch with holeaf and we can help you figure out the sensible version.