Sifter Update: Building a Practical Shopify Theme Testing Workflow

Idea

Sifter (or SFTR) started as a way to test Shopify themes offline. The decision now is to push it closer to a product that can sit inside a real client workflow.

The release of WTProj (our tool) and the GPT-5.6 family of models made this a good test of using AI models like a small development team. Sol acts as our senior developer, taking on architectural changes and work that crosses a large part of the codebase. Terra is our seasoned developer for complex, well-scoped tasks. Terra takes the junior role for smaller jobs and acts as sprint manager, opening Codex threads with the right model for each task. WTProj now supports suggested models, which makes that division of work much easier to repeat.

The useful question was never whether a model could write code. It was whether a group of model-led threads could keep a long-running project pointed in roughly the right direction. “Roughly” is doing some work there.

This post is useful for developers and small teams exploring AI-assisted development, Shopify tooling, or a practical route to testing client themes before deployment.

Status

SFTR is 13 Liquid filters away from being something I would use to test client themes inside a CI/CD pipeline. It does not cover every edge case, and it does not match Shopify Plus feature for feature. For most Shopify users on lower plans with a B2C store, though, it is already a useful test environment.

Two related projects have now been merged into SFTR. The first is a theme editor that lets you modify JSON files in the browser and see preview updates live. The second is a store cacher that uses a developer's logged-in Shopify CLI session to make the GraphQL calls needed to cache store data in an SFTR-compatible format.

That cacher is the part that changes the shape of the project. SFTR is no longer only a generic offline theme test. It can now test a client’s theme against a snapshot of that client’s data, which is much closer to the conditions that cause real bugs.

The two projects had not been published as standalone posts because they were not ready. They are now good enough to be part of the wider workflow, which is a more useful milestone than pretending every first draft deserves a launch announcement.

Problems Faced

The largest problem was testing across three operating systems. With VM setup being the easy part, the harder job was making the Linux development environment build and test for all three without turning the build process into its own full-time project.

The Windows build can be created on Linux and sent over for testing without too much drama. macOS was less cooperative. Libraries exist that could move more of the work onto Linux, but the simpler route was to let the build script perform the macOS build on macOS before running its tests.

Windows exposed a different class of problem. The GPT-5.6 models were less reliable with Windows and PowerShell over SSH than with the rest of the stack. A few failures looked like process timeouts but were really binding or connection problems. One process bound to one interface while the test tried to reach another. Scripts were rewritten several times to separate those cases and report them properly.

That debugging loop consumed around 40% of the weekly usage remaining. The model was capable of making progress, but it could also spend a long time revising its own script without quite noticing that the original assumption was wrong. Computers remain very good at following a plan they invented five minutes ago.

Techniques That Helped

Manual intervention and monitoring helped most. Long-running agent threads are useful for keeping several pieces of work moving, but they still need a person checking what is happening and whether the latest result makes sense.

The practical pattern was simple: let the agents handle well-defined tasks, watch the tests and build output, then step in when the evidence did not match the model’s explanation. That kept the work moving without allowing one mistaken assumption to become an afternoon-long side quest.

Lessons Learned

An agent that can run for hours and spawn Sol, Terra, and Luna threads is genuinely useful for managing work within a fixed subscription. It can split tasks, keep smaller jobs moving, and make better use of the models available for each type of work.

It also needs supervision. On one task, an unchecked sprint agent could easily have spent the rest of our usage circling the same Windows and PowerShell problems. More runtime does not automatically mean more progress; sometimes it just gives a wrong assumption a larger office.

The strongest setup so far is a mixed one: agents for repeatable implementation work, a clear task boundary for each thread, and a human watching the output when builds cross operating systems or touch live project data. That combination is less exciting than handing over the keys, but it is considerably more useful.

If you are building a similar workflow and want a second pair of eyes, get in touch with holeaf.uk and we can help you work out a sensible next step.