Debugging Token Burn with Claude Code: A Spotify-Inspired Experiment

Introduction

The growing reliance on AI models to handle routine coding tasks raises questions about efficiency, especially regarding token consumption. A tech enthusiast experimented with Claude Code, attempting to replicate Spotify's success in reducing token costs through a specific setup and usage patterns. Despite high expectations influenced by Spotify's reported savings on API costs, the results revealed that the anticipated efficiency gains were not entirely realized.

Understanding the Token Burn Challenge

Token costs can accumulate rapidly when using AI models for common coding tasks like generating test files or extracting information from extensive codebases. Unlike subscription models, API-driven projects can lead to unexpected expenses if not managed carefully. Spotify addressed this issue by creating two new AiKA Modes on its Portal—a developer platform designed for internal use, later made available for other developers.

Replicating Spotify's Setup

Although lacking access to Spotify's proprietary tools, the experiment utilized Claude Code along with other large language models (LLMs) to reproduce the resource-saving mechanisms implemented within Spotify's architecture. The engineer's adaptation involved creating a system to automatically redirect coding tasks to less expensive models when feasible. This included a detailed examination of the plugin known as Shunt, which allows Claude Code to offload the grunt work to a cheaper model.

How Shunt Works

Shunt's primary function is to efficiently handle reading tasks: instead of Claude Code processing lengthy files itself, it delegates this to a less expensive model capable of summarizing content. For instance, when confronted with a file exceeding 350 lines, Shunt links Claude to a bulk-reader service that processes the file and returns a concise summary.

Initial Setup Challenges

While configuring the environment, several technical hurdles arose, each critical to understanding the subsequent performance discrepancies:

• The structure of Shunt’s hooks proved incompatible with the current version of Claude Code, causing numerous errors until adjustments were made.
• The local model used, Qwen3-Coder, initially produced erroneous function names and had variable performance, validating the need to ensure reliable backend support.
• Moreover, attempts to use the headless version of Claude Code led to complications with permission prompts, necessitating adjustments to allow proper functioning.

Measuring Performance: Token Usage and Results

To quantify the benefits of utilizing Shunt, the engineer set up a comparative analysis between runs with and without it using an extensive codebase. The analysis primarily focused on the agency/app.py file, consisting of 3,079 lines. The goal was to determine how effectively Shunt could minimize token consumption during reading and writing tasks.

Token Consumption Breakdown

In this examination, it was clear that both delegated runs using Shunt consumed more tokens compared to running Opus 5.5 independently. Specifically, using Qwen3-Coder increased token consumption by 23% and had extended runtimes. Similarly, Claude Haiku's usage resulted in a 34% increase in tokens spent, despite yielding better test outputs.

Efficiency Observations

Shunt's intended efficiencies only surfaced when examining larger files. Delegating tasks may have been advantageous in specific scenarios, but the corresponding costs negated much of the anticipated savings. For example, repetitive reading tasks—particularly those handled efficiently by Opus without hitting the read hooks—demonstrated that any potential savings from delegation were not realized.

Write Delegation Performance

Writing tasks revealed additional discrepancies in token consumption. The delegated runs’ quality results varied significantly, driven primarily by the local model's handling of the passed specifications versus Opus 5.5’s direct work.

Problematic Outputs

Qwen3-Coder frequently failed critical checks identified in the passed specifications, leading to extensive losses in test completion rates. In its specifications, Opus proficiently identified vital components regarding how CLI commands functioned, highlighting potential issues with temporary directory handling, yet Qwen3-Coder mismanaged these specifics, resulting in substantial failures in the final tests.

Increased Latency Issues

Extent of delays experienced during the writing tasks was another critical factor in assessing Shunt’s effectiveness. On the Vulkan backend, Qwen3-Coder processed about 12 tokens per second, while Haiku also introduced latency by initiating an entire new Claude Code session with each task. The combination of high token consumption and increased latencies shifted many functions to become unmanageable, effectively doubling downtime.

Spotify's Successful Approach Versus the Experiment

Meanwhile, Spotify’s benchmarks show impressive percentages in savings (up to 90% for reads) due to its structure and model efficiencies inherent to their setup. If Claude Code operated similarly—especially in a setup linking to large files—it’d likely achieve similar success. Yet, the test results indicated that not all setups aligned similarly or worked efficiently with Claude Code.

Conclusions and Future Considerations

While aiming to mimic Spotify's success with Claude Code and the Shunt plugin yielded insightful results, it also reinforced the importance of careful planning and execution when managing token burn. The direct comparison between setups illustrated that increase in token consumption can often offset the intended benefits of delegation, suggesting that teams should evaluate their architecture closely before opting to introduce additional tools like Shunt. Furthermore, future explorations into model performance against varying backend configurations could yield potentially optimal strategies for developers aiming for efficiency.

Key Takeaways

• The token burn associated with AI models, especially in API-based setups, can accumulate quickly.
• Implementing structures similar to Spotify's Shunt plugin led to increased token consumption, rather than savings.
• Delegating reading and writing tasks didn’t yield expected efficiency gains when tested against established processes.
• Direct operations by Opus 5.5 proved faster and cheaper than delegated runs with external models.
• Future improvements in setup depend on careful assessment of backend performance and integration with existing workflows.