Kog Aims for Faster AI Inference with GPU Optimization

Pioneering GPU Optimizations for Faster Inference

The demand for faster AI inference is rapidly escalating as businesses seek to improve efficiency and performance. French startup Kog is stepping into this competitive landscape with a unique proposition: enhancing the capabilities of conventional GPUs, such as the AMD MI300X and Nvidia H200. Unlike specialized hardware solutions, Kog is focusing on software optimization to unlock new levels of performance from the existing machinery already utilized by enterprises.

CEO Gaël Delalleau's ambition for Kog was recently highlighted during a tech preview that garnered significant attention on Hacker News. The demonstration showcased the potential for "extremely fast single-request decoding" on standard datacenter GPUs, tapping into the hardware enterprises already own. While some observers voiced disappointment that this optimization doesn’t yet extend to laptops, the reaction to Kog's initial promise was overwhelmingly positive. Delalleau noted, “We had 200 tangible business leads,” indicating robust market interest in the technology.

Market Insights and Customer Demand

Delalleau also shared insights on why Kog's software optimization is timely: speeding up inference processing is becoming a critical factor for businesses relying on AI workflows. Users of systems like Claude Code are acutely aware that results can sometimes take hours, leading to potential revenue loss. Recognizing this gap, Kog aims to reduce these wait times, particularly targeting customers who depend on AI for professional tasks.

The CEO explained the core strategy: "We’re targeting professionals put off by latency that rely on AI workflows for their work. A faster outcome via our Kog Inference Engine (KIE) means more revenue for these users, which drives demand for our solutions." Kog has also engaged with design partners to create opportunities for generating games and apps directly from prompts, whereby enhanced inference time has financial implications for their operations.

Understanding the Challenge with Larger Models

As Kog ventures into the realm of larger AI models, the path forward requires fundamental shifts in focus. Delalleau noted the prevailing demand for larger models among potential clients. However, many prospective users lack the willingness to fine-tune smaller models. This led to Kog’s decision to concentrate heavily on accelerating larger model development.

The company's early demonstrations indicated impressive capabilities, achieving 3,000 tokens per second (TPS) during processing requests with a relatively small model—the Laneformer 2B, which includes around 2 billion parameters. However, to fulfill promises of delivering up to 30x faster LLM inference, Kog must demonstrate similar advancements with larger models that present additional challenges in inference speed.

Performance Metrics

Unlocking GPU Potential Through Software Optimization

Kog’s focus is not merely about quick decoding; it’s about what can truly be achieved with the hardware that already exists. Delalleau remains confident that GPUs hold a substantial potential often underestimated. He states, “GPUs have a bright future,” arguing that advancements in memory bandwidth and optimization techniques can enable GPUs to achieve levels of performance previously thought impossible. He challenges the misconception that GPUs are ill-suited for decoding. Kog differentiates itself from other companies like ZML, which released hardware-agnostic software to improve inference speed by bypassing Nvidia's CUDA. Kog seeks a deeper understanding of how to maximize GPU capabilities, similar to research initiatives at Stanford’s Hazy Research lab.

Kog's Unique Approach and Team Background

Delalleau's background is especially relevant to his approach to Kog. Having a foundation in solid-state physics from France’s École Polytechnique and experience in offensive cybersecurity, he encourages a mindset within his team that prioritizes thorough understanding. He emphasizes the importance of recognizing the laws of physics as they apply to GPU architecture.

His previous experience also shapes Kog's methodology, where understanding how to reverse-engineer technology is key—transforming challenges into solutions. However, he admits this approach is labor-intensive, stating, “For every new GPU, we’ll dedicate several weeks or even months to really dig into the details and conduct GPU engineering research on that hardware.” Given Kog’s current team size of just 11 members, this translates to focused exploration but also sets limitations on the number of GPUs that can be optimized concurrently.

Future Directions and Funding Needs

Looking ahead, Kog aims to integrate its innovative methodologies into agent-based pipelines that could expand its support for more GPU models and diverse AI tasks. As the European technology landscape pushes for greater independence in AI capabilities, Kog is poised to be part of this shift, backed by investments from Scaleway, Bpifrance, and the French Tech 2030 program.

Kog’s immediate challenge lies in proving its optimization works on larger language models (LLMs). “Once we’ve implemented our first major model at 10x speed, which I think will be in September,” Delalleau forecasts, “We will begin demonstrating customer traction.” This would position the startup well for its Series A funding round, allowing it to realize its growth ambitions in an expanding AI market.

Key Takeaways

• Kog aims to improve AI inference speeds significantly using existing GPU technology.
• The company achieved 3,000 TPS with a small model and targets 30x faster speeds for larger models.
• Kog is backed by substantial funding, including support from Scaleway and Bpifrance.
• Future developments may enhance the company’s capability to support more GPU models.
• Marketing efforts have already led to over 200 business leads for Kog's inference solutions.

With a dedicated focus on GPU optimization and an ambitious roadmap, Kog stands at the forefront of driving efficiency and innovation in AI inference. The challenge remains in proving these capabilities on a larger scale to fulfill market expectations.