Together AI tripled Step 5 throughput with agents tuning its stack

Felix Laurent 8 Nov 2026 4 min read

Models

Together AI says it more than tripled the throughput of StepFun's Step 5 Preview on its servers. The company shared the result on X on November 8, 2026. AI agents searched through possible changes to the serving stack, and each change was benchmarked before the next one.

Together's chart shows eight optimizations that add up to 3.47 times the starting throughput. The work aimed at targets StepFun set for time to first token (TTFT) and tokens per second (TPS). Together has not published those targets or any absolute speeds.

What Step 5 Preview is

Step 5 Preview is StepFun's open-weight flagship. The Shanghai lab built it as a mixture-of-experts model with about 600 billion parameters, 27 billion of them active per token. It handles a 1 million-token context and image input, and it was trained for long agentic tasks such as coding.

We covered its launch pricing in October. StepFun charges $1 per million input tokens and $2.70 per million output tokens. Weights in BF16 and FP8 went up on Hugging Face on October 15 under the StepFun Model License, which adds attribution and commercial terms.

Together joined as a launch partner for the open weights. The two companies co-hosted a launch event in San Francisco on October 22. Together's model page lists Step 5 on dedicated endpoints, with pricing on request.

A model this size is expensive to serve, which is where the new result comes in.

The eight optimizations behind 3.5x

Together's chart lists each step with its own gain and the running total:

Together AI raised Step 5 Preview throughput 3.47x in eight steps, led by a new MoE backend at +40% and scheduling at +28% — Figures from Together AI's post on X, November 8, 2026

The two biggest gains came first. A new mixture-of-experts backend with a fused router added 40%, and better scheduling and batching added 28%. Together then fused the decode kernels and turned on speculative decoding, where a fast draft predicts tokens for the model to verify.

The second half worked on attention and prefill, the stage where the model reads the prompt. It spread sparse attention across GPUs with context parallelism and ran prefill in FP8. It also tuned the prefill kernels and added what it calls block-union sparse attention. Each of these steps gave between 8% and 15%.

The gains multiply, so eight modest steps reach 3.47x. Together has not said which hardware it used or how fast the model ran before or after. The more unusual claim in the post is who did the searching.

How AI agents helped optimize Step 5

In the X post, Together says agents searched across "moe routing, scheduling, kernels, spec decode + sparse attention" for gains. It does not name the agents or say how many experiments they ran.

The timing points to Together Compiler, which the company launched on November 4. It is a system where parallel AI agents tune a model's serving setup and share findings through a message board. A cost model drops weak ideas before they are tested. Together says engines that stay loaded cut restart time from 467 seconds to 9 seconds, so agents can run far more experiments.

On launch, Together reported up to 2.6x more tokens per second on Kimi K3, GLM 5.3, MiniMax M3.1 Flash and DeepSeek V4.1 Flash. That work took under 24 hours per model. The Step 5 post does not mention the Compiler by name, but it describes the same method.

Together has used agents in this way before. In August 2025 it used OpenHands and Claude Code agents to train speculative decoders, gaining 22% to 37%. Its ATLAS speculator, presented in October 2025, raised decoding speed by up to 4x. In July 2026 the company raised $800 million at an $8.3 billion valuation.

What developers asked about the result

The post drew about 7,000 views and a short thread of replies. StepFun's own account thanked Together. Several engineers asked what the headline number leaves out.

One reply pointed out that wider batches and longer speculative drafts can raise total throughput while slowing the slowest requests. Another noted that higher total throughput does little for the speed of a single long agent session. Others asked which of the eight changes the agents found on their own, and whether TTFT improved too.

Those questions matter for anyone choosing a host for Step 5.

Why Step 5 throughput matters

Throughput decides how many users one server can handle, which drives the price per token. Our October analysis found that Step 5 is cheap per token but uses more tokens per task than rival models. Faster serving helps hosts offset that.

The result also shows how much speed the serving software holds back. Together started from a stack that already ran Step 5 and ended with 3.5 times the throughput on the same model. StepFun's own engineering post gives a sense of scale for single tricks. Speculative decoding with three draft tokens gave about 30%, and native FP8 experts added about 20%.

The claim comes only from a chart on X. Together has published no blog post, hardware details or latency figures. Until it does, 3.5x is a vendor number, and per-request speed is still an open question. For teams that pay for dedicated Step 5 endpoints, the method matters as much as the figure. If agents can keep tuning a stack this way, older models will keep getting cheaper to run after launch.