AI physics benchmark: GPT-6-Astra solved 49 of 50 unpublished problems

Felix Laurent 26 Oct 2026 4 min read

models

A group of 34 physicists has tested AI agents on 50 unpublished theoretical physics problems from their own research. GPT-6-Astra running in Codex got 49 of them right, at about $2.33 per problem. Opus 5.5 in Claude Code solved 46. The AI physics benchmark is called TheoPhysEval, and the paper went up on arXiv on October 9, 2026. We read it to see what exactly was tested: physics research, or calculation.

What the TheoPhysEval benchmark tested

The problems came from working theorists at ICTS, TIFR, Harvard, Penn, UT Austin, KU Leuven and other institutes. Praneeth Netrapalli of Google DeepMind and Suvrat Raju of ICTS coordinated the project. Each contributor wrote a problem from their own research area. The authors say a world expert could solve a typical one in about a day, and experts in neighboring fields would need much longer.

The paper sits in hep-th, the high-energy theory section of arXiv. It is cross-listed in gravitation, statistical mechanics, strongly correlated electrons and AI. The problems are spread across eleven themes.

Theme	Problems
Quantum field theory	7
Gravitation and cosmology	6
Classical physics	5
Stochastic processes and biological physics	5
Quantum phases of matter	5
Scattering amplitudes	5
Holography and entanglement	5
Electronic structure theory	4
Nonequilibrium statistical mechanics	3
Lattice theories	3
Particle physics	2

One example asks for the rate at which axions convert into photons by scattering off electrons in a dielectric. Another adapts a Page time calculation from a five-dimensional spacetime to four dimensions. The full set is public on Hugging Face.

Physics problems, judged by a number

The authors call these "turn-the-crank" problems. A known technique is used to extend a result that already exists. The answer is new and sometimes publishable, but no problem is an open question. The authors say contributors were told to publish real open problems themselves.

Grading is automatic and looks only at the final answer. Where the natural answer was a formula, the authors gave the agents numerical values for the parameters. Every answer became a number or a list of numbers that cannot be guessed. This removed the need for an LLM judge. It also means the benchmark checks whether the calculation is right, not how the agent set it up.

So the subject is physics, and the skill under test is research-level calculation. The paper says this directly: solving these problems does not require great originality or new physical theories. It requires knowing the current literature and adapting its techniques accurately.

The results

Five agents ran each problem once, on two tracks. The realistic track allowed web search and local tools. The isolated track ran in a Docker sandbox with the internet off. Each task had 8 hours, and there was no cap on tokens or cost.

Agent	Realistic track	Isolated track	Cost, realistic
GPT-6-Astra in Codex	49/50 (98%)	46/50 (92%)	$116.28
Opus 5.5 in Claude Code	46/50 (92%)	46/50 (92%)	$240.43
Gemini 3.8 Flash in Antigravity	34/50 (68%)	31/50 (62%)	$95.01
DeepSeek-V4.1-Flash in Codex	27/50 (54%)	25/50 (50%)	$37.42
GLM 5.3 in Claude Code	21/50 (42%)	26/50 (52%)	$180 for both tracks*

*GLM ran on a coding plan. The API price would have been $311.71 for the realistic track. Costs are the authors' estimates from token usage. Per-problem figures in this article are our division by 50.

Google's entry was Gemini 3.8 Flash, not a larger Gemini model. The authors set reasoning effort to the maximum for every model. For Gemini 3.8 Flash that is "high."

The one problem nobody solved

Astra's single miss was an inverse problem. The agent had to find a differential equation that produces a given curve. No agent solved it on either track. The authors think it should be easy to build more problems like this if someone wanted to lower the scores.

Astra's score was at first lower. The authors marked several answers as wrong, then checked with the experts. In each case the expert's reference solution had a subtle error. After the experts fixed their solutions, Astra's answers matched.

Opus 5.5 made a few subtle errors. In one case it took initial data from a cited paper instead of the data in the prompt. In another, it kept hitting the output token limit and gave no answer. The weaker agents made errors an expert would spot quickly. Gemini and GLM used five-dimensional formulas in a four-dimensional problem.

The memorization finding

Turning off the internet barely changed the scores. The authors say the gap is within the statistical noise of a single run. Many problems rely on formulas from specific papers that were not given to the agents. The authors conclude that the leading models have memorized large parts of the physics literature. They can reproduce key formulas without looking them up.

Each agent ran only once, and the models sample at nonzero temperature. The authors say scores could shift on a rerun. They invite others to rerun the benchmark from the Hugging Face files.

What the authors say agents still cannot do

The authors write that frontier agents may now match the best humans at concrete calculations, even in narrow topics. They also list what the benchmark does not measure. A calculation comes late in research, after someone picks a theory and asks the right question. In their experience, agents do not switch to a simpler related problem when the first one is too hard. They also lack a physicist's "taste" for which problems are interesting.

The authors see two practical effects. Problems like these were once given to new graduate students as first projects. The paper says that no longer makes sense, unless the goal is only to learn the method. The authors also expect harder problems to need more compute, which may make theoretical physics less equal across institutes.