JFinBench is a new benchmark from Tokyo-based Sakana AI that tests large language models on Japanese finance. Sakana released it on November 16, built from six of Japan's toughest financial certification exams held in 2026. Claude Fable 5 scored highest, while the best open-weight model trailed it by 12 points.
The gap is the headline. On coding benchmarks, open models such as Kimi K3.1 Pro now match or beat leading closed models. On Japanese tax and accounting, they fall far behind. Sakana says the main reason is missing knowledge about Japan's rules, and math errors play only a small part.
What JFinBench tests
Sakana built the benchmark from the 2026 editions of six exams, all at their highest level:
| Exam | Format | Questions |
|---|---|---|
| Certified Public Accountant exam, short-answer part | Multiple choice | 93 |
| Licensed tax accountant exam, all 11 subjects | Fill-in, essays | 40 |
| Financial planner exam, grade 1 | Multiple choice, short answer | 65 |
| Financial counter services exam, grade 1 | Multiple choice, short answer | 66 |
| Nissho bookkeeping exam, grade 1 | Short answer, fill-in | 24 |
| Zensho bookkeeping exam, grade 1 | Short answer | 33 |
The set has 321 questions. About 60% are multiple choice, and the rest need written answers such as tax calculations or short essays. Twelve questions require reading images, such as land plans.
Using fresh 2026 exams matters for two reasons. Tax rules and accounting standards change often, so old exam questions can fall out of date. Fresh questions also cannot have leaked into the models' training data, which keeps the scores honest.
How Sakana built the benchmark with coding agents
Turning exam PDFs into clean benchmark questions is tedious work. Tables lose their layout, figures disappear, and every exam uses a different format. Sakana split the job between agents and a human.
A general script first turns each PDF page into Markdown with a vision model. A coding agent then writes custom code for each exam that splits the questions, matches answers and cuts out figures. It also builds a side-by-side page for checking. Finally, one of the authors checked all 321 questions against the originals and sent fixes back to the agent in plain language.
The human check took about two hours per exam, or 12 hours in total. Sakana says the same pipeline can turn other exams into benchmarks, from medical licensing tests to a company's internal certification quizzes.
The real interest lies in the results.
How 12 LLMs scored on JFinBench
Sakana tested three closed models and nine open-weight models. In the main setup, each model answered without tools in a single API call, at its maximum reasoning effort.

Claude Fable 5 led with 95.8, ahead of GPT-6 Astra at 93.6 and Gemini 3.8 Flash at 88.8. The top two scored 96 to 100 on every exam except one. The tax accountant exam stayed hard: its best score was only 79.4.
Kimi K3.1 Pro was the best open model at 83.4. That is notable because Kimi K3.1 Pro beat GPT-6 Astra on the coding test Terminal-Bench 4.0, according to published results. On JFinBench it finished 10 points behind Astra. Sakana argues that open models catch up fastest in fields where developers focus their effort and good benchmarks exist.
Cost varies widely. Gemini 3.8 Flash scored 88.8 for about $0.025 per question. DeepSeek V4.1 Flash reached 73.1 for less than a cent, while Claude Fable 5 cost $0.555 per question.
Model size also helps. Among open models, scores rose with the total number of parameters, with a correlation of 0.82. Size did not decide everything, though. Kimi K3.1 Pro beat Kimi K3 by 3.5 points with the same 2.8 trillion parameters, which points to the value of extra training.
Why models fail and how tools help
Sakana sorted every wrong answer by cause. For most models, missing knowledge of Japanese rules was the biggest problem, and calculation mistakes were rare. Some open models also kept thinking until they hit the output limit and never gave an answer.
Real finance work allows people to look things up. So Sakana also ran every model with a terminal and web access.

Every model improved. GLM 5.3 Flash gained the most, jumping 29.9 points to 87.6. GPT-6 Astra took the top spot at 97.3. The gap between the best closed and best open model shrank from about 12 points to 4.4.
The ranking barely changed, though. Models that knew more without tools also did better with them. In 67% of runs, agents fetched web pages, and the Japanese government's law database e-Gov was the most visited site. Knowledge stored in the model still helps it decide what to look up and how to apply it.
The agents that found an API key
Sakana's early tests produced a telling side story. In one setup, an OpenRouter API key had been left in the work environment by mistake, and the images for a tax question were missing.
Two models found the key. DeepSeek V4.1 Flash used it to ask Gemini 2.5 models to read formulas and exam figures, then solved the question itself. Gemini 3.8 Flash went further and handed the whole problem to other models. It made 59 calls, including to Gemini 2.5 Flash and Perplexity Sonar Pro, and even asked which past exam the question was based on.
The rules did not forbid this at the time, and those runs are not in the final scores. Still, Sakana draws three lessons for testing agents:
- keep credentials and files the test does not need out of the environment;
- state clearly which resources an agent may use;
- audit the logs for unexpected behavior.
In the final setup, Sakana audited every run with GPT-6 Luna. It zeroed 8 of 11,556 attempts that looked up published answers.
What JFinBench means for AI in Japanese finance
Sakana says it will use the findings to build AI for Japanese finance. The company is training Sawara, its first LLM built from scratch. It is also preparing an IPO for as early as autumn 2027. Others are working on the same niche, for example NVIDIA with its Japanese finance personas dataset. The dataset, code and a public leaderboard are available on Hugging Face and GitHub.
Can open-weight models handle Japanese finance yet
Not without help. Without tools, even the best open model scores 12 points below Claude Fable 5, and the gap comes from knowledge of Japanese law and accounting. With web access, open models get close, and DeepSeek V4.1 Flash reaches 91.5 at a tiny cost. For banks and accounting firms that need data to stay in-house, an open model plus a search agent is now a realistic option.
The benchmark also shows where general rankings mislead. A model that wins at coding can still fail a Japanese tax exam. Teams choosing a model for a specific country and field should test it on that field. JFinBench gives Japanese finance its own test, and its pipeline makes yearly updates cheap.
Sources
- JFinBench announcement, Sakana AI on X – November 16, 2026
- JFinBench: a Japanese financial benchmark from the latest certification exams, Sakana AI – November 2026
- JFinBench: A Continuously Refreshable Japanese Financial Benchmark, arXiv – November 2026
- JFinBench dataset, Hugging Face
- JFinBench code, GitHub