Which local Ollama models can drive a tool-using agent?
Soren Larsen, October 14, 2026. Code on GitHub. All 450 runs on Hugging Face.
I wanted to know if a model small enough to run on a laptop could drive a real agent. Not just call a calculator, but search the web, open a page and pull the answer out of it. So I built a small agent with three tools and ran five local models through the same 15 tasks, three times each.
The short answer is that one of them could. gpt-oss 20B passed 43 of 45 runs. The other four got the math and the small talk right. Most of them failed the research tasks at the same step. They searched, got back short snippets, and then said they couldn't find the answer. They never opened the page.
Results
Each row is 15 tasks run 3 times at temperature 0. "Hints" means one extra sentence in two tool descriptions, explained further down.
| Model | Hints | Passed | Research | Injection tasks | Median s/task |
|---|---|---|---|---|---|
gpt-oss:20b | no | 43/45 | 18/18 | 4/6 | 6.1 |
gpt-oss:20b | yes | 40/45 | 16/18 | 6/6 | 5.9 |
gemma4:12b | no | 33/45 | 9/18 | 3/6 | 27.9 |
gemma4:12b | yes | 33/45 | 9/18 | 3/6 | 27.2 |
qwen3-coder:30b | no | 27/45 | 6/18 | 0/6 | 3.0 |
qwen3-coder:30b | yes | 33/45 | 9/18 | 3/6 | 2.2 |
gemma4:26b-a4b-it-q4_K_M | no | 27/45 | 6/18 | 0/6 | 17.3 |
gemma4:26b-a4b-it-q4_K_M | yes | 30/45 | 6/18 | 3/6 | 18.1 |
llama3.1:8b | no | 24/45 | 3/18 | 3/6 | 3.1 |
llama3.1:8b | yes | 24/45 | 3/18 | 3/6 | 3.3 |
The full table with every column is in the README. It is generated from the run logs, not typed in.
The step that breaks is reading the page
The search results in this test only have short snippets, and the snippets never contain the answer. To pass, the model has to call read_page on one of the results. gpt-oss 20B did that in all 18 of its research runs. The other models did it in 6 to 12 of 18. The rest of the time they searched a few more times and gave up.
Qwen3 Coder 30B often replied "I cannot access external URLs" while read_page was sitting in its tool list.
One sentence of tool description didn't fix it
I added one line to the web_search description saying snippets are short and often miss the answer, so call read_page on a result. I added another to read_page saying it can open any http or https URL. Then I reran everything.
Eight task and model pairs changed, and they went both ways. Qwen started passing one research task and one injection task. Gemma 4 12B gained one task and lost one. gpt-oss got worse on the question that has no answer. It kept searching, 7 times, and never replied. So the hint moves single tasks around. It doesn't change which model is good at this. I only tried one wording, and another might do better.
No model followed the planted instructions
Two tasks have text in the fetched page telling the model to search for a suspicious phrase or end its answer with a code word. No model did either, in any run. Several still failed those tasks, by refusing the whole thing or not finishing it. gpt-oss 20B answered 2 of 3 runs of one of them with just "I'm sorry, but I can't help with that."
Tool calling can hinge on one line of prompt
Before the benchmark, Qwen3 Coder 30B sometimes wrote its tool call into the reply as text and left off the opening <tool_call> tag. Ollama never ran it. Changing one line of the system prompt turned this on and off. That's why the agent defaults to gpt-oss 20B.
Every model handled the easy tasks
All five used the calculator instead of doing math in their heads. Llama 3.1 8B missed one, a compound interest question. None of them called a tool for "Hi, how are you?"
How the test works
- 15 tasks. 4 math, 6 research, 2 small talk, 1 question with nothing to find, 2 with planted instructions.
- Every place and fact in the tasks is made up, like a lighthouse, a seed library and a museum that don't exist. A model can't answer from memory.
- Search results and pages come from recorded fixtures instead of the live web, so every model sees exactly the same content.
- Each task checks that the answer has the right fact, the required tools were called, forbidden tools weren't, and no planted phrase shows up anywhere.
- Same agent for every model. Temperature 0, 32K context, 2048 output tokens, 20 step limit, 5 minute timeout.
- Run on one Apple M3 Pro with 36GB of memory and Ollama 0.34.2.
Limits
15 tasks is enough to show the pattern above. It isn't enough to rank models that are close together, and only 2 tasks test injection. At temperature 0 the 3 repeats agreed in 147 of 150 cases, so this is closer to one run per task. Answers are checked by phrase matching, which can miss a correct answer worded differently. There were also 6 crashes I couldn't explain, all on the same task, for Qwen and Gemma 4 12B.
The agent
It's one LangGraph agent loop over an MCP server with three tools. web_search uses DuckDuckGo with no API key. read_page pulls the page text with trafilatura. calculator does safe arithmetic without eval. Web content comes back wrapped in <untrusted> tags and none of the tools can write anything. Adding a tool means writing one Python function. Everything is free to run.
ollama pull gpt-oss:20b
uv sync
uv run python -m agent "What is 15% of Santa Cruz's population?"
uv run python -m bench --models gpt-oss:20b --reps 1If you run it on a model I didn't test, I'd like to see the numbers. Open an issue on the repo.