Lifeboat blog

If you are writing more powerful tools than add_two(a, b) or web_search(query_string) for LLM to call, you know that even the best models of today struggle with them. The typical approach upon seeing an invalid tool call in a run-of-the-mill harness (Claude Code, Codex, etc) is to add a tool result with an error and loop into the LLM again. This hides the problem from the user, but I do not like it for being terribly slow. I want my "AI" to be snappy and I hate making user stare at the "ruminating..." line.

Here are some tips that help:

1. Just accept the alternative format. If I have a parameter that should be a list (e.g. JsonArraySchema.builder().items(new JsonStringSchema()).build()) LLM often double-encodes it and produces a tool call with a json like this: {"field": "[\"a\", \"b\"]"}, which is almost correct, except for JSON list being extra-encoded into a string. Instead of giving it an error and waiting for another LLM-loop it is much faster to just try parsing the parameter one extra time and if that succeeds - make the user happy.

2. Avoid common prefixes on adjacent parameters. For example, avoid tool call schema with .addStringProperty("value").addStringProperty("valueHint"). Instead go with .addStringProperty("value").addStringProperty("hintForValue"). The reason is LLM being token completion machine and already outputing {"value tokens sees as equal candidates both " and Hint" completions. And you would not want to see a JSON like {"value": "a", "value": "hint for a"} - even though it is a valid JSON, good luck getting both the value and the hint from this one with all the major parsers.

3. Don't be shy of asking LLM to call the same tool N times on a single response. You may even suggest LLM do that in the tool description. It is better to have a list of tool calls than to have a tool call with a list. First of all, see the first hint on why LLM struggles with those on occasion. Second of all, you have an option to ignore invalid calls when you have valid ones to work with if that fits your tool's logic.

4. If there was an invalid call and you had to resort to the "error and loop again" approach, you can delete both the troubled call and the error from the chat history upon seeing a valid tool call. You save a bit on tokens, but more importantly invalid tool calls in the history make it more likely for LLM to screw other tool calls. You could see it as LLM overcompensating for its earlier mistakes. So, it is better to just pretend those never happened.

5. Metrics and alerts are crucial. Logs are vital (beware of capturing PII and other sensitive info in prod). You'd be surprised at the kinds of "creative" tool calls payloads your LLM will produce. Especially when you switch models.

Remember: LLM is just a tech and we understand exactly how it works.