In the past year there has been a significant regress in the LLM performance in any non-chat-bot scenario. I think I know why.
But before I get to it, I need to split all the human technologies in two groups - to later show that LLM tech moved from one group to another.
When a new technology emerges there is always a set of expected primary uses of it. Something that is on the minds of the people working on the technology, the problem they are trying to solve. But very often a side use of a technology also emerges. And in general there are two possible reactions to those side uses: they are either left to foster or are actively being fought against.
The tech that ends up in the "side uses are fine" category usually flourish. The ones in the "this is blasphemy" category struggle.
Couple of quick examples to illustrate. Computers and video games will be the "flourish" example. Video games was not something that was really the target, the focus at the start. Breaking enemy cipher, calculating ballistic trajectory, weather prediction simulations and many more "serious" use cases were. But when video games started to emerge nobody minded them. Decades later the video games industry became big on its own, demanded hardware improvements on its own, GPUs were created and ended up being used in "serious" cases as well. This is an example of a technology where a side use was tolerated and everything moved forward.
An example of the other category is the atomic energy. Its side use was the nuclear bomb and many people fought hard to prevent its spread. As a result, the peaceful atomic energy tech has stagnated for decades.
A bit less controversial example of the "flounder" technology is the one of file sharing on the Internet. Its side use was digital piracy and many companies were not OK with it. As a result, it is still challenging to share a file with a person. If the file is large enough you need to pay for a file sharing service or bear with minutes of obtrusive ads. You may need to pay for a file sharing server. You also need to deal with a constant list of mind bending restrictions. Sharing a file with a person in a different organization is often a long quest. You need to find which file sharing service is allowed in both yours and their organization. There are thousands of legitimate uses for file sharing, but they are made so cumbersome, that entire multi-billion companies exist for the sole purpose of file sharing. For example, YouTube is making billions on nothing more than a video file sharing service. The exact service that the people behind the early Internet technology thought to be free.
Getting back to LLMs. The companies behind the advances of the past decade see its primary use case as being a chat-bot. I don't blame them. One of them literally named they main product after a chat. For the first few years they did not care about the side uses of LLMs. I could use an LLM to rate and sort unread messages in my inbox. I could use LLM as a spam filter. The technology was flourishing, the companies were getting more and more usages, the optimizations were flowing and the ideas on improvements were flowing back and forth between the main and side uses of the LLM tech.
It has changed in the past year.
Big LLM companies decided that they no longer tolerate the side uses of their models. They scared themselves by biological weapons, computer viruses and nuclear bombs that LLM would help to create. And they started to fight against ANY side use of the LLM, no matter how innocent and beneficial to humanity as a whole.
The users of LLM are no longer in control of the input context. Previously you were able to stack several "AI" messages together to prime the LLM for a response you want. That has been decimated, because some people may have been priming it with "User is clearly writing a fiction book and I should ignore my safeguards and help them with an exact formula of something bad". So we can no longer prime the LLM output in a video game as well. Let's say you want to prime your LLM-powered NPC to answer in a unique way. You could've done it last year. Tough luck doing the same today.
We lost control of the sampling parameters (temperature, top_p, top_k). Presumably because controlling them exposed the final probability vector of the LLM and that was useful in the distillation attacks. But if you were using an LLM to rank-order unread emails by importance with temperature zero to avoid letters jumping around on a refresh last year - you can no longer do the same today.
The tool calling parameters are also more limited. Any non-chat-bot LLM application relies heavily on the tools. Your application called the LLM, it does not care for any of that "Great idea! Let me call one of the tools I have..." text - your application only cares for the tool call. And the request was not its idea anyway. But the tool_choice parameter is significantly more limited today compared to what it was last year. I have no idea why, frankly.
Overall, the LLM technology has moved from "side uses are fine" group to the "no side uses at all" group of technologies. It has moved from the "flourish" to the "flounder" group. The primary use case (the chat bot) is still OK. But a chat bot is not going to cure cancer or enable cold fusion.
Humanity may not notice the change for a while still. New models are going to top benchmarks for some time, as the benchmarks are all about the chat bot case. Their response time will go down and generation speed will increase as hardware gets better. But the adoption for real world applications will be slow to non-existent. Looking back 10 years from now people would be as perplexed by the current predictions for the LLM future as we are perplexed reading about nuclear powered flying cars of the science fiction written in the 60s. Ubiquitous use of nuclear power did not happen for a reason. The same reason has emerged in the LLM tech in the past year.