Web Architecture is moving back in time because of AI
For the past 3 years I've been responsible for architecting, writing and maintaining a Serverless backed for a SaaS solution. It is not always trivial, there are queues and notifications. But it works, it is fast, it scales well and our AWS bill is not killing us.
At the core of it is a GraalVM-compiled Micronaut application, that takes less than a millisecond to cold boot. This involves resolving the environment variables, bunch of placeholders and even testing the liveliness of a DB connection with a quick check. To be frank, that DB connection check is what takes the most of its cold boot time.
But I am about to start migrating this to an ECS or EC2 instances. Not all at first, but there will be an always-on server to handle persistent connections soon. And the reason is: AI.
Despite its respectable age, Serverless is still a niche way to run a web service. It was not well represented in the LLM training materials. As a result, any example you see of an AI-generated code is never compatible with being serverless. What's worse, any AI-related networking protocol, such as MCP, is also not compatible with Serverless implementation. Well, MCP authors hallucinated an OAuth version for its authentication, so it is not like this incompatibility is the biggest problem with MCP. But it is just an example. Anything that is AI-generated or is expected to work with AI is hostile to anything Serverless.
Another example is AWS Bedrock. There are many ways to submit a query to a Bedrock-hosted LLM, but all of them involve waiting for a response to arrive back over the same TCP connection that was used to send the request. It may take LLM an hour to generate a response and you have to maintain that TCP connection for an hour. That is pretty hostile to anything Serverless. If something is alive for 15 seconds it means something is not right.
To be fair, there is a "batch" mode that allows a programmer to submit a request and not wait for the response on the same connection. The result will arrive into a dedicated S3 bucket. That'd be awesome for a Serveless
application, but that mode has a limitaiton: the minimal size of a batch is 100 requests. What am I supposed to do if over a couple of seconds I did not get enough user requests? Fill the rest of the batch with garbage? I'd be paying for
it dearly.
The architecture of the SaaS application I have been building for the past three years is pretty advanced and is pretty good. It may not be the architecture of the future, but it is much better than what we used to have a decade ago. I do not have a fleet of servers or instances to manage and orchestrate. The monitoring is a breeze. Uptime is awesome. Latency is low and stable. It costs very little when there is no traffic and the cost scales reasonably when there is a burst.
But the way the modern AI is being written by people I can not influence forces our SaaS architecture back to 2005.
Many people talk about "jagged frontier" of AI. Not many realize that it is jagged not because AI pushed things forward unevenly, but because it pushed some things forward and other things back.