A friend of mine, who studies Chinese history, recently told me a story about Chinese generals and their wars. Apparently the greatness and power of a general depended not on the victories he had, but on the number of solders he could command. Which lead to the Chinese style of war - where the armies move around each other a lot, but avoid any actual fighting at any cost. That was because no matter the outcome, both generals would loose some solders in a fight and their influence would diminish. Because of that they mastered those forms of wars where the outcome is decided without a single engagement.
Something like this is happening in the DevOps world right now. But instead of soldiers, we have servers.
I was chatting to one DevOps engineer, who used Kafka and claimed to get a decent performance out of it (which is impossible). Over the course of conversation it turns out that his organization runs hundreds and hundreds of Kafka servers. Considering this, their per-server traffic was nothing out of usual and could've been handled by anything - from MySQL to Cassandra. The conversation went on and I realized that having 400+ Kafka servers is not seen by that DevOps engineer as a problem. He takes a great deal of pride in it. If I were to come up with a solution that would've handled his usecase on 2 or 3 servers (just for HA) he would've rejected such an offer. He wants his big cluster of terrible applications.
I think this is an outcome of organization pushing the decision-making down the chain of command. It is great that a team can choose whatever technology it likes. But the developer making a decision needs to advance his career. And what would she prefer on her resume: "400 Kafka" or "3 RabbitMQ" servers? Who would you hire to manage your swarm in the cloud, a person who has managed thousands of docker containers already or an engineer who use to know the IP address of each box she runs?
We now have a new form of mastery, not dissimilar from the old Chinese stratagems, where some people write terribly inefficient code and wrap it in tons of marketing BS, while others pretending to eat it by the spoonfull and jumping on the bandwagon only ever happy to see their clusters ballooning beyond reason.
I wonder if studying "The Art of War" by Sun Tzu should be part of a modern CS course.