For many ears I cared about the bigger picture of Software. Supporting the technologies I like, highlighting the issues in the ones I despise. I was hoping that eventually my efforts will be part of a bigger push and will make everything better. That we have the power to change the world to the best.
It did not work.
The world of software is not getting better. It is getting worse and instead of improving atop of already solved problems we keep reinventing the wheel. And the real problem is that the new wheels we invent are not at all better.
You may wonder what was the final straw. It was this project.
It is a metrics collection library. The one where you'd want to track the number of 200 responses on your API
and you call something like metric("get.200").inc() every time it happens.
It all starts with the basics. Guess the underlying datatype used? If you guessed u64, which can be safely
and atomically incremented with a single LOCK XADD CPU instruction and you would be wrong. It is a
double precision floating point number (float64). Which uses exactly the same number of bits, but looses
precision way sooner.
So the inc() function actually calls inc(1d) and adds a double to a double. Non atomically.
This is a problem, you may think. But feat not, the wheel is not complete yet. In order to achieve atomicity
of the operation, the code re-casts the double as u64 and uses the CAS CPU operation to atomically update
it in memory.
That fixes the correctness, but we are still struggling with performance here. So it continues. Instead of all threads accessing the same 64-bit value, lets write a hashmap which would map each thread to its own instance of the counter. And to avoid the false sharing of it, we pad it with enough memory on each side. This will make sure each instance of the counter takes on entire CPU cacheline.
After all of that, the inc() performance is on par with the AtomicLong (backed by u64). Of course
our get operation (which we got for free previously) is way more complex now as it has to iterate over all
instances of the counter and add them up. And it is not atomic. But who cares. This wheel is good enough
already.
In the end we reinveted a wheel - a counter. It is all different, it takes up way more memory, it pollutes
the CPU cache, it is way slower (on get()) and... I have nothing to say anymore.
Here is the reason I am giving up. My capacity to fix things is way lower than the capacity of smart fresh graduates picked up by large corporations and never exposed to any of the existing solutions to break everything.
They are not stupid. They are smart. And determined. They come up with smart solution to the problems that were solved years ago. Decades ago. And the improved upon. They make junior level programming mistake (such as using double for a counter) and instead of listening up and fixing them, they blame the "old ways" and go out of their way to make their solution work. If anyone told me he wrote a double counter whose performance is comparable with the long long one out of the blue, I would be "that is cool, why do you need a double counter?"
I decided I do not need to worry about that anymore. The tech centre is in a bubble at the moment and all sort of crazy technologies is being written and paid for now. I am not angry anymore. I am not sad about the great software those google ad facebook hires could've written if they had a chance. They won a lottery of the sort.
I am willing to accept it and let it be. Good luck.