Abstraction is Kinda Beautiful

When I was a kid, my dad, who studied computer science, used to tell me he had trouble keeping up with all the new technology coming out and of course I didn't get what he meant. I'd just see him watching YouTube tutorials made by some guy with a thick Indian accent and think,

"That's a pretty calm life. You don't need to know much, you just learn it on the job."

Now that I'm doing my second internship and have been trying to keep up with the industry, I finally understand him. With my internship, tech twitter and a few developer community events, I've had the chance to keep up with the new AI models, new startups, what's hot, what's not, and in the process, I've come to realize how much I don't know, sometimes even about the most basic and trivial aspects of software engineering. Things that I never even thought about, had utmost intricate detail and the most minuscule performance optimization.

Lately I've been reading Designing Data-Intensive Applications (aka the boar book), which some people call the holy grail of software engineering. The premise is almost that this one book is all you really need to design software at the highest level, no formal education required. I'm halfway through, on chapter 7 of 12, and I've learned so much despite it being released nearly 10 years ago.

The problem however, is that in order for me to learn things properly and not forget, I need to apply the knowledge and build it... and that's basically impossible because it would take forever.

In a lot of my CS classes in school I had to build something that already existed, which is a little silly but the point was to understand how it works and you do that by actually going through the process of building it. I built an POP3 email server, QuickSort in assembly, a DNS router, rebalanced AVL trees and some search thing using the DICT protocol just to name a few.

Similarly, in DDIA, there's a lot of things to understand and build: B-trees, LSM trees, encoding, serialization, deserialization, write-ahead logs, replication, partitioning, fixed partitioning, dynamic partitioning. And guess what? Every database and service does it differently with different trade-offs and different use cases. By the time you learn all of it, you're already behind, there's newer, "better" way of doing things, with its own set of trade-offs to learn from scratch.

I'm only on chapter seven and I can't lie I've already forgotten some of what I read in chapters one and two even though I'm already building a project alongside the book. Catching up on all these things is hard and very time-consuming, never mind keeping up to date with all the latest technologies.

If you're not in tech, all this is amazing because it's all abstracted for you. If you're in tech and working on and building these systems, it doesn't feel good to have to also abstract this knowledge and not necessarily know exactly what's going on.

What I've realized is that most of software engineering is exactly this. A lot of the time, you kinda just have to accept the abstractions that already exist and let go of knowing how everything works under the hood. Sometimes it's worth digging in, the internal trade-offs and design decisions can genuinely matter for your use case, but you can't do that for everything. There's simply too much to fully understand and your job is to focus on cutting your side of the lawn and not the whole neighbourhood.

So that's my takeaway, halfway through the book: there's far more in software engineering than anyone can learn, so accept the abstractions and choose where to go deep. If you want to truly be a great engineer, your goal isn't to know everything, but instead is to understand one specific problem better than anyone else, and then abstract it away so no one else has to worry about it.

Because I'm unoriginal, here's a quote I searched up because of confirmation bias from the GOAT himself:

"The purpose of abstracting is not to be vague, but to create a new semantic level in which one can be absolutely precise."

~ Mr. Dijkstra