When people talk about infrastructure of large internet services, the term distributed systems comes up pretty fast. It's basically the umbrella term for "the backend stuff that have to keep running even when something inevitably breaks". When I first herd about about distributed systems it all sounded like a black magic. You know, all of those complex protocols after node crashes, data formats that don't corrupt everything when a network switch glitches, or time that sometimes goes backwards.
The funny part is that the core ideas behind this aren't new or mysterious. Most of the key ideas were developed in the 1970s, mostly in academic and research labs. Cool tech, but not something most people ever had to think about. One example is ARPANET (baby internet) was already a distributed system, just tiny. So yeah, distributed computing has basically existed as long as computing has been a thing.
Catch: Why do we need distributed systems anyway? 🤔
What web brought though is that around 2000s the consumer internet services came up and suddenly needed to handle millions of users. At that point, scaling out across many machines stopped being a research topic and became the way to build real-world systems. Companies that hit scale early had to turn decades of ideas into running code and infrastructure. Over time, many of those ideas have been packaged into open source software and cloud service abstractions.
That's great for making us ship fast, but at time same time, I feel lost in those abstractions and marketing terms.
What is Getrafty?
Getrafty is just a regular engineer's journey to demystify distributed systems by building one from scratch, layer by layer.
Open source, exploratory, and fun.
If that is something you feel into, welcome aboard.
What's Inside?
We're going to build a tiny distributed file system - BarkFS. Layer by layer, starting with the absolute basic such as operating system primitives all the way down to consensus algorithms 🍿.
Contributing
The project is open to anyone. You are free to just walk through exercises or you can join getrafty project and contribute tasks, docs, or suggest new ideas ❤️.