Or, happy new year (2⁶⁻¹)(2⁶−1).

We just wrapped up a long-running big data project. While I didn’t find the project itself particularly interesting, I learned quite a bit of interesting stuff doing it.

On the skills side, I learned Go, learned how to navigate AWS, and gained a good understanding of how cluster data stores work.

But probably the most important takeaway from last year is summarized neatly in this blog post from 2010, recently sent to me by a coworker. In a nutshell, there are a lot of big data systems that are far larger and more complicated than they need to be for the data that they are designed to process.