With sessions running on eight parallel tracks, it is totally hopeless to give a comprehensive overview of the QCon conference held in New York between June 11 and 13, so I'm not even going to try. Instead, I'll just highlight a few sessions I especially enjoyed and thought it would worth sharing.
Given Kinja's current state and the transition our tech stack goes through, I was really looking forward to see talks about software architecture, specifically microservices. Moving away from a monolithic system is tough enough and we could use as many real world examples as we can find.
First off, Foursquare's Jon Hoffman, whose company is also in the the middle of reorganizing their services gave us a rundown on how Foursquare gradually replaced their a monolithic PHP application with a more modular Scala based platform (the same programming language that powers Kinja).
As it can be expected it was a huge undertaking that took several years and is still ongoing. Also, one of the most obvious trade-offs of having several small services is the need to pay a lot more attention to details that were not really a concern before. As the complexity of the infrastructure increases, it also becomes a more difficult task to deploy, run and monitor different parts of the system.
To solve this problem most companies use a mix of third party tools (Zookeeper and Kafka kept popping up among others) and roll their own solutions on top of those tools, with a definite emphasis on these in-house solutions. Even basic tasks like debugging and logging can become a concern that the developers have to spend time dealing with. I know it's not exactly original or even a recent development. Still, it was interesting to see how much effort goes into building up parts of the platform that's historically been as far from the product development as possible therefore being neglected in the day to day operation of most startups. Even though, beyond a certain size, it is obviously crucial, not just for the success of the product, but simply to keep it above water.
This became even more apparent on Yoni Goldberg's talk from Gilt (slides). They went all in on the microservices approachby running 400 services in production. Hearing that, it wasn't surprising the least to learn that they not only spend a lot of their time figuring out monitoring and deployment, but even creating a development environment that would work for all their developers (running every service on every single machine would be quite a challenge, and with a plethora of data storages it's borderline impossible), or, for example, the right visualization tools to track dependencies inside the system.
The next day, Eric Evans talked about a different aspect of evolving the architecture of your codebase, how you should embrace imperfections. My takeaway was that instead of trying to find an all-encompassing model that neatly describes everything, it's better to just embrace, or at least accept, the imperfections of your design (future, current and legacy) and rather focus on the procedure of moving from one design to another. So on one hand, don't try to resolve all problems at once, and on the other hand you should never try to apply those solutions, immediately and everywhere. Just focus on gradually cleaning legacy code up, while keep things running.
Another often overlooked point that kept coming up is that it's relatively easy to build a system with a current team structure in mind, and simply ignore the fact the code usually sticks around a lot longer than people do. Even if the people stay, teams are created and folded into each other all the time. So instead of trying to create an architecture that's in perfect sync with the team that's currently working on it, it's actually way more important to figure out the procedures to transfer ownership of that code to someone else whenever it's necessary.