Guus Bosman

Scalable Internet Architectures provides a good introduction to scalability and performance engineering for large internet applications. The book has useful high-level discussions and interesting real-world insight but could have benefited from better editing. The book would have been even stronger with more focus on theoretical aspects -- which the author explains well -- and less emphasis on specific tools and code-snippets. Overall, even though the book is from 2006 it is worth a read, especially for engineers new to the field.

The author of the book, Theo Schlossnagle, is principal at a consulting company and his real-world experience with scalability and other aspects of large-scale engineering clearly shows in the book. He excels at outlining the challenges and possible solutions on a high-level, giving the reader a good background to make informed choices.

Still relevant 6 years later

The book was written in 2006 but most of the material is still relevant; the architectures and concepts that are described are still valid today. The code examples and the recurring emphasis on the author's favorite tools, Spread and Whackamole, are less useful for a book on this level.

The book is almost exclusively focused on the ‘back-end’ server architecture and doesn’t talk much about ‘front-end’ items except for mentioning that cookies make an excellent 'super local' cache for web applications. Most of the development in the field since 2006 has been client-side, with the possible exception of experimental things like SPDY, Google’s new protocol. It would be interesting to read more about the impact of increased Ajax use and streaming partial page-rending such as Facebook’s on the back-end architecture.

"Developers have no qualms about pushing code live..."

The excellent first three chapters introduce the field of scalability and performance engineering and explain the challenges that occur once an internet application reaches a large scale. The classic tension between flexibility and stability is summarized succinctly, where "developers" are really a proxy for the demands of the business to deal with a changing internal and external world:

"In my experience, developers have no qualms about pushing code live to satisfy urgent business needs without regard to the fact that it may capsize an entire production environment at the most inopportune time. [...] My assumption is that a developer feels that refusing to meet a demand from the business side is more likely to result in termination than the huge finger-pointing that will ensue post-launch".

For me this is a very familiar discussion -- part of being an engineering manager is to make these types of judgment calls: when will we push back, when will we take risk, what is the risk/benefit trade-off.

High-level problems and solutions

The author is at his best when explaining high-level problems and their possible solutions. The author explains the need for horizontal scaling and introduces various techniques that make this possible. He goes into advanced topics but doesn’t forget to cover the basics. For example, there is an excellent walk-through on the performance gains from serving static content vs dynamic content. This is a good description for people new to the field and it is well illustrated, including the slowness of the initial TCP handshake and the dramatic difference in memory footprint of Apache 'bare-bones' versus Apache with Perl or PHP compiled in.

An interesting piece of real-hand knowledge is the author's claim that on web servers (in clusters > 3 servers) one can expect up to 70% resource utilization. That's a good benchmark to have.

I also liked the explanation on caching semantics. The author illustrates the problems of having shared, non-scalable resources (such as databases) and explains how introducing caches can provide the ability to create a more scalable architecture. The sample PHP code is helpful in explaining caching and two-tier execution. The book discusses transparent caches, look-aside caches and distributed caches.

The descriptions of the various types of database replication were good to – master-master, master-slave, and even cross-vendor database replication, where an expensive Oracle master is used in combination with open source PostgreSQL slaves. The latter definitely has its pros and cons and would introduce quite a bit of extra maintenance, but author is right that is opens the mind to think about possibilities like that.

Peer-to-peer

Throughout the book Schlossnagle discusses peer-to-peer high availability software. The tools Spread and Whackamole are being pushed quite a lot; they are part of a project the author worked on at John Hopkins University. This peer-to-peer concept brings in an interesting perspective – for me looking at these solutions makes sense, although it is not something I have worked with yet. However, the author gets too specific in the last chapters of the book, and instead of high-level discussions he delves into the specifics of using Spread for logging, which is a missed opportunity to really discuss the various architectures in that area.

The book is clearly written by someone who has been in the trenches, although the tone is a little cynical at times: "And yes, 1 fault tolerant and N-1 fault tolerant are the same with two machines, but trying to make that argument is good way to look stupid". The book could have benefited from a stronger editor who would have kept those things in check. The book is woolly, especially chapters 4 and 5, and could have been a bit shorter.

Recommended

The book provides a good high-level discussion of concepts such as various caching models, fail-over and scalability, combined with real-world experiences of the author. The book would have been stronger if it had had a better editor but is worth a read, especially for engineers new to the field of large scale websites.

There are very few books out there that discuss all these aspects on a high level. Perhaps a second edition can fix some of the minor shortcomings, but the book is recommended.

More info: http://scalableinternetarchitectures.com