Building Internet-scale Distributed Systems for Fun and Profit
Today there are many examples of successfully deployed Internet-scale distributed systems, such as search engines, content delivery networks and peer-to-peer applications. These systems consist of thousands of computers world-wide and must provide a continuous reliable service to millions of users. To achieve this, they must be scalable, fault-tolerant and yet resource-efficient. As one can imagine, designing, building and deploying Internet-scale applications remains a challenging task.
In this talk, I give an overview of our research on the design and engineering of scalable and robust global applications. Our research projects investigate different aspects of this challenge and I present three solutions to problems faced at different layers. At the network layer, our work explores how to improve best-effort Internet communication by permitting applications to take detour routing paths with better quality-of-service properties. At the application layer, we develop new abstractions for designing scalable distributed algorithms. I discuss how geometric relationships between Internet hosts can be used to build a scalable content delivery network. Finally, in terms of data management, we examine new processing models that make the degradation of data quality due to failure explicit in large-scale systems. Such models can be used to implement a dependable Internet-scale service for processing sensor data.