Why Designers Need to Worry about Open vs. Closed
People
Adam Wierman
Bianca Schroeder
Mor Harchol-Balter
| |
| Figure 1. An illustration of the structures of the open and closed system models. |
Motivation
Workload generators are an invaluable resource when evaluating the performance of proposed system designs. Most workload generators for web server and database workloads assume a closed system model, where new job arrivals are only triggered by job completions (followed by a think time). In contrast, whenever a trace is used to generate the workload, an open system model is implicitly assumed, i.e. new jobs arrive independently of job completions. Though every systems researcher is well aware of the importance of setting up one’s experiment so that the system being modeled is “accurately represented;” system designers generally pay little attention to whether a workload generator is closed or open.
| |
| Figure 2. An illustration in the difference in performance between the open and closed system models under FCFS scheduling. While variability is disastrous for FCFS in open system models, the effect under closed systems is much smaller. |
Results & Impact
In a paper in NSDI 2006, we used implementation and simulation experiments in order to illustrate that there is a vast difference in behavior between the open and closed models in real-world settings. Not only is the measured response time different under the two system models, but the two systems respond fundamentally differently to variations in parameters and scheduling policies, e.g. the impact of scheduling is far more dramatic in the open model than in the closed model. Further, these differences are present across a range of applications, including static and dynamic web servers, a database backend, and an auctioning web site. The differences between the open and closed models motivate the need for system designers to be able to determine how to choose if an open or closed model is appropriate for evaluating new designs. Thus, we also develop a simple recipe for how to make this choice.
Publications
-
Co-recipient of the CMU SCS Distinguished Dissertation Award
Finalist receiving Honorable Mention for the INFORMS OR in Telecommunications Dissertation Award Ph.D. Thesis, 2007. -
Proceedings of NSDI, 2006.Workload generators may be classified as based on a closed system model, where new job arrivals are only triggered by job completions (followed by think time), or an open system model, where new jobs arrive independently of job completions. In general, system designers pay little attention to whether a workload generator is closed or open. Using a combination of implementation and simulation experiments involving static and dynamic workloads, this paper illustrates that there is a vast difference in behavior between these open and closed models in real-world settings. In addition, this work synthesizes many of the differences in behavior between the models into eight simple principles.