counter stats

Why Designers Need to Worry about Open vs. Closed

 

People

Adam Wierman
Bianca Schroeder
Mor Harchol-Balter

Figure 1.  An illustration of the structures of the open and closed system models.
 
Motivation

Workload generators are an invaluable resource when evaluating the performance of proposed system designs. Most workload generators for web server and database workloads assume a closed system model, where new job arrivals are only triggered by job completions (followed by a think time). In contrast, whenever a trace is used to generate the workload, an open system model is implicitly assumed, i.e. new jobs arrive independently of job completions. Though every systems researcher is well aware of the importance of setting up one’s experiment so that the system being modeled is “accurately represented;” system designers generally pay little attention to whether a workload generator is closed or open.

Figure 2.  An illustration in the difference in performance between the open and closed system models under FCFS scheduling.  While variability is disastrous for FCFS in open system models, the effect under closed systems is much smaller. 


Results & Impact

In a paper in NSDI 2006, we used implementation and simulation experiments in order to illustrate that there is a vast difference in behavior between the open and closed models in real-world settings. Not only is the measured response time different under the two system models, but the two systems respond fundamentally differently to variations in parameters and scheduling policies, e.g. the impact of scheduling is far more dramatic in the open model than in the closed model. Further, these differences are present across a range of applications, including static and dynamic web servers, a database backend, and an auctioning web site. The differences between the open and closed models motivate the need for system designers to be able to determine how to choose if an open or closed model is appropriate for evaluating new designs. Thus, we also develop a simple recipe for how to make this choice.


Publications