Model Validation

The topic of how to validate a simulation model has been well-studied and oft written about in academic circles. We’ll defer to existing texts on that topic as it is beyond the scope of a simple blog entry. (Averill and Law, in their excellent general simulation text, for example, have an excellent discussion of the topic).

Here are a few of the tips that we rely on for our simulation projects.

1. Set Realistic Expectations. First and foremost, spare yourself a few headaches and begin the validation process with realistic expectations. Real systems are typically messy systems. Often, the best you can do is determine the likely range of important benchmarks (such as average wait time, service levels, server utilization, etc.). You cannot reasonably expect to predict all benchmarks exactly. Why? Because the real system doesn’t work that way either; two days with identical summary level inputs will produce slightly different outputs. To better understand this issue, think back to the performance of your real system. Better yet, if you have it handy, look at some actual past performance data. If you look carefully, you will be able to find examples of days with nearly identical volumes, staffing and process times, yet different performance metrics. The reason is simple – it is variation that causes all of the trouble in the system, not the averages. If you are running close to threshold on server utilization, and happen to randomly get a few long process times in a row, it can cause you to run behind for hours. If, on the other hand, those process times are spread out, you will never get behind, and will therefore have better service levels. Because a simulation is creating demand and process times according to statistical distributions, you might get a simulated day with a run of closely spaced long process times or you may get a simulated day with nicely spaced short duration process times - just as in reality. As a result, sometimes you’ll get a service level that is almost exactly the actual observed, sometimes only close, but both cases will be an accurate prediction of reality because both situations can happen in reality.  Does this mean that it isn’t worthwhile to create a simulation model in the first place? No, quite the contrary. This degree of ‘messiness’, this amount of sheer complexity, is all the more reason to build a model. The purpose of a simulation model is to let you explore the full range of potential outputs that you can expect with a given set of inputs, and moreover, how potential strategy changes would impact that range of outputs.
2. Do an approximate ‘sniff test’. Does it look about right? Have a quick look at summary benchmarks for all demand types. Which types are off, if any? In which direction?
3. If the numbers don’t seem reasonable, start troubleshooting. There’s always a reason. Usually, it is a simple matter of correcting or refining incorrect input data. Sometimes it is because there is an important element that has not yet been modeled. For example, are customers prioritized in your system? Have you modelled them that way? Is there any volume sharing that you forgot to specify? Does a particular group of servers also take a customer type that hasn’t yet been included? Finally, don’t forget to consider the possibility that the unexpected results you are seeing are non-intuitive – but correct. Read on for more detailed troubleshooting tips.
4. Start validating the simplest parts first. If you have a model with many interdependencies, there could be a lot of moving parts to nail down in the validation process. It can be helpful to get the simpler parts working well first before tackling the more complicated sections.
5. Double-check the major parameters – demand volumes, staffing levels, process times. If those aren’t right, there’s no point in fine tuning other elements.
6. Review intra-day patterns. If your overall daily performance metrics are off, are you at least getting your high and low performance periods at about the right times of the day? This can be a clue to problems with your demand volume and staff period allocation percentages.
7. With the obvious checks out of the way, start looking into details. For example, could your processing times have wider variation than you’ve entered? Remember it is variation that causes all of the trouble in a system.
8. Review your operating parameters. If your system involves any complex routing, did you check the simulation results to ensure that your customers are being sent to the right locations in the model.
9. Be patient! Having a reasonably-validated model will be incredibly useful, and it is worth the effort to fine tune it. One you’ve got a solid baseline you can run any number of interesting scenarios.