How do you know if the results of a simulation analysis are accurate and reliable?

This is an important and fundamental question. If a simulation produces invalid results, yet the client accepts and acts upon them, then they’re heading into a world of pain. Just as bad is the situation in which the results are valid, but are distrusted by the client and so are disregarded. In both cases, the outcomes can be catastrophic.

Clearly, the accuracy and reliability of a simulation model and the client’s trust in its results are key factors. As a simulation expert, it’s my job to realize both—if I achieve just one and not the other, or neither, then my client and I are both going to be in trouble. So how do I address this?

UPDATE 4/1/2016: I have clarified and updated some elements of this guide.

Minting Money With Simulation

Before I get started, though, it’s necessary to point out that not everyone using simulation has formal training or qualifications in the discipline. (There is, alas, no accreditation body for our profession.) While it seems that just about anyone can develop a simulation model these days—a claim promoted by many simulation product vendors—there is still a right way and a wrong way to do a simulation study, and, unfortunately, it appears that there are still too many of the latter.

While it’s maybe pushing things a little to compare simulation to brain surgery—it’s unlikely that anyone will die as a result of a failed simulation project—there are some things the two professions have in common. For instance, you wouldn’t expect to buy a box of user-friendly surgical tools, take a two-day training course and expect to be able to perform a craniotomy on a live patient with any guarantee of success—yet many seem to have a similar expectation when purchasing an easy-to-use simulation software product. Similarly, you probably don’t want to shop around for the cheapest brain surgeon and nor should you use that criteria to select your simulation expert. The costs of a failed simulation project, of both the mistakes and the missed opportunities, can easily run to millions of dollars…

Part of my motivation for writing this article is to make it abundantly clear that simulation, when executed correctly, is an accurate, reliable and credible tool that will save you millions of dollars in capital and operating expenditure. If you have ever been involved in a simulation study that turned out badly, I hope that reading this article will help you determine where things went wrong and encourage you to give it another try; you’ll be glad you did!

Setting A Course for Success

The first step I take in any simulation study is to listen carefully to my client’s requirements, clarify them, then craft a set of objectives—with which the client formally agrees—that the simulation study must satisfy to address those requirements. Without carefully defined objectives, the simulation study is doomed to failure. (This is true of just about any project, not just a simulation study, so it’s no doubt an issue that you’re very familiar with; I’ll simply state the fact and not dwell further on the downfalls of failing to define appropriate objectives.)

These objectives form the foundation of the subsequent simulation study.

What Should Be Measured?

Since they must be measurable, each objective will suggest one or more statistics, sometimes also referred to as metrics, outputs or responses—and which may also be classified as key performance indicators (KPIs)—that the simulation model must produce as it runs.

For example, if a simulation study has the objective “Determine whether the new facility can achieve its target net production rate of 80 jobs/hour”, then it must clearly measure net production statistics with a high degree of accuracy.

So, given a set of agreed objectives, I next compile a list of required statistics.

Defining the Conceptual Model

It now becomes time to consider the nature of the model that will produce these statistics. I’ll start by interviewing the client’s process experts, including manufacturing engineers, controls engineers, planning & logistics functions, material handling system suppliers, etc. in order to build a conceptual model of the system. (A conceptual model defines the characteristics and behavior of the computer model that will be developed later in the project.)

Interestingly, many of the benefits of undertaking a simulation study come to light during construction of this model—before a computer model has been built and before any simulation runs have taken place—simply by subjecting the system design to intense scrutiny.

At the same time, any model is a simplification of the system whose behavior it attempts to replicate. The statistician, Dr. George E. P. Box, once famously declared “essentially, all models are wrong, but some are useful”—a statement that is accurate, but which I think is also a little misleading since there are varying degrees of wrongness and since usefulness depends upon what you’re attempting to do with the model. It’s essential, therefore, that the model is designed with the job in mind.

Controlling Model Complexity

The model’s complexity is defined by two factors: its scope and its level-of-detail. The scope specifies the set of system features that will be included in the model, while the level-of-detail describes the representations of the included features and their interactions.

The model needs to have a large enough scope and be sufficiently detailed to ensure the accuracy of the generated statistics, while avoiding adding unnecessary features and details that do nothing to improve statistical accuracy, but which would otherwise lengthen the study dramatically. I like to bear in mind advice attributable to Albert Einstein: “Everything should be made as simple as possible, but not simpler”. When in doubt, I typically err on the side of including additional scope or detail.

The key point is that the model’s complexity is dictated by the required statistics and their necessary accuracy, which in turn are dictated by the study’s objectives. Given a different set of objectives, I’ll construct a different model.

State Assumptions

Whenever I remove features from the scope or simplify how included features are modeled, I’ll state the rationale for doing so as a modeling assumption. Clearly, the construction of the conceptual model is a subjective process—best undertaken by an expert, particularly if the associated system is high-value or mission critical. I need to be aware of this subjectivity and so I do all I can to check the validity of these assumptions—and the resulting conceptual model—as the project progresses.

Agree Specification

Once the modeling has been completed, I’ll document the resulting conceptual model of the system in a specification, and agree its content with the client—making any required corrections and modifications—before proceeding.

That Sounds Boring—I Want To Develop Models!

Too many simulation practitioners simply dive into building simulation models on their computer without taking any of the steps outlined above.

In particular, there are many whose conceptual models are based more upon the out-of-the-box capabilities of their simulation software than on the nature of the problem at hand. After all, if all you have is a hammer, everything looks like a nail. I could blog at length on that topic alone!

If you have failed to follow these steps, then your project has pretty much already failed—primarily because the model being built bears no relation to what is required of it. What is the model for? Has the design been scrutinized? Have the system’s features been analyzed? Is the model detailed enough? Are the statistics produced relevant? Who knows! Such models typically would not likely be classified as useful by Dr. Box…

Developing the Computer Model

Only once the specification of the conceptual model has been approved is it time to start converting the conceptual model into a computer simulation model—a process termed model development.

I’ll develop the model iteratively, with features added (and thoroughly tested—see the next section) incrementally. However, development of the model is not the primary focus of this article, so I’ll constrain the discussion to making a couple of points:

• I develop simulation models in a defensive manner. For example, if my model code detects an error, then the model will complain about it loudly then fail, providing as many details as possible. This forces me to address the problem so that the model can run to completion. If instead I quietly ignore such problems, and simply paper over the cracks, then I’ll compromise the accuracy and reliability of the model.
• Wherever possible, I limit the interactions between different parts of my models to small, simple, well-documented interfaces. This simplifies the design of the computer model and makes it easier to understand and maintain—and, therefore, less buggy. If any part of a model can access and interfere with every other part, then I’ll end up with a complex ball of spaghetti code that is difficult to understand, time-consuming to work with and, as a result, very buggy. Simulation products that support object-oriented modeling features (in particular, a feature called encapsulation) can be extremely useful in this regard. If a team of people is working on the same simulation, this approach is even more valuable.

Killing The Bugs…

Translating a conceptual model specification into a computer simulation model can result in a lot of errors, commonly known as bugs. No-one is immune from this. Even the very best simulation modelers unintentionally put large numbers of bugs into their models and I am certainly no exception.

The solution is to perform continual verification during model development so that the bugs are quickly identified and fixed. Verification is the process of ensuring that the simulation’s execution matches the specification of the conceptual model that it is intended to implement. If done correctly, verification eliminates every bug affecting the simulation’s accuracy.

Assume The Model Is Buggy, Verify Operation

I have observed that many simulation practitioners do not implement a formal verification process. Instead, many work on the assumption that their models run bug free and only take action when the impact of a bug becomes obvious. So, here’s a simple thought experiment: if you do not verify the operation of a simulation model, how accurate and reliable would you expect the output of that simulation model to be?

Even when verification is routinely performed, more often than not it takes the form of manually watching an animation of a recently-developed portion of the model and mentally comparing its apparent behavior to that expected by the modeler; if there’s a difference, the bug is tracked down and fixed. But there are problems here too. Many models can look right even when they’re wrong. Consequently, these latent bugs can spring into life when subsequent model development exposes them. The process of verifying the entire model after every change appears to be such a chore than many—if not most—practitioners subconsciously choose not to bother. Guess what? The result is buggy, unreliable simulation models.

A far more defensive approach is required: assume that the model contains bugs until we’re satisfied that it is operating correctly.

From a philosophical perspective, it is not possible to prove that a simulation model is bug free—we can only prove that specific bugs exist. This is akin to the scientific method, in which theories can be disproven, but can never be proven; our faith in a theory increases as it survives attempts to disprove it. In the same way, our faith that a model is accurate and reliable increases as the model passes more and more tests designed to verify its operation. However, we can never demonstrate that it has no bugs.

So how should we proceed?

A good first step is to draw up a list of acceptance criteria that capture the expected behavior of the model, check the accuracy of its statistics, etc. We must then verify that the model satisfies those criteria.

Track Known Bugs

If a bug is found (whether through failed acceptance tests or simply through use), it is imperative that those bugs are logged. There are plenty of bug tracking software products that can be used for this purpose.

For each bug that is found, ensure that there are acceptance criteria that expose it, and add new criteria as required. When the model passes all such criteria, the bug can be declared as fixed. Furthermore, these criteria ensure that the bug does not come back to life.

Failing to track bugs is a great way to undermine the success of any simulation study!

Automated Verification: The Power of Unit Testing

So we should now have a long list of acceptance criteria. Do we have to verify each one every time we make a change to the simulation model? Yes, of course we should!

How do we do that? I base my approach to verification upon the software engineering practice of unit testing. In the context of simulation model development, unit testing consists of creating scripts that test the behavior of the simulation under different conditions—often at a very basic level of operation—and executing all such scripts automatically after each change (which takes mere seconds on a modern PC). In essence, this involves expressing our acceptance criteria as unit tests.

For example, if a process within the simulation is supposed to take 5 minutes to complete, then one of my acceptance criteria should be that this process takes 5 minutes. To verify this, I write a unit test that measures how long the process takes and checks that its duration is 5 minutes. Not 2 minutes, not 10 minutes, not 4.9 minutes or 5.001 minutes.

Similarly, if the model needs to output mean work-in-progress (WIP) for a process, and I have an acceptance criteria that requires this statistic to be accurate under all circumstances, then I will write a set of unit tests (also known as a test fixture) that populates the process with known jobs at specific times and then verifies that the reported mean WIP is the value I know to expect, no more, no less.

A large simulation model may have thousands of such tests.

Does it take time to write these tests? Sure! But it takes a lot less time to write these tests than it does to manually test the model, or to observe, track down and kill bugs in the traditional way.

Simulation Software & Automated Verification

I find it very interesting to note that, at the time of writing, there is not a single simulation software product that supports automated model verification testing out-of-the-box. Or, at least, none that I am aware of.

The situation is different if simulations are written in a high-level programming language; most modern languages have unit testing frameworks, with varying degrees of sophistication, freely available. (For example, simulations written in Java can be verified during development with the assistance of JUnit.) Most simulation modelers, however, do not use high-level programming languages. Still, I have found that less sophisticated unit-testing can still be performed in products such as AutoMod, Demo3D, Plant Simulation, Witness, etc. (I once developed a full-featured unit-testing framework for SLX, but that’s primarily because SLX is more of a programming language than an application.)

Validating & Approving the Simulation Model

So, eventually, I end up with a simulation model that does what it is supposed to do and implements the specified conceptual model. But there is one more step that I need to undertake before I can start to use the simulation for its intended purpose: I need to validate the conceptual model. The conceptual model, if you recall, was developed using a subjective process that stated a lot of assumptions. However, as it has not been possible to execute the conceptual model until now, there has been no way of knowing whether that model is valid or not. Once again, I take the defensive position that the model is assumed to be invalid until I have taken steps to determine otherwise.

Validation, which is often confused with verification, is the process of checking that the conceptual model is an accurate representation of the intended system, taking into account the model’s objectives. That latter point is important: a simulation model that is perfectly valid for one set of objectives may be lacking in required detail and accuracy for a different set of objectives. One size does not fit all.

Models are typically easier to validate if the modeled system already exists; the model can be configured to represent, say, the current operation of the system and the system’s output and behavior can then be directly compared to the model’s.

If the modeled system does not yet exist, then validation is more of a challenge, but there are still many approaches that work well:

• Showing the model to a roomful of experts (which would typically include the system’s designers, manufacturing engineers, industrial engineers, controls engineers, production staff, material handling experts, etc.) and focussing on each aspect of its operation and decision-making will yield a good deal of feedback. If the model’s behavior is inappropriate for its objectives, I’ll soon know about it!
• In the latter case, it is also a good idea to re-state the modeling assumptions identified during construction of the conceptual model, and included in the specification, and see if there is still broad agreement. For some reason, people seem to pay more attention to such details when watching an animation of the consequences than they do when reading the specification. This process also helps to remind attendees that the model does not have to be exhaustive to achieve its objectives.
• The model can be compared to similar existing systems, if any are available.
• etc.

Using the Simulation to Drive The Project

In the first two cases above, it’s interesting to note that, typically, many attendees—upon seeing the new system in operation for the first time—will identify defects that the simulation reproduces perfectly well; they will then want to go away and fix those defects while the design is still on the drawing board. That’s understandable, but I strongly recommend that the simulation be used to evaluate any such fixes (in the experimentation phase—see later), before they are incorporated into the official design, for a number of reasons:

1. The simulation and the proposed system will always remain in sync. Otherwise, if changes start being made to the system without being simulated first, then the simulation will enter a catch-up mode that will prevent it from being used to drive the project subsequently—a major benefit of simulation will then disappear.
2. The fixes may not work, or may not be the best options available. In many cases, far more radical better options exist that result in considerably simpler, less costly processes. Using the simulation to test the fixes allows them to be validated (and ensure that they have no unintended, harmful side-effects), while also allowing alternative potential fixes to be evaluated and compared objectively.

Correct the Model Before Experimenting

If, during validation, any significant discrepancies between the simulation model and the modeled system are identified, then the conceptual model will need to be revised and the computer simulation model updated accordingly.

Before continuing to perform any experiments upon the simulation, I require my client to agree that the simulation is a reliable, accurate model of their system, and they will not do that until they are satisfied that their model has attained that status. By this stage, after all the attention to detail, and after all the verification and validation, I typically find that my clients have no problems trusting the simulation whatsoever.

A common mistake that many simulation practitioners make is to fail to validate their models before starting to experiment with the model. This is probably the biggest single cause of failed simulation projects! If you conduct experiments upon the—unproven, unvalidated—simulation and present the results as facts, then those “facts” will have a way of influencing subsequent decisions in a very prejudicial manner. Furthermore, if the model is then modified (say to fix a bug or to correct a flaw in the conceptual model), then it may subsequently issue very different results which will—quite rightly—undermine the confidence of any rational person who has seen both the current and prior set of results. “If the results can change that much, how can I have faith that the results will not change beyond recognition again when another bug is fixed?” they will ask themselves.

I will not start experimenting with the model, and issuing results, until after the model has been formally validated and approved by my client. That way, I know—and they know—that any results they receive have bona fide credibility.

Long Shelf-Life Models

Many simulations outlive the original project and go on to lead healthy, successful, productive lives. In such cases, should we still specify objectives, build conceptual models, state assumptions, seek approvals, verify and validate the model as each change takes place? Absolutely! If not, then the model will become less focused, contain more bugs and demonstrate more invalid behavior over time.

In other words, I personally follow a similar process to the above when updating existing simulation models. However, if the objectives, or the subject of the model, have changed too radically, it’s worth considering whether it would be better to start over with a fresh model.

What’s Your View?

I hope it’s clear that, with the right approach, simulation is an accurate, reliable, credible tool. Feel free to contact me personally if you would like to discuss any of the issues that I have raised.

In the meantime, what’s your view on simulation quality control? Do you have any war stories you’d like to share? Feel free to comment below.

Mike Allen
President, Hindsight Consulting, Inc.