A checkbox simulation is a simulation study that is done purely to comply with a company policy that new facility designs must be simulated—but typically without any criteria specifying the quality of that simulation. What follows is a fictitious story—yet one that’s all-to-common in my experience—about the limitations of such simulation studies, and the accompanying race-to-the-bottom by consulting firms competing to perform them. I’m glad to point out that Hindsight is not one of them…
Once upon a time, an executive—let’s call him Rob—was assigned a project to bring his company’s new West Coast production facility on-line. He had recently attended a lean symposium at which he had been impressed by a case study demonstrating the benefits of facility simulation studies. The consultant giving the presentation, James, stressed the importance of making time on the critical path to evaluate and improve the design prior to procurement, installation & commissioning. “If you don’t make time to learn from your mistakes and improve the design while its still on the drawing board,” James had claimed, “you’re just committing yourself to spending a lot more time—and money—to putting things right once it’s operational, assuming that that is even possible.” While a little skeptical, Rob decided to hire James to work on his project, and took his advice.
He was glad he did. The simulation study was viewed by all concerned as a huge success. It highlighted a number of serious design defects and inefficiencies that, had the original design been realized, would have seriously impacted the viability of the new facility. Instead, the entire project team used the simulation model to understand their original design’s deficiencies, simplify the processes involved, eliminate non-value-added equipment and process steps, suggest alternative solutions to the various problems, and test how well these various “solutions” actually performed.
Rob came to realize that the simulation provided an egalitarian, evidence-based approach to evaluating the new facility—one that harnessed everyone’s contributions, while not relying upon gut-feel or office politics. (With some chagrin, Rob recalled how one of his own ideas—which no-one on the team was brave enough to challenge—was revealed by the simulation to have made matters a lot worse, rather than better.) The simulation had provided them the precious benefit of hindsight, without their having to experience the costs and complexity of the original design in operation.
Finally, what was to become the company’s most efficient, simple, elegant and profitable facility was commissioned, then ramped-up to full production, without a hitch. Rob shuddered to think how things might have turned out had they not used simulation. After all, the original design had been approved and signed-off by all involved, per their then standard procedure, prior to the simulation study. Things could have turned out very differently.
Indeed, Rob received so much recognition for his role in the project’s success that he was promoted. His first act in his new role was to decree a change in their procedure for designing a facility to ensure that the design was subjected to a simulation study.
Over time, Rob moved on to bigger and better things elsewhere, and new people came into the organization. The company determined that it needed a new manufacturing plant on the East Coast and assigned one of their newest executives, Brian, to head the project.
Brian had joined the company after Rob’s West Coast project had completed, so he had no idea why he was required to simulate the design of his new plant. Not wanting to rock the boat too much on his first big project, he decided to go ahead with the simulation, but addressed it from his project management perspective.
First off, Brian googled a number of simulation consulting companies and invited them, as well as James, to submit bids for a simulation study. It turned out that James’ quotation was at the higher end of the spectrum of bids received, and was promptly discarded. When James called to follow-up on the bid, Brian explained the situation to him. James pointed to the success of his West Coast facility study and highlighted the thorough nature of his proposed study, to no avail. Brian just needed to do a simulation study, and that was what he was going to do. After all, a simulation study is a simulation study, right? Brian awarded the contract to the lowest bidder, and congratulated himself on reducing his simulation budget by a few thousand dollars.
Secondly, it made no sense to Brian that the simulation should be undertaken on the project’s critical path. His manufacturing engineers had assured him that they’d learned lessons from the West Coast project, and that their East Coast design would be good to go. The simulation was simply going to rubber-stamp its approval, so why delay the whole project while waiting for the study’s results? Brian agreed, and scheduled the project so that procurement and installation would commence as soon as possible after (pre-simulation) design approval. As a result, Brian was able to shorten the lead-time for the project by a couple of weeks.
The project continued, and the design was approved before completion of the simulation study. Orders were placed with suppliers and installation commenced.
Everything was going well. The simulation study had completed, with only a minor hiccup being highlighted: the gross line rate of one process was a little low and needed to be increased to make rate. This involved some re-engineering of the design, and cost a few thousand dollars in extra equipment and site re-work, but nothing too serious and the project could proceed more-or-less as originally planned.
A couple of weeks into the commissioning phase, Brian was approached by Frank, the company’s Controls Engineering Director. His on-site engineers were struggling to make rate commissioning a critical part of the material handling system; instead of achieving the required gross rate of 120 jobs/hour, they could only squeeze out around 95 jobs/hour. They believed that it might be possible to solve the problem and achieve the target rate—but only if the facility layout was radically re-designed (the facility’s building profile had been shrink-wrapped around the design’s floor-plan and there was almost no room for maneuver). Meanwhile, their attempts to resolve the situation were cranking up their time, exceeding their budgeted hours, while also causing them to fall behind on their schedule. Frank expressed his puzzlement. Hadn’t the design been simulated and proven to operate at the required rate?
Brian called in his simulation consultants, represented by a young graduate named Josh, so that he could personally review the simulation model together with his manufacturing engineers and Frank. As it turned out, Frank had been actively involved in James’ West Coast simulation project, but—due to promotion—had not been directly involved with the East Coast simulation until now. It rapidly became clear to him that the simulation he was looking at was a lot less detailed than the simulation that James had developed for their West Coast plant.
For one thing, the material handling system had not been modeled in any great detail, and was represented in the simulation by a series of buffers and processes. Josh explained this by pointing out that their proposal clearly stated an assumption that the material handling system had been fully engineered and, therefore, did not need to be modeled in detail. For whatever reason, that assumption had proved to be invalid, but it had allowed Josh’s company to lower the price of their proposal significantly. (Brian noted that his signature was clearly visible in the acceptance box at the bottom of the proposal.) As a consequence, the detailed sequence of operations for controlling the flow of jobs in the material handling system was completely absent from the simulation! It was, therefore, no surprise that the simulation had failed to identify the problems that Frank’s team were struggling with. Josh volunteered to consider the matter and submit a price later that day, with timing, for adding a detailed model of the material handling system to the simulation.
But other problems were becoming apparent. Josh didn’t appear to be very familiar with the simulation’s details and was unable to answer specific questions that Frank, Brian and the engineers were putting to him. “I’ll get back to you on that,” he kept saying. When pressed about this, Josh explained that it was his company’s policy to offshore simulation development work to a subsidiary based in Asia, so he wasn’t personally familiar with the details of this simulation; the simulation engineer on the other side of the planet who could answer the questions was likely in bed asleep. Josh was also unfamiliar with material handling systems in general and the system that the company were installing in particular. Frank was dubious that Josh would be able to comprehend it in sufficient detail—and in the time available—to price it correctly, never mind explaining how it should be modeled to his Asian colleagues.
Still, as good as his word, Josh got back to them later that day. His price for adding a detailed representation of the material handling system was almost double what his company had proposed for their first simulation, bringing the total, Brian noted, above the price originally submitted by James. Furthermore, the work would take an additional two weeks to complete.
Brian was stunned.
His manufacturing engineers had failed to design a working system, and his insurance policy—his simulation consultants and their simulation—had failed to point that out. As a result, he was busy constructing a crippled plant, and his project was creeping over budget while also slipping behind schedule. And the decisions that had led to this state of affairs were his and his alone.
He discussed matters with Frank, then called Josh and told him to price up just that part of the plant in which the material handling problems had been found. If Josh could address and resolve the problems they were having there, they would consider giving him an order to update the rest of the simulation too. Josh agreed.
A few days later, Josh came in with the updated simulation, but was unable to reproduce the problems Frank and his team were experiencing. To help validate the model, Frank carefully went through the simulation’s logic with Josh and pointed out a number of instances in which the behavior of the simulation differed from the behavior of the real-life system.
It appeared that the simulation software that Josh’s company employed, which was aimed at more abstract, high-level modeling situations, made it difficult to model the precise, detailed behavior of the material handling system to the required degree of accuracy; Josh kept trying to get Frank to agree to further modeling assumptions that Frank doubted where valid. For example, Josh’s software did not have any support for job acceleration and deceleration, so he wanted to make an assumption that omitting full motion modeling wouldn’t adversely affect the simulation’s output statistics. However, Frank knew that doing so would shave around 15% off station cycle times, allowing the simulation to process jobs significantly faster than the real-life system; he reasoned this would make it difficult for the simulation to reproduce the plant’s reduced throughput. In the end, Josh agreed that they would add delays to the simulation to fully account for job motion and address Frank’s other concerns.
Finally, a few days later, over a week after Frank had raised his concerns with Brian, Josh had a simulation that reproduced the problems Frank’s commissioning team had encountered. The team pored over the simulation, but could find no solution that didn’t involve moving already installed equipment and adding an extra column to one side of the building to accommodate it. Brian reluctantly sought approval for the extra $2M for the required modifications, put a halt to installation and commissioning and had Josh simulate the proposed changes before allowing the project to recommence.
The facility finally went live around three months late, and over $2M over budget. It had the highest year-on-year OpEx of all their plants.
At the project’s conclusion, Brian held a post-mortem with his team to look at where they had gone wrong. Together, they concluded:
- They had made a big mistake in assuming that their original design would just work. It was a complex system and his engineers—who were, on the whole, good at their job—had been unable to foresee all of the problems that they were going to encounter. Subjecting the design to a detailed simulation study had proven essential.
- They had been wrong to proceed to procurement immediately after completing the design; instead, they should have awaited the results of a detailed simulation study. Fixing the design on the drawing board would have been a lot less costly, far more timely and they would not have ended up investing millions of dollars in an underperforming plant. Simulation must in future be acknowledged as a critical path activity.
- They had erred in believing that any old simulation study would do. In future, instead of just checking the “Submit the design to a simulation study” checkbox, they would need to pay a lot more attention to defining the objectives of the simulation study, and consider its content—and, in particular, the validity of the stated modeling assumptions—in received proposals. Clearly, in this case, a detailed representation of the material handling system should have been included from the start.
- They had been wrong to place the simulation contract with a company more interested in securing the winning, lowest bid than in ensuring the success of the project. Offshoring the simulation resulted in a number of cultural misunderstandings, and the simulation engineers involved had no direct contact with any of the project team, nor had they even seen a facility of the type being constructed. Brian’s company would no longer participate in a race to the bottom; in future, their criteria would favor trusted, experienced simulation partners, such as James, who they knew could handle the work.
- Frank pointed out that, in the West Coast facility simulation, James had included an allowance for them all to work together on improving the design of the facility, not just merely approving a working design. Doing so had saved them significant dollars off the project’s CapEx budget and had, by comparison to their other plants, led to much lower year-on-year OpEx costs. From now on, Brian agreed, they would employ a similar phase on future jobs.
- The cost savings from going with Josh and his company instead of with James had proven to be a false economy; the difference in fees evaporated over the course of the project and, in any case, paled into insignificance compared to the project’s budget overrun. Brian now regarded James’ price as a bargain.
- Brian also decided to write-up a detailed rationale for the use of simulation, with the conclusions of the post-mortem and their lessons learned, so that his successors wouldn’t make the same mistakes he had.
What do you think? Do you have a “checkbox” simulation horror story to tell?
Mike Allen
President, Hindsight Consulting, Inc.
