By Pat Dixon, PE, PMP
President of DPAS, (DPAS-INC.com)
One of my favorite podcasts is "Causality". About every 2 to 3 months, an Australian engineer named John Chidgey produces a 30-60-minute podcast addressing an industrial accident, the chain of events, what went wrong, and how it could have been prevented. It is very well done, and I strongly recommend this podcast as required listening for anyone in industry.
Episode 59 addresses the August 28, 2008 explosion at a Bayer CropScience facility in Institute, West Virginia. This accident killed 2 people, injured 8 others, advised 40,000 residents to shelter in place, resulted in a $143K fine from OSHA, payment of $5.6M in legal settlements, $975K in civil penalties, and $4.23M in required improvements to operate.
This is not the worst industrial accident of all time, but it is an example of what happens when people make assumptions and ignore proper procedures. Industrial accidents are not always the result of one simple and obvious cause. As in this case, the causes are complex and multivariable. However, as the subject of this newsletter is automation in the 4th industrial era, I need to call attention to a glaring issue.
One of the problems is the way they handled the migration of a distributed control system (DCS). They had a Honeywell TDC 3000 control system. I am one of the few people still around that know how to work on that system. In 2006 Bayer began migrating the system to Siemens PCS7. When doing a migration, it requires not only experience in the destination system but the original system. If you want to translate German to Spanish, you need to know both languages. By the time of the incident, the DCS upgrade had not been completed or fully tested. Starting up a new DCS without thorough commissioning is very bad and very dangerous. The PID loops had default tuning, instead of translating the tuning from the loops in Honeywell to Siemens. There were undocumented changes in control schemes implemented in the migration. Perhaps the most alarming problem was the inability of operators to see alarms and interlock status effectively. The HMI implementation did not follow good design practice, resulting in operators being overwhelmed with detailed information at the instrumentation level but not able to see the process overview views that would have helped operators notice alarms, overridden safety interlocks, and cause and effect.
I have done a bunch of migrations in my career, and each one has risk. There is great danger in treating a migration casually, either from the implementation side or the operation side. In the case of Bayer, it seems incredulous that would even think of starting up operation before commissioning the migration. Even though industry has been in a digital age for about 50 years, it seems way too common for industry to ignore the risks of control systems. The high capital steel and concrete projects are noticed, but lower capital DCS migration efforts can sometimes hide in the background with the assumption that risks are low. While the investment in control systems can be lower, the impact of a bad migration can be deadly.
This should not be an impediment to migration. Honeywell TDC 3000 reached end of life a long time ago. I know people in the business of stockpiling old TDC parts because the only other place you can find it is eBay. Facilities hanging on to these legacy systems are one IO card failure away from a lengthy unplanned shutdown. Migrations must happen, and you need the right people to deliver a well-designed and commissioned system as well as responsible operations people that will not start production until it is ready.
I ask you to make "Causality" a part of your routine. Decision makers in industry need to recognize how dependent we are on automation, and the dangers of doing automation badly.
