They Lost Their Cars to the Floods — But in Doing So, Saved Crucial Computer Servers
As flood waters from Tropical Storm Irene swamped the Waterbury state office complex, seven employees from the Vermont Agency of Human Services rushed inside to rescue computer servers that are critical for processing welfare checks and keeping track of paroled prisoners living around the state.
Two AHS employees —network administrator Andrew Matt and deputy chief information officer Darin Prail — parked their cars behind the AHS building at around 6 p.m. that Sunday, August 28, and rushed in to save the equipment. When they came back outside, giant trees were floating by and the entire parking lot was under water. So were their cars.
"We didn't know how much time we had," Matt said, "and our job was to save the servers."
The employees' quick thinking is being credited with saving the state's largest agency from disaster. AHS oversees not only the Department of Corrections but runs programs that serve thousands of Vermont children, families, senior citizens and individuals with disabilities. Within days, AHS was up and running again — its servers installed at an alternate site. The AHS servers run nearly 300 distinct applications — from desktop software like Excel and Microsoft Word to more specialized software that tracks prison inmates and handles applications for food stamps and health care.
After last spring's record flooding, AHS developed a server rescue plan to save critical data systems in the event of future flooding. This was the first test of that plan, said Angela Rouelle, the agency's chief information officer.
"We realized that we didn't have a good enough plan in place to handle a real emergency, so as we watched the river we talked about what we would need and what we would do," said Rouelle.
As Irene was lashing Vermont, AHS staff drove by the complex routinely throughout that day to check on the rising water levels. Staff also kept in touch directly with the National Weather Service for up-to-the-minute information on how high the river might crest.
By 6 p.m., the waters were rising fast and the team launched a rapid response effort to shut down the agency servers, unplug them and move them to dry land.
In the days leading up to the storm, officials with the Vermont Department of Buildings and General Services and Vermont Emergency Management believed that the complex wouldn't face the kind of dramatic flooding that occurred. Rather, they anticipated the site would see flooding similar to what happened in May — when some of the fields behind the site were flooded. What occurred was a flash flood of epic proportions.
"I don't think anyone that I know of contemplated the severity of the event — especially at that campus." said BGS Commissioner Michael Obuchowski.
As a precaution, however, the state cleared out dozens of state-owned vehicles in the days before the flood, and several BGS custodians and security staff were at the building until nearly 11 p.m. the night before, powering down the entire complex and making sure everyone was evacuated, said Obuchowski.
For Rouelle, taking a more proactive action saved the state — and her team — months of work and the cost of replacing tens of thousands of dollars in equipment and applications.
Rouelle had rented a truck and had it waiting at the loading dock so staff could load computers onto the back of the truck and spirit them away to dry land. But the water rose much faster than anticipated and Rouelle realized that by the time the truck was loaded, they would be unable to drive it out of the complex.
So, the AHS employees began to drag the servers from the basement to the second floor. Days after the flood waters subsided, Vermont National Guard trucks hauled the servers to Montpelier where they were reconnected to the state system.
"Had we listened to what others were saying — that this wasn't going to be a big deal and we wouldn't flood — we would have been sunk," said Rouelle.
Matt, who lost his car, recounted: "My first instinct was to park as close to the data center as I could because I had tools in my car that we might need as we pulled the servers, so I parked right near the data center in alleyway. We started working and after Darin showed up and parked his car in front of mine, someone else said, 'You guys might want to move your cars,' but unfortunately at that point I didn't really have time to move my car."