When the Inglewood Park Cemetery network kept crashing, Sanity Solutions looked at systems integrations to uncover—and fix—the problem.
Network outages forced Miji Pitts, IT operations manager at Inglewood Park Cemetery, to miss his own Father’s Day celebration.
The Inglewood Park organization, which has around 400 users spread across several cemeteries and satellite offices in Southern California, was experiencing multiple IT outages most days. Pitts, whose own father and grandparents are buried at Inglewood Park, knew that the holiday would be particularly busy. And so, even though his wife and mother had planned a special dinner to honor him, he instead sat chained to his laptop, waiting to restore the network when it inevitably went down.
“It got to the point where we knew we were going to have a disconnect every eight to 12 hours, and even more often during busy periods,” Pitts says. “At night, I was working on configurations in my sleep. We were completely on fire.”
The outages stemmed from a growing mismatch between the organization’s expanding data demands and aging solutions. Inglewood Park’s IT ecosystem relied on a mix of HP storage, a VMware virtualization environment, and 1GB networking infrastructure. But as the cemetery expanded its operations and added more users, traffic began overwhelming parts of the system, repeatedly knocking critical services offline.
“The connection between our storage and our VMware environment was dropping arbitrarily,” Pitts says. “The system was getting overloaded, and it just couldn’t sustain the connection.”
Pitts and his team initially worked with hardware vendors to upgrade key components of the environment, including network adapters and switches. The upgrades improved performance, but the outages continued.
At the same time, the organization’s VMware licenses were coming up for renewal. Pitts reached out to Sanity Solutions, hoping that the jump from VMware 7.0 to VMware 8.0 would help solve the problem. Rather than simply processing the license renewal, the Sanity team began asking deeper questions about the broader IT environment, determined to get to the root of the problem.
“Before, there wasn’t anybody to bring all of these products together,” says Justin Tolzmann, a senior field engineer at Sanity. “Vendors are going to try to solve the problems within their own products. Our job is to bring our knowledge of the overall infrastructure to make sure everything works together.”
A Growing Infrastructure Challenge
Founded in 1905 near the current site of SoFi Stadium, Inglewood Park Cemetery is one of the oldest and largest cemeteries in Los Angeles County, serving as the final resting place for cultural luminaries like Betty Grable, Ella Fitzgerald, Cesar Romero, and Ray Charles.
When Pitts was a child, his family often visited the cemetery for picnics near his grandparents’ gravesite. But it wasn’t until he came to work for the organization that he fully understood the IT demands of the nonprofit behind the peaceful grounds.
“When the job opportunity came about, the first thing I thought was, ‘Inglewood Park Cemetery is hiring for a network administrator? What is going on there?’” Pitts recalls. “But we now have a dozen physical sites that I oversee and manage. There’s a lot going on behind the scenes.”
The organization’s outages required Pitts and his team to manually reset the network several times a day. Full restoration took up to 30 minutes, locking hundreds of employees out of resources they needed to do their jobs effectively.
The organization’s first touch with Sanity was with account executive Benjamin Chadrick. Although Chadrick knew that Inglewood Park wanted to upgrade its VMware licenses, he paused to learn more about the outages. “We don’t want to just be renewal takers,” Chadrick says. “When a customer comes to us, we always ask why they need this product. We want to make sure that we’re building out a real solution to their problem.”
Sanity began with a full assessment of the organization’s infrastructure, reviewing the storage environment, networking architecture, and VMware virtualization stack. The team identified several bottlenecks, including limited network throughput and configuration issues within the virtualized environment. Sanity helped upgrade the cemetery’s VMware platform, expand network capacity from 1-gigabit to 10-gigabit connections, and refine the configuration of the storage and virtualization systems so that the different components could operate reliably together.
“Sometimes, our role is to work with the vendors and get the correct people on the phone to help resolve an issue,” says Tolzmann. “In this case, it was more about getting all of these different solutions to work together.”
Putting Out the Fire
Working closely with Pitts and his team, Sanity’s engineers began implementing fixes to restore network stability.
“The Sanity team is very detail-oriented,” says Pitts. “Their solution architects very quickly got a grasp on our environment and what was happening. They’re very hands-on and very technical, and they built out a plan and executed on it. I view it as a true partnership.”
Pitts notes that the organization couldn’t take its network down until 5 p.m. Pacific Time, and that Tolzmann worked after hours to stand up the new environment. “Justin was very flexible,” he says. “He made himself available on our schedule, and he was extremely methodical.”
In addition to upgrading the VMware environment, Tolzmann investigated why new networking cards had not reduced the frequency of outages. He discovered that the VMware environment hadn’t been configured to take advantage of the increased capacity, and he tweaked configurations to ensure compatibility between the systems. As Tolzmann completed his work, the environment gradually became more reliable, until finally the outages ceased.
“Justin showed me tidbits and tricks during the process,” Pitts says. “So, I was strengthening my skills with VMware, as well, thanks to his knowledge and insight. It was a great experience.”
Chadrick says knowing that Pitts was losing sleep over the outages made the Sanity team want to go “above and beyond” to solve the problem. “We want to help people get back to focusing on their organization,” Chadrick says. “We want them focused on how to grow their company, and not have to worry about all these firestorms.”
“Before, I was constantly paranoid that the system was going to disconnect,” Pitts says. “Now, I can go weeks without even looking at it. It’s so nice to have that peace of mind.”
+++