How NLCHI fixed their vaccine portal in time for dinner

For more than a decade, the Newfoundland and Labrador Centre for Health Information (NLCHI) has worked to enhance the health and well-being of citizens of the Canadian province through the planning and delivery of information systems, analytics, and all things eHealth.

The Centre’s goal is to facilitate the best possible care by ensuring that wherever citizens present for healthcare in the province, providers have the most accurate and up to date information.

NLCHI’s original mandate was to deliver eHealth solutions. But in recent years, the mandate expanded to include supporting regional health authorities with their technological needs. This meant more responsibility for the Centre, but also more opportunity to support healthcare providers.

When COVID-19 hit Canada, these responsibilities expanded once again. Vaccines rolled out and the government announced citizens would need proof of vaccination for entry into venues.

It was up to NLCHI to deliver personal vaccine records to the province’s 500,000 citizens.

We sat down with NLCHI’s Director of Solution Development and Support Andrew Roberts to discuss how it went. And as we found out, it was no easy feat.

“We were asked to turn the vaccine portal around really quickly. From when they told us they needed vaccine certificates, we had about a month until the system had to go live,” says Roberts. “It was a pretty heavy lift. One of our developers was working seven days a week, 10-to-12-hour days for that whole month. We had to sprint to get to the finish line."

The sprint to the finish line worked, and NLCHI’s vaccine portal was fully operational and ready for launch day.

The team knew demand for vaccination certificates would be high, so they launched the portal two weeks before the certificates would be required by venues.

“In our media announcement we said everyone doesn't need to go here the first minute the portal goes live. You have two weeks before the ban is going to take effect. Relax, don't panic.”

But as we often see at Queue-it, online visitor behavior is unpredictable. While NLCHI tried to calm the rush with their long lead time and public announcement, they weren’t the only ones sprinting towards vaccine certificates.

“Of course, no one listened,” says Roberts. “Everyone went there first thing in the morning and tried to get their certificate.”

“When we went live, we saw huge amounts of traffic all at once. We had thousands and thousands of people every second trying to come in and get their vaccine records. Our go live lasted for about two hours and then our backend systems became overwhelmed. We were definitely in a high stress situation the day we went live.”

Andrew Roberts, Director of Solution Development & Support, NLCHI

After a month of hard work and heavy lifting, this was a worst-case scenario for NLCHI. And the public was unforgiving.

“The media was posting a bunch of things about the site crashing. Social media was blowing up about it. So, you know, there was a lot of pressure on us that day.”

The team was trying to pivot quickly looking for a solution, they quickly scaled their servers on AWS, but “this just brought the bottleneck from the frontend to the backend.”

“When we saw that, we realized we didn’t have a technical fix in our hand that we could implement. We needed to figure out some other way to throttle that traffic into our system altogether.”

Having seen virtual waiting rooms used for other public sector services, Roberts reached out to Queue-it. An hour later, he was on a conference call with the team.

“You could tell this wasn't Queue-it’s first rodeo reacting quickly to a customer in a crisis situation. When it came to the financial contract, it was like ‘don’t worry about that, we’ll deal with that later. The most important thing now is to get you back up and running,’” says Roberts. “That was really refreshing and something you don’t often hear from vendors. It let us focus on what really mattered to us that day, which was getting our system back up and online.”

The Queue-it team worked alongside NLCHI throughout the day to implement the virtual waiting room solution in front of the vaccine portal.

“We called maybe at 10 o'clock in the morning and by dinnertime the virtual waiting room was up and running. It was polished and professional and it integrated very seamlessly. As soon as it went live, we didn't have one hiccup. It worked right away and it did exactly what it was supposed to do.”

With the virtual waiting room in place by the evening of day one, NLCHI’s vaccine portal worked exactly as it was designed to.

“After all the panic and the media attention, it was such a sense of relief to have Queue-it in place. We could see that traffic was going to the virtual waiting room and the number of people downloading their vaccine records was climbing quickly. It just felt like ‘Okay, we’re back up and running.’”

Traffic inflow graph from Day 2 of the vaccination certificates being available

In the few weeks that followed, over 300,000 citizens successfully downloaded their vaccine certificates. The negative headlines and social media complaints eased up, and the core government team was pleased with the launch.

“Queue-it is a fantastic product. The time to implement was phenomenal, to take a solution like that and turn it around in less than half a day. When you’re in an operation situation like that, to have a solution that we could lean on to save the day was such a relief. It really saved our go live. It’s a great solution that I'd recommend for anyone who's having those types of performance concerns.”

Even after the vaccine mandate was removed and demand died down, Roberts told us they were so happy with the solution they kept Queue-it in place.

“It's just nice to know it’s there. It’s a safety net for us. It gives us the confidence that if there’s any spike, for any reason, the virtual waiting room is there to protect our systems."

After the success of their vaccine certificate launch, NLCHI is already hard at work on their next big project: a personal health record for the citizens of Newfoundland and Labrador

They’ve learned releasing digital services to the public is unpredictable and aren’t looking to repeat the crisis they faced the morning they rolled out their vaccine portal. After their experience with Queue-it, they have confidence there’s a robust solution close at hand if demand is high.

“When we go live with the personal health record next year, much like the vaccine portal, we're probably going to generate a lot of buzz with the public to be able to go in and access all their own health information. So there’s definitely another solution coming that we’re already considering using Queue-it for.”