How to Mitigate Network Outages on Your College Campus

Today’s higher education landscape is a far cry from what it was 20 years ago. Among the most significant changes is the continual evolution in how technology is utilized for teaching, learning and operating an institution. Information technology (IT) networks have become the backbone of the modern learning environment.

The number of internet-connected devices has quadrupled since 2012. In fact, it’s projected that there will be an estimated 30 billion devices online by 2020 – that’s nearly three times the number of humans on earth. Even just a decade ago, the average student would show up to their college campus with one or two devices they’d intend to connect to the internet. Today’s average student owns at least five devices that require connectivity, according to the Refuel Agency’s 2018 College Explorer Market Research Study.

For example, all students need to connect to the internet in order to log onto their school’s Learning Management System (LMS), which gives them access to assigned readings, class discussion boards and other course material. Some instructors have students collaborate with classmates through synchronous video and web conferencing. Students may also watch Netflix videos in their downtime, connect their gaming systems in their dorm rooms, use the web to power their wearable fitness trackers, access social media sites on their phones and more.

It’s also true that an increasing number of college campuses are beginning to utilize a variety of Internet of Things (IoT) enabled technologies, such as security cameras, thermostats, door locks and lighting systems. With these statistics in mind, campuses are faced with a mounting question: How can we avoid network failures and ensure the growing surge of connectivity needs is met?

Join us as we explore some of the most common causes of campus network outages, analyze the true impact network failure can have on an institution and consult expert advice on preserving the integrity of your network.

Why do network outages occur on college campuses?

In addition to students operating a number of different devices at once, classroom instruction has also evolved to include an array of virtual resources. If an institution isn’t accounting for this vast device base and its ramifications in terms of bandwidth, problems with network connectivity can increase due to overtaxed network equipment.

Aside from a sweeping increase in network demand, there are a handful of common culprits behind campus network outages. One of the most prominent among them is weather-related events. Hurricanes, thunderstorms, floods and heat waves can all increase the likelihood of outages due to power failures and power surges that can destroy network equipment.

Another driving factor seen commonly in unexpected network outages is aging infrastructure. The unfortunate truth is that many colleges find they can’t afford the infrastructure upgrades they need. In fact, the 2018 Campus Computing Survey revealed that 67 percent of participating institutions reported their IT funding had not fully recovered from harsh budget cuts experienced over the last four to six years.

Network outages can also stem from poorly implemented change activity. Most universities and colleges don’t have a full-scale ‘test’ network where technologists can test their changes in advance. Instead, when changes are made to the network – such as upgrading equipment or changing a firewall rule to allow communication between systems – they’re not always fully thought-out, resulting in unanticipated impact.

Finally, network outages can occur as a result of general equipment failure, as electrical components do degrade over time. Without properly designing for equipment redundancy, there can be several single points of failure within a network environment. A single failed network device can bring down an entire campus network.

How do network failures impact college campuses?

Regardless of what triggers an outage, the effects of network downtime can have a sizable impact. It’s been found that just one hour of stoppage can cost the average data center $260,000. In addition to being costly, outages can disrupt class schedules, prevent access to buildings, and some institutions could even lose valuable research and student data.

And while a sweeping network outage may seem farfetched for the average institution, we’ve seen a number of colleges stuck dealing with the fallout of such events in recent years. California State University, Sacramento, for example, experienced a campus-wide power outage on the first day of its 2018 spring semester. The network outage resulted from old switchgear failing. Classes were canceled and full power wasn’t restored to the campus for several days.

But most notable in recent memory is the network outage experienced by Amherst College in early 2019. Aging infrastructure and wiring issues left the entire campus without access to online services for five full days. This meant faculty and students were unable to access everything from Wi-Fi, email and online learning platforms to accounting and card-scanning systems. All content hosted by the Amherst.edu website was rendered inaccessible.

Many were left questioning how an elite college with a $2.2 billion endowment could fail to provide services as seemingly basic as internet access. As it turned out, the network the college was operating on was designed approximately 20 years prior, with the last hardware upgrade occurring a decade ago.

During the five-day outage, students took to social media with quips like, “Hell is real, and it’s Amherst College.” The institution took the outage as a call-to-action, having since expedited plans to upgrade its network. The IT leaders on campus have also prioritized moving the college to increasingly cloud-based IT solutions in an effort to prevent any future problems from interfering with access to the institution’s server.

How can you strengthen your college campus’ network?

While institutions can’t always prevent the causes of network outages – such as natural disasters – proper network design and planning can help mitigate the potential impact. In an effort to learn more about how universities can keep their networks operating smoothly, we consulted the expertise of Todd Pombert, vice president of IT and campus CIO for Collegis Education.

“Collegis helps its partner universities to keep their networks operating with 99.999 percent of availability,” Pombert explains. “This keeps their students, faculty and staff connected to the important resources needed to learn, teach and operate the university.” He identifies a number of attributes he and his team consider key to operating a well-functioning network.

Single points of failure

Part of what Pombert and his team at Collegis do to support universities’ networks is identify single points of failure within the network, along with what those points of failure impact. This assessment, he says, should include technology (e.g. routers, switches, firewalls and telecom carrier equipment) as well as facility and utility design maintenance (e.g. air conditioning equipment for network closets and data centers, UPS batteries and diverse cable paths).

“Sometimes it’s prohibitively expensive or infeasible to have full redundancy of all technology and facility components,” Pombert admits. “But understanding where your single points of failure are will help you identify where your response to a failure might require more urgent incidence response processes.”

Test redundancy

Put simply, technology components do sometimes fail, and Pombert notes that a redundant design often works well – on paper. “Where you have redundant technology components, test the failover rigorously during and after installation,” he urges.

Whether your redundancy solution is having stand-by hardware in a storage closet, utilizing active-active clustering, employing diverse network paths or another solution, there are complexities involved in performing a failover, and Pombert advises that these recovery steps be rehearsed, practiced and tested.

Incident response

One thing that kept Amherst College afloat during its 2019 outage was a well-functioning crisis communication plan. Despite the lack of access to the network, the college had ways to keep students, faculty and administrators in the loop regarding updates, closures, cancelations and backup plans.

In addition to keeping the pathways of communication open with those impacted by a network failure, Pombert points toward the importance of quick response time from the IT teams that help campuses function.

“Having a 24/7 incident response process in place to respond to high priority and urgent issues is critical to keeping your technology systems running with a high level of system availability,” he says.

Pombert maintains that responding with urgency – whether at three o’clock in the morning or in the middle of the work day – can be a critical aspect to keeping a university running when an unexpected outage occurs.

Change management

“Most system failures result from change,” Pombert reveals. If a rigorous change management process is not in place within an IT team, failures become all the more likely.

He explains the multilayered process in place with his team at Collegis by highlighting the following critical aspects of change management:

• Multiple sets of eyes reviewing the change
• Rigorous testing
• Impact analysis by multiple parties
• Stakeholder, end-user and impacted party communication
• Well-orchestrated execution of change activity

These elements, Pombert maintains, are critical to keeping technology systems running and continually meeting the needs of an institution. “Many IT organizations feel this level of rigor is too much to accommodate. But the level of effort to avoid an outage can be a fraction of the amount of time spent resolving and recovering from an outage,” he explains.

Systems monitoring

“It’s paramount to have 24/7 systems monitoring to know when and where there are failures,” Pombert explains. Detailed systems monitoring, he adds, is often undervalued by IT teams. When this is overlooked during the design and implementation portions of a project, it can become detrimental later on.

“At Collegis, we include the operational monitoring of a network from design through production,” Pombert explains. “You can’t design a 99.999 percent available network without incorporating component, system and end-user simulated monitoring into how you operate a network.”

Internal technology expertise

What’s one of the most prominent elements of keeping a college’s network afloat? Pombert says it can all come down to employing experts. “At Collegis, we have multiple layers of technology expertise working as one big team,” he divulges.

“This allows us to see how making a change in one area of the technology ecosystem impacts another area. That isn’t to say we don’t use external resources, but we do so selectively – hiring consulting skill while training our internal team members to acquire those important skills.”

Accountability

Finally, an undeniably critical aspect to what Pombert and his team offer colleges and universities is unfailing accountability. “Our partner universities are our customers, and they count on us to provide a valuable technology service,” he says. “We instill trust by providing service level agreements, and then meeting and exceeding these goals.”

Transparency, Pombert maintains, is key. “If everything is operating as expected, we show the metrics. And when something goes wrong, we provide full visibility into the root cause analysis and what we are doing to make it better.”

Prepare your campus network for anything

While it’s true that most college campuses will never be faced with the mounting fallout of a sizable network outage, incidents like the ones at Sacramento State and Amherst College have made it clear that all institutions should be prepared. You’re now armed with more information about the driving forces behind network outages, the impact such disturbances can have and a handful of measures your college’s IT team should have in place to ensure your network is prepared for anything.

If you’re hoping to build upon what your campus IT leaders already have in place with the expertise of teams like ours,  contact us to learn more about your options.