Redundancy In Live Streaming

Director of Photography Steve Miles sets up a redundant camera rig utilizing two Sony FX3s for a recent live stream.

Are you redundant? If you’re in the UK, you may be scratching your head when I mention redundancy and relate the term to video/film/live streaming production equipment rather than employment. In the UK, and in some circles in the U.S., being made redundant is a polite way of saying that a person is losing their job, being fired or being laid off. In the world of production though, the term redundancy takes on a wholly different meaning.

According to Wikipedia, “In engineering, redundancy is the duplication of critical components or functions of a system with the intention of increasing reliability of the system, usually in the form of a backup or fail-safe, or to improve actual system performance.” I’ve worked in video and film production for quite a while and I’ve found that as gear has become better while becoming less costly, I’ve often wondered why more of my colleagues don’t think about redundancy in their gear packages?

Zoom Engineer Damian Christie setting up a Zoom drone farm on a recent live stream project that utilized 20 laptops to bring hundreds of participants into the live stream event.

How Redundant Are You?

Since the beginning of the pandemic, I’ve spent an increasing amount of time working on live stream projects. I still produce and shoot lots of non-live stream content as well, but as the pandemic has worn on, there has been a distinct uptake in the number of live stream projects that I’ve become involved in. In live production, as you probably know, there’s usually no chance of “retakes,” second chances or repeats. Whatever you sent out on your live stream, you almost always only get one shot at making it happen and making sure that it happens right.  

Sound Engineer Gary Wright utilizing our Allen & Heath SQ5 digital audio mixer for a recent live stream. We didn’t have access to a second SQ5, but we did bring a simpler, lower-end mixer as a redundant audio mixer in case the Allen & Heath went down during the telecast.

Will There Be A Reshoot Option?

As a producer, I really don’t like doing reshoots. Reshoots are sometimes done because of talent issues. Talent got sick or had a change of heart or mind and their part must be recast. At times, though, a reshoot must be done because of personnel or equipment failure. The majority of time when a reshoot must be done, the clients aren’t happy about it. In live production, though, there’s usually little to no option to do a reshoot. Usually, the audience gathers to see and experience the live program, period. “Failure is not an option” is attributed to NASA’s Gene Kranz, flight director of Gemini, Apollo and Space Shuttle missions, but it’s very applicable to live streaming and live television production. Besides minimizing the variable with talent and crew, what else can you do to minimize the chances of failure? There are two words you should become familiar with: systems redundancy.

Why Do Some Ignore Systems Redundancy?

In a perfect world, you’d have at least two of everything you bring into a production. In the real world, though, there are severe limitations that prevent you from having full systems redundancy:

  • It adds significant expense. If you opt for 1:1 systems redundancy, it doubles your equipment costs whether you own or rent the redundant gear.
  • It significantly increases setup and breakdown time.
  • It significantly increases engineering, wiring/cabling and hook-up costs in gear and time.
  • It’s easy to add complexity when you multiply gear and hookups, which can actually increase the chances of gear failure.

Systems Redundancy…

  • Reduces downtime in cases of equipment failure.
  • Increases how switch or swappable gear reduces the time until your program can get back online for your audience.
  • Increases peace of mind and confidence in the production team.
This is a basic hookup of our main vMix system for a typical live stream event. While it’s not always possible to have full systems redundancy for every item used, it’s important to have backups of the most likely-to-fail components.

My Approach To Systems Redundancy

In the real world, we simply cannot acquire, engineer, transport or afford full systems redundancy. It’s just not practical. We’ve taken an approach to provide our clients with major component redundancy that significantly increases the chance that if a major piece of gear fails, we can be back up and running in as little time as possible. We present systems redundancy to every client. If clients want the added security that major systems redundancy provides, they MUST be willing to pay for that.

What are the most likely systems failure points? Based upon our experience live streaming over the past few years, we usually make the following recommendations.

Camera Systems

Notice how I use the term “camera systems”? This is because cameras need batteries, media, support (tripods, gimbals, sliders, etc.), lenses, filters and often other components to function as a camera system. 

Live Streaming Systems

We utilize a variety of systems and platforms, depending on the client’s needs and if we are talking about our main system or a sub-switched system that feeds into our main system. Our most commonly used live streaming platform is vMix and vMix Live. This is a sophisticated live streaming platform that we run on high-end PCs. PCs, inherently, are less reliable and have more vulnerable points than dedicated video switchers. However, vMix running on a PC also gives us a feature set that’s nearly impossible to replicate with traditional production switchers feeding an encoder without adding dozens of additional pieces of gear.

Because we like the performance and options we can provide to clients using vMix, we can massively increase the reliability of vMix by always bringing at least two systems on each live stream shoot. If the main system goes down, it can quickly be swapped with the backup system. For a few jobs, we’ve even included two redundant backup systems, giving us three fully functioning systems at the ready. vMix itself is inherently reliable. Unfortunately, PCs running Windows can’t claim the same track record. Microsoft’s incessant insistence on mandated software and firmware upgrades means that a system that was reliable and perfectly functional one moment can become absolutely useless right after a required software or firmware update.

To take this to the next level, we recently acquired a Blackmagic Design 40 x 40 Matrix Router system that allows us to run every single source (cameras, PowerPoint laptops, Zoom laptops, etc.) into the router and the router sends identical outputs to the SDI inputs on our vMix computers. In the event of a system failure, we could switch the stream from the failing computer to a fully functional backup vMix system in a matter of seconds. This solution isn’t perfect, but it’s a viable backup should a system fail in the middle of a live stream.

Behind the scenes from a recent high-end live stream Zoom drone farm bringing in hundreds of participants to appear in the program as a “live” virtual audience.

Internet Access

Internet access is the lifeblood of any live stream. Without reliable, fast Internet, our clients’ shows can’t reach their audience. We do everything possible to ensure that we have good quality, reliable Internet access at the client’s location via hardwired Ethernet connections to their access points and/or servers. We’ve discovered, though, that even the most reliable Internet access can go down anytime no matter what precautions are taken. We’ve acquired a Pepwave wireless bonded router that bonds together up to four wireless Internet providers at a time to provide our backup Internet access should a client’s venue or facility have a network outage. This has happened more times than I like to recall and the Pepwave has saved quite a few live streams when the clients’ “bulletproof” Internet access at the live stream venue has gone down.

All Of The Other Stuff

Besides these three biggest categories, we always bring spare cabling, UPS (uninterruptable power supplies) in case there’s a power outage or brownout at the venue. Spare adapters, spare tripods and monitors have also been pressed into service on more than one live stream.

Me, technical directing a recent live stream. Notice I have two main monitors in front of me? The third monitor on my right is hooked up to a redundant vMix system that’s run into a Blackmagic Design 40 x 40 Matrix Router. Should the main system fail, I can switch to the redundant system and keep the show going.

What If I Don’t Produce Live Streaming?

Even if you only produce “regular” productions, you should still provide at least some level of gear redundancy for your clients. I always bring along a spare camera body, lens, batteries and media. I always bring a spare tripod when doing tripod-based shoots. I usually don’t have the room or gear to bring spare grip, lighting or audio package, but I do try to provide replacements on site for at least the most basic components to complete a shoot. Bring a tool kit that you can use to repair a critical piece of gear if something simple goes wrong with it.

In the end, this mindset is what separates amateurs from professionals. Professionals are always prepared and think about their client’s needs. It’s not practical to bring two of everything on every shoot, not to mention that would be exceedingly expensive in a world of ever-decreasing budgets. If you cannot bring a spare camera, microphone and light, at least do some legwork and find out if you have friends, colleagues, ShareGrid or a rental house near the location that could bail you out if a crucial piece of gear fails. Redundancy is almost as much a mindset as it is a lot of spare gear. Always be thinking, what would I/could I do if a major piece of gear goes down during production?