Using Health Metrics to Troubleshoot Streaming Issues

 

Monitoring the health of a live stream can be critical in preventing a poor viewer experience. Analyzing your stream health metrics allows you to accurately predict what will happen so you can quickly correct any issues with minimal impact to the viewer. For these reasons, stream monitoring should be a top priority for video publishers.

Why? Playback issues don’t just cause viewers to tune out. A disruptive viewing experience can impact your brand, subscriptions, and damage your advertising relationships. Both viewers and advertisers expect a high-quality streaming experience and have little tolerance for interruptions.

Studies show that 40 percent of viewers will click away or drop out if buffering happens more than once during a video stream. In addition, advertisers who pay to reach a certain number of viewers during a stream will go elsewhere should playback issues cause a significant number of viewers to leave. The consequences of a poor streaming event can have long-term impact, so you’ll want to monitor your stream health metrics and avoid a potential streaming catastrophe. Time is of the essence during a live event and your metrics can quickly point you to where the issue is.

Streaming issues can be caused by the:

• Encoder hardware or software
• Video and audio sources
• Internet and network
• Stream target destination

To help you monitor your streaming events, both Wowza Streaming Engine and Wowza Streaming Cloud provide a stream health dashboard, but quickly resolving an issue often depends on your ability to interpret the metrics.

In this blog, we’ll review the top health metrics to closely monitor during a live stream. We’ll also examine some examples of poor stream health and discuss the recommended troubleshooting tips.

 

Stream Health Metrics Overview

For this tutorial, we’ll refer to the Streaming Cloud dashboard under the Stream Health tab.

 

 

Here’s some terminology to get started:

• Inbound Bitrate – The actual rate of the stream traveling from the source encoder to Wowza Streaming Cloud, in kilobits per second (Kbps).
• Outbound Bitrate – The actual bitrate of all outbound video renditions, in kilobits per second (Kbps).
• Frame Rate – The rate of the video, in frames per second, at the transcoder.
• Keyframe Interval – The number of video frames compressed in a group of pictures (GOP) between keyframes at the transcoder.

 

Healthy Inbound Bitrate

It can be perfectly normal to see a small amount of bitrate variance in a stream based on the action within the video. For example, in a football game, if the announcer is on the screen talking, you would expect to see a lower bitrate in your health metrics due to the decrease in data required to broadcast that part of the stream. However, as soon as the camera switches to the action on the field and the players are all running around, you would notice a sudden increase in the incoming bitrate. The same rule applies if you’re streaming a countdown clock to the moment the live event begins. The moment the stream switches from the clock to the live event with more action, you’ll notice a sudden increase in bitrate. When the event ends and the camera fades to a still image, you will see the bitrate drop again.

Here is an example where a stream switches from a “Stream Starting Soon” image to the live event at around 9:43. Notice the bitrate drop just before 11:00 when the camera momentarily switches to a still image and then back again to the camera. This is an example of a healthy stream. The encoder is working well as the keyframe interval and frame rate metrics are stable throughout.

 

High Variance in Inbound Video Bitrate

If you see high variance in the video bitrate, but a stable frame rate and audio bitrate, this could indicate the encoder and network are working fine, but either the action is changing drastically in the live stream, or you haven’t set constraints for an upper bitrate limit within the variable bitrate parameter.

Not sure whether to use constant or variable for your use case? Constant bitrate (CBR) encoding is faster than variable bitrate (VBR), but will result in a lower-quality video for high-motion content. VBR encoding adjusts the bitrate depending on the complexity of the stream content, which produces higher-quality playback. When using VBR for live streaming, there is a setting to constrain or limit the data rate of VBR source encode to 110% of the target bitrate to avoid buffering or dropped frames. If you’re unsure of the bandwidth or network conditions for your event, using constant bitrate may be a safer bet.

For a stream with variable bitrate setting, you would expect to see a large fluctuation in the bitrate. If you are using a constant bitrate, however, high variance could indicate a potential problem.

Here is another example of a healthy stream that is using a variable bitrate, so the variance fluctuation is normal and based on the complexity of the stream. Some additional metrics here that point to a healthy stream are the steady keyframe interval (GOP) and frame rate (FPS).

 

 

If the inbound bitrate has high spikes and drops despite a constant bitrate setting or the stream drops completely and then reconnects, the bitrate may need to be lowered.

 

Possible causes:

• The encoder is struggling to keep up
• Network or bandwidth issue
• The video bitrate is set too high

   

Unhealthy Spikes in Inbound Video Bitrate

If you see a high number of spikes in the video bitrate when there is low activity in the stream, your bitrate is potentially unstable and, as a result, you may see additional problems with frame rate and frame loss.

Here is an example of an unhealthy stream and an encoder that is overloaded. Notice the difficulty in maintaining a healthy inbound bitrate (green) and keyframe interval (gray). Eventually, the entire stream disconnects.

 

Possible causes:

• Lack of bandwidth
• Incorrect encoder settings
• CPU load is too high
• Encoder hardware issue

 

Variance in Outbound Bitrate

If you see an unusual variance in the outbound video bitrate, this could be caused by changes from the incoming stream, e.g., a change in the camera or audio capture. It could also indicate drops in the inbound bitrate, keyframe, and frames per second (fps). Once again, it’s good advice to always check the first mile of streaming when you see playback issues. The problem very often originates at the source, so you may want to review your encoder settings first.

Little to no outbound bitrate could mean your streams are not configured properly to reach the content delivery network (CDN) or stream targets. It could also mean an issue with the CDN itself, although that is rare.

Here is an example of severe issues with the outbound bitrate (orange), where viewers experienced poor playback as a result. Video quality can be affected for the viewer if the outbound bitrate drops too low, especially for a high-definition stream.

 

 

Possible causes:

• Network congestion or lack of bandwidth in the first mile
• Stream targets are not properly receiving the chunks

 

Drop in Frame Bitrate or Frame Loss

High-definition devices typically capture video at 30 or 60 frames per second (fps), but the standard for video streaming is 29.97 fps in the U.S. and Japan, 25 fps in the rest of the world. For remote cameras used to capture security footage, a frame rate of about 10–15 fps works fine.

If your stream health metrics show a drop in the frame rate or intermittent frame loss, viewers may experience a choppy stream.

 

 

Possible causes:

• The network is congested and frames are being forced to drop
• The video bitrate is too high for the encoder
• Lack of bandwidth
• The CPU usage is too high (perhaps from too high of bandwidth)
• The transcoder is overloaded and can’t set the proper frames per second

   

Fluctuating Keyframe Interval 

For live streams, keyframe intervals should be two seconds. So, if your frame rate is 30 fps, the keyframe interval would be every 60 frames. Note that too frequent of keyframes can result in stuttered playback, so a two second keyframe interval is optimal. It’s also worth noting that for HTTP Live Streaming (HLS) playback, video chunks must start on a keyframe.

A warning appears in Wowza Streaming Cloud if the keyframe interval varies from the recommended rate of being equal to or twice the frame rate.

Should you notice inconsistencies in your keyframe interval when analyzing your stream health, be aware this can result in disruptive playback for the viewer.

Here is an example where the keyframe interval was not properly set to a fixed number, resulting in unhealthy fluctuations. Keep in mind that adaptive bitrate streaming requires a set keyframe interval as a seek point for the player.  

 

 

Possible Causes:

• The keyframe interval isn’t set properly in the encoder
• You are not using an even numbered keyframe
• The keyframe interval is set above 4 seconds
• The encoder doesn’t have the bandwidth needed to deliver data to the media server

   

Summary

Monitoring stream health metrics can help you quickly diagnose a streaming issue and determine if the issue originates at the video source. You should always check your encoder settings first when troubleshooting a problem with playback.

We recommend the following settings to create source encoding presets in Wowza Streaming Cloud.

   

For additional optimal encoder settings in Streaming Cloud, visit our documentation and learn how to create high-quality encoded source video.

If you’re unable to see your stream health metrics in the Streaming Cloud dashboard, your stream may not be playing due to a configuration error. You can review the steps for sending an RTMP test stream from OBS in our video tutorial here.

If you have any additional questions about analyzing your stream health metrics, feel free to post a question in our Wowza Community Forums. Finally, if you need a Streaming Cloud Service license to begin streaming today, you can get started with a risk-free 30-day trial.