Identifying causes of packet loss in StarLeaf calls
Managing Troubleshooting StarLeafLast updated September 9, 2020
This topic regards advanced packet loss troubleshooting for StarLeaf admins and network administrators. Choose a topic below for more information:
- About packet loss
- Examples of different severities of packet loss
- How to tell if packet loss is affecting your calls
- How to troubleshoot the causes of packet loss
Live video calling is a demanding application for any network. For first-time video customers, it can highlight problems in their local network or Internet connection that they may not have been aware of. There are two main differences between live video network traffic, and “traditional” network traffic (web surfing, email, file downloads etc). One is the sustained high bit rates that video calling requires. The other is the requirement for very low latency (i.e. minimal delay in packets getting from one endpoint to the other). When viewing a web page, a couple of seconds delay while some HTML gets retransmitted by the server has no effect on the content that you see.
On a video call, a half second of latency is enough to reduce your conversation to “walkie talkie” level, and the loss of three or four consecutive audio packets means that the speaker gets asked to repeat themselves. Any form of voice or video over IP communication requires the network to deliver the packets quickly; so you need a good Internet connection. By default, StarLeaf room systems use 2.5Mbit/s but can be configured up to 6Mbit/s. Apps use 1.5Mbit/s download and 1.5Mbit/s upload. So to have 4 apps and 2 rooms in concurrent calls: 4 x 1.5 + 2 x 2.5 = 11Mbit/s of network bandwidth in each direction.
StarLeaf endpoints use a speed testing mechanism when registering with the StarLeaf service to choose the right bit rate for the bandwidth of the local network and Internet connection they are connected to (for more details about this refer to Video bandwidths and resolutions used by StarLeaf). However, this mechanism isn’t always successful. Network configuration problems or temporary congestion can mean that the network does not have enough capacity to transport all the media packets being generated by StarLeaf endpoints, or received from far end endpoints via the internet. When this happens, the “excess” packets are discarded or delayed by routers and switches in the network. The packets either never arrive at their destination, or arrive too late. This is called “packet loss”.
The recipient of a media stream that is subject to packet loss experiences anything from a slight degradation of media quality, to unexpected call drops. When packet loss is extreme, it even affects StarLeaf endpoints that are not on calls: responsiveness of phone user interfaces might be slow; they might be seen deregistering from StarLeaf, restarting, or displaying a flashing exclamation mark symbol at the top left of their screen.
How badly packet loss affects a StarLeaf call depends on the percentage of media packets being lost. This table describes what the recipient of a lossy media packet stream might experience on a call between two StarLeaf systems.
|Loss||Phone behavior||Audio||Video||Screen shot|
|Moderate (0-5%)||Normal||Very slight “roughness” to the audio, hardly noticeable||No discernible difference|
|Heavy (5-20%)||Response to key presses is slow. Calls take longer to connect and establish media||Possible to maintain conversational flow, but sound is interrupted by noticeable breaks that require the speaker to repeat themselves||Slight blockiness in areas of high motion. Static picture looks fine.|
Hard to use the phone interface
Phone might go offline for periods of time.
Connected calls might drop.
|Speaker has to be frequently asked to repeat themselves||“No remote video” message at start of call. Video might establish after several seconds but frequently freezes. Video in an established call is smeary|
|Extreme (30%+)||It is not possible to connect calls. Phones might deregister from the StarLeaf. In-progress calls drop.||Unusable. Periods of complete silence interrupted by short bursts of tinny sound.||Heavily broken, frozen image in call; StarLeaf hardware endpoints might display “No remote video” StarLeaf app might switch to audio-only style view||No remote video|
Watch the packet loss symptoms video here:
When StarLeaf detects packet loss for a prolonged period of time, it will reduce the bandwidth used by the call. When received packet loss is detected, a warning message appears at the top of your screen:
When sent packet loss is detected, other participants in the meeting will see an icon indicating there is a problem.
You can see detailed packet loss stats for a call through the portal or directly on a room system. For more information, see Call statistics.
- Transmit: refers to the stream of packets going from your room system unit or app to the far end
- Receive: refers to the packets that the far end is sending you
- Cumulative loss: shows the number of packets of each type (audio, video, and content) lost since the start of the call
- Current loss: shows the percentage of packets lost in the last 5 seconds of the call. It may take a few seconds for these numbers to accurately reflect the video/audio conditions being experienced
If you are experiencing media problems and the Call Statistics show significant numbers in the Cumulative Loss and Current Loss values for the “problem” media stream, then packet loss is the cause. If the “current packet loss” statistic is zero, as in the screen shot above, but noticeable media quality problems are still happening, the problem is likely a StarLeaf app local device issue (a problem with the camera/microphone/speaker setup or PC capability of whoever is sending the bad media) or a third-party interop problem (a bug in the way that StarLeaf and video endpoints from other manufacturers decode each others’ media streams). Refer to Troubleshooting StarLeaf for more information.
After you’ve established that packet loss is responsible for your call quality problems, you need to work out which end of the call is causing the problem. In a point-to-point call between two individuals, it is hard to determine where the problem is. In a multi-party meeting, all data goes through the StarLeaf platform so you can view the packet loss statistics between each individual and the StarLeaf platform. With a third person in the call, the problem will be easier to identify.
When you know which connection has the problem, examine the local network and internet connection on that site to identify possible causes. You may need to enlist the help of your IT administrator, as their knowledge of your network setup may help.
Below is a list of possible fixes for various types of network:
If the device is connected using WiFi, then check the following:
- Try switching to a wired connection. If the issue is fixed this indicates a problem with your WiFi connection
- Check the WiFi signal strength. If it is weak, try moving your device closer to the access point. If this fixes the issue then this indicates that you have a WiFi blind spot. Either upgrade your existing access points or add additional access points to cover the area
- If there are many devices using the same WiFi network, it’s possible that the access points are overloaded and are therefore dropping packets. Try replacing your access point with one that can handle more throughput, or increase the number of access points so that each one handles fewer devices
Check to see whether the internet connection at your site is good enough for your internet usage, including both concurrent StarLeaf video calls and other internet activity:
- You IT admin or ISP may be able to monitor the amount of internet bandwidth used throughout the day to determine if at any time the internet connection is overloaded. If your internet connection is not sufficient for the volume of the traffic, then speak to your ISP about upgrading your available bandwidth
The problem might be further upstream – within your ISP’s network or beyond.
A good tool to figure out which router within a network is dropping packets is WinMTR. You can download WinMTR from Source Forge .
The host that you should run it
against is your organization’s call.sl subdomain name – for example, example.call.sl.
This shows if any of the routers in the path between your network and StarLeaf is dropping packets.
Your ISP or StarLeaf can help further, when you have the results. Example screen shot:
If any of the values in the Loss column shows a significant percentage of packets being dropped, that is a possible cause of the problem.
If you experience problems with point-to-point calls between wired devices within the same site, then this indicates a problem with your local network.
Some StarLeaf endpoints set the DSCP. This tells your network to prioritize StarLeaf video over less critical traffic, therefore in general, we recommend enabling Quality of Service (QoS) on DSCP for your network equipment. However, it is also possible that bugs in your network equipment could incorrectly handle the DSCP, making it worse, so you could try disabling it as well. For more information, see Quality of Service with StarLeaf.
An example of a local network problem is an Ethernet duplex ismatch on a connection between devices within your network. This causes heavy packet loss on video calls, but may not be noticeable problems with other network applications. A mismatch could be found between a StarLeaf controller itself and the switch that it is connected directly to a core switch and a router.
If you are using a MPLS network, you must make sure that it has sufficient bandwidth to handle all of the video calls that go through StarLeaf.
By default, if two endpoints for a peer-to-peer call are directly routable, then they will attempt to connect to each other using StarLeaf’s direct media technology. If there is a MPLN network between the two endpoints, then this means that calls will be routed over that network, so you must ensure that it has enough bandwidth to handle video call traffic. If your MPLN network is bandwidth limited compared to your internet connection, then contact your reseller to disable direct media for your organization.