Identifying causes of packet loss in StarLeaf calls
Managing Troubleshooting the CloudLast updated October 9, 2018
Live video calling is a demanding application for any network. For first-time video customers, it can highlight problems in their local network or Internet connection that have so far gone unnoticed. There are two main differences between live video network traffic and “traditional” network traffic (web surfing, email, file downloads etc). One is the sustained high bit rates that video calling requires. The other is the requirement for very low latency (i.e. minimal delay in packets getting from one endpoint to the other). When viewing a web page, a couple of seconds delay while some HTML gets retransmitted by the server has no effect on the content that you see.
On a video call, a half second of latency is enough to reduce your conversation to “walkie talkie” level, and the loss of three or four consecutive audio packets means that the speaker gets asked to repeat themselves, causing frustration all round. Any form of voice or video over IP communication requires the network to deliver the packets quickly; so you need a good Internet connection. For the best quality StarLeaf calls, your Internet connection needs to provide 1.5 mbps of bandwidth upstream and downstream for each concurrent call that you want to be able to have, over and above bandwidth in use by your existing network applications. So to have 6 concurrent calls: 6 x 1.5 = 9Mbps of network bandwidth in each direction.
StarLeaf endpoints use a speed testing mechanism when registering with the StarLeaf Cloud service to choose the right bit rate for the bandwidth of the local network and Internet connection they are connected to (for more details about this refer to Video bandwidths and resolutions used by StarLeaf Cloud). However, this mechanism isn’t infallible. Network configuration problems or temporary congestion can mean that the network does not have enough capacity to transport all the media packets being generated by StarLeaf endpoints, or received from far end endpoints via the Internet. When this happens, the “excess” packets are discarded or delayed by routers and switches in the network. The packets either never arrive at their destination or arrive there so late as to be useless. This is packet loss.
The recipient of a media stream that is subject to packet loss experiences anything from a slight degradation of media quality, through media that is so garbled, robotic or frozen as to be unusable, to unexpected call drops. When packet loss is extreme, it even affects StarLeaf endpoints that are not on calls: responsiveness of phone user interfaces might be slow; they might be seen deregistering from the StarLeaf Cloud service, restarting, or displaying a flashing exclamation mark symbol at the top left of their screen.
How badly packet loss affects a StarLeaf call depends on the percentage of media packets being lost. This table describes what the recipient of a lossy media packet stream might experience on a call between two StarLeaf Personal Telepresence systems.
|Loss||Phone behavior||Audio||Video||Screen shot|
|Moderate (0-5%)||Normal||Very slight “roughness” to the audio, hardly noticeable||No discernible difference|
|Heavy (5-20%)||Response to key presses is slow. Calls take longer to connect and establish media||Possible to maintain conversational flow, but sound is interrupted by noticeable breaks that require the speak to repeat themselves||Slight blockiness in areas of high motion. Static picture looks fine.|
Hard to use the phone interface
Phone might go offline for periods of time.
Connected calls might drop.
|Speaker has to be frequently asked to repeat themselves||“No remote video” message at start of call. Video might establish after several seconds but frequently freezes. Video in an established call is smeary|
|Extreme (30%+)||It is not possible to connect calls. Phones might deregister from the StarLeaf Cloud service. In-progress calls drop.||Unusable. Periods of complete silence interrupted by short bursts of tinny sound.||Heavily broken, frozen image in call; StarLeaf hardware endpoints might display “No remote video” Breeze might switch to audio-only style view||No remote video|
Watch the packet loss symptoms video here:
If you or the people you’re talking to on video are experiencing the symptoms described above, the first thing to do is “double-tap” the Home key on your StarLeaf touchscreen, or click the “graph” icon in Breeze, while the problem is actually happening:
The “Transmit” column refers to the stream of packets going from your Telepresence unit or Breeze client to the far end, and “Receive” refers to the packets that the far end is sending you.”Cumulative Loss” shows the number of packets of each type (audio, video, and, optionally, content) lost since the start of the call. “Current Loss” shows the percentage of packets lost in the last 5 seconds of the call. In conditions of high latency, it might take a few seconds for these numbers to actually reflect the real video/audio conditions being experienced.
If you are experiencing media problems and the Call Statistics show significant numbers in the Cumulative Loss and Current Loss values for the “problem” media stream, then packet loss is the cause. If the “current packet loss” statistic is zero, as in the screen shot above, but noticeable media quality problems are still happening, the problem is likely a Breeze local device issue (a problem with the camera/microphone/speaker setup or PC capability of whoever is sending the bad media) or a third-party interop problem (a bug in the way that StarLeaf Cloud and video endpoints from other manufacturers decode each others’ media streams). Local device issues can be worked around by trying different devices; Breeze has device selection options at the bottom right of its window. If a third-party interop problem is suspected, we at StarLeaf would like to hear about it – you can contact us via starleaf.com/support or through your reseller.
After you’ve established that packet loss is responsible for your call quality problems, you need to work out which end of the call is causing the problem.
- If the problem call involved two endpoints on the same site (i.e. on the same Local Area Network) it is possible that the bottleneck was that site’s Internet connection. The default setting is for all media between StarLeaf endpoints to be sent out across the Internet to the StarLeaf Cloud service, so this requires double the bit rate of a single StarLeaf call to be sent upstream and downstream through your Internet connection.
- If the problem call was between two different sites, both endpoints involved should make test calls to a StarLeaf endpoint in a completely different location (for example, your StarLeaf reseller or StarLeaf Support). The problem is likely to be on the network of whichever endpoint also shows packet loss on the test call (it could be that both sites have a problem).
When you know which site has the problem, examine the local network and Internet connection on that site to identify possible causes. You probably need to enlist the help of your IT department or provider at this stage – their knowledge of your networking setup may help. Here are some general points to check:
- Is the Internet connection at your site good enough for the number of video calls that you are trying to place? You can check out the speed of your connection by going to https://speedtest.net . If the connection is not performing at the expected speed, contact your Internet Service Provider.
- If the packet loss problem is intermittent, could anything have been going on (for example, a large number of PCs backing files up to the Cloud) that would have taken up a lot of network capacity at the time you were having problems on your video calls?
If either of the above is true, you might need to consider whether your current Internet connection is sufficient for the amount of video calling that you need, and upgrade it if necessary.
- Do all StarLeaf endpoints in your network have packet loss problems? If not, the problem could be localized within the network. A good example of this would be an Ethernet duplex mismatch on a connection between devices within your network – this causes heavy packet loss on video calls, but might not cause noticeable problems with “traditional” network applications. A mismatch could be found between a StarLeaf controller itself and the switch that it is connected directly to, or deeper within your network, say between a core switch and a router.
- Do you have an MPLS network? This introduces other factors. Not only should the network connection at your local site have sufficient bandwidth for your StarLeaf calls, but the connection between the MPLS network and the Internet, which could be in a different geographical location, also needs to be adequate. So MPLS network needs to provide sufficient bandwidth to allow the number of concurrent StarLeaf calls that you want to have (using the same calculation as in the ‘Introduction’ section).
The problem might be further upstream – within your ISP’s network or beyond.
A good tool to figure out which router within a network is dropping packets is MTR.
Windows users can download a version here – https://sourceforge.net/projects/winmtr/ . The host that you should run it
against is your organization’s call.sl subdomain name – for example, example.call.sl.
This shows if any of the routers in the path between your network and the StarLeaf Cloud service is dropping packets.
Your ISP or StarLeaf can help further, when you have the results. Example screen shot:
If any of the values in the “Loss” column shows a significant percentage of packets being dropped, that is a possible, although not a guaranteed, cause of the problem.