Troubleshooting Guide for unstable network connection
This troubleshooting Guide is targeting Axis customers to be used as a self-help. The guide should help our customers to find possible mitigations to common issues they might encounter. The guide covers the data as well that is needed to be collected and provided when opening a support case. This should help the Axis Support Team to quickly assist them with the issue they are facing by providing them the correct and needed information.
Axis cameras will operate as intended if installed as specified by the Installation Guide. Generally, the dome of the device should be facing the ground when operating or testing the devices.
Please note that Axis does not take any responsibility for how the modification of configuration described in this guide may affect your system. If the modification fails or if you get other unexpected results, you may have to restore the settings to default.
2. The Infrastructure
The network is the most unpredictable source of disconnections. Switches, routers, cables proxies…everything in the network between senders and receivers will at some point affect the connection.
2.1 Network Management
Unpredictable factors can cause issues in the network.
The link between the camera and view device needs to have a guaranteed throughput.
- In a LAN (Local Area Network), this could be done by making sure there are as few hops as possible in the link. The link should not by shared with other traffic such as Voice over IP (VoIP) or other protocols that will be prioritized over video by default, or other demanding services that will overload the link.
- If the link is over a WAN (Wide area network), the QoS needs to be guaranteed in each hop, i.e. routers and switches. This can also be accomplished by leasing a point-to-point route through your local internet provider.
- Use of shielded cables is recommended. Depending on the application, cables can be adversely affected by EMI/RFI/ESI (electromagnetic interference, radio frequency interference, electrostatic interference) also known as ‘signal interference.’ Insulation alone provides no protection from signal interference – so to combat the effects of signal interference, proper shielding is vital.
Network video products utilize network bandwidth based on their configuration. If you are requesting a high-quality video using a very low compression, the bitrate will increase sensibly. If the bandwidth is limited, the device will have to compensate the quality of the stream (reduce bitrate) to match the available network infrastructure bandwidth, in some cases it can cause packet loses.
- In a small surveillance system involving 8 to 10 cameras, a basic 100-megabit (Mbit) network switch can be used without having to consider bandwidth limitations.
- When implementing 10 or more cameras, the network load can be estimated using a few rules of thumb:
a) A camera that is configured to deliver high-quality images at high frame rates will use approx.
2 to 3 Mbit/s of the available network bandwidth.
b) With more than 12 to 15 cameras, consider using a switch with a gigabit backbone. If a gigabit-supporting switch is
used, the server that runs the video management software should have a gigabit network adapter installed.
3. Device Overload
The streaming might be interrupted if the device crashes due to an overload. The following are some of the factors that can cause a device overload:
3.1 Factor 1 - Chipset Performance
It is hard to predict exactly how many simultaneous, individually configured streams with maximum resolution and full frame rate a video product(chipset) will deliver in all situations. The overall performance depends on:
- CPU load or video compression load (Depending on Chipset) in the video product
- Total data throughput (bandwidth) from the product and network infrastructure
- Client performance
- The complexity of the image
- The lighting conditions
- From a user perspective, the following items are the most common parameters that could affect the overall performance negatively:
- High image resolution
- Low image compression level
- Mixing Motion JPEG and H.264/H.265 streams
- Many clients accessing the server simultaneously
- Clients accessing different image settings (resolution, compression, etc.) simultaneously
- Heavy usage of event settings
- Motion detection enabled
- Poorly performing client PC which does not decode all images
- Limited or poor network infrastructure. Frames will be dropped when network is congested.
- When the image scene, compression level, and compression method are kept constant, an increase in resolution will decrease the number of full frame rate streams or cause the frame rate per stream to decrease.
- Educating about the chipset will help you understand the limitations: If we take an ARTPEC-7 as an example. The test results for that chipset shows that it can deliver almost six simultaneous H.265 streams in HDTV 1080p at 30 fps, or two H.265 streams in 4K at 23 fps, or one H.265 stream in 3712x2784 at 20 fps. Alternatively, it can deliver one H.265 stream in 4K at 30 fps and one H.265 stream in 1080p at 30 fps, simultaneously.
3.2 Factor 2 - Several distributed or edge analytics
When there is a lot of ACAPs used the CPU processing is at the edge and may affect heavily the performance of the device and cause unwanted crashes/interruptions. This issue can also be seen when an ACAP is overused and when you add several high-quality streams to the mix which can also cause unwanted crashes and interruptions.
- Limit the number of ACAPs used.
- For Multisensor/Multi Channel cameras like AXIS F series and AXIS P3707-PE products, if motion detection is required on several channels, it is recommended to use the embedded Motion Detection (sometimes referred as VMD1) instead, for the desired channels. VMD1 requires less processor power.
3.3 Factor 3 - High-frequency HTTP requests
Incorrectly configured Video Management Systems sending non-stop requests that the devices cannot handle, and the resources used by the devices.
- Isolating the camera from the VMS would be in this case the first step in troubleshooting.
- Once the camera is isolated then there will be a need for a test which includes pulling a stream from the camera as simple as using the live view from the web interface of the device.
4. Troubleshooting Using Wireshark or Vapix Commands
The preferred application for taking a network trace to attach to a support case is Wireshark. If you don't have it installed on your PC it can be downloaded for free from http://www.wireshark.org . Wireshark gives you the possibility to filter traffic both when recording and reviewing it. Both can be useful in different situations.
Additionally, when troubleshooting connectivity issues with Axis devices (firmware v. 5.x or later), the following Vapix commands can also be issued via a web browser:
— Generates a packet capture file taken from the device itself, for the specified amount of time in seconds. The capture is in standard *.pcap format, so it can be reviewed in Wireshark.
— Displays a list of currently open TCP/UDP connections/ports.
— Pings the specified IP address once and then reports the result.
5. Opening a support case
It might happen that the above factors and mitigations do not help identify and solve the network issues. You are welcome to open a support case and provide the below needed information and data.
5.1 Data and Information Needed
NOTE: Please do not forget to add the Server Report of the device!
- A detailed description of the problem and when it occurs and how often?
- Was there an internal investigation of the network? Network devices?
- A description of the network and if possible, a network topology drawing?
- Detailed information about all the devices on the network?
- Firewall and antivirus suites and if anything, has been updated?
- Compare with other working cameras installed?
- Wireshark from the failing and other working units in the network?
- Check if the problem is the same if the unit is isolated from the network?
- Information about how the unit is powered.
- The name of any third party (VMS,EMS,…) application in use?
- If it’s an old or new installation, and if any changes have been made in the network?
[Please contact technical support with all questions answered and files attached]
Considerations and limitations
The guide does not cover all possible issues but rather the most seen through the support cases. As an Axis Customer you are always welcome to contact Axis Support whenever you need with any type of issues listed or not in this troubleshooting guide.