Docs > Troubleshooting

Troubleshooting

This section goes through some of the most typical special situations that the users can face when performing measurements with Qosium. Hints and solution proposals are given on how to proceed.

1. Dealing with a Failure #

Failing to get sensible measurement results is not a reason to feel ashamed. We do it too, on a rather regular basis. While we have worked hard to make Qosium as easy and intuitive to use, the concept of passive network measurement remains a challenging topic.

In practice, this means that eventually, you’ll run across an error notification, or the received measurement results are not as expected. When this happens, the most important step is to take a deep breath and orient yourself into problem-solving mode. The solution might be just around the corner, but hasty actions can lead you further away from it.

To start with, Qosium has some inbuilt intelligence for detecting incorrect parameterization. However, some parameterization errors are still not (yet) detected. For example, selecting wrong interfaces is something that Qosium does not recognize as an error. Thus, it helps a lot if you are well familiar with the use of Qosium, and, in general, the philosophy of passive measurements.

If something peculiar is happening in the measurement, start by considering the following:

What do I exactly want to measure?
Does Qosium even support my needs?
Have I built my measurement setup accordingly?
Have I parameterized Qosium accordingly?
Is it possible that the, in my opinion peculiar, results are, in fact, real?

Errors of measurement setup can often be detected by observing the measurement results. In particular, if the statistic Sent Info Not Found shows continuous non-zero values, it often is a sign of false parameterization. Also, duplicated packets can cause problems to the QoS calculation of Qosium. That is not necessarily related to false parameterization but to what you want to measure and how.

If it seems likely that Qosium is to blame, try first to disconnect the measurement controller and connect back. That reconstructs the whole measurement context and can help if something went wrong in the measurement process itself. Measurement controller restart can help in some rare cases, but Qosium Probe restart is hardly ever needed, but, of course, can be tried too. Reinstalling Qosium is not recommended since it can help only in some very rare license-related issues.

Try also studying both the log output of Scope and the console output of Probe. They can lead you to the source of the problem.

Next, we go through some common special situations. If you don’t find relief to your problem in this section, contact Kaitotek’s support.

2. Startup Problems #

2.1. Missing wpcap.dll (Windows) #

2.2. Qosium Probe Does Not Start #

Probable cause and solution

The got error means that Qosium Probe cannot set up its QMCP server. This happens if some other program has already booked the TCP port the Probe is trying to use. The most typical situation is that you already have Qosium Probe running in the machine. Thus, check it and check also that is Qosium Probe running already as a system service. If so, there is no need to start another instance of it.

Sometimes, especially in Linux-based systems, a premature termination of Probe execution may leave the socket up for a couple of minutes. This happens when the Probe is shut down by other means than CTRL-C while there are measurement controllers connected. If this was the case, try to start the Probe a bit later again.

If Qosium Probe is and was not running, it can be some other program using the same TCP port. Check from Qosium Probe’s log screen which port it is trying to use. Then use, e.g., netstat to check if another program has booked the intended Probe port already. You can change Qosium Probe’s port in the initialization file (QosiumProbe.ini).

2.3. License Activation Issue #

Probable cause and solution

It is very likely that Qosium Probe’s license activation is tried, but it fails. License activation will be carried out online when the license has ended or when you are using the Probe for the first time, having a license with a limited number of installation rights.

Check first the internet connection in the device where the Probe is installed. The problem can also be that there is a strict firewall in your organization blocking outgoing connections of Qosium Probe. You can contact Kaitotek support to discuss the details of the required license server connection. Please note that the license server access is required only during activation. During normal operation, no Internet connection is required.

You can also check if Qosium Probe’s log gives more thorough information on the issue.

3. Common Special Situations #

3.1. Negative Delay Values #

Probable cause and solution

It is very likely that the clock synchronization between the measurement points is not accurate. This problem is common when measuring high-speed networks without an accurate enough synchronization method. The effect of inaccurate synchronization causes a mirrored delay behavior, e.g., 10 ms delay to one direction but -10 ms to the other.

To verify this assumption, see in the average results view that the mirrored delay behavior exists. If so, the synchronization problem has been verified. The actual synchronization error can be evaluated by equation

synch_error = (|delay_ul| + |delay_dl|)/2 + delay_ul + delay_dl

As a solution, check first that your synchronization method is working as supposed to. If this does not help, try better synchronization methods (e.g., replace NTP with PTP or even GNSS).

If the average results show that the delay to both directions is negative, the measurement is set up incorrectly (the next solution proposal).

3.2. QoS Results Disappear #

Possible cause and solution 2

The QMCP connection between the measurement nodes might be broken or temporarily corrupted.

Qosium measurement controllers will detect a broken QMCP connection as long as Robust QMCP mode is in use. Thus, check your measurement controller’s status views and/or logs. Bear in mind that detecting a broken QMCP connection can take some time, e.g., 10 – 30 s. If a broken QMCP connection if verified to be the cause, see Giving Commands Results in Timeout to fix this.

If it seems that the QMCP connection is ok, check the packet loss statistic. If that shows 100 %, no other QoS statistics can be calculated. If you are sure that not all the packets are lost, try to increase the Loss Timer parameter value. It will give more time for packet loss detection (to separate it from highly delayed packets). Notice that increasing Loss timer will delay the packet loss detection accordingly.

3.3. No QoS Results #

3.4. Giving Commands Results in Timeout #

Possible cause and solution

Likely the QMCP connection from the controller or between the measurement points is broken. Typically, the measurement controller will inform about such situations, but sometimes it may not be possible (especially, if Robust QMCP mode is not activated). Check that the Qosium Probes are still up and running. Next, check that the connection to the Probe platforms is also ok. If so, you can try to disconnect and connect back the measurement controller. If the connection to be measured is generally of bad quality, consider routing the QMCP traffic elsewhere if just possible.

It is recommended to use the Robust QMCP Mode: it handles the QMCP connection reconnection automatically. Once the network problem is over, the connection will get back up unless the Robust QMCP timeout is exceeded.

3.5. Correct Packet Filter but No Packets #

3.6. Pcap Drops Packets #

Possible cause and solution

If you did increase the Pcap buffer already, but it didn’t help, the platform running Qosium Probe might not have enough resources to perform the intended measurement.

The higher is the packet rate of the measured traffic, the more it requires computational effort in the platform. In the Qosium Probe platform, take a look at the processes. That can be carried out in Windows by Task manager and in Linux by top command. If the Qosium Probe process consumes a lot of processor capacity, likely, it is not able to process the measured traffic fast enough, leading to drops.

Consider if you can filter the measured traffic part smaller to relieve the burden of the Probe. If not, consider moving the Probe to another platform having enough computational resources for the measurement.

3.7. Excessive Qosium Control Traffic #

3.8. New Interface Not Appearing #

3.9. Missing Interface #

3.10. Mouse Issue with GNSS in Windows #

4. Inconsistent Statistics #

4.1. The Statistic Sent Info Not Found Grows #

Possible cause and solution: Altered traffic

Sometimes the network path is such that it alters the content of the packets. For example, a video stream can go through a device that transcodes the video, e.g., to a higher compression level to save bandwidth. This will cause a mismatch of the packet context between the measurement endpoints, and the QoS measurement will fail. If this is the case, the only way to proceed with the measurement is to disable the transcoding (or similar packet content alteration) or to perform the measurement in two separate parts.

Then, it could be that there is a device in the network path that has a smaller MTUMaximum Transmission Unitm
This determines the largest packet size that can be communicated in a single network-layer transaction. than the traffic endpoints. That can cause a single large packet to be segmented as multiple smaller ones. However, depending on the NIC’sNetwork Interface Card
A piece of hardware which offers a device a networking interface. capabilities, the segmentation can be masked from the measurement perspective, causing no problems. If not, however, the QoS measurement will fail at least partially. In such a case, the solution is to lower the MTU in the communicating endpoints in such a way that packet segmentation does not occur. Unfortunately, typically, the only way to do this is just to test which MTU value is low enough. Please note that MTU, generally, should not be set too low since it can cause inefficiencies in network resource usage.

4.2. The Statistic Duplicate Packet Count Grows #

Probable cause and solution

Most likely, there is nothing wrong with such, but it is just that the traffic you measure has interesting properties. By default, Qosium identifies packets based on payload. Thus, if the payload from one packet to another is exactly the same, Qosium counts them as duplicates. That is normal – some applications just generate similar content to packets in some situations. Even normal TCP connections can cause duplicate packets. If information is lost, TCP resends the lost data, appearing as a duplicated packet for Qosium, if the packet context matches suitably.

Since Qosium Probe version 1.4.15.0, QoS will not be calculated for duplicated packets any longer. That is because there is a danger that QoS will be calculated incorrectly especially if packet losses are occurring over the measured network path. Thus, if you are using newer Qosium versions, this explains the absence of QoS results, and with an older one, the mentioned strange QoS results are likely caused by this. Remember, however, that this happens only for those packets that are duplicates. Other packets are calculated normally. In addition, if the duplicate packet interarrival time » Qosium’s Packet loss timer, they are calculated normally as individual packets.

If you wish to measure QoS also for the duplicated packets, try different Packet identification methods. For example, if the traffic is of IPv4, using the IPv4 ID often relieves this problem. Be aware, however, that if you measure high traffic loads, the IPv4 ID field rolls over within Qosium’s averaging interval, and, again, there will be duplicated packets. Moreover, some (fortunately rare) applications do not respect the IPv4 ID field’s purpose. Instead, they always put the same value to this field. As understood, in this case, the packet identification will fail.

It is also possible that a network device is operating incorrectly, causing duplicates. In this case, changing the Packet identification method makes no sense. Instead, Qosium provides you valuable information about a misbehaving network device. Thus, try to locate the device and see what is wrong.

4.3. High Packet Losses #

Glossary >

Virtual LAN

A LAN seqment which has no physical hardware, but instead is carried over network.

Maximum Transmission Unitm

This determines the largest packet size that can be communicated in a single network-layer transaction.

Network Interface Card

A piece of hardware which offers a device a networking interface.

Troubleshooting

Contents

1. Dealing with a Failure #

2. Startup Problems #

2.1. Missing wpcap.dll (Windows) #

2.2. Qosium Probe Does Not Start #

2.3. License Activation Issue #

3. Common Special Situations #

3.1. Negative Delay Values #

3.2. QoS Results Disappear #

3.3. No QoS Results #

3.4. Giving Commands Results in Timeout #

3.5. Correct Packet Filter but No Packets #

3.6. Pcap Drops Packets #

3.7. Excessive Qosium Control Traffic #

3.8. New Interface Not Appearing #

3.9. Missing Interface #

3.10. Mouse Issue with GNSS in Windows #

4. Inconsistent Statistics #

4.1. The Statistic Sent Info Not Found Grows #

4.2. The Statistic Duplicate Packet Count Grows #

4.3. High Packet Losses #

Further Reading

Glossary >