Auditing measurement accuracy

In a virtualized environment, the load generating capacity can be degraded significantly for throughput, CPU utilization, and measurement accuracy. For instance, in a cloud environment, measuring the response time can vary, depending on factors such as the location of data center, type of host, and life-span of agent virtual machines. It is difficult to get an accurate response time every time because not all factors can be controlled by HCL OneTest™ Performance. But, you can do a statistical comparison of the reported measurements against a trusted control.
Important: For a cloud run, it is very important to audit measurement results using the control techniques described in this topic.
HCL OneTest Performance provides two options to audit the reported response time accuracy:
  • Apply a known fixed delay with the unknown response times and observe deviations from the known correct value as a 'proxy' for response time measurement accuracy.
  • Distribute a subset of the full load on a non-compromised controlled agent virtual machine that runs in parallel and measure the deviations.

The first approach automatically generates an audit test in the cloud schedule. You can assign extra audit users to run the audit test on every control and load agent locations. These audit users are 'dummies' that do not apply load to the system under test. You do not have to account for them when filling out the Stage page of the wizard.

The second approach is recommended for auditing measurement accuracy because a control agent is expected to be run on a dedicated computer. For a control agent that is provisioned with BareMetal, the internet latency and congestion will be the same as the other cloud agents from the same data center. So, the differences will be only due to CPU utilization, memory, and Network Interface Card.
  • Cloud-based control agents cannot detect data center or physical host bandwidth bottlenecks.
Consider using an on-premises location for the control agent. For a control agent that is on-premises the internet latency and congestion factors count for response time. The number of virtual users in a control agent should be low enough to be able to have data to audit measurement, typically 10-20% of the total capacity. The load should not be more than 30% of the capacity. At the same time, the number of users should be high enough to get sufficient statistical sampling for each stage, typically 100 samples for every counter of interest.
Note: If some of the user groups have low percentage Group sizes, the number of control users running the tests is reduced. You can, however, counterbalance the reduction by running longer stage durations to get more samples for each user.