Benchmarking Speed TestersA comparison of Internet connection speed testers
In the table below you can see the results of the author's evaluation
of various widely-used Internet broadband speed testers:
|Speed Tester||Run nr 1||Run nr 2||Run nr 3||Run nr 4||Run nr 5|
|CNET Bandwidth meter 2||3.43/---||3.22/---||3.14/---||3.18/---||3.22/---|
'Mbps / Kbps' for various Internet speed testers
As discussed before, a credible speed tester must provide both consistent and accurate results. From the table above you can see that only MySpeed provided consistent and accurate test results.
Why are some speed testers not as accurate and consistent as compared to other testers? Unfortunately, many companies write their own bandwidth speed test without understanding how easy it is to create a speed tester which contains hidden incorrect assumptions, resulting in a speed test which produces inaccurate test results. Writing an accurate speed tester at a minimum requires:
- a detailed working knowledge of certain aspects of TCP/IP
- a knowledge of timer units vs accuracy in the programming language used
- an understanding of basic mathematical concepts, like precision
- a server with an appropriate connectivity for multiple simultaneous client speed tests
- an ability to validate one's own speed test results
Let's examine some of the speed testers and analyze the test results.
Evaluating Speed Tester Two
TIME = (time in seconds for image to download)There are a couple of issues with this code. The first is an error in the KBYTES (in 1000 byte units) of the JPEG image. It is 518.095, not the 500 seen in the code. The second is the 1.02 multiplier, which could be an attempt by the test author to account for TCP/IP header overhead, but as that multiplier is not accurate the true intent of this multiplier is not known. For testing purposes it is possible to create a set of images using an online image optimizer, and use the resulted jpegs to test the download of ligher and heavier images.
KBYTES = 500;
kbps = Math.round(KBYTES/TIME*8*10*1.02) / 10;
Timing how long it takes to download an image is a classic, well known and published speed test method. However, this technique is flawed in that it incorrectly assumes that:
- downloading a moderately sized image utilizes 100% of the download bandwidth immediately
The reason why is left as an exercise for the reader (requires a moderate understanding of how TCP/IP works).This initial under-utilization can clearly be seen in the bandwidth utilization over time graph in Fig.3, of the actual image download during this speed test. This initial short under-utilization is one of the reasons that this speed test is not accurate. This speed test assumes 100% bandwidth utilization at all times, when in reality that is not the case. The result (as can be seen in the table above) is a large error (because of the relatively small image file downloaded) by this speed tester in underestimating the true bandwidth speed (a speed of 3.2 Mbps reported for a 4.2 Mbps link).
Understanding Precision: Your boss gives you a digital thermometer which displays the temperature accurate to the degree. He asks you to calculate average temperature outside your office at high noon in May. Every day in May you take a measurement. At the end of the month you calculate the average and present your results to your boss. Do you report the average with any decimal digits, like 85.47°, or do you just report the temperature rounded to the degree, like 85°.
The answer is 85° because you can never have more digits of precision in a calculated answer than the minimum digits of precision in any value used to calculate the answer. So if all temperatures have two digits of precision, you can only present an answer with two digits of precision.
Another way of understanding the answer is to look at 85.47°, which is pretty much saying that the average could not have been 85.45° or 85.49°. But given the error possible in each daily temperature measurement, you really don't know that. So reporting 85.47° as the average temperature is implying a precision that does not exist.
Evaluating Speed Tester Three and Four
This speed test is making very similar incorrect assumptions that timing a moderately sized web page will provide you with a measure of your bandwidth. For all of the same reasons that Speed Tester Two is not accurate, neither is this speed test.
start = new Date().getTime();
(614405 bytes in the HTML)
end = new Date().getTime();
elapsed = (end-start)/1000;
KBps = Math.round(614405*100/(elapsed*1024)) / 100;
Moreover, this speed test reports a Mbps value that has been calculated incorrectly. The text on this speed tester's web page appears very clear on the issue of bits vs bytes:
- Kbps - kilobits per second, or thousands of bits per second
- KBps - kilobytes per second
- Mbps - megabits per second, or millions of bits per second
- MBps - megabytes per second
As you can see in Fig.4 (ignoring mathematical precision issues for
now), downloading 614405 bytes in 1.609 seconds is 372.91 kilobytes
per second, but that is not 2.913 megabits per second
(it is that many megabytes per second). It is instead 3.055
megabits per second (614405*8/1.609/1000000).
The bottom line is that Speed Tester Three/Four is not accurate, and the Mbps value reported to the end user is calculated incorrectly.
Evaluating Speed Tester Five
When the author ran some tests with this speed tester a couple of days later, the results seemed much better, but they were still were not accurate. After reviewing the HTML source code, the test download files were found.
Once again, when using an authenticated download speed program the download results were 4.23 Mbps. This confirms that this speed tester also incorrectly assumes 100% download utilization during the entire download.
Evaluating Speed Tester Six
When the author tested the download speed from this tester's server using a speed test verification program, results were mostly in the upper 3 Mbps range, sometimes above 4 Mbps, and once around 1 Mbps, confirming that this speed test is assuming 100% bandwidth utilization during the entire download, resulting in a bandwidth measurment that is a lot more incorrect than it should be. Also, since 4.23 Mbps results were not obtained every time (except once), this confirms that either the web server or network of Speed Tester Six is over utilized.
In Fig.6, this speed tester places "Cable" below that of T1, which is around 1.5 Mbps. Given that almost all cable companies provide cable modem download speeds above 3.0 Mbps, it appears that this speed tester was written some time ago and not updated to deal with today's true broadband connections.
How Accurate is Your Speed Tester?
The key benchmark for any bandwidth speed tester is: Can it deliver
consistent and accurate results for a highly accurate broadband connection
of known speed? If not, the speed tester cannot be trusted.
The results of this benchmark analysis determined that only the MySpeed speed tester provided accurate and consistent Internet bandwidth speed test measurements.
- Visualware MySpeed: 4,233,464/370,888, 4,233,336/370,912, 4,232,408/370,648, 4,232,984/370,600, and 4,232,008/370,440. These results are fully auditable and can be viewed in the online enhanced reporting database. See tests #190078, #190080, #190081, #190083, and #190084.
- CNET bandwidth meter: 3434.3, 3225.3, 3148.1, 3182.5, and 3225.3 kbps.
- McAfee Speedometer (restart via web browser restart): 2.858, 2.885, 2.856, 3.001, and 2.804 Mbps.
- McAfee Speedometer (restart via the 'retest' button): 2.856, 3.296, 4.223, 4.167, and 4.481 Mbps.
- speedtest.dslreports.com 3,359,457/368,459 3,093,440/370,851 3,329,373/369,480 bps from server 1 and 679,936/363,967 620,765/363,967 bps from server 2.
- Intel Broadband Speed Test: 732.2 Kbps, 767.9 Kbps, 1.3 Mbps, 339.4 Kbps, 700.6 Kbps.
- Client computer was a Windows XP Professional, 3.20 Ghz Pentium 4, 1.0 GB RAM system. The ethernet card (Realtek RTL8169/8110 Gigabit) was connected at 100 Mbps to a LinkSys 10/100 Workgroup Switch, connected to a LinkSys WRT54G router, connected to a LinkSys BEFCMU V3 cable modem, connected to Comcast; see 8 below.
- All tests were performed on a broadband cable modem with a known download speed right around 4.23 Mbps and a known upload speed right around 370 kbps. In the test environment, the cable network amazingly provides virtually 100% consistent bandwidth all the time, which is very unusual for a broadband cable system, but which is great for running benchmarks. This speed was validated by downloading and uploading large files. 51,726,024 bytes were downloaded in 97,786 milliseconds (4,231,800 bps) and 8,626,342 bytes were uploaded in 186.36 seconds (370,310 bps).