VoIP Stage 2: Results
During evaluation we thoroughly inspected the source code of changes proposed by the contestants. Please refer to Сontest.com for short summaries and our comments on each of the submissions.
To test call quality, we launched two different instances of each library between 2 different hosts. Custom reflectors (if needed) were launched in the same network. The following network conditions were used (we included some of the conditions that the contestants used to test their libraries):
| Alias | Conditions |
|---|---|
| WiFi | ->networkType(‘wifi’) |
| 3G1 | ->loss(9, 20)->rateControl(‘44kbit’)->networkType(‘hspa’) |
| 3G2 | ->loss(17)->rateControl(‘29kbit’)->networkType(‘3g’) |
| 3G3 | ->loss(12, 3)->rateControl(‘32kbit’)->networkType(‘3g’) |
| 3G4 | ->loss(18)->rateControl(‘32kbit’)->networkType(‘3g’) |
| 3GDelay | ->loss(17, 5)->delay(500, 50)->networkType(‘3g’) |
| 3GOutage | ->loss(3, 10)->rateControl(‘64kbit’)->after(3)->loss(20)->rateControl(‘8kbit’)->after(3)->rateControl(‘64kbit’)->networkType(‘3g’) |
| EDGE1 | ->loss(11)->rateControl(‘24kbit’)->networkType(‘3g’) |
| EDGE2 | ->loss(15, 5)->rateControl(‘19kbit’)->networkType(‘edge’) |
| GPRS1 | ->loss(20, 5)->rateControl(‘17kbit’)->networkType(‘gprs’) |
| GPRS2 | ->loss(19, 5)->rateControl(‘14kbit’)->networkType(‘gprs’) |
| GPRS3 | ->loss(40, 5)->delay(500, 50)->rateControl(‘8kbit’)->networkType(‘gprs’) |
To get numerical ratings, we used tgvoiprate provided in the tgvoip-test-suite, as well as some other raters from the winners of the first round. The final rating for each call was a weighted sum of several ratings (1.0-5.0). Failed calls resulted in the score 1.0.
Mean scores for each contest entry by Network Alias:
We also collected and analyzed traffic and latency metrics in a similar way. They might help further improve the library or show the cause of failure. Full analysis was performed in Jupyter Notebook which is provided here: PDF Version 1.1MB | HTML Version 2MB
For your convenience, raw CSV and the Notebook source are also provided here:
Download Call Metrics, Ratings and Notebook: VoIP-Stage2-Dataset.tar.gz (2MB)
Audio samples are provided in a separate archive due to its large size:
Download Call Audios: VoIP-Stage2-Audios.tar.gz (2.7GB)