Info

Download ZIP (41 MB)

Testing and Issues

You can test this entry and submit issues during the testing period of the VoIP Contest, Stage 2. $75,000 contest.

Entries with serious issues will not be able to win the contest, but even minor issues might be important for overall results.

Voting

Comments

@Fair Wombat one more note - you can not run 2 or more callers as well as 2 or more callee at the moment. There are hardcoded caller/callee tokens, signaling server will reject new connections with the same online tokens.

2020.01.30 20:51:01.108: [tgwss] [THR#2] [DEBUG] wscbService: data received, client 0x7f0c3c0092d0, message: {"type":"logon","token":"callee_123456789"}, size 43, remaining size: 0
2020.01.30 20:51:01.108: [tgwss] [THR#2] [DEBUG] closeWithErrMsg: {"error": "'token' is already online"}
@Fair Wombat
it looks like you are running caller without any online callee.
logs from my singling server - 
2020.01.30 20:50:10.982: [tgwss] [THR#2] [DEBUG] wscbService: data received, client 0x7f0c3c000b20, message: {"type":"logon","token":"caller_123456789"}, size 43, remaining size: 0
2020.01.30 20:50:10.982: [tgwss] [THR#2] [DEBUG] write: client 0x7f0c3c000b20, message: {"type": "logon", "status":true}, size 48
2020.01.30 20:50:11.039: [tgwss] [THR#2] [DEBUG] closeWithErrMsg: {"error": "'token' is offline"}
You have not added any comments yet...
by rating

Issues

Fair Wombat Mar 9, 2020 at 19:24
A promising implementation of an ambitious task.
Huge room for improvement in terms of audio quality.

We expect the contestant to take part in round 3. Suggested priorities:
1. Focus on improving audio quality on different connections.
2. Build library for Android (real device I/O etc).
3. Support updated tgvoipcall #issue10339 , correct support for preprocessed callback #issue10597, remove 5 seconds wait during init #issue10337, fix periodical freezes during call #issue10737.
Debian 5.2.9
10
Fair Wombat Jan 30, 2020 at 19:34
Is signalling server up now? I get

Timeout while establishing the connection

every time, while it seemed to work about 3 hours ago.

tcpdump shows packets in both direction with 35.214.230.40.8080 TCP, but nothing over UDP or 3478.

Is there any way to enable debug output for library?
Debian 5.2.9
Bossy Gnu Jan 30, 2020 at 21:34
It works for me and I can not understand where is the problem. There are two options. 

1. Please download my submission with enabled debug messages - https://drive.google.com/open?id=1IoHzQ2LngP7p1y1Z6s7_XQ_sZuZxyIcN 

2. You could comment out lines 48 & 50 at src/tgvoip/tgvoip/webRTCPeer.cpp and lines 58 & 63 at src/tgvoip/tgvoip/TgVoip.cpp. You will need to rebuild the project following the README.md instruction at src/tgvoip/README.md. Sorry for the inconvenience...

3. Look at my other comments please
Fair Wombat Jan 31, 2020 at 13:35
Managed to make signalling work thanks to the debug output.
Now it seems that it relies on P2P connection, which was forbidden by rules (https://contest.com/docs/voip2#notes) and doesn't work. Is it intended? is it possible to disable P2P?

Log looks like this:

...
[133:945] [21626] (connection.cc:454): Conn[5c0e3690:0:Net[eth0:142.93.96.x/20:Ethernet:id=1]:VSqWLjsL:1:0:local:udp:142.93.98.x:58630->FuUPzR7Z:1:1853693695:prflx:udp:35.214.230.x:53961|CRxW|-|0|0|7961553801070525951|-]: Received STUN ping, id=6443723133586c3745345942
[133:946] [21626] (connection.cc:646): Conn[5c0e3690:0:Net[eth0:142.93.96.x/20:Ethernet:id=1]:VSqWLjsL:1:0:local:udp:142.93.98.x:58630->FuUPzR7Z:1:1853693695:prflx:udp:35.214.230.x:53961|CRxW|-|0|0|7961553801070525951|-]: Sent STUN ping response, to=35.214.230.x:53961, id=6443723133586c3745345942
[135:251] [21621] (audio_device_buffer.cc:432): [PLAY: 10003msec, 48kHz] callbacks: 992, samples: 476160, rate: 47602, rate diff: 1%, level: 0
[136:320] [21626] (tcp_port.cc:478): Conn[5c0c5370:0:Net[eth0:1:10.19.0.x/16:Unknown:id=2]:MeIxT3y8:1:0:local:tcp:10.19.0.x:52583->lhQ7/uO3:1:1518083839:local:tcp:142.93.98.x:59005|---W|-|0|0|6520120444127690239|-]: Connection closed with error 110
[136:320] [21626] (connection.cc:1092): Connection deleted with number of pings sent: 0
[136:320] [21626] (p2p_transport_channel.cc:1943): Channel[0|1|RW]: Removed connection 5c0c5370 (6 remaining)
[136:321] [21626] (p2p_transport_channel.cc:1755): Channel[0|1|RW]: Transport channel state changed from 1 to 2


Here connection 10.19.0.x (internal ip address) -> 142.93.98.x (external ip address) looks odd.
Debian 5.2.9
Bossy Gnu Jan 31, 2020 at 14:22
P2P mode is disabled. My coturn sever runs with "--no-stun" flag. All STUN requests are ignored. So all traffic goes through TURN relay on 35.214.230.40. Please look at https://drive.google.com/file/d/1D_wPRRfL9oUSKbqrYYzP9Y9FquuoGD12/view?usp=sharing (my coturn's log)

Also you see on your side log lines like:
[005:954] [9731] (connection.cc:282): Conn[f5016200:0:Net[en0:192.168.88.x/24:Unknown:id=1]:4WZODoDQ:1:0:relay:udp:35.214.162.x:65362->N7MBxd3N:1:2122260223:local:udp:192.168.88.x:49815|C--W|-|0|0|179896594928123390|-]: Connection created
[005:954] [9731] (connection.cc:662): Conn[f5016200:0:Net[en0:192.168.88.x/24:Unknown:id=1]:4WZODoDQ:1:0:relay:udp:35.214.162.x:65362->N7MBxd3N:1:2122260223:local:udp:192.168.88.x:49815|C--W|-|0|0|179896594928123390|-]: Connection pruned

You can change TURN settings at lines 156-158 of src/tgvoip/tgvoip/webRTCPeer.cpp
Fair Wombat Jan 31, 2020 at 14:55
For now I use the .so file with debug you provided in #issue10325, I suppose it doesn't need to change turn settings if I use your reflector.
Found another interesting piece of logs:

[005:181] [25111] (connection.cc:282): Conn[c801b5c0:0:Net[v-peer1:10.201.202.x/24:Unknown:id=3]:M9MhHXN6:1:0:local:udp:10.201.202.x:33446->n2zCqs3a:1:2122260223:local:udp:10.203.204.x:40478|C--W|-|0|0|9115038255631187454|-]: Connection created, total: 1
[005:181] [25111] (p2p_transport_channel.cc:1755): Channel[0|1|__]: Transport channel state changed from 0 to 2
[005:181] [25111] (jsep_transport_controller.cc:1428): 0 Transport 1 state changed. Check if state is complete.
[005:181] [25111] (jsep_transport_controller.cc:1428): 0 Transport 1 state changed. Check if state is complete.
[005:181] [25111] (p2p_transport_channel.cc:1566): Channel[0|1|__]: Have a pingable connection for the first time; starting to ping.
[005:182] [25111] (connection.cc:1044): Conn[c801b5c0:0:Net[v-peer1:10.201.202.x/24:Unknown:id=3]:M9MhHXN6:1:0:local:udp:10.201.202.x:33446->n2zCqs3a:1:2122260223:local:udp:10.203.204.x:40478|C--W|-|0|0|9115038255631187454|-]: Sent STUN ping, id=4e726a692b79552b39435554, use_candidate=1, nomination=05] (webRTCPeer.cpp:288): onCandidate: ICE candidate added
[005:183] [25111] (turn_port.cc:1413): Port[c801ad60:0:1:0:relay:Net[v-peer1:10.201.202.x/24:Unknown:id=3]]: Received TURN allocate error response, id=6b456e593935574852737a36, code=401, rtt=1560:0:1:0:relay:Net[v-peer1:10.201.202.x/24:Unknown:id=3]]: TURN allocate request sent, id=53544d52336455486a567168rn_port.cc:1367): Port[c801ad60:0:1:0:relay:Net[v-peer1:10.201.202.x/24:Unknown:id=3]]: TURN allocate requested successfully, id=53544d52336455486a567168, code=0, rtt=14: Port[c801ad60:0:1:0:relay:Net[v-peer1:10.201.202.x/24:Unknown:id=3]]: Gathered candidate: Cand[:2567287068:1:udp:41885439:35.214.162.x:60052:relay:142.93.98.42:33446:zfMK:RxabOZjWVbbSWSqJ6zs08iQS:3:50:0]n:id=3]]: Port ready.
[005:197] [25111] (physical_socket_server.cc:557): Socket::OPT_DSCP not supported.
[005:197] [25111] (p2p_transport_channel.cc:830): Port[c801ad60:0:1:0:relay:Net[v-peer1:10.201.202.x/24:Unknown:id=3]]: SetOption(5, 0) failed: 005:198] [25111] (turn_port.cc:1636): Port[c801ad60:0:1:0:relay:Net[v-peer1:10.201.202.x/24:Unknown:id=3]]: TURN create permission request sent, id=6870544f64454e6e4b6c4d69cc:282): Conn[c801e2d0:0:Net[v-peer1:10.201.202.x/24:Unknown:id=3]:kmGCzXFp:1:0:relay:udp:35.214.162.x:60052->n2zCqs3a:1:2122260223:local:udp:10.203.204.x:40478|C--W|-|0|0|179896594928123390|-]: Connection created2, total: 2



Namely:
Received TURN allocate error response, id=6b456e593935574852737a36, code=401.
Any idea why this error can happen?
Debian 5.2.9
Bossy Gnu Feb 1, 2020 at 00:27
I think these are rejected STUN requests.

And you are right, WebRTC is trying to use "localhost" or "host" by default, does not matter "relay" connection is established or not. It was a surprise to me, it's my bad, sorry. I was testing my submission from machines without a direct access to each other networks...
"localhost" and "host" ICE candidates must be ignored, there is a simple patch - kindly add the following code to src/tgvoip/tgvoip/webRTCPeer.cpp, at line 283:

if (iceCandidate->candidate().type() != "relay") {
RTC_LOG(INFO) << "onICE: candidate ignored, not a relay";
return true;
}

Also you can download the updated submission here - https://drive.google.com/open?id=1lW5e3BEN3hQqDmbB-AedvG-DTw9deWUk

I've tested everything with Wireshark and now I am sure P2P is 100% disabled.

Thank you in advance.
Fair Wombat Feb 1, 2020 at 16:15
#issue10333 I tried to patch and rebuild as you described. Now the output PCM files seem to be generated. But the call doesn't stop after input file duration is reached. I waited about minute for 6 seconds file. Here are the logs for caller and callee.

https://drive.google.com/file/d/1bLHkCnOXhzQ4uJH8JIPShJk7QsSPE3St/view?usp=sharing
https://drive.google.com/file/d/1LifK2QYbvmEe4VNXZ_xxxjNoE33KdWvP/view?usp=sharing

Can you check if you see any problems here?

Also it seems the call freezes for about 5 seconds in the beginning according to log:

[000:000] [960] (wsClient.cpp:53): wsClient: initializing SSL/TLS...
[000:000] [960] (wsClient.cpp:56): wsClient: launched
[005:002] [960] (wsClient.cpp:96): wsClient: SSL/TLS connection will be used
[005:062] [965] (wsClient.cpp:172): cbService: connected to server
Debian 5.2.9
Bossy Gnu Feb 2, 2020 at 16:20
I've fixed some synchronisation issues on "callee" hangup event. You can find the updated submission here - 
https://drive.google.com/open?id=109YVHBNk6Ukl1Se_fbnO9Xp9oWlL5DNo

I hope it's the final issue, thanks for your work.

Regarding 5 sec delay, please look at src/tgvoip/tgvoip/TgVoip.cpp lines 78-80. I used initializationTimeout from TgVoipConfig to be sure "callee" has enough time to register on the singling server.
Fair Wombat Feb 3, 2020 at 11:49
#issue10337 Still with updated library the call doesn't stop automatically.
Debian 5.2.9
Bossy Gnu Feb 3, 2020 at 13:35
Make sure please you are testing the latest version: https://drive.google.com/open?id=109YVHBNk6Ukl1Se_fbnO9Xp9oWlL5DNo

- the latest libtgvoip.so MD5 sum:
$ md5sum ./libtgvoip.so 
4d97b39670101ffe2b1786c1b5b2ddaa  ./libtgvoip.so
- the latest tgvoipcall MD5 sum:
$ md5sum ./tgvoipcall
5dfdfd6dc92d1ef78dcdad752c9844ad  ./tgvoipcall
- correct dynamic linking:
$ ldd ./tgvoipcall
...
libtgvoip.so => ./libtgvoip.so (0x00007ff94d36f000)
...

I've tested the latest version on a few hundreds calls without any issues.

My test script for your reference: https://drive.google.com/open?id=1-sjUaLbJnBVeZFxkHCqSAeRddSQE5y4P
Fair Wombat Feb 4, 2020 at 19:06
#issue10339 Was fixed with your custom tgvoipcall. Now it works.
Debian 5.2.9
Fair Wombat Feb 5, 2020 at 10:59
This is not an issue, just a request. If there were any specific network conditions which you used to optimize the initally provided library, you are welcome to share test sequences as well as your results. e.g.:

->loss(12)->rateControl('32kbit')->networkType('3g')
Stable version: 3.0
My submission: 3.3

->loss(3)->rateControl('64kbit')->after(3)->loss(20)->rateControl('8kbit')->after(3)->rateControl('64kbit')->networkType('3g')
Stable version: 3.2
My submission: 3.4

We will re-check your results and may use the sequences you provide to test other submissions.
Debian 5.2.9
Bossy Gnu Feb 5, 2020 at 15:59
There is nothing special related to network conditions.

I found out "tgvoiprate" utility calculates a lot of wrong similarity rates. So I used Dejavu library in my calculations - https://github.com/worldveil/dejavu
I do not have a redy to use script, my idea was: create a "fingerproints" DB from the original samples, get libtgvoip's output files and find "confidence" values for each of these files. Confidence values are the base metric in my similarity estimation tests.
I'd like to show why relying on Dejavu library could be wrong. As I'm aware, a fingerprint solves different tasks: instead of reacting to, say, downsampling (which makes an impression to users) fingerprint is insensitive to it. In 1st round guessing the user's rating was required, so the general impression has sense.

Unfortunately, you didn't participate in 1st round, but a lot of hating about tgvoiprate from you I've read. I can say: heuristics in my original submission have a physical meaning and gives adequate estimation in comparison with other submissions. First of all, I say about the second score, named ScoreOutput. Probably you are confused about ScorePreprocess, but the aim of it is to show the distortion on preprocessing (say, to prevent converting the source audio into silence and earn high ScoreOutput). It's absent from the original submission, so I can't comment on the details.

I believe you'll get answers to your questions soon. Don't hating.
Bossy Gnu Feb 5, 2020 at 22:10
What a hating are you talking about?
I noted tgvoiprate from the provided tgvoip-test-suite is not good on my tests and I replaced it by the dejavue library for my convinience.
Take it easy.
There are a few examples of tgvoiprate fails -
https://drive.google.com/open?id=1bO2O5rhPLlU7YESRKjTRArL62vV_4TuR
First of all look at the 2nd example. tgvoiprate esimates it in 1,24... While Dejavu's confidence is 242 (it's about the same as the original sample)
As you're aware WebRTC has multi-layer architecture. What objectives of using a high level of WebRTC in your solution instead of the Voice Engine layer? In my view it has some lacks:
- more external dependencies (such as WebSockets);
- limited control of network streams (peer IP can become disclosed);
- (probably) built-in encryption algorithm is weaker than in MTProto.

Does it relate to the complexity of WebRTC native API or has other reasons?
I'm exploring your source code and see the issue in fileAudioDevice.cpp:550:
_cbPreAudioData(reinterpret_cast<int16_t *>(_recordingBuffer), _recordingFramesIn10MS * kRecordingNumChannels * 2 / sizeof(int16_t));

You call the preprocessed callback with exactly the same data as input one. This isn't what it takes. In your solution, it produces preprocessed.pcm exactly the same to input.pcm (but since _recordingFramesIn10MS=0, preprocessed could be empty at all).

preprocessed.pcm should contain an audio stream after preprocessing by NetEQ, Echo Canceler, etc. It should contain the stream which is going to transmit through the network to split expected losses (preprocessing) from non-expected ones (while transmitting). It's because of tgvoiprate can't tell the difference between these types of distortions for sure.
Bossy Gnu Feb 10, 2020 at 18:06
Thank you for the note. It's the really useful feature request to test both preprocessed and received audio streams. But there is nothing like "preprocessing" requirements at the context description. I hope it is enough to match sources and received streams side the terms of the contest.
Fair Wombat Feb 28, 2020 at 10:03
Sometimes the call via poor connection won't stop at all.
Caller http://paste.debian.net/1132676/
Callee http://paste.debian.net/1132677/
(the issue is just for future reference)
Debian 5.2.9
Nobody added any issues yet...