Repeater Gives Me Raspberries!

AllStarLink
41

Great, thanks for that insight Chuck.

So are you pretty sure that the unexpected introduction of an ast_null_frame into the stream doesn’t cause the Asterisk JB/PLC mechanism to try to fill that the space with something? I ask that because I think that’s what Asterisk would try to do if the missing frame was from the network side.

(I note also from testing on a different track that it sometimes takes a short period of time for the ALSA driver to recover from a capture underrun. So it wouldn’t be very surprising to me if a few ast_null_frames might be presented in this case. Of course, I have no evidence that this case is happening, I’m just looking at the code.)

42

I don't know if this will help anyone studying this issue but the SimpleUSB channel driver allows you to save the rx and tx PCM data.

  • For RX capture, use sudo asterisk -x 'susb tune rxcap'.
  • For TX capture, use sudo asterisk -x 'susb tune txcap'
  • Turn off the captures with sudo asterisk -x 'susb tune nocap'

... and the captures are stored in the /tmp directory.

1 Like
43

Yup, and here's a quick bash one-liner to monitor your connection quality:

while true; do cls; /usr/sbin/asterisk -x 'iax2 show netstats'; /usr/sbin/asterisk -x 'iax2 show channels'; sleep 1; done

Use a wide terminal.

44

Yup, the only "fill" from a null frame should be silence.

45

OK cool, and is the same true for a missing frame from the IAX side?

46

Yes, as far as app_rpt goes.

47

OK, thanks very much for educating me. It does seem to me that the behaviors around gap filling would be addressed inside of Asterisk itself, i.e. app_rpt just gets the result of what Asterisk decides to do. But I’m no expert and I won’t belabor this.

Either way, there’s no sense speculating, I’ll do some testing of this theory and report back. 73s!

2 Likes
48

as a unsolved follow up on the raspberry issue

I need to get this thing off my bench
and dont have time to chase untraceable weirdness
So much to my chagrin I stuck a fresh HamVoip install on the same Pi4
using the same wiring, same audio interface, same repeater,same level settings, running simple_USB

and it exhibits NO audio artifacts what so ever

I did not want to do that
but it will be off my bench for the time being

49

Yeah… HamVoIP’s simpleusb is definitely smoother, both RX and TX, and doesn’t have the aliasing present in ASL3’s version.

I recently migrated a radioless node using a UCI120 from HamVoIP to ASL3, and even in just using cop 4 to generate a test tone, I can hear what sounds like individual frames occasionally. It’s not very loud, but if I can hear it with my naked ears (and I’m about 70% deaf due to a genetic condition), then there is something going on. This was not present using HamVoIP on the same rPi4 with the same UCI120, and is also not present if I pipe the audio direct from app_rpt using outstreamcmd to the rPi’s built-in audio codec. It’s more latent that way, but it’s also much cleaner, both in terms of the lack of aliasing and lack of random little frame boundaries being audible sometimes. What I'm hearing isn't quite as severe as what is being described/what I have heard in the wild, but it's probably related.

If I generate a 20 millisecond loop of pink noise and loop it inside a DAW, that is a much more extreme version of what I am hearing from simpleusb on this particular node.

Usbradio with all the filters wide open seems to be a little more well behaved, but I miss some of the low-end response I can get from simpleusb in both directions.

I will, at some point, try capturing PCM data to see if what I hear is captured as output. I will also, eventually, do some testing, replacing the simpleusb and usbradio driver in ASL3 with Analog_Processor from DVSwitch to compare things. That’s a USRP I/O with USB support, among other things, I.E. additional audio processing, such as EQ and AGC if desired.

By the way, nothing interesting comes up in Asterisk’s debug logs when I hear the frame boundaries.

I will probably also try Ampersand in a radioless configuration. AFAIK, it isn't really ready for radio use, unless your radio is using speaker/microphone audio, there is no PL tone present in the audio path, and you don't mind having a squelch tail at the end of your transmissions if not using DCS, but I haven't really looked at it too closely yet.

50

I asked for node numbers exhibiting this phenomenon. Crickets.

I asked people having the issue to listen to a known good node and see if they hear anything unusual. Crickets.

I listened to a node supposedly having this issue and heard mostly bad mic technique. Crickets, other than a weak attempt to convince me blowing air was the issue.

I'll be interested in this again when I see a node number where I can hear the issue.

51

I strongly disagree with your oversimplification of this topic.

Right now we don’t know where the issue is actually coming from. It could be in the remote RX, the local TX, or somewhere in between. It could be in the analog chain, digital chain, audio codec, IP transport, or something else entirely. Without knowing that, I don't see how it would be possible to point to specific node numbers and say definitively whether they do or don’t "have the issue".

Some people are actively working on reproducing this in a controlled setup so we can isolate the source. That will give us much more reliable information than trying to depend on hearing it intermittently on live nodes.

Once the source and conditions are better understood, it will be much easier to identify the root issue and move toward a fix.

52

You folks know from the infamous echolink issue that once we can (even if barely) reproduce an issue we can fix it. The real issue is that, so far, I've seen no useful troubleshooting info in this thread. That includes the blog with the supposed sample of the problem from the OP, which could be a cellphone beaconing, a noise blast unlocking the codec, or any number of other things.

50 posts into this thread, there seems to be more grousing than useful info. I'd be Real Happy if that trend turned.

EDIT: Hopefully you all are checking things like

  1. possible environmental issues (esp. with repeaters)
  2. Determining whether RX or TX channel is involved
  3. Does this happen on a hub, as well as with a (yet to be determined) channel?
  4. Does a system exhibiting the issue have any sort of unusual combination of config parameters?
53

Sharing one observation that may or may not be relevant to the original problem.

I’ve been curious about the behavior of Asterisk/app_rpt during underruns of various kinds. Earlier in this thread there was some speculation about what the system does when an audio frame goes missing. I’m testing that scenario from the perspective of the network (i.e. IAX network frame missing) and the USB interface (i.e. underrun causes a USB frame to get missed).

In my tests of the IAX (network) case it appears that the system interpolates the missing frame. The picture below shows what it looks/sounds like when an IAX frame goes missing. It shows a pure sinusoid (400 Hz test tone) during most of the test. However, the 20ms gap shaded area in the middle shows the period of the single missing frame. During this interval we see what appears to be a continuation of the test tone, but with a linear ramp down of the amplitude and a phase discontinuity at the start/end of the 20ms “gap.” Once the data stream resumes the tone goes back to normal. There is no silence in this period, only an audible pop which I’m assuming is a result of the phase change.

Longer gaps show the same behavior out to about 5 missing frames, at which point the amplitude had decayed away.

image
image651×453 30.3 KB

My Test Setup

My test node opens an IAX call to an ASL3 server and injects a synthetic tone. I’ve modified my test node to route every 100th outbound audio mini-frame to the wrong address. That equates to a dropped frame every 2 seconds. I listen to the audio (and record) using a second node dialed to the same server.

Explanation of This Behavior (My Guess)

I think what I’m seeing/hearing is the normal function of the packet loss concealment (PLC) feature of Asterisk. PLC is designed to fill any sudden gaps in an attempt to improve audio quality. The reason the 20ms “gap” contains the 400 Hz tone is that the PLC algorithm fills gaps with copies of earlier audio frames. The amplitude ramp-down is designed to “smooth” the transition in and out of the gap period and to limit ringing in the case of a longer gap. Unfortunately, this simple algorithm doesn’t match phase. A very similar (same?) PLC algorithm is described in detail in the G.711 documentation.

The USB version of this test is pretty interesting. I’ll share a write-up when I’m finished.

2 Likes
54

It also looks like there's a phase change at the beginning of the missing frame, then the ramp down. That should be a jitter buffer frame inserted for dropout correction, but the ramp is a little puzzling.

Are you using default jitter buffer settings? Global ones are in iax.conf.

55

I believe the ramp is fully expected and is part of the design of the PLC. It looks to me like Asterisk has implemented a variation of the G.711 (app I) algorithm which uses a 20%/10ms ramp.

image
image1059×209 42.5 KB

And yes, default settings.

56

Yep, just found that part of the asterisk repo. That's a correctly handled dropped frame, then. You might hear a tick from that one frame insert at ~3 dB down at the end.

EDIT: You could probably generate a pretty good sawtooth envelope with a 3:1 or 4:1 dropped:good frame ratio. Just thinking that might sound interesting, esp. if the ratio were changing randomly.

57

I did not want to stick my foot back in this till I had another repeater on the bench
but I wanted to make clearer that the raspberries only appear in receive audio
whether it is from the actual receiver or plumbed in from another source
it does not occur on incoming network audio
or audio telemetry being generated by Allstar

when I get the next bench patient (will be soon)
I will grab some more data

1 Like
58

@k9wkj and @kc1fsz. are you using simpleusb or usbradio?

59

Yup, fully understood. The description of my network test above wasn’t trying to suggest that the problem you are pointing out is related to network audio, I was just probing at the PLC mechanism via the network because it is easier.

But I’m fairly certain that the PLC behavior will step in for USB (receive) audio as well and I have some tests that show what that looks like. It’s harder because it’s more difficult to generate an underrun on the USB interface than it is the IAX interface. 73s.

60

Agreed. I'm thinking of a way to abuse an AIOC with some test equipment. Kinda Rube Goldberg, but...