Master RTCM Timeout Regression in ASL3 (Build 22.10.1+asl3-3.10.5-1)

WB6ZSU · September 24, 2026, 9:48pm

The Issue: After a recent update to the above build, my server periodically loses connection with the master timing RTCM, forcing a disconnect of all remaining clients. This setup was perfectly stable on prior ASL3 builds.

The Symptoms:

Standard ICMP pings are solid, but voter pings drop exactly 5 packets during the event.
The server and the RTCM log the timeout at the exact same millisecond, indicating a thread block rather than a network drop.
"OFFLINE" and immediate "OK" CW messages sent every 5 -10 minutes on all clients
- Temporarily have disables RTCM CW messages
Buffers are padded (buflen = 480, Tx Buffer = 1760) and hardware is running on an SSD.

Server Side Log:

[2026-09-24 00:40:49.206] NOTICE[277377]: chan_voter.c:4927 voter_timer: VOTER 42833: Client Home_Rptr disconnect (timeout)

[2026-09-24 00:40:49.206] WARNING[277377]: chan_voter.c:4933 voter_timer: Lost master timing client, disconnecting remaining clients.

[2026-09-24 00:40:49.465] NOTICE[277376]: chan_voter.c:5489 voter_reader: VOTER 42833: Mast

RTCM Side Log:

09/24/2026 07:40:54.800 ERROR! Host response timeout

09/24/2026 07:40:54.800 Host Connection Lost (Pri) (10.4.73.22)

09/24/2026 07:40:55.840 Host Connection established (Pri) (10.4.73.22)

System Info:

asl3-asterisk: 2:22.10.1+asl3-3.10.5-1.deb12

W5GLE · September 24, 2026, 11:01pm

#1169 — landed in the 3.10.x line before 3.10.5:

Wrapped the TX audio sendto() loops in voter_xmit with voter_lock.
Moved unlock so pings and keepalives also run under that lock (pre-#1169 keepalives were outside it).
So every 20 ms TX wake can hold the global lock across many UDP sends. While that holds:

voter_reader cannot process inbound master packets / update lastheardtime
voter_timer cannot run the timeout check until the lock is free, then may see a stale master and tear everything down

This would be the most likey culprit IMO. Only way to confirm would be to roll it back and do a manual build to confirm.

VE7FET · September 26, 2026, 2:38pm

Interesting.

Tell me more about your configuration.

Confirm you have only one "master" RTCM?
How is the master RTCM via network to wherever your host is? (They should be on the same network switch.)
What is the Ethernet duplex setting on all of your RTCMs, especially the master?
How are any remaining RTCMs configured (normal voting clients, mix-mode, etc.)?
What firmware version are your RTCMs running?
How often does this "event" happen? Is it consistent? (So I can try and replicate the problem)
How did you arrive at a TX Buffer on the RTCM of 1760 (it shouldn't be related to this issue, but seems like an excessive value).

Some thoughts...

The channel driver has been upgraded in recent releases to provide more robust reporting of what clients are doing. There was a lot of stuff "buried in the weeds" that wasn't being adequately or consistently reported.

The channel driver was also updated to (as you have seen) dump clients upon the loss of the master timing source. That was done because continuing to try and use clients when you don't have a master clock is an irrational situation. If the connection between the host and the master client is lost for more than 100ms, a timer expires (that has always been the case), and the master is kicked out due to the timeout. All other clients (non-master) have a much longer timeout before they are kicked.

There are some known issues with the Ethernet chip used in the RTCM/VOTER that relate to it being used at 10Mb/s and specific duplex settings (half duplex appears to be a problem). I believe there also might be a bug in the current firmware that doesn't necessarily properly set the duplex on the Ethernet chip. However, that hasn't changed in forever, so has always been present. There is work underway to update the TCP/IP stack in the firmware, and correct some of those issues.

I'm not ruling out a bug or regression at this point, however, it is also equally plausible that the new reporting and client handling has uncovered a previously unknown or hidden network issue in your configuration.

If you can provide some more details, I will try and replicate the setup to see if we can narrow things down. I have the current ASL and RTCM firmware running on a production system with three satellite receivers, and have not had reports of this issue from users.

Lee

WB6ZSU · September 27, 2026, 1:29am

Here are some details:

First of all, I have been on ASL3 since it was released and my configuration hasn't changed, with the exception of more voters being added over the years. I only noticed the issue as my last update.

I have only one Master and currently eight RTCMs. Some of the RTCMs are in a voter role and some are in a mix configuration.
The Master and server are local to each other and on the same VLAN. The only reason they are physically separated is due to the availability of the GPS antenna connection for the Master.
I checked the Master and the eight RTCMs for their duplex settings. The Master is set for Half Duplex. The RTCMs seem to be a mix with five as Half and three as Full.
I believe four of the RTCMs are in voter configuration and four are in mix.
Seven the RTCMs (including the Master) are running 3.00BEW 3/24/2021 firmware and one of the remote RTCMs is running 1.51CHUCK 08/08/2017.
The event appears to happen randomly, but generally every 5 to ten minutes.
Tx Buffer and buflen were derived using voter ping per the Wiki..

If you have any other questions, I will endeavor to answer them.

Thanks,

Greg

VE7FET · September 27, 2026, 3:28am

Great, thanks for the further information, Greg.

The first quick thing to try would be to set the Master to full duplex. I did some research recently based on another user that has been having some random disconnects, and that indicated that there is potentially a problem when the Ethernet is in half duplex that can cause packets to be lost/delayed. If I recall correctly, it was along the lines of the Ethernet gets busy receiving traffic, and forgets that it needs to send packets out too. This could potentially be caused by something on the LAN sending broadcast traffic that goes to all the devices. I believe there are some mitigations for this that should hopefully be in the next firmware release, but I'm still working on rolling in some other changes before that is ready to release for testing.

For clarity, the Master RTCM is told to "send audio always", so it is effectively sending a continuous stream of audio packets (regardless of actual receive state), at a rate of one packet every 20ms (one frame of audio).

The channel driver is watching to see when it "last heard" the clients, and for the Master, it times out in 100ms (which would be 5 audio packets).

That's why, if something potentially throwing broadcast traffic at the Master RTCM, causing its Ethernet receive to be busy, it isn't going to take much for it to delay sending its audio packets and hit that 100ms timeout timer pretty quick.

I am not saying that IS the answer, but it is something to consider.

I am going to go back and review some of the recent channel driver updates, and see if I can narrow down what the cause might be.

Lee

WB6ZSU · September 27, 2026, 5:42am

Lee,

I set the duplex to Full for the Master, but the disconnects/reconnects are still happening.

On a somewhat side note (which doesn't make sense to me).... As I mentioned previously, occurrence of this seemed to be approximately every 5 -10 minutes. After I gave you details of my setup, I updated my server to include the Allmon 1.10 release. It seems like after that, the occurrence rate dropped. In fact, the first occurrence after that update took 45 minutes. Perhaps this is coincidental, as I can't fathom a connection.

Greg

VE7FET · September 27, 2026, 2:19pm

Thanks for the additional information, Greg.

It looks like there could be a regression that was introduced back in chan_voter: Review usage of voter_lock by ve7fet · Pull Request #1169 · AllStarLink/app_rpt · GitHub that may be responsible for what you are observing.

While that change improved code robustness that would prevent chances for corrupting data while it was being operated on, it moved some code "under lock" that previously was not.

In particular, there is some code related to I/O, CODEC translation, and ping/keepalives that is now done while a global lock is held. Those operations can have a variable delay due to such things as processor load (to name one), which could prevent incoming packets from the Master client from being processed in a timely manner, which could then lead to us hitting the master timeout, triggering the response you are seeing (since we have such a short timeout to work with).

The problem can potentially be exacerbated when there are a number of clients to perform operations for, since the locks are held longer due to the additional clients that need to be processed sequentially. Since you're operating on 8 clients, that is going to compound the potential delays, which could explain why you are seeing this behaviour, while my system with four may not.

I am working on a patch that will refactor how that locking is done, to minimize the time the locks are held, as well as move the operations noted above back out from being under lock (after we have done the critical tasks while holding our lock).

I hope to have that patch submitted in the next couple of days for wider testing, after I do some additional work and testing myself.

Lee

W5GLE · September 27, 2026, 3:32pm

I have a patch if you would like it Lee. I have not done a PR.

VE7FET · September 29, 2026, 2:58am

I've got it patched, just working on some testing, before I submit the PR.

I'll add you as a reviewer, and you can see if you would do anything different.

WB6ZSU · September 29, 2026, 3:36am

Thanks Lee. I take it you were able to duplicate the issue?

VE7FET · September 30, 2026, 12:51am

I don't have THAT many clients to throw at it, but some code analysis pointed to some potential causes of processing delays that could lead to the observed behaviour.

I've just submitted a fix for review: chan_voter.c: Fix Regression Introduced in PR#1169 With `voter_lock` Usage by ve7fet · Pull Request #1294 · AllStarLink/app_rpt · GitHub

I have tested it locally with the testbed I have, and everything seems nominal. Once the code is reviewed, and we complete any finishing touches, it will make it's way into the next set of beta packages for wider testing.

If you'd need to apply the fix more urgently, you'd have to build from source, and replace chan_voter.c with the patched version.

Lee

VE7FET · October 1, 2026, 7:57pm

The (hopeful) fix has been pushed to the development/beta channel:

See the following guide on how to install the beta release:

Lee

WA3WCO · October 1, 2026, 8:16pm

Note: the 3.10.7 packages are currently in the "devel" stream. They'll be moved to the "beta" stream after we see a few more installs and a bit more testing.

WB6ZSU · October 1, 2026, 10:35pm

Sorry for my ignorance on this, but I've never done any ASL beta testing. I enabled the beta channel and performed the update/upgrade, however even after a restart , I still see the old version. Shouldn't this reflect the beta version:

WB6ZSU · October 1, 2026, 10:37pm

I guess I should have read this before I tried to use the beta stream.

N8EI · October 1, 2026, 10:52pm

You can try devel. It PROBABLY won't break anything but no guarantees.

WB6ZSU · October 2, 2026, 12:00am

I tried devel, but unfortunately it looks like it's still happening (slightly truncated logs):

[2026-10-01 15:49:47.120] AllStar restarted and all clients connected

[2026-10-01 15:58:50.051] chan_voter.c: VOTER 42833: Client Home_Rptr disconnect (timeout)
[2026-10-01 15:58:50.051] chan_voter.c: Lost master timing client, disconnecting remaining clients.
[2026-10-01 15:58:50.051] chan_voter.c: VOTER 42833: Master changed from client Home_Rptr to NONE
[2026-10-01 15:58:50.134] chan_voter.c: VOTER 42833: Master changed from client NONE to Home_Rptr

[2026-10-01 15:58:55.491] chan_voter.c: VOTER 42833: Client Home_Rptr disconnect (timeout)
[2026-10-01 15:58:55.491] chan_voter.c: Lost master timing client, disconnecting remaining clients.
[2026-10-01 15:58:55.491] chan_voter.c: VOTER 42833: Master changed from client Home_Rptr to NONE
[2026-10-01 15:58:56.424] chan_voter.c: VOTER 42833: Master changed from client NONE to Home_Rptr

[2026-10-01 16:19:56.971] chan_voter.c: VOTER 42833: Client Home_Rptr disconnect (timeout)
[2026-10-01 16:19:56.971] chan_voter.c: Lost master timing client, disconnecting remaining clients.
[2026-10-01 16:19:56.971] chan_voter.c: VOTER 42833: Master changed from client Home_Rptr to NONE
[2026-10-01 16:19:57.143] chan_voter.c: VOTER 42833: Master changed from client NONE to Home_Rptr

[2026-10-01 16:20:02.711] chan_voter.c: VOTER 42833: Client Home_Rptr disconnect (timeout)
[2026-10-01 16:20:02.711] chan_voter.c: Lost master timing client, disconnecting remaining clients.
[2026-10-01 16:20:02.711] chan_voter.c: VOTER 42833: Master changed from client Home_Rptr to NONE
[2026-10-01 16:20:03.636] chan_voter.c: VOTER 42833: Client Home_Rptr connected.
[2026-10-01 16:20:03.642] chan_voter.c: VOTER 42833: Master changed from client NONE to Home_Rptr

[2026-10-01 16:25:57.771] chan_voter.c: VOTER 42833: Client Home_Rptr disconnect (timeout)
[2026-10-01 16:25:57.771] chan_voter.c: Lost master timing client, disconnecting remaining clients.
[2026-10-01 16:25:57.771] chan_voter.c: VOTER 42833: Master changed from client Home_Rptr to NONE
[2026-10-01 16:25:58.008] chan_voter.c: VOTER 42833: Master changed from client NONE to Home_Rptr

[2026-10-01 16:25:58.951] chan_voter.c: VOTER 42833: Client Home_Rptr disconnect (timeout)
[2026-10-01 16:25:58.951] chan_voter.c: Lost master timing client, disconnecting remaining clients.
[2026-10-01 16:25:58.951] chan_voter.c: VOTER 42833: Master changed from client Home_Rptr to NONE
[2026-10-01 16:25:59.123] chan_voter.c: VOTER 42833: Master changed from client NONE to Home_Rptr

[2026-10-01 16:26:00.151] chan_voter.c: VOTER 42833: Client Home_Rptr disconnect (timeout)
[2026-10-01 16:26:00.151] chan_voter.c: Lost master timing client, disconnecting remaining clients.
[2026-10-01 16:26:00.151] chan_voter.c: VOTER 42833: Master changed from client Home_Rptr to NONE
[2026-10-01 16:26:00.291] chan_voter.c: VOTER 42833: Master changed from client NONE to Home_Rptr

[2026-10-01 16:26:02.892] chan_voter.c: VOTER lost master timing source!!
[2026-10-01 16:26:02.911] chan_voter.c: VOTER 42833: Client Home_Rptr disconnect (timeout)
[2026-10-01 16:26:02.911] chan_voter.c: Lost master timing client, disconnecting remaining clients.
[2026-10-01 16:26:02.911] chan_voter.c: VOTER 42833: Master changed from client Home_Rptr to NONE
[2026-10-01 16:26:03.836] chan_voter.c: VOTER 42833: Client Home_Rptr connected.
[2026-10-01 16:26:03.843] chan_voter.c: VOTER 42833: Master changed from client NONE to Home_Rptr

WB6ZSU · October 2, 2026, 12:12am

The only thing I haven't explicitly checked is whether the Master RTCM was losing data from the GPS Receiver... but wouldn't that show-up on the Master RTCM? Below is what the MAster RTCM shows in that same time period:

VE7FET · October 3, 2026, 12:23am

Ok, interesting. A few things here... because I am confused as to what you've got configured.

First, what does "sudo asl-show-version" show? Let's confirm you're on the right (updated) code.

Second, what timezone is your host in, so I can correlate those logs a bit (I know the RTCM is going to be in UTC) and make sure we're looking at the same logs at the same time period.

Third, these logs look like you only have the one (master) client connected? I thought you said there were 8 clients on this system? I am not seeing the disconnect messages on the console after, "Lost master timing client, disconnecting remaining clients." that shows all the rest of the clients being disconnected? I am expecting to see, "Forcing disconnect of client: " for each of the rest of the clients getting dumped.

Yes, if you lost GPS on the client, you would see it on the client console.

Right now, if I correlate some of the timestamps, I get this:

[2026-10-01 16:19:57.143] chan_voter.c: VOTER 42833: Master changed from client NONE to Home_Rptr

23:20:02.560 ERROR! Host Response Timeout
23:20:02.560 Host Connection Lost
[2026-10-01 16:20:02.711] chan_voter.c: VOTER 42833: Client Home_Rptr disconnect (timeout)
[2026-10-01 16:20:02.711] chan_voter.c: Lost master timing client, disconnecting remaining clients.
[2026-10-01 16:20:02.711] chan_voter.c: VOTER 42833: Master changed from client Home_Rptr to NONE

That tells me that the RTCM didn't hear ANY traffic from the host for 6 seconds (that is the timeout timer on the RTCM), so IT initiated the timeout and dumped the connection at xx:20.02.560.

That was then followed by the host responding with it's own disconnect at xx:20:02.711, since the RTCM had already dumped the connection, so it naturally hit the 100ms timeout on the host side.

When idle, the host is sending keepalive packets every 1.5 seconds towards the RTCM to keep the connection open. The RTCM resets its watchdog on any traffic it receives from the host (keepalive, audio, voter ping).

So, something strange appears to be going on here.

Note that there is a new firmware version for the RTCM that was released today. It upgrades the TCP/IP stack in the firmware, as well as fixes a bug in the full/half duplex setting that can be a contributing factor to Ethernet issues.

Lee

WB6ZSU · October 3, 2026, 1:50am

Lee,

Here's the version dump:

I'm in the PDT timezone.

I truncated the log files and didn't show must of the client activity, so maybe that confused you. Here's a new occurrence of the process:

I don't have the matching log from the Master RTCM. If you need that, I'll grab another occurence when it happens.

Greg