Allstar registration not syncing to backend DNS not updating

AllStarLink
1

I’ve been watching this for about four hours.
From what I am seeing on my end, it appears that registration events are not updating part of the backend and what eventually makes it to DNS.
I moved a node earlier today to a different Internet connection.
It is registering properly with Allstar but is not updating DNS and thus the nodes list is not being updated properly.
My node (27298) was intentionally changed to a different public facing IP address earlier today.
And is on a different Internet provider.
It continues to register and provide stats but cannot be connected to from the public facing nodelist/DNS.

DNS is still showing it’s old IP address and port.
Anything I have tried on my end, restarts reboots, re-applying settings in the server/node allstar portal do not fix this.

Also any DNS lookups return results from over 12 hours ago.
If I am interpreting the query correctly: “RT=2024-11-17 12:39:22”
Assuming last update was 11-17-2024 12:39 UTC.
Lookups on other (besides my own) nodes return what looks like results from 12+ hours ago as well.

dig @ns-436.awsdns-54.com 27298.nodes.allstarlink.org txt

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> @ns-436.awsdns-54.com 27298.nodes.allstarlink.org txt
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 26963
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;27298.nodes.allstarlink.org. IN TXT

;; ANSWER SECTION:
27298.nodes.allstarlink.org. 60 IN TXT “NN=27298” “RT=2024-11-17 12:39:22” “RB=1” “IP=98.224.X.X” “PIP=0” “PT=4579” “RH=register-central”

;; AUTHORITY SECTION:
nodes.allstarlink.org. 172800 IN NS ns-1467.awsdns-55.org.
nodes.allstarlink.org. 172800 IN NS ns-1708.awsdns-21.co.uk.
nodes.allstarlink.org. 172800 IN NS ns-436.awsdns-54.com.
nodes.allstarlink.org. 172800 IN NS ns-868.awsdns-44.net.

;; Query time: 19 msec
;; SERVER: 205.251.193.180#53(205.251.193.180)
;; WHEN: Sun Nov 17 18:09:46 EST 2024
;; MSG SIZE rcvd: 294

RIP: Dude

dig @ns-436.awsdns-54.com 2001.nodes.allstarlink.org txt

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-9.P2.el7 <<>> @ns-436.awsdns-54.com 2001.nodes.allstarlink.org txt
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 38077
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;2001.nodes.allstarlink.org. IN TXT

;; ANSWER SECTION:
2001.nodes.allstarlink.org. 60 IN TXT “NN=2001” “RT=2024-11-17 12:42:07” “RB=1” “IP=162.248.X.X” “PIP=0” “PT=4569” “RH=register-west”

;; AUTHORITY SECTION:
nodes.allstarlink.org. 172800 IN NS ns-1467.awsdns-55.org.
nodes.allstarlink.org. 172800 IN NS ns-1708.awsdns-21.co.uk.
nodes.allstarlink.org. 172800 IN NS ns-436.awsdns-54.com.
nodes.allstarlink.org. 172800 IN NS ns-868.awsdns-44.net.

;; Query time: 16 msec
;; SERVER: 205.251.193.180#53(205.251.193.180)
;; WHEN: Sun Nov 17 18:13:00 EST 2024
;; MSG SIZE rcvd: 289

2

sounds like this might be the same issue effecting my node, i can not connect to anyone

3

Looks like it just got fixed.
Been monitoring this is almost realtime.
-SteveG

4

i am able to connect to some nodes again but not others

5

Looked fixed for a minute but problem came right back after a few minutes.
Maybe they are working on it.
dig @ns-1467.awsdns-55.org 27298.nodes.allstarlink.org txt

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> @ns-1467.awsdns-55.org 27298.nodes.allstarlink.org txt
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62041
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;27298.nodes.allstarlink.org. IN TXT

;; ANSWER SECTION:
27298.nodes.allstarlink.org. 60 IN TXT “NN=27298” “RT=2024-11-17 12:39:22” “RB=1” “IP=98.224.X.X” “PIP=0” “PT=4579” “RH=register-central”

;; AUTHORITY SECTION:
nodes.allstarlink.org. 172800 IN NS ns-1467.awsdns-55.org.
nodes.allstarlink.org. 172800 IN NS ns-1708.awsdns-21.co.uk.
nodes.allstarlink.org. 172800 IN NS ns-436.awsdns-54.com.
nodes.allstarlink.org. 172800 IN NS ns-868.awsdns-44.net.

;; Query time: 47 msec
;; SERVER: 205.251.197.187#53(ns-1467.awsdns-55.org) (UDP)
;; WHEN: Sun Nov 17 21:23:37 EST 2024
;; MSG SIZE rcvd: 294

6

@n8lbv Is node 27298 an ASL3 node or something else? The RT field is the last time the registration database itself saw your registration ping which should be ~180 seconds when a node is online. Looking at the registration database directly right now (11/18 at 1:22pm UTC), it’s reporting the last registration from node 27298 was at unix timestamp 1731909255 which is Nov 18 at 5:54AM UTC. Is/was the node supposedly online at at 1:22pm UTC? Other nodes that I know of to be online and active all appear to be properly updating their regseconds entry in the database.

7

Hi N8EI this is my “mobile” node.
I often move it between Internet connections and sometimes park it at home.
It is an up-to-date ASL3 node.
This has been working without issues for weeks (moving it around to different internet connections).
I noticed the problem started happening yesterday when I moved it from one connection to another (and rebooted).
Now it is just stuck with the old DNS entry from Early morning yesterday and this has not updated to reflect it’s actual public IP address.
It is registering with Alstar as it should but DNS never gets updated to show the actual IP address it is coming from.
The DNS entry shows a date from Yesterday at 12:39UTC
And the old IP address:

27298.nodes.allstarlink.org. 60 IN TXT “NN=27298” “RT=2024-11-17 12:39:22” “RB=1” “IP=98.X.X.123” “PIP=0” “PT=4579” “RH=register-central”

It is presenting as a backend problem from my perspective and tests.

8

I assume you’re using HTTP registration? What does rpt show registrations report? From the node database perspective, your node just came online again with a new IP address 17 minutes ago for the first time since yesterday per the above.

9

And do you happen to be using the beta or development channel for ASL3 software?

10

Yep, registration is working fine:
34.105.111.212:443 27298 174.X.X.175:4569 179 Registered
1 HTTP registration.

DNS is not getting updated with this IP address.
It shows the old IP address from yesterday morning.
The DNS entry also appears to have a time stamp from that time:
“NN=27298” “RT=2024-11-17 12:39:22” “RB=1” “IP=98.X.X.123”
This is the problem I am reporting.
This node is not able to connect to other nodes because the old DNS and current node database
does not reflect the IP change.

dig @ns-436-awsdns-54.com 27298.nodes.allstarlink.org txt
dig: couldn’t get address for ‘ns-436-awsdns-54.com’: not found
root@pinode:/etc/asterisk# dig @ns-436.awsdns-54.com 27298.nodes.allstarlink.org txt

; <<>> DiG 9.18.28-1~deb12u2-Debian <<>> @ns-436.awsdns-54.com 27298.nodes.allstarlink.org txt
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 57274
;; flags: qr aa rd; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 1
;; WARNING: recursion requested but not available

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;27298.nodes.allstarlink.org. IN TXT

;; ANSWER SECTION:
27298.nodes.allstarlink.org. 60 IN TXT “NN=27298” “RT=2024-11-17 12:39:22” “RB=1” “IP=98.224.236.123” “PIP=0” “PT=4579” “RH=register-central”

;; AUTHORITY SECTION:
nodes.allstarlink.org. 172800 IN NS ns-1467.awsdns-55.org.
nodes.allstarlink.org. 172800 IN NS ns-1708.awsdns-21.co.uk.
nodes.allstarlink.org. 172800 IN NS ns-436.awsdns-54.com.
nodes.allstarlink.org. 172800 IN NS ns-868.awsdns-44.net.

;; Query time: 28 msec
;; SERVER: 205.251.193.180#53(ns-436.awsdns-54.com) (UDP)
;; WHEN: Mon Nov 18 09:35:34 EST 2024
;; MSG SIZE rcvd: 294

11

Not using beta on this node.
It is up to date release (3-days ago).
I’ll run (can run) update again in case this has changed in the past 3 days.
Currently installed:
OS : Debian GNU/Linux 12 (bookworm)
│ OS Kernel : 6.6.51+rpt-rpi-v8
│
│ Asterisk : 20.9.3+asl3-3.0.6-1.deb12
│ ASL [app_rpt] : 3.0.6
│
│ Installed ASL packages :
│
│ Package Version
│ ============================== ==============================
│ allmon3 1.4.1-1.deb12
│ asl3 3.5.0-1.deb
│ asl3-asterisk 2:20.9.3+asl3-3.0.6-1.deb12
│ asl3-asterisk-config 2:20.9.3+asl3-3.0.6-1.deb12
│ asl3-asterisk-modules 2:20.9.3+asl3-3.0.6-1.deb12
│ asl3-menu 1.9-1.deb12

12

Interesting. The node database is showing your 174 IP address but the change timestamp doesn’t match up. I guess the IPTime field in the data isn’t always being set accurately. The use of that field in the registration database for the DNS view is new as we’re trying to optimize the load on the systems. But there must be some subtle bug in the system where that column isn’t always being updated properly.

13

Rand update/upgrade (Rpi)
One of the updates was an RPi eeprom update.
rebooted,
Then shutdown.
Power cycled (reboot).
Fully up to date as of (today).
Allstar nodes DNS not getting updated issue remains.

14

Roger that.
The DNS thing has been stuck since 12:39 yesterday (assuming that is UTC).
Registration and internal database as you mention appear to be fine.
But public facing node resource is stuck with the old IP due to the DNS issue.
I suspect this is a global service affecting issue for anyone changing IP addresses.
But is generally un-noticed right now for most people as their IP address is not changing frequently.
So this is not “taking everyone down” but might be for anyone with frequently changing connections.
I also tried changing server/node settings (then saving) in the allstar portal then changing them back(and saving) reapply in case that would kick something to update and work, but it did not.

15

Note: I am querying an authoritative DNS for my tests:

dig @ns-436.awsdns-54.com 27298.nodes.allstarlink.org txt

and not a local DNS just to asure myself I’m looking at the most recent data.
Although I think you have the DNS cache time set to only 60 seconds.
So local DNS should not be answering anything cached if it’s older than 60s

16

I’ve changed some of the internal logic on the DNS system to account for this variance. It’s a lot less efficient and will take longer to publish updated node chnages. But I believe your issue should be resolved?

17

Yep it’s updated now.
Let me know if you need any help or testing over the next few days/weeks etc.
What is the variance you are referring to?
Is it on my end?
To the best of my knowledge I am not doing anything strange or different with node 27298.
Thanks and happy to help out /test.
I do need to get to work on that beta node soon as work settles a bit here.
Tuesday is nuts, but I think I can get to work on it Wednesday.
In the current state of things (I will be changing IP address on 27298) frequently.
Do you know how long the updates should take now?
Thanks for fixing it!
Take care man.
-Steve LBV

18

No it’s not something with your node per-se unless it’s a bug in the app_rpt registration code. The node registration system is a spaghetti mess of 15+ years of iterative code.

There’s two fields in the registration database - regseconds which is a Unix timestamp of the last registration checking and IPTime which is supposed to be updated when your IP address changes as perceived by the registration system. In your case at least, it’s definitely NOT updating that field properly with your IP changes as you described. With the new DNS system, I’d been trying to limit the API work on Route53 so that we could propagate node IP changes really fast. If IPTime had been right, there was only < 50 records of change every 15s cycle of the run. However now that I’ve discovered that, at least in some instances, that field is not right I have to swing the big hammer and just say “give me all the nodes that checked in within the last 60 seconds” and now I’m just running that every 30 seconds. Took the changeset from 50 records to a few thousand. :frowning:

19

It’s all been working well since this last change.
I’m guessing the lighter ~50node/15second node update plan is a work in progress.

20

Happy to test - moving that node around Internet connections with any future changes etc.
Thanks!
SteveG