Allmon3 memory usage

Allmon
1

Is it typical to see the resident memory of allmon3 creep up and up over time? Another system admin on our end is questioning the RAM usage of allmon3.

I have two rpi3 running raspbian trixie + ASL3. Both have uptimes of ~ 2 days and the RES ram for allmon3 is at a crazy 412MB and 452MB and seems to keep climbing.

Both pi’s have light to no allmon3 actual usage (the web page is not particularly used).

pi3 (1)

********** AllStarLink [ASL] Version Info **********

OS : Debian GNU/Linux 13 (trixie)
OS Kernel : 6.12.62+rpt-rpi-v8

Asterisk : 22.7.0+asl3-3.7.1-1.deb13
ASL [app_rpt] : 3.7.1

Installed ASL packages :

Package Version
============================== ==============================
allmon3 1.7.0-1.deb13
asl3 3.15-1.deb13
asl3-asterisk 2:22.7.0+asl3-3.7.1-1.deb13
asl3-asterisk-config 2:22.7.0+asl3-3.7.1-1.deb13
asl3-asterisk-modules 2:22.7.0+asl3-3.7.1-1.deb13
asl3-menu 1.16-1.deb13
asl3-update-nodelist 2.0.0-1.deb13
asl-apt-repos 2.0-1.deb13
dahdi 1:3.1.0-2.1
dahdi-dkms 1:3.4.0-10.asl.deb13
dahdi-linux 1:3.4.0-10.asl.deb13
dahdi-source 1:3.4.0-10.asl.deb13

pi3 (2)

********** AllStarLink [ASL] Version Info **********

OS : Debian GNU/Linux 13 (trixie)
OS Kernel : 6.12.62+rpt-rpi-v8

Asterisk : 22.8.2+asl3-3.8.2-1.deb13
ASL [app_rpt] : 3.8.2

Installed ASL packages :

Package Version
============================== ==============================
allmon3 1.7.0-1.deb13
asl3 3.17-1.deb13
asl3-asterisk 2:22.8.2+asl3-3.8.2-1.deb13
asl3-asterisk-config 2:22.8.2+asl3-3.8.2-1.deb13
asl3-asterisk-modules 2:22.8.2+asl3-3.8.2-1.deb13
asl3-menu 1.18-1.deb13
asl-apt-repos 2.0-1.deb13
dahdi 1:3.1.0-2.1
dahdi-dkms 1:3.4.0-11.asl.deb13
dahdi-linux 1:3.4.0-11.asl.deb13
dahdi-source 1:3.4.0-11.asl.deb13

2

AFAICS the memory usage will step up for each logged in monitoring console. The amount of resident memory will fluctuate with the system state, normally.

Valgrind shows no memory leaks in the current version of allmon3.

3

Hmm. It would be safe to say that there are zero or one logged in users max on these two devices at any time for allmon3. If there are any users of Raspbian Trixie + ASL3 + allmon3 that can say what their RES values for memory are after a few days that would be interesting.

4 
top - 16:01:16 up 2 days, 21:25,  1 user,  load average: 0.19, 0.26, 0.26
Tasks: 175 total,   1 running, 174 sleeping,   0 stopped,   0 zombie
%Cpu(s):  2.7 us,  2.6 sy,  0.0 ni, 94.7 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem :   7820.7 total,   6644.4 free,    911.8 used,    363.6 buff/cache     
MiB Swap:   2048.0 total,   2048.0 free,      0.0 used.   6908.8 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                          
   2077 allmon3   20   0  815452 558620  14808 S   0.7   7.0   8:57.37 allmon3
5

That number there looks high to me. I would assume that the system will trim out the RAM from allmon3 if it is needed elsewhere?

The above example that machine has 8GB of RAM, but a pi3 will have 1GB. Seven percent RAM vs 50+% heh.

6

that's what SWAP is for .. beware the default swap size delivered by a standard Raspi OS install

7 
top - 17:24:29 up 9 min,  1 user,  load average: 0.08, 0.15, 0.14
Tasks: 181 total,   1 running, 180 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.3 us,  0.3 sy,  0.0 ni, 99.2 id,  0.0 wa,  0.0 hi,  0.1 si,  0.0 st 
MiB Mem :   7820.7 total,   7209.2 free,    411.6 used,    291.7 buff/cache     
MiB Swap:   2048.0 total,   2048.0 free,      0.0 used.   7409.0 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                          
   2131 allmon3   20   0  224120  65244  14684 S   1.0   0.8   0:05.24 allmon3
8

I see anywhere from 500k to 800k bytes of virtual memory used after a week or two, so I don't worry as long as the resident memory is <= that value. :slightly_smiling_face:

Just because the number looks big doesn't mean it's necessarily a worry. You may have noticed that allmon3 uses Python, so what's loaded in memory in part is whatever the language pulls in during execution.

Let us know if you observe an unbounded memory usage increase.

9

I have seen unreasonable memory inflation if you have someone on the internet who does a drive-by on the websockets and blast the crap out of them. Something that was doing that to Node 2004 a week or so ago. Finally went away.

Allmon3 is all Python which is not a memory-lightweight. IT also stores the entire “Allmon DB” in memory.

10

previous was a raspi4, now this from a raspi3 1GB replacement

top - 15:07:33 up 57 min,  1 user,  load average: 0.16, 0.17, 0.12
Tasks: 154 total,   2 running, 152 sleeping,   0 stopped,   0 zombie
%Cpu(s):  0.7 us,  1.3 sy,  0.0 ni, 95.9 id,  0.0 wa,  0.0 hi,  2.1 si,  0.0 st 
MiB Mem :    906.0 total,    342.3 free,    352.1 used,    270.2 buff/cache     
MiB Swap:    905.0 total,    905.0 free,      0.0 used.    553.9 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                      
   3155 allmon3   20   0  324624  66744  14860 S   2.0   7.2   0:25.11 allmon3
11

That looks like you just fired up and have one login if I'm reading the tea leaves correctly. I see 800k/400k, more like your first posting, on an RPi5 after 24 hr and ~1000 keyups on a net with 100-250 nodes.

Fortunately I have no pathologically connecting nodes, so @N8EI's caveat does not apply.

12

yes, freshly configured raspi3 .. will let that simmer a few days and then report once more

13

raspi3 with two interfaces connected to ECR most of the time (last report)

top - 14:42:37 up 3 days, 20:46,  1 user,  load average: 0.25, 0.37, 0.41
Tasks: 161 total,   1 running, 160 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.5 us,  3.5 sy,  0.0 ni, 94.8 id,  0.0 wa,  0.0 hi,  0.2 si,  0.0 st 
MiB Mem :    906.0 total,    280.9 free,    393.7 used,    308.9 buff/cache     
MiB Swap:    905.0 total,    905.0 free,      0.0 used.    512.3 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                           
    866 allmon3   20   0  329556  72332  14844 S   3.6   7.8 171:37.79 allmon3

memory usage appears to be nothing much to worry about

14

Quite odd, my pi3’s were up to 600M RES memory for allmon3. Not sure if there is a significant difference between the pi image from asl3, and going the raspbian+asl3 way, since I do the raspbian+asl3 way. I am able to keep the memory down by adding this to the allmon3 systemd override:

sudo systemctl edit allmon3

[Service]
Environment="MALLOC_TRIM_THRESHOLD_=131072"

sudo systemctl daemon-reload
sudo systemctl restart allmon3

15

I think you're seeing confirmation bias rather than a real fix. The default value for malloc trimming is 128k. So you actually made the trim value slightly bigger. Also, this doesn't really trigger anything with Allmon3 directly, it's changing the glibc library value which impacts the Python3 runtime underlay. As I said before, I know there is memory growth issue when allmon3 gets a lot of abandoned connections to a websocket listener because the process will still be trying to deliver content to things that are no longer listening. I've got a note to look at that at some point, but it only seems to be an occasional problem so it's way down the priority list. If you look at your webserver logs, are you getting arbitrarily pounded by something on your Allmon3 web interface? Especially something directed to a URI beginning /allmon3/ws?

16

No these two nodes are not accessible via WAN other than through a wireguard tunnel, and allmon3 would sit idle otherwise.

17

I will keep digging, there has to be a reason heh. hmm

18

Here is what I am relying on for my ‘fix’. Some Google AI output:

The developer is technically correct that 128 KB is the starting default. However, the reason your "fix" works is that modern glibc (like the version in Trixie) automatically scales that number upward during runtime to improve performance. By manually setting it to 128 KB in the service file, you are disabling that auto-scaling.

Here is the "smoking gun" you can share with them to explain why your observed fix is real:

1. The "Dynamic Mmap Threshold"

In glibc (the C library), the trim threshold is tied to the mmap threshold.

  • The Rule: If an application allocates a chunk of memory larger than the current mmap threshold, glibc uses mmap instead of sbrk.

  • The Result: When this happens, glibc automatically increases the trim threshold to twice the size of that allocation.

  • The Bloat: If Allmon3/Python 3.13 makes a few 8MB or 16MB allocations (common for web buffers or SSL contexts), your "128 KB default" can silently grow to 32 MB or more.

2. Why the Override Works

According to the official glibc manual:

"If M_TRIM_THRESHOLD is set explicitly (via mallopt or an environment variable), dynamic adjustment is disabled."

By setting MALLOC_TRIM_THRESHOLD_=131072, you aren't just "restating the default"—you are locking the door. You are telling the system: "I don't care how big your allocations get, never raise the threshold above 128 KB."

3. Verification for the Developers

They can verify this themselves on a Trixie/3.13 system by running this command while Allmon3 is under load:

bash

sudo gcore $(pgrep -n allmon3) && strings core.* | grep -A 1 "malloc"

Use code with caution.

This often reveals the "internal" state of the allocator.

Summary for your Co-Admins:
The "fix" works because it strips away the allocator's "intelligence." In a high-performance server, that intelligence is good. On a Raspberry Pi radio node, that intelligence leads to the 600MB "bloat" you observed.

19

I still thing it's something with your setup. Here's a long-running example:

root@node460180:/home/jdm# ps auxw | grep allmon3
allmon3      925  0.1  4.0 232124 78296 ?        Ssl  Mar09  33:47 /usr/bin/python3 /usr/bin/allmon3

232M allocated, 78M res, 12.5M shared.

20

FYI those are k displayed in top, not M.

From the man page:
"Regardless of which of these forms memory may take, all are managed as pages (typically 4096 bytes) but expressed by default in top as KiB (kibibyte)."