After kernel update: Debian12 ASL3 Distortion/choppy audio on SimpleUSB TX audio especially telemetry

n8lbv · January 13, 2025, 4:08pm

I recently brought up an AUSink SHARI-Pi clone node on Debian12 and ASL3 on a PC.
It has significant choppyness/distortion on TX audio regardless of any settings/adjustments.
I initially have been chalking this up to hardware quality.
However I have just as of yesterday fully updated my main node which also runs on a Debian12 PC but no cheap or questionable hardware.
This most recent OS update included a new kernel.
I am now noticing audio distortion on it’s telemetry on TX audio.
At this point I am just posting this as a footnote or starting point as much more data and details are expected to be needed.
And a bit of a heads up for anyone else who may be updating Debian12.
I will continue on this post as I learn more.
I was not experiencing this on my main node before the recent update.

AI3I · January 13, 2025, 10:06pm

Thanks, it’s not just me then, either…I have similar issues where Asterisk is calling rpt localplay and repeating snippets of words, but is very infrequent and inconsistent–I have yet to find a means by reproducing it.

n8lbv · January 14, 2025, 4:54am

Yep, something’s up at least for me since yesterdays’ OS update.
All telemetry playback, time, tones etc. have a definite repeat choppiness in them that was not there before the update.
Listened to it all day today while using the system.
Same after reboots.
URGH.
It’s not terrible, can still use the system but it’s very much there and noticeable.
I’ve been updating frequently – about every two weeks.
Other’s may not notice until they do update Debian12 to the latest stuff.
I think the majority of folks do not update if everything is working.
I’m doing this myself just to keep in touch and in tune with what’s going on etc.
And when something breaks I get an opportunity to learn something new usually.
I may stick a link to a recording of it happening here to see if anyone relates as my description alone is not all that great.
I may also need to try spinning this up on a pi to see if it is gone.

n8lbv · January 14, 2025, 7:18am

My formal investigation has started
I am now noticing this problem on a number of Debian12 (Intel CPU) ASL3 systems that have been updated recently.
I suspect the problem may have been present before but it is certainly much worse now.
Here is a video showing it in action.
Telemetry audio has jitter and many very short duration holes in the transmit audio.
The benchmark is to issue a *904 test tone command which should generate a clean test tone.
Which it very much is not.
The there is also a huge variation in the frequency and pattern of the audio interruptions depending on what the software may be doing or where it is at.
I have hear it on rare occasion go “clean” for a number of seconds then come back to holepunched as it usually is.
Sometimes after an Asterisk “core reload” it would go clean for a bit, but usually not.
But in this video you can see me do this and the audio “holepunch” pattern changes significantly.
This test with the test tone being generated by asterisk/rpt will be my basis for further testing and investigation or a bug report to continue work on it.

—LINK—

n8lbv · January 14, 2025, 7:59am

Just as a starting point:
I expect that 904 = cop,4 ; Test tone on/off (toggle)
when ran locally should generate a clean test tone from the USB sound device.
No radios involved for testing and simplification.
The test as demonstrated has transmit audio that is being interfered with in software.
I suspect interrupts and causing the what generates these tones and audio to be interfered with at regular or semi regular intervals.
Something appears as to not being serviced properly here.

kb4mdd · January 14, 2025, 12:47pm

What is the result of the command dahdi show status in the cli?

n8lbv · January 14, 2025, 3:21pm

Span Description                              Alarms  IRQ    bpviol CRC    Fra Codi Options  LBO
   1 DAHDI_DUMMY/1 (source: HRtimer) 1        UNCONFI 0      0      0      CAS Unkn          0 db (CSU)/0-133 feet (DSX-1)

n8lbv · January 14, 2025, 3:21pm

Span Description Alarms IRQ bpviol CRC Fra Codi Options LBO
1 DAHDI_DUMMY/1 (source: HRtimer) 1 UNCONFI 0 0 0 CAS Unkn 0 db (CSU)/0-133 feet (DSX-1)

n8lbv · January 14, 2025, 3:24pm

Hoping to learn something new-
You didn’t ask but just poking around.
lsmod:
dahdi_dummy 16384 0
dahdi_transcode 16384 0
dahdi 258048 18 dahdi_dummy,dahdi_transcode

n8lbv · January 14, 2025, 3:29pm

At the same time of trying to learn something new.
I tried turning off HPET (High Precision Event Timer) in the motherboard bios, this did not seem to make any difference.
I’m also under the impression that the OS would still use it or enable it even if it is turned off in the BIOS, but I really have no real idea what I am doing here.
HPET is a curiosity thing right now.

n8lbv · January 14, 2025, 3:38pm

Also tried this alongside while distorted/chopped audio is playing.
dahdi_test
Opened pseudo dahdi interface, measuring accuracy…
99.992% 99.995% 99.994% 99.997% 99.997% 99.996% 100.000% 99.995%
99.996% 99.999% 99.998% 99.995% 99.997% 99.998% 99.994% 99.997%
99.989% 99.997% 99.998% 99.997% 99.998% 99.997% 99.998% 99.998%
99.997% 99.997% 99.998% 99.998% 99.998% 99.997% 99.998% 99.997%
99.998% 99.997% 99.997% 99.997% 99.997% 99.997% 99.998% 99.997%
99.997% 99.996% 99.998% 99.998% 99.997% 99.997% 99.998% 99.965%
99.992% 99.991% 99.996% 99.997% 99.996% 100.000% 99.984% 99.998%
99.988% 99.976% 99.991% 99.988% 99.997% 99.998% 99.993% 99.995%
99.977% 100.000% 99.993% 99.992% ^C
— Results after 68 passes —
Best: 100.000% – Worst: 99.965% – Average: 99.994983%
Cumulative Accuracy (not per pass): 99.997

n8lbv · January 14, 2025, 5:33pm

I can no longer edit the TITLE of the post… seems to be a time limit.
I wish it to reflect that this is with using simpleusb.

WA3WCO · January 14, 2025, 8:48pm

I added SimpleUSB to the title. Please reach out if you would prefer some alternate wording.

n8lbv · January 14, 2025, 9:52pm

That’s great! Thank you.
I’m at a bit of a standstill here not sure what to try next.
The problem is pretty bad as shown in the video, it gets a little worse then goes back to the same,
sometimes it seems to clear up somewhat for a very brief time maybe 20-30 seconds ish?
I think I may have uncovered a significant problem with the latest Debian updates and kernel.
I am experiencing this on a couple of newly installed systems as of yesterday.
And am experiencing this on our production system/ on the air repeater controller just after updating it.
It may be possible to revert to the previous kernel on the production system but I am not exactly sure how to do that and I risk taking it off the air if an attempt to revert fails.
Assuming the previous kernel and any associated things are still present on the system and has a method to revert…?
The newly installed test systems have no way to revert of course as they installed fresh up to date.
I had a friend check on his PC Intel system that has not been updated since maybe 3-weeks ago and he is not having this problem. Command *904 produces a steady clean test tone for him.
And he does not want to risk updating and breaking it right now.
I also do not have access to this system for realtime testing and comparing to the newer broken ones.
So I’m a bit stuck and not knowing what to try next.
Not the end of the world but I am anxious to try some stuff and frustrated that I’m not more knowledgeable!
It’s certainly an opportunity for me to learn something.
I also don’t want to be too trigger happy on the bug report route and give this some time for others to weigh in on it.
I think we can live with the “distortion” on our production system’s transmit audio while this is figured out.

N8EI · January 14, 2025, 10:06pm

Please paste in the output of the following from the console:

uname -a
dpkg -l | grep -E ^linux
lsusb
dmesg | tail -n100

N8EI · January 14, 2025, 10:21pm

So I’ve tested this on 2 systems, both running Debian 12 Arm64 with the latest RPI kernel package:

$ uname -a
Linux node460181 6.6.62+rpt-rpi-v8 #1 SMP PREEMPT Debian 1:6.6.62-1+rpt1 (2024-11-25) aarch64 GNU/Linux

This is also running all latest versions of asl3-asterisk, DAHDI, etc. I cannot replicate this on two different systems using SimpleUSB audio.

My point being it doesn’t seem related to the recent DAHDI package rev…

n8lbv · January 14, 2025, 11:30pm

Will do also to re-iterate and make sure this is clear:
These are not RPI ARM64 installs.

 uname -a
Linux ASL3 6.1.0-29-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.123-1 (2025-01-02) x86_64 GNU/Linux

~# dpkg -l | grep -E ^linux
root@ASL3:~#

(returns blank)

~# lsusb
Bus 004 Device 002: ID 8087:8000 Intel Corp. Integrated Rate Matching Hub
Bus 004 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 8087:8008 Intel Corp. Integrated Rate Matching Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 002 Device 002: ID 0d8c:0013 C-Media Electronics, Inc. USB Audio Device
Bus 002 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub

~# dmesg | tail -n100
[    1.834123] systemd[1]: modprobe@dm_mod.service: Deactivated successfully.
[    1.834232] systemd[1]: Finished modprobe@dm_mod.service - Load Kernel Module dm_mod.
[    1.834363] systemd[1]: systemd-repart.service - Repartition Root Disk was skipped because no trigger condition checks were met.
[    1.834930] EXT4-fs (sda1): re-mounted. Quota mode: none.
[    1.835662] systemd[1]: Finished systemd-remount-fs.service - Remount Root and Kernel File Systems.
[    1.835755] systemd[1]: systemd-firstboot.service - First Boot Wizard was skipped because of an unmet condition check (ConditionFirstBoot=yes).
[    1.835784] systemd[1]: systemd-pstore.service - Platform Persistent Storage Archival was skipped because of an unmet condition check (ConditionDirectoryNotEmpty=/sys/fs/pstore).
[    1.836462] systemd[1]: Starting systemd-random-seed.service - Load/Save Random Seed...
[    1.837180] systemd[1]: Starting systemd-sysusers.service - Create System Users...
[    1.853513] ACPI: bus type drm_connector registered
[    1.854352] systemd[1]: modprobe@drm.service: Deactivated successfully.
[    1.854477] systemd[1]: Finished modprobe@drm.service - Load Kernel Module drm.
[    1.861100] systemd[1]: Finished systemd-random-seed.service - Load/Save Random Seed.
[    1.861191] systemd[1]: first-boot-complete.target - First Boot Complete was skipped because of an unmet condition check (ConditionFirstBoot=yes).
[    1.864188] systemd[1]: Finished systemd-sysusers.service - Create System Users.
[    1.864965] systemd[1]: Starting systemd-tmpfiles-setup-dev.service - Create Static Device Nodes in /dev...
[    1.877895] systemd[1]: Started systemd-journald.service - Journal Service.
[    1.891405] systemd-journald[250]: Received client request to flush runtime journal.
[    1.892868] dahdi: loading out-of-tree module taints kernel.
[    1.892935] dahdi: module verification failed: signature and/or required key missing - tainting kernel
[    1.894641] dahdi: Version: 3.4.0-4+asl
[    1.894777] dahdi: Telephony Interface Registered on major 196
[    1.896733] dahdi_transcode: Loaded.
[    1.897891] dahdi: Warning: Span DAHDI_DUMMY/1 didn't specify a spantype. Please fix driver!
[    1.897943] dahdi_dummy: Trying to load High Resolution Timer
[    1.897944] dahdi_dummy: Initialized High Resolution Timer
[    1.897944] dahdi_dummy: Starting High Resolution Timer
[    1.897945] dahdi_dummy: High Resolution Timer started, good to go
[    1.973437] audit: type=1400 audit(1736676484.080:2): apparmor="STATUS" operation="profile_load" profile="unconfined" name="lsb_release" pid=289 comm="apparmor_parser"
[    1.973891] audit: type=1400 audit(1736676484.084:3): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe" pid=290 comm="apparmor_parser"
[    1.973895] audit: type=1400 audit(1736676484.084:4): apparmor="STATUS" operation="profile_load" profile="unconfined" name="nvidia_modprobe//kmod" pid=290 comm="apparmor_parser"
[    1.974768] audit: type=1400 audit(1736676484.084:5): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/bin/man" pid=292 comm="apparmor_parser"
[    1.974772] audit: type=1400 audit(1736676484.084:6): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_filter" pid=292 comm="apparmor_parser"
[    1.974773] audit: type=1400 audit(1736676484.084:7): apparmor="STATUS" operation="profile_load" profile="unconfined" name="man_groff" pid=292 comm="apparmor_parser"
[    1.994323] audit: type=1400 audit(1736676484.104:8): apparmor="STATUS" operation="profile_load" profile="unconfined" name="tcpdump" pid=307 comm="apparmor_parser"
[    2.002407] audit: type=1400 audit(1736676484.112:9): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-client.action" pid=291 comm="apparmor_parser"
[    2.002411] audit: type=1400 audit(1736676484.112:10): apparmor="STATUS" operation="profile_load" profile="unconfined" name="/usr/lib/NetworkManager/nm-dhcp-helper" pid=291 comm="apparmor_parser"
[    2.024882] input: Power Button as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0C0C:00/input/input1
[    2.038693] ACPI: button: Power Button [PWRB]
[    2.038749] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input2
[    2.040087] sd 5:0:0:0: Attached scsi generic sg0 type 0
[    2.040924] ACPI: button: Power Button [PWRF]
[    2.049728] input: PC Speaker as /devices/platform/pcspkr/input/input3
[    2.062783] at24 0-0050: supply vcc not found, using dummy regulator
[    2.064657] at24 0-0050: 256 byte spd EEPROM, read-only
[    2.094339] dcdbas dcdbas: Dell Systems Management Base Driver (version 5.6.0-3.4)
[    2.097470] mc: Linux media interface: v0.10
[    2.099180] iTCO_vendor_support: vendor-support=0
[    2.112218] iTCO_wdt iTCO_wdt.1.auto: Found a Lynx Point TCO device (Version=2, TCOBASE=0x1860)
[    2.112298] iTCO_wdt iTCO_wdt.1.auto: initialized. heartbeat=30 sec (nowayout=0)
[    2.113628] input: Dell WMI hotkeys as /devices/platform/PNP0C14:00/wmi_bus/wmi_bus-PNP0C14:00/9DBB5994-A997-11DA-B012-B622A1EF5492/input/input4
[    2.116242] RAPL PMU: API unit is 2^-32 Joules, 4 fixed counters, 655360 ms ovfl timer
[    2.116244] RAPL PMU: hw unit of domain pp0-core 2^-14 Joules
[    2.116245] RAPL PMU: hw unit of domain package 2^-14 Joules
[    2.116246] RAPL PMU: hw unit of domain dram 2^-14 Joules
[    2.116247] RAPL PMU: hw unit of domain pp1-gpu 2^-14 Joules
[    2.124831] cryptd: max_cpu_qlen set to 1000
[    2.132649] Adding 998396k swap on /dev/sda5.  Priority:-2 extents:1 across:998396k SSFS
[    2.151537] AVX2 version of gcm_enc/dec engaged.
[    2.151562] AES CTR mode by8 optimization enabled
[    2.153448] r8169 0000:02:00.0: firmware: direct-loading firmware rtl_nic/rtl8168g-2.fw
[    2.161735] usbcore: registered new interface driver snd-usb-audio
[    2.177863] Generic FE-GE Realtek PHY r8169-0-200:00: attached PHY driver (mii_bus:phy_addr=r8169-0-200:00, irq=MAC)
[    2.178326] snd_hda_codec_realtek hdaudioC1D2: autoconfig for ALC3220: line_outs=1 (0x1b/0x0/0x0/0x0/0x0) type:line
[    2.178328] snd_hda_codec_realtek hdaudioC1D2:    speaker_outs=1 (0x14/0x0/0x0/0x0/0x0)
[    2.178330] snd_hda_codec_realtek hdaudioC1D2:    hp_outs=1 (0x15/0x0/0x0/0x0/0x0)
[    2.178332] snd_hda_codec_realtek hdaudioC1D2:    mono: mono_out=0x0
[    2.178332] snd_hda_codec_realtek hdaudioC1D2:    inputs:
[    2.178333] snd_hda_codec_realtek hdaudioC1D2:      Front Mic=0x1a
[    2.178335] snd_hda_codec_realtek hdaudioC1D2:      Rear Mic=0x18
[    2.249315] input: HDA Intel PCH Front Mic as /devices/pci0000:00/0000:00:1b.0/sound/card1/input5
[    2.249348] input: HDA Intel PCH Rear Mic as /devices/pci0000:00/0000:00:1b.0/sound/card1/input6
[    2.249374] input: HDA Intel PCH Line Out as /devices/pci0000:00/0000:00:1b.0/sound/card1/input7
[    2.249407] input: HDA Intel PCH Front Headphone as /devices/pci0000:00/0000:00:1b.0/sound/card1/input8
[    2.464663] i915 0000:00:02.0: [drm] PipeC fused off
[    2.464799] i915 0000:00:02.0: vgaarb: deactivate vga console
[    2.465934] Console: switching to colour dummy device 80x25
[    2.466969] i915 0000:00:02.0: vgaarb: changed VGA decodes: olddecodes=io+mem,decodes=io+mem:owns=io+mem
[    2.487011] [drm] Initialized i915 1.6.0 20201103 for 0000:00:02.0 on minor 0
[    2.488251] ACPI: video: Video Device [GFX0] (multi-head: yes  rom: no  post: no)
[    2.488491] input: Video Bus as /devices/LNXSYSTM:00/LNXSYBUS:00/PNP0A08:00/LNXVIDEO:00/input/input9
[    2.488582] snd_hda_intel 0000:00:03.0: bound 0000:00:02.0 (ops i915_audio_component_bind_ops [i915])
[    2.490297] r8169 0000:02:00.0 enp2s0: Link is Down
[    2.500254] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    2.527852] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    2.530262] i915 0000:00:02.0: [drm] Cannot find any crtc or sizes
[    2.543450] input: HDA Intel HDMI HDMI/DP,pcm=3 as /devices/pci0000:00/0000:00:03.0/sound/card0/input10
[    2.543546] input: HDA Intel HDMI HDMI/DP,pcm=7 as /devices/pci0000:00/0000:00:03.0/sound/card0/input11
[    2.543643] input: HDA Intel HDMI HDMI/DP,pcm=8 as /devices/pci0000:00/0000:00:03.0/sound/card0/input12
[    2.553948] intel_rapl_common: Found RAPL domain package
[    2.553951] intel_rapl_common: Found RAPL domain core
[    2.553952] intel_rapl_common: Found RAPL domain uncore
[    2.553953] intel_rapl_common: Found RAPL domain dram
[    2.553956] intel_rapl_common: RAPL package-0 domain package locked by BIOS
[    2.553961] intel_rapl_common: RAPL package-0 domain dram locked by BIOS
[    5.434261] r8169 0000:02:00.0 enp2s0: Link is Up - 1Gbps/Full - flow control off
[    5.434267] IPv6: ADDRCONF(NETDEV_CHANGE): enp2s0: link becomes ready
[22792.945452] perf: interrupt took too long (2503 > 2500), lowering kernel.perf_event_max_sample_rate to 79750
[28040.057505] perf: interrupt took too long (3129 > 3128), lowering kernel.perf_event_max_sample_rate to 63750
[65519.171950] perf: interrupt took too long (3918 > 3911), lowering kernel.perf_event_max_sample_rate to 51000

Thanks!

N8EI · January 15, 2025, 12:17am

Yes I understand. As I said I was pointing out its not the latest DAHDI package.

Also, my bad on the dpkg. Do dpkg -l | grep linux-image

Also can you install the latest kernel please - “-30”.

n8lbv · January 15, 2025, 2:00am

Cool, just wanted to make sure and I didn’t totally understand your message exactly
And pardon my not knowing how to install the latest kernel - “-30”.
If you could give me a pointer on how to do that I’ll go right at it!
Thanks!!

~# dpkg -l | grep linux-image
ii  linux-image-6.1.0-22-amd64      6.1.94-1                       amd64        Linux 6.1 for 64-bit PCs (signed)
ii  linux-image-6.1.0-23-amd64      6.1.99-1                       amd64        Linux 6.1 for 64-bit PCs (signed)
ii  linux-image-6.1.0-26-amd64      6.1.112-1                      amd64        Linux 6.1 for 64-bit PCs (signed)
ii  linux-image-6.1.0-27-amd64      6.1.115-1                      amd64        Linux 6.1 for 64-bit PCs (signed)
ii  linux-image-6.1.0-28-amd64      6.1.119-1                      amd64        Linux 6.1 for 64-bit PCs (signed)
ii  linux-image-6.1.0-29-amd64      6.1.123-1                      amd64        Linux 6.1 for 64-bit PCs (signed)
ii  linux-image-amd64               6.1.123-1                      amd64        Linux for 64-bit PCs (meta-package)

n8lbv · January 15, 2025, 3:28am

I’m a bit lost on this, and if all I would need to do is install it? And are there more steps to do?
Also there is no -30

:~# apt search linux-image
Sorting... Done
Full Text Search... Done
linux-headers-6.1.0-21-amd64/stable-security 6.1.90-1 amd64
  Header files for Linux 6.1.0-21-amd64

linux-headers-6.1.0-21-cloud-amd64/stable-security 6.1.90-1 amd64
  Header files for Linux 6.1.0-21-cloud-amd64

linux-headers-6.1.0-21-rt-amd64/stable-security 6.1.90-1 amd64
  Header files for Linux 6.1.0-21-rt-amd64

linux-headers-6.1.0-23-amd64/stable-security 6.1.99-1 amd64
  Header files for Linux 6.1.0-23-amd64

linux-headers-6.1.0-23-cloud-amd64/stable-security 6.1.99-1 amd64
  Header files for Linux 6.1.0-23-cloud-amd64

linux-headers-6.1.0-23-rt-amd64/stable-security 6.1.99-1 amd64
  Header files for Linux 6.1.0-23-rt-amd64

linux-headers-6.1.0-26-amd64/stable-security 6.1.112-1 amd64
  Header files for Linux 6.1.0-26-amd64

linux-headers-6.1.0-26-cloud-amd64/stable-security 6.1.112-1 amd64
  Header files for Linux 6.1.0-26-cloud-amd64

linux-headers-6.1.0-26-rt-amd64/stable-security 6.1.112-1 amd64
  Header files for Linux 6.1.0-26-rt-amd64

linux-headers-6.1.0-27-amd64/stable 6.1.115-1 amd64
  Header files for Linux 6.1.0-27-amd64

linux-headers-6.1.0-27-cloud-amd64/stable 6.1.115-1 amd64
  Header files for Linux 6.1.0-27-cloud-amd64

linux-headers-6.1.0-27-rt-amd64/stable 6.1.115-1 amd64
  Header files for Linux 6.1.0-27-rt-amd64

linux-headers-6.1.0-28-amd64/stable-security 6.1.119-1 amd64
  Header files for Linux 6.1.0-28-amd64

linux-headers-6.1.0-28-cloud-amd64/stable-security 6.1.119-1 amd64
  Header files for Linux 6.1.0-28-cloud-amd64

linux-headers-6.1.0-28-rt-amd64/stable-security 6.1.119-1 amd64
  Header files for Linux 6.1.0-28-rt-amd64

linux-headers-6.1.0-29-amd64/stable,now 6.1.123-1 amd64 [installed,automatic]
  Header files for Linux 6.1.0-29-amd64

linux-headers-6.1.0-29-cloud-amd64/stable 6.1.123-1 amd64
  Header files for Linux 6.1.0-29-cloud-amd64

linux-headers-6.1.0-29-rt-amd64/stable 6.1.123-1 amd64
  Header files for Linux 6.1.0-29-rt-amd64

linux-image-6.1.0-11-amd64/stable-security 6.1.38-4 amd64
  Linux 6.1 for 64-bit PCs (signed)

linux-image-6.1.0-11-cloud-amd64/stable-security 6.1.38-4 amd64
  Linux 6.1 for x86-64 cloud (signed)

linux-image-6.1.0-11-rt-amd64/stable-security 6.1.38-4 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT (signed)

linux-image-6.1.0-12-amd64/stable-security 6.1.52-1 amd64
  Linux 6.1 for 64-bit PCs (signed)

linux-image-6.1.0-12-cloud-amd64/stable-security 6.1.52-1 amd64
  Linux 6.1 for x86-64 cloud (signed)

linux-image-6.1.0-12-rt-amd64/stable-security 6.1.52-1 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT (signed)

linux-image-6.1.0-17-amd64/stable-security 6.1.69-1 amd64
  Linux 6.1 for 64-bit PCs (signed)

linux-image-6.1.0-17-cloud-amd64/stable-security 6.1.69-1 amd64
  Linux 6.1 for x86-64 cloud (signed)

linux-image-6.1.0-17-rt-amd64/stable-security 6.1.69-1 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT (signed)

linux-image-6.1.0-20-amd64/stable-security 6.1.85-1 amd64
  Linux 6.1 for 64-bit PCs (signed)

linux-image-6.1.0-20-cloud-amd64/stable-security 6.1.85-1 amd64
  Linux 6.1 for x86-64 cloud (signed)

linux-image-6.1.0-20-rt-amd64/stable-security 6.1.85-1 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT (signed)

linux-image-6.1.0-21-amd64/stable-security 6.1.90-1 amd64
  Linux 6.1 for 64-bit PCs (signed)

linux-image-6.1.0-21-amd64-dbg/stable-security 6.1.90-1 amd64
  Debug symbols for linux-image-6.1.0-21-amd64

linux-image-6.1.0-21-amd64-unsigned/stable-security 6.1.90-1 amd64
  Linux 6.1 for 64-bit PCs

linux-image-6.1.0-21-cloud-amd64/stable-security 6.1.90-1 amd64
  Linux 6.1 for x86-64 cloud (signed)

linux-image-6.1.0-21-cloud-amd64-dbg/stable-security 6.1.90-1 amd64
  Debug symbols for linux-image-6.1.0-21-cloud-amd64

linux-image-6.1.0-21-cloud-amd64-unsigned/stable-security 6.1.90-1 amd64
  Linux 6.1 for x86-64 cloud

linux-image-6.1.0-21-rt-amd64/stable-security 6.1.90-1 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT (signed)

linux-image-6.1.0-21-rt-amd64-dbg/stable-security 6.1.90-1 amd64
  Debug symbols for linux-image-6.1.0-21-rt-amd64

linux-image-6.1.0-21-rt-amd64-unsigned/stable-security 6.1.90-1 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT

linux-image-6.1.0-23-amd64/stable-security 6.1.99-1 amd64
  Linux 6.1 for 64-bit PCs (signed)

linux-image-6.1.0-23-amd64-dbg/stable-security 6.1.99-1 amd64
  Debug symbols for linux-image-6.1.0-23-amd64

linux-image-6.1.0-23-amd64-unsigned/stable-security 6.1.99-1 amd64
  Linux 6.1 for 64-bit PCs

linux-image-6.1.0-23-cloud-amd64/stable-security 6.1.99-1 amd64
  Linux 6.1 for x86-64 cloud (signed)

linux-image-6.1.0-23-cloud-amd64-dbg/stable-security 6.1.99-1 amd64
  Debug symbols for linux-image-6.1.0-23-cloud-amd64

linux-image-6.1.0-23-cloud-amd64-unsigned/stable-security 6.1.99-1 amd64
  Linux 6.1 for x86-64 cloud

linux-image-6.1.0-23-rt-amd64/stable-security 6.1.99-1 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT (signed)

linux-image-6.1.0-23-rt-amd64-dbg/stable-security 6.1.99-1 amd64
  Debug symbols for linux-image-6.1.0-23-rt-amd64

linux-image-6.1.0-23-rt-amd64-unsigned/stable-security 6.1.99-1 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT

linux-image-6.1.0-25-amd64/now 6.1.106-3 amd64 [installed,local]
  Linux 6.1 for 64-bit PCs (signed)

linux-image-6.1.0-26-amd64/stable-security 6.1.112-1 amd64
  Linux 6.1 for 64-bit PCs (signed)

linux-image-6.1.0-26-amd64-dbg/stable-security 6.1.112-1 amd64
  Debug symbols for linux-image-6.1.0-26-amd64

linux-image-6.1.0-26-amd64-unsigned/stable-security 6.1.112-1 amd64
  Linux 6.1 for 64-bit PCs

linux-image-6.1.0-26-cloud-amd64/stable-security 6.1.112-1 amd64
  Linux 6.1 for x86-64 cloud (signed)

linux-image-6.1.0-26-cloud-amd64-dbg/stable-security 6.1.112-1 amd64
  Debug symbols for linux-image-6.1.0-26-cloud-amd64

linux-image-6.1.0-26-cloud-amd64-unsigned/stable-security 6.1.112-1 amd64
  Linux 6.1 for x86-64 cloud

linux-image-6.1.0-26-rt-amd64/stable-security 6.1.112-1 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT (signed)

linux-image-6.1.0-26-rt-amd64-dbg/stable-security 6.1.112-1 amd64
  Debug symbols for linux-image-6.1.0-26-rt-amd64

linux-image-6.1.0-26-rt-amd64-unsigned/stable-security 6.1.112-1 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT

linux-image-6.1.0-27-amd64/stable 6.1.115-1 amd64
  Linux 6.1 for 64-bit PCs (signed)

linux-image-6.1.0-27-amd64-dbg/stable 6.1.115-1 amd64
  Debug symbols for linux-image-6.1.0-27-amd64

linux-image-6.1.0-27-amd64-unsigned/stable 6.1.115-1 amd64
  Linux 6.1 for 64-bit PCs

linux-image-6.1.0-27-cloud-amd64/stable 6.1.115-1 amd64
  Linux 6.1 for x86-64 cloud (signed)

linux-image-6.1.0-27-cloud-amd64-dbg/stable 6.1.115-1 amd64
  Debug symbols for linux-image-6.1.0-27-cloud-amd64

linux-image-6.1.0-27-cloud-amd64-unsigned/stable 6.1.115-1 amd64
  Linux 6.1 for x86-64 cloud

linux-image-6.1.0-27-rt-amd64/stable 6.1.115-1 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT (signed)

linux-image-6.1.0-27-rt-amd64-dbg/stable 6.1.115-1 amd64
  Debug symbols for linux-image-6.1.0-27-rt-amd64

linux-image-6.1.0-27-rt-amd64-unsigned/stable 6.1.115-1 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT

linux-image-6.1.0-28-amd64/stable-security 6.1.119-1 amd64
  Linux 6.1 for 64-bit PCs (signed)

linux-image-6.1.0-28-amd64-dbg/stable-security 6.1.119-1 amd64
  Debug symbols for linux-image-6.1.0-28-amd64

linux-image-6.1.0-28-amd64-unsigned/stable-security 6.1.119-1 amd64
  Linux 6.1 for 64-bit PCs

linux-image-6.1.0-28-cloud-amd64/stable-security 6.1.119-1 amd64
  Linux 6.1 for x86-64 cloud (signed)

linux-image-6.1.0-28-cloud-amd64-dbg/stable-security 6.1.119-1 amd64
  Debug symbols for linux-image-6.1.0-28-cloud-amd64

linux-image-6.1.0-28-cloud-amd64-unsigned/stable-security 6.1.119-1 amd64
  Linux 6.1 for x86-64 cloud

linux-image-6.1.0-28-rt-amd64/stable-security 6.1.119-1 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT (signed)

linux-image-6.1.0-28-rt-amd64-dbg/stable-security 6.1.119-1 amd64
  Debug symbols for linux-image-6.1.0-28-rt-amd64

linux-image-6.1.0-28-rt-amd64-unsigned/stable-security 6.1.119-1 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT

linux-image-6.1.0-29-amd64/stable,now 6.1.123-1 amd64 [installed,automatic]
  Linux 6.1 for 64-bit PCs (signed)

linux-image-6.1.0-29-amd64-dbg/stable 6.1.123-1 amd64
  Debug symbols for linux-image-6.1.0-29-amd64

linux-image-6.1.0-29-amd64-unsigned/stable 6.1.123-1 amd64
  Linux 6.1 for 64-bit PCs

linux-image-6.1.0-29-cloud-amd64/stable 6.1.123-1 amd64
  Linux 6.1 for x86-64 cloud (signed)

linux-image-6.1.0-29-cloud-amd64-dbg/stable 6.1.123-1 amd64
  Debug symbols for linux-image-6.1.0-29-cloud-amd64

linux-image-6.1.0-29-cloud-amd64-unsigned/stable 6.1.123-1 amd64
  Linux 6.1 for x86-64 cloud

linux-image-6.1.0-29-rt-amd64/stable 6.1.123-1 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT (signed)

linux-image-6.1.0-29-rt-amd64-dbg/stable 6.1.123-1 amd64
  Debug symbols for linux-image-6.1.0-29-rt-amd64

linux-image-6.1.0-29-rt-amd64-unsigned/stable 6.1.123-1 amd64
  Linux 6.1 for 64-bit PCs, PREEMPT_RT

linux-image-amd64/stable,now 6.1.123-1 amd64 [installed]
  Linux for 64-bit PCs (meta-package)

linux-image-amd64-dbg/stable 6.1.123-1 amd64
  Debugging symbols for Linux amd64 configuration (meta-package)

linux-image-amd64-signed-template/stable 6.1.123-1 amd64
  Template for signed linux-image packages for amd64

linux-image-cloud-amd64/stable 6.1.123-1 amd64
  Linux for x86-64 cloud (meta-package)

linux-image-cloud-amd64-dbg/stable 6.1.123-1 amd64
  Debugging symbols for Linux cloud-amd64 configuration (meta-package)

linux-image-rt-amd64/stable 6.1.123-1 amd64
  Linux for 64-bit PCs (meta-package)

linux-image-rt-amd64-dbg/stable 6.1.123-1 amd64
  Debugging symbols for Linux rt-amd64 configuration (meta-package)