First, verify which pin you are using. On the 6-pin mini-DIN connector there are two audio output pins available: Pin 5 “PR1” which is meant for 1200bps packet applications, and Pin 4 “PR9” which is meant for 9600bps applications.
The circuitry in the radio is different for each due to their intended applications. This is a voice application and should be using Pin 5.
Secondly, make sure the settings in the radio are correct. It appears you are familiar with the Data Band menu item. This obviously should be set to the side of the radio you are intending on using.
Next you will want to verify the Data Speed menu item (#518) matches the chosen pin on the connector. If using Pin 5 this setting should be “1200”.
Lastly the squelch method needs to be set in Squelch Source menu item (#520). This should almost always be set to “SQL”. In this mode, if the channel is programmed for CTCSS on receive, the “active” signal state will only be output if the proper tone is present on a received signal, just as if operating the radio from the front panel. Similarly, if a channel is programmed without CTCSS, “active” will be output on any received signal.
Pin 6 “SQC” provides this output and the High/Low logic can be configured using Kenwood’s MCP-2A programming software.
I have a TM-V71A, which I understand to be the same hardware just without the TNC/APRS control head, that I am using in a remote type configuration similar to this and have confirmed correct operation through testing. When CTCSS is programmed on the selected channel, only valid signals are relayed through. As expected, a CSQ channel will relay any received signal, even noise if the squelch is broken. I have also confirmed AM modulated aircraft transmissions are handled appropriately.
Since I am using the squelch output as intended to control downstream equipment in my application, I cannot say exactly how the output on the receive audio pin behaves. It is possible the audio is always present when a signal is received regardless of valid CTCSS/DCS as it is intended for data applications. The SQC output does act as expected and using it properly should eliminate any problems.