I installed a new repeater recently which used an RTCM. Almost immediately I noticed periodic disconnections between the RTCM and my server. Associated with these disconnections, I observed poor results when performing both voter pings to my server and also pings from other onsite devices to the RTCM. The frustrating thing was the behavior was inconsistent and frequently did not occur when I was onsite to troubleshoot.
After several weeks of fighting the issue, I procured another RTCM and replaced the original troublesome device. Once this swap was made the problem went away, so the issue was obviously a problem with the initial RTCM.
Once I brought the original RTCM home, I performed some initial troubleshooting. I was able to reproduce the problem, but it seemed to take a couple of days of powered-up operation to materialize. This led me to believe it was a temperature related issue. More detailed evaluation pointed me to the behavior of the Ethernet Controller IC, which includes a 25 MHz oscillator based on an external 25 MHz crystal.
I found that under some temperatures that the oscillator would jump from from the nominal 25 MHz frequency by approximately 180 kHz. This offset frequency exceeds the +/- 50 ppm requirement of the ENC28J60 IC.
I procured some 25 MHz crystals from DigiKey (P/N ECS-250-18-9X) and replaced the part on the RTCM. After the new part was installed, the frequency of the oscillator has remained stable. At this point I have been monitoring the frequency for about a week, and have just integrated the RTCM with an at-home repeater. I initially performed a 10,000 voter ping test with 100% success. I will continue to monitor performance, but it seems like a bad crystal was the source of my problem.
I wanted to post my findings in case anyone else experiences this type of issue.