The clock that rang like a bell under load

Last time, I caught a GPS receiver’s pulse-per-second (PPS) edge on a PRU core, a small deterministic coprocessor on the chip that Linux never touches: a polling loop watched the pin and grabbed a free-running 200 MHz counter within one 5 ns cycle of the transition. A userspace daemon then converted each captured tick count into CLOCK_REALTIME, the clock chrony steers, a step called a clock-domain crossing. On the bench, it held: chrony’s sourcestats settled at an estimated standard deviation of one nanosecond.

That kind of crossing has a design flaw baked in. The fix wasn’t a better crossing. I removed the crossing from chrony’s reference path, the pipeline that feeds chrony the number it steers by, entirely.

The bug in the crossing#

I fit PRU ticks directly against CLOCK_REALTIME in one step: the design the last post showed. CLOCK_REALTIME is exactly the clock chrony steers, so that single fit measured the PPS edge against the very clock disciplined from its own output. Every correction chrony applied perturbed the next measurement: a feedback loop, one fit wide.

My own code comments call this the original design, and they record what happened when I tuned it tighter for more precision. The refclock’s poll interval had been acting as the loop’s damping, without my ever meaning it to. Shortening that interval (poll 0, filter 4) removed the damping and drove the loop unstable within minutes: skew went from 0.006 ppm to 11.9 ppm, root dispersion from 3 to 56 microseconds, pulse offsets swinging plus or minus 16 microseconds. That’s the same crossing behind the last post’s one-nanosecond number: quiet at the poll rate I’d left it at, audibly circular the moment I touched it.

I split the crossing into two steps in response. The first step fits ticks against CLOCK_MONOTONIC_RAW, a clock chrony never adjusts. Call it the hardware step: hardware against hardware, with only bus-read noise as error, clean enough to average over an 8-second window and hold to 11 to 18 ns RMS. The second step computes CLOCK_REALTIME minus CLOCK_MONOTONIC_RAW, and adds that gap to the hardware step’s answer to place it on CLOCK_REALTIME. Call it the tracking step: it has to follow chrony’s live correction, so it uses a short window instead.

Splitting removed the circularity from the precision-critical term. It did not remove it from the tracking step, because that gap is still chrony’s own correction, read back. The daemon wrote the tracking step’s output to chrony’s NTP shared-memory refclock every pulse, and chrony read that back as its reference: a feedback loop, closed once a second, just narrower than before.

Tuning can’t remove a loop built into what you’re measuring. The fix was not a better tracking step.

Removing the daemon from the loop#

Two separate problems made the numbers worse than the hardware could do: capture jitter, and the feedback loop. I’d already fixed the first one in firmware. The old firmware ran on the PRU, not on Linux, so it was already immune to interrupt and scheduler jitter, but it still found the PPS edge by polling: a polling loop read the pin every cycle, compared it to the last cycle’s value, and grabbed the counter on the cycle it saw a change. The current firmware doesn’t poll the pin for the edge at all:

static void ecap_init(void) {
  ECAP_ECCTL1 = 0x0100; /* CAPLDEN; CAP1 rising, absolute, no prescale */
  ECAP_ECCTL2 = 0x0010; /* TSCTRSTOP=run; continuous; wrap after CEVT1 */
  ECAP_TSCTR = 0;
  ECAP_ECCLR = 0xFFFF;
}
...
if (ECAP_ECFLG & 0x0002) { /* CEVT1: an edge was captured in hardware */
  pps_data.iep_lo = ECAP_CAP1;
  pps_data.seq++;
  ECAP_ECCLR = 0x0003; /* clear CEVT1 + INT */
  pru_rpmsg_send(&transport, arm_dst, arm_src, &notify, 1);
}

The eCAP unit captures ECAP_CAP1 in hardware, the instant the pin moves; the code above only checks a flag and publishes what’s already in the register, so no instruction can smear the timestamp.

Fixing capture jitter didn’t touch the feedback loop. That was never in the capture. It was in the tracking step, downstream. I gave chrony a reference that never depends on the daemon’s crossing at all, neither step of it: a small kernel module, ptp_pruss, that turns the same 200 MHz counter into a real Linux PTP (Precision Time Protocol) hardware clock, a PHC exposed as /dev/ptpX. Its phase base is set once, when the module loads, to place the clock on TAI, atomic time with no leap-second jumps. Its frequency is trimmed toward GPS separately. When the counter’s capture register changes, the module fires a PTP external timestamp event, extts, directly from a kernel work item. No daemon involved:

static void unwrap_fn(struct work_struct *w)
{
	...
	raw = readl(ecap);
	ext_ticks += (s32)(raw - last_raw);
	last_raw = raw;
	fold();
	cap = readl(ecap + CAP1_OFF);
	if (extts_on && cap != last_cap1) {
		ev.type = PTP_CLOCK_EXTTS;
		ev.index = 0;
		ev.timestamp = scaled_ns(ext_ticks + (s32)(cap - last_raw));
		fire = true;
	}
	last_cap1 = cap;
	...
	if (fire)
		ptp_clock_event(clk, &ev);
	schedule_delayed_work(&unwrap_work, HZ / 4);
}

Nothing here reads CLOCK_REALTIME or CLOCK_MONOTONIC_RAW, and nothing here knows chrony’s state. The module only knows PRU ticks, a phase base, and a frequency trim. chrony reads the event through its own name for the same thing, extpps, and applies its own PHC-to-system-clock model, the same one it uses for any other PTP hardware clock:

# the real reference: hardware PPS capture read off the PHC
refclock PHC /dev/ptp1:extpps:pin=-1 refid PPS precision 1e-9 poll 1 prefer trust lock GPS offset -0.0000047

pin=-1 matters: without it, chrony asks the driver to set a pin it doesn’t have, and chronyd exits at startup. offset compensates a fixed ~4.7 microsecond antenna-and-capture delay; the rest is ordinary chrony refclock syntax.

The daemon’s crossing left chrony’s reference path. Not improved. Removed.

The daemon, demoted#

pru_pps_shm still runs. It still does the same two-step crossing described above and writes the result to NTP SHM, but that SHM refclock is configured noselect: chrony logs it and never steers by it.

The daemon still earns its keep, though not entirely clear of chrony. Once a minute, it trims the PHC’s own frequency through clock_adjtime. While GPS is locked, that trim target comes from chrony’s own frequency correction, the tracking step’s slope, read back into the PHC on a much coarser timescale than the per-pulse loop that rang. In holdover, with GPS gone, the trim switches to a temperature model learned for the DS3231, a temperature-compensated real-time clock chip, which reads chrony nowhere. The daemon also applies a per-pulse timing correction from the GPS receiver’s own quantization error, also chrony-free. None of it sits in the path chrony reads for pulse timing; only the frequency trim, not the timing reference, still touches chrony at all.

A separate problem remained: the served NTP packets. CPTS, the on-chip Ethernet MAC’s own timestamp engine, stamps PTP frames only, not NTP, so fixing the reference didn’t fix the timestamps chrony writes into the packets it serves. I added a second PRU for that: PRU1 polls the same MAC’s frame counters and, each time one moves, latches the same 200 MHz counter. A second kernel module, cpsw_pruts, matches those entries to packets and delivers them through the kernel’s standard timestamping API. Reference and served packets now read the same counter.

Getting those receive timestamps into chrony, I found one more bug. On this hardware, the kernel reports the timestamp’s interface index as 0. chrony matches that index against its configured interfaces; a zero index matches nothing, so chrony silently fell back to kernel timestamps, the software-stamped kind, instead of the hardware ones PRU1 provides. I patched chrony to treat a zero index the same as no index, keeping the existing fallback to the packet’s own interface. Miroslav Lichvar, chrony’s maintainer, accepted it for the next release.

What that bought#

The torture test matters most. The original crossing routed CPU-load latency straight into its own next sample, and the poll 0, filter 4 test above shows how little margin that left even without added load. The new path has no tracking step in the reference to stretch. Ninety seconds at full CPU load, and the reference held 88 ns RMS, nothing past 424 ns.

The bigger change is external. The last post could only report what the box said about itself. Now the served packets carry real hardware timestamps, and two other GPS-disciplined machines can measure this one from outside: under 160 nanoseconds mean, under 450 nanoseconds at the 95th percentile over an hour. That comparison didn’t exist before. This box is no longer grading its own work.

The daemon is not gone. It stopped feeding chrony the pulse-by-pulse correction it was measuring in the first place. The only thing it still hands back, once a minute, is how fast a crystal is running.

The code, firmware, kernel modules, and daemon are on GitHub.