MESHWATCH Lab

Every node in the mesh timestamps its detections from its own disciplined clock, and fusion has to decide which of those timestamps it still believes. The obvious gate is to check each node's clock offset and reject the ones that are out. That gate is unbuildable, and the reason is the whole point: a node that has lost its time reference is in holdover, and holdover is precisely the state of not knowing your own offset. Ask the box and it will answer honestly and wrongly.

So the gating quantity is not the offset. It is the bound: the error the node can prove it has not yet exceeded, which is exactly what chrony's Skew and Root dispersion and PTP's clockClass ladder already encode:

The consequence is enforced structurally. The simulator carries a god-view field for a node's true error because the physics needs one, and no code path may render it into a tool-shaped line. A leak test pins that. What a holdover box may print is what it can still measure locally, plus the bound it can compute. Everything else would be a lie the model was telling on the hardware's behalf.

Because the bound grows at the oscillator's rate, the same event has three visibly different outcomes. The lightweight node class carries a TCXO and spends its budget in about twenty seconds; the large fixed class carries an OCXO and lasts roughly an hour; the grandmasters carry rubidium and last about sixteen hours. A fenced node keeps scanning and keeps painting its own scope: nothing is wrong with the radar. Only fusion stops admitting its plots.

That idea survives the trip out of the simulator. On the real Linux deployment it becomes a systemd unit: meshwatch-timesync-gate is a Type=oneshot gate that the fusion service and every templated meshwatch-node@ instance carry a Requires= on. It asserts that chronyd answers on its command socket (not that the unit is active), which source was selected (reference ID 7F7F0101 means "syncing to itself" and fails unless this host is the designated rack master), leap status, offset inside budget, and then a role-aware PTP check. Mask chrony on a node and the radar does not degrade. It refuses to start.

The simulator is the easy half. The interesting half is that the same artifact gets deployed four times into environments that are progressively harder to lie to. Each rung exists because the one above it hit a wall the substrate could not move, and naming that wall out loud is most of the lesson.

Rung two is where the timing story stops being a model. Three node hosts and an aggregation host sit on their own bridge; plots are real UDP datagrams crossing it, registration is a real TCP handshake fusion can really reject, and the nodes' chrony really measures the rack master across the network: 2.85 µs, an actual NTP measurement, not a rendered one. Real ptp4l runs a real BMCA across hosts and elects a grandmaster. What the containers cannot do is steer anything: there is one CLOCK_REALTIME on the machine, shared by every container, so chronyd runs with -x and ptp4l with free_running 1. Servo lock, makestep, and recovery from an injected offset are untestable there, and the README says so rather than faking them. That is the entire justification for rung three.

Rung four is about the boundary rather than the box. The workstation becomes the control node and the source of truth for the code; the VM becomes the rack. nginx terminates TLS so exactly one socket is reachable from outside, and the playbook proves the exposure model on every deploy by asserting from the workstation that 443 answers and that the console, control and telemetry ports do not. Adding a radar is one inventory line plus systemctl enable --now meshwatch-node@RL-23; the new site walks the same join pipeline as everything else, and removal deliberately does not notify fusion: at the fusion layer a decommissioned radar and a destroyed one are the same event.

rack_check is a pytest suite split along one seam: collectors shell out to systemctl is-active, ss -tuln, df -P, chronyc -c tracking, docker ps and ip -j addr and return plain data; the assertions are parametrized directly over a declarative config.yaml. Adding a required service or port to the config adds a test case with no code change, so a per-site variant is a config file rather than a fork.

The seam also buys the thing most validation tooling never has: a test story of its own. Set RACK_CHECK_SNAPSHOT and the collectors read captured JSON instead of a live machine. The healthy snapshot passes all eleven checks. The degraded snapshot fails exactly three, and the three are the deliverable:

Run live on the deployed rack it comes back ten green and one red, and the red is deliberate. systemctl is-active docker asks whether the container runtime is healthy. But the runtime is the layer below this host; it is what is running it. Putting a dockerd inside a workload container to make the check pass would be a worse lie than the failure, so the row stays red and is documented as a bare-metal item.

The chrony assertion carries a subtler trap. On the aggregation host it passes with an offset of exactly zero, because that host is the rack's own reference. A perfect zero there is not health, it is a node measuring itself, and the shared timing validator encodes the rule directly: read the reference ID before you read the offset.

The bring-up sequence exists twice on purpose: once inside the simulator as a model with shaped evidence, and once as a shell script running the actual commands on the deployed host. Both stop at the first failure and mark everything after it BLOCKED rather than failed, because only the first failure is actionable, and a checkout runner that turns eleven gates red for one dead unit has told you nothing you can act on.

A simulator that flatters itself is worse than no simulator, because someone will eventually cite it. So the repository carries an adversarial self-audit of the model against real behaviour, produced by a line-by-line survey, in which every finding lands in one of four buckets: FAITHFUL (matches reality, with evidence), ACKNOWLEDGED (simplified, and the code already says so), SILENT (simplified or wrong, and nothing on screen flags it), and ABSENT (no analog here at all).

Six labs build the deployment layer by hand (units, sockets, containerisation, an air-gapped chrony master/client pair, a device-file reader under systemd, md RAID on a VM), and four of them close by pointing at the production artifact that does the same thing properly. That order is deliberate: running someone else's bring-up script before you can read a unit file teaches nothing.

On top of that sit two scored fault-injection trainers, and their interesting property is the verification rather than the faults. linux-dojo injects twelve real systemd failures into a rootless user unit: a stripped exec bit, a masked unit, a decoy process holding the port, a crash loop that has already hit its start limit. And check never trusts the edit: it requires the unit active, stable rather than flapping, and actually echoing a UDP probe on loopback, plus a drill-specific assertion. netlab does the same for ten Docker networking faults across a three-container topology (a DNS divorce, a one-way conntrack rule, an MTU black hole, a name thief), and its check demands the page load from the host port and an in-container fetch by DNS name to return valid telemetry repeatedly and the UDP time echo to answer. An imposter does not get lucky three times.