TELEMETRY SOURCE REVIEW — source semantics, not experimental validation

Scope and evidence limits
All cited paths are relative to package source/. Unchanged Stage_01 files were carried forward into the complete 91-file archive. Line ranges refer to this exact snapshot. Active functions, rather than the many historical commented copies, were inspected. No code was edited, no hardware was accessed, and no run output or executable identity was established. Full-source addendum: Module/ is now supplied; PowerSanity admits finite 0.30–12.0 W and raw rejection uses 24 W. See collector_review.txt for actual admission/selector semantics, and lifecycle_review.txt for configured merge wait. The Lynsyn header has a closing class declaration at line 503; a missing final newline makes wc -l report 502. No truncated-class defect is asserted.

Names used below:
A = Stage_01/Concretes/SystemMetricsAggregatorConcrete_v3_2.h
L = Stage_01/Concretes/LynsynMonitorConcrete_new.h
S = Stage_01/Concretes/SoCConcrete_new.h
T = Stage_01/SharedStructures/allModulesStatcs.h
D = Stage_01/Concretes/DataConcrete_new.h
G = Stage_01/Concretes/AlgorithmConcrete_new.h
V = Stage_01/Concretes/SdlDisplayConcrete_new.h

1. Active data path and clock domains
The active ISystemMetricsAggregator includes allModulesStatcs.h (Interfaces/ISystemMetricsAggregator.h:131-139); A includes it too (83-86). T defines PowerStats, CameraStats, AlgorithmStats, DisplayStats and SystemMetricsSnapshot. Standalone SharedStructures/PowerStats.h and SystemMetricsSnapshot.h are alternative/legacy declarations, not the definitions on this inspected path. In particular, the standalone PowerStats initializer {4,0.0} would yield a two-element vector [4,0], but the active T:130-134 uses explicit size/value construction and avoids that bug. Do not report the legacy 4-W default as an active defect.

Camera calls beginFrame(frameId, CameraStats) before frame distribution (D:1537-1568). CameraStats.timestamp is capSys, derived from accepted V4L2 monotonic/monotonic-raw driver time, falling back to steady time near dequeue. D:1012-1019 establishes a system/steady offset at stream start; D:1384-1448 identifies/plausibility-checks driver clock basis, converts raw if required, and maps capMono to system time. This is a driver capture/dequeue proxy, not externally validated sensor-exposure time.

Algorithm passes its explicit frameId to mergeAlgorithm after processing and before publishing its normal processed-frame output (G:1028-1051). AlgorithmStats.startTime/timestamp are system-clock times; inferenceTimeMs is a steady-clock elapsed duration. The algorithm window begins at G:770-774 before output preparation; ends G:962-965 after CPU/GPU work and output-frame construction G:951-959, but before stats construction, merge and output queue push. It excludes input queue waiting/draining and downstream display. GPU launch wrappers synchronize their work before returning; this supports a completed-work interval in these source paths, not a proof of correct numerical output or a pure kernel interval. See pipeline reviewer for algorithm-specific details. G:967-991 computes actual processed count / elapsed steady time in a roughly one-second or partial window, not 1000/inferenceTimeMs. A:1193-1194 prefers that algorithm FPS over camera FPS.

Display drains queues, retains latest original/processed frames, uploads each new frame ID once, renders and calls SDL_RenderPresent, then stamps DisplayStats.timestamp with steady_clock::now (V:925-990). Stats use retained processed frame ID where valid, otherwise original ID; renderTimeMs/latencyMs are left at their defaults zero in this active display method. Thus display-derived E2E ends after the SDL_RenderPresent call, not measured physical scanout or photon appearance. An original-only frame can reach mergeDisplay without an algorithm record and be rejected by the normal expectations.

SoC reads one tegrastats stdout line per loop, timestamps it with system_clock::now at parse time, forwards a cached copy to aggregator, optionally appends direct CSV, then sleeps pollIntervalMs (S:218-298). It uses a pipe reader, not the hardware timestamp of a temperature or utilization measurement. A custom tegrastats command gets --interval added only when absent (S:78-97); an explicit mismatched --interval is left unchanged. Parsing/CSV time plus the subsequent sleep can still put a consumer behind the producer. Pause/resume can also retain buffered lines. Consequently fresh parse timestamps are not sufficient proof of fresh acquisition. CPU util/frequencies, GR3D utilization, EMC and temperatures are parsed independently/conditionally; unavailable fields remain defaults. Temperature/GR3D failures are logged but a partially parsed struct is still returned and pushed (S:355-449).

Lynsyn uses raw device cycles converted to seconds. In the default legacy hardware branch, it free-runs and drains FIFO once per host sampleRateMs wake (L:196-240,403-429). It decimates by averaging voltage and current separately over accepted raw samples and emits a single PowerStats at the newest raw sample's device time (L:293-323). First emitted device time is anchored to host system_clock::now; later stamps use the original host anchor plus elapsed device time. Backward device steps or consecutive gaps >5s re-anchor; HALT resets first-sample anchoring (L:246-257,432-459). No measured USB/FIFO latency or drift correction is present. The direct CSV system_time_us is a separate host receipt/write-time sample (L:325-334), not the PowerStats mapped timestamp. T:120-124 contains receipt_steady_ns/device_ts_raw fields, but this monitor does not populate them.

2. Latency definitions actually computed
A:1054-1060 converts the display steady stamp to system time using a fresh pair of now() calls at finalization. Camera uses a stream-start offset and algorithm uses wall-clock stamps directly; therefore the path is not uniformly monotonic and wall-clock changes or offset-estimation error can affect cross-stage durations.
processingLatencyMs = algorithm end - algorithm start when hasAlgTime (set unconditionally by mergeAlgorithm at A:399-401), otherwise inferenceTimeMs (A:1073-1094).
displayLatencyMs = converted post-present timestamp - algorithm end (A:1108-1128).
endToEndLatencyMs = converted post-present timestamp - camera stamp (A:1131-1145). If display is absent and expectations permit a partial row, it falls back to camera->algorithm end, then processing latency (A:1146-1154).
Negative or >10,000ms latencies are clamped to [0,10,000]; an E2E < processing+display by more than 0.25ms logs an invariant violation but the row is retained (A:1089-1094,1113-1118,1136-1141,1157-1174). Missing/invalid timing is therefore not represented by an explicit quality flag or NaN.
T:954-963 defines avg_latency_ms = 0.6*E2E + 0.3*processing + 0.1*display. This is a within-row weighted blend of overlapping intervals, not an average across frames or a distinct end-to-end measurement. weighted_end_to_end_latency_ms simply copies E2E. No reward/scoring implementation is available in these files. SampleTestIntegrationCode_v35.cpp:1020-1042 merely modifies Scheduler objective weights; it does not establish which latency/power/energy fields the missing selector consumes.

3. Power and energy semantics — two distinct products
Active PowerStats sums per-sensor power over four configured slots, with sensorPower=V*I only when V and I are each >0.001 (T:132-170). isValid requires a nonzero timestamp, sensors, nonnegative values and total >0.01W (T:189-202). Monitor and aggregator both apply the hrl::PowerSanity gate (L:311-322; A:519-540); its numeric range is finite 0.30–12.0 W in Module/PowerSanity.h. Raw monitor filtering rejects nonfinite/negative total and >2*kMaxW (L:266-277). The electrical rail topology is not encoded: summing sensors proves neither board input power nor non-overlapping rails, and no idle-baseline subtraction or per-engine attribution is performed.

Per-frame path: A:1802-1891 performs ZERO-ORDER-HOLD duration weighting. Each valid power sample holds until the next; the first valid sample may backfill to window start (even when its time is after window end), and the last sample is held through end. A defensive clamp handles backward timestamps. It divides per-sensor weighted sums by covered seconds, returning mean power over algorithm [start,end]. This is not trapezoidal integration. Histories are not consumed, so samples may be reused by multiple frame windows. The same whole-device power can be attributed to overlapping frame windows if callers have such overlap; summing the resulting joules is not a conserved device-energy total. The visible main algorithm worker is sequential, so overlap across that one worker's algorithm windows should not be asserted as an observed defect. Idle and non-algorithm portions are omitted from the sum regardless.

A:1216-1244 then sets joulesPerFrame = average summed power * processingLatencyMs/1000, with inference-time fallback. Power <=0.1W or unavailable timing yields zero. This is system/rail energy attributed to an algorithm-processing interval, not isolated CPU/GPU energy, total energy per delivered frame, or run-energy/frame-count. Missing power history returns zero-valued PowerStats; mergeDisplay or attachWindowIntegratedAsync still sets hasPower/hasSoC true (A:442-454,862-868), so expectations can call the row COMPLETE even without valid power/SoC. Old history can remain indefinitely when a producer stops because pruning occurs only on push, by newest-minus-oldest duration; the integrator has no staleness/coverage limit. No sample count, coverage fraction, fallback indicator, raw acquisition span, or quality flag is emitted with the per-frame row.

The decimated monitor power is sum(mean(V_i)*mean(I_i)), not sum(mean(V_i*I_i)); their difference is summed V-I covariance, with sign/magnitude dependent on actual data. Raw trapezoidal energy is not passed to PowerStats or the aggregator. No source-only claim of precise micro-window energy is justified when windows are shorter than the emission cadence or anchor uncertainty.

Separate direct Lynsyn CSV: L:279-290 computes trapezoids on accepted raw samples using device-clock deltas >0 and <0.5s. Previous raw sample is retained across emission windows; energy_j resets after each emitted/processed window (L:355-358), so it is a per-window contribution, not cumulative run energy. avg_power_w_trap divides that contribution by last-current-window raw time minus first-current-window raw time (L:329-342). CSV includes 7 I/V pairs, but only sensors 0..3 are averaged and included in power/energy; sensors 4..6 are copied from the latest raw record (L:303-307,335-342). Rejected decimated windows are not logged, and their accumulated energy is reset. Rejected raw samples may be bridged by the next accepted trapezoid when gap<0.5s. HALT discards partial-window energy and resets continuity.

4. Snapshot lifecycle, retention and output
beginFrame inserts a pending record keyed by frame ID, guards duplicate begin calls with frameStarted_, and invokes retention (A:365-387). mergeAlgorithm drops a record when no pending entry exists, records timing, then tests readiness (A:390-410). tryFinalize requires expected camera/algorithm/display; it does not gate SoC/power readiness (A:593-620). Since merge handlers reset pf.arrived immediately before this call, its age timeout is normally near zero. The flush thread instead attempts stale finalization after 2*mergeWaitMs since the last merge (A:318-345). Finalize drops records missing an expected module (A:981-1007), rather than exporting a consistently tagged partial row.

mergeDisplay waits for beginFrame, attaches async data and unconditionally finalizes/removes the pending entry, even before independently testing whether algorithm data has arrived (A:412-465); normal algorithm output ordering reduces that race for normal output paths. A finalized row enters batchBuffer; getLatestSnapshot reads the last batched row first, else last history row, and marks it valid if any of FPS/inference/power/energy is nonzero (A:637-704). This neither checks age nor proves all expected fields are valid. getAggregatedAt ignores its timestamp and returns latest (A:2104-2105). getAllSnapshots forces a batch flush, then returns only retained history, not all run records (A:227-231).

The constructor opens append-mode CSV/NDJSON and starts its own periodic flush thread (A:298-345); start() is empty (2115-2118). Flush swaps batch under batchMutex, serializes under ioMutex, then commits useful snapshots to history (A:727-766). 'Flushing' here does not call stream.flush(); the writes use newline, not std::endl (A:846-856,938). forceFlushBatch only drains batchBuffer (824-826). stop() stops/joins the thread but does not finalize pending or flush batch; the destructor force-flushes batch and closes files (A:353-360,2120-2126). Pending frames at shutdown are not finalized by those methods.

Output channels:
• Aggregator CSV/NDJSON: default output/realtime_metrics_010.csv and output/realtime_metrics010.ndjson; camera, algorithm, display, derived latency/energy, selected RAM/CPU and four power channels (A:298-299,829-939). These writers omit GPU utilization, temperatures, top-level fps/avg_power_w_alg/avg_latency_ms, per-stage timestamps, power timestamp, receipt/device timestamps, validity/coverage, most camera provenance and display FPS. SystemMetricsSnapshot.toJson has a different broader schema; these writers do not call it. Thus internal field presence does not prove its inclusion in these files.
• Direct SoC CSV: append per sample, timestamp to seconds, richer RAM/CPU/EMC/GR3D/temp fields (S:477-510). Stream errors are not explicitly checked; catch handles exceptions but default ofstream errors need not throw.
• Direct Lynsyn CSV: truncate at monitoring start; device_time_s, host system_time_us, pc0..3, I/V0..6, n_avg, energy_j, avg_power_w_trap; flush every 10 rows and at stop, and detect failbit (L:112-125,325-352,398).
• Entry point assigns unique metrics, telemetry, Lynsyn and profiler paths (SampleTestIntegrationCode_v35.cpp:621-640) and later emits flat hrl::MetricsSnapshot mission telemetry. Actual ConfigManager bridge and fitness consumer are absent, so exact propagation into controller scoring is unverified.

Runtime defaults are constructor/struct defaults only, not established run configuration:
A:282-290: retention_window_sec=2000, prune_max_age_sec=5 (min1), max_pending_frames=2000, max_history_size=10000, merge_wait_ms=2000 (clamp50..5000), flush_period_ms=1000 (clamp100..10000), json_flush_threshold=2000 (min1), drop_empty_compat_rows=true, drop_empty_flush_rows=true. expectsCamera/Algorithm/Display/Power/SoC all true (A:2049-2055). Async sample history window is fixed5s (A:142) plus maxHistorySize cap, pruned only at ingress (506-550). Pending/history enforcement uses counts only (1675-1684). retentionWindow_, pruneMaxAge_, flushThreshold_, flushCount_ and BATCH_SIZE_LIMIT=500 have no active enforcement/use beyond declarations/initialization/logging. Lynsyn default sampleRateMs=1000, durationSec=10, coreMask=0xF, periodSampling=true, outputCSV empty (SharedStructures/LynsynMonitorConfig.h:29-60). SoC defaults poll500ms, tegrastats --interval500, CSV enabled and jetson_nano_tegrastats.csv, Nano board, partial parsing allowed (SharedStructures/SoCConfig.h:29-59).

5. Important source defects/limitations with clear triggers
M1 — Repeated display records can block display for the whole merge wait. V:978-990 submits metrics for the retained frame on every presentation; deduplication at 947-955 applies only to upload. The first merge removes frameStarted/pending at A:459-461. A repeat then waits for frameStarted_.count at 422-423, times out at the configured merge wait; ConfigManager derives a capture-dependent value (often clamped to50ms), while the aggregator standalone default is2s, and returns frame-not-found at425-428. It does NOT create duplicate snapshot rows. It can stall rendering and cause latest-frame draining/skips; occurrence and magnitude in an actual run require logs.
M2 — Missing/stale sensor information can look complete: A:453-454 or862-868 sets presence true even if history is empty/invalid window. A:981-1010 validates flags, not PowerStats.isValid or freshness; missing power becomes J=0 (1216-1245). Tail holding without age limit can attribute a disconnected meter's last value indefinitely. Neither CSV exposes this quality distinction.
M3 — Lynsyn avg_power_w_trap denominator mismatch: energy includes preceding-window last sample -> current first sample (L:282-290; prev persists), but winStartDevS is current first sample and denominator is current last-first (329-332). For multiple samples and positive inter-window gap, constant positive power yields inflated reported window-average power by that extra boundary interval/denominator. energy_j itself retains the boundary area; do not 'fix' energy by discarding it.
M4 — HALT accumulator contamination: HALT clears accN at L:248 but does not clear accV/accC. If accepted raw samples accumulated earlier in that same drain, accN becomes0 so emission/reset at302-358 is skipped; next wake divides old+new sums by only new accN. This can corrupt an emitted voltage/current mean or trigger a sanity rejection. Reset both sums with count when discarding a partial window.
M5 — Incomplete bounded-memory contract: stale finalization/tryFinalize erase pending entries without removing arrivalOrder IDs (A:336-338,617-620); count-cap eviction removes pending only and leaves frameStarted (1675-1679). Runs with missing display or expectations disabling display can accumulate auxiliary bookkeeping even while pending/history caps hold. batchBuffer has no active cap (1264-1266,2087-2091); a blocked I/O flush permits growth. The configured time-retention and batch-threshold knobs are unused.
M6 — Persistence assurance is weak: periodic flush does not flush C++ streams to the OS, no disk durability request exists, and append writers simply return if streams are bad while snapshots still enter history (A:749-765,830,880). Export methods open the already-open member stream without first closing and iterate history without its mutex (708-723), so the public re-export path is unsafe/unreliable during normal operation. These do not prove data loss in any supplied experiment.
M7 — Overlay metadata is not reliably retained: applyOverlaysForRow iterates/modifies pending with overlay_mtx only (1760-1794); ordinary display finalization has already removed current pending and released mutex before calling it (459-465,1046-1047). Overlays are not copied into snapshot or these output writers. It therefore does not establish reliable timestamp-aligned output for camera/raw provenance.

6. Old README claims cross-check
Stage_01/README.md:26,52,65 claims accurate/precise joules per frame and specifically 'time-weighted trapezoidal integration over PowerStats'. Active per-frame PowerStats integration is ZOH, not trapezoidal. The monitor's separate raw CSV trapezoids do not feed per-frame snapshots. A defensible statement is 'estimated measured-rail energy attributed to each algorithm-processing interval, from timestamped decimated power with a zero-order-hold model'; accuracy needs cadence, electrical topology, clocks, gaps and provenance evidence.
README:61 gives '<5ms capture-to-display latency'; README:127-133 lists 41–52ms latency and performance/power gains. Source can identify latency endpoints but proves none of those numeric outcomes. The present endpoint is API return, capture may fall back to dequeue, values are clamped, and only successfully merged/expected frames are output (survivorship/selection matters). Claiming exact optical end-to-end latency or supplied numerical results is unsupported.
README:67,27 claims bounded queues/memory, survival of SD-card removal/no OOM. Aggregator count caps and separate flush locks exist, but its unbounded batch/auxiliary bookkeeping and unchecked stream failures do not substantiate that absolute guarantee. SoC's default cadence is500ms and Lynsyn's struct default1s, not the aggregator banner's claimed100Hz poller (A:27); final configured cadence is unavailable.

7. What an evaluation can and cannot conclude
This source inspection supports a telemetry architecture and formulas, plus the source defects above. It cannot authenticate the executable used for historical runs, confirm accepted camera/CPU/GPU mode, validate numerical equivalence of CPU/GPU outputs, measure real power/thermal sensor correctness, establish sample coverage/clock offset, or substantiate speed/power gains. For any full evaluation, preserve source/binary/config hashes; raw/decimated meter timestamps and sample-count/gap/rejection evidence; clock-anchor/acquisition uncertainty; frame ID and raw start/end/present stamps; per-stage drop counts; parser success/freshness; and explicit definitions of total-run versus algorithm-window energy. Retain missing/invalid data as such rather than treating zeros, COMPLETE, or a latest snapshot as proof of experimental validity.
