anomalies
last poll 2026-10-12 23:32:26 UTC · – ago · polls every 5m 0s
detector · rows through ago · 30m 0s grid — the poll above and this are different clocks
| host — the host reporting the field | service — the graph the anomalous field is drawn on | field — the field whose values ran outside their expected range | first seen — when the excursion began — the link zooms to that window | last seen — the latest sample seen outside the expected range — the link zooms to that window | observed — the value at the excursion's peak | expected ± σ — what the seasonal model expected at that peak, ± its σ in the field's own units | impact — bounded symmetric impact: |observed − expected| ÷ (|observed| + |expected|), shown from 0% to 100%; unlike expected-relative effect it cannot become infinite at a zero or near-zero baseline; this is the default rank, with alternatives in the control above | peak σ — the peak's deviation from expected, in σ of the seasonal model — a large multiple says that hour is normally very steady, not that the event was near-impossible (the σ pools few same-hour samples; the rule below has the numbers), which is why it is no longer what the table ranks by; a row marked unconfirmed had no weekly model to be put to; a row marked earlier rule was judged by a rule no longer in force, listed below |
|---|---|---|---|---|---|---|---|---|
| open (31) | ||||||||
| 28 fields began together — one host, one 30m 0s detector row, starting ago: read them as one event, ranked here by its worst | ||||||||
| localhost | netif_eth0 | up | ago | ago | 10.19M | 21.74k ± 1.93k | 100% | 5278.3σ above the expectation |
| localhost | ip-traffic | IPv4 outbound | ago | ago | 20.38M | 60.40k ± 2.17k | 99% | 9359.2σ above the expectation |
| localhost | nft_counters-bytes | inet/myfilter/cnt_output_estrel | ago | ago | 2.55M | 7.64k ± 266.85 | 99% | 9519.8σ above the expectation |
| localhost | nft_counters-bytes | inet/myfilter/cnt_output | ago | ago | 2.55M | 7.79k ± 274.60 | 99% | 9251.8σ above the expectation |
| localhost | nft_counters-bytes | inet/myfilter/base_checks/established - related | ago | ago | 1.82M | 8.25k ± 897.96 | 99% | 2015.8σ above the expectation |
| localhost | nft_counters-bytes | inet/myfilter/cnt_input | ago | ago | 1.82M | 8.42k ± 927.57 | 99% | 1951.6σ above the expectation |
| localhost | ip-traffic | IPv4 inbound | ago | ago | 9.30M | 60.91k ± 1.23k | 99% | 7491.6σ above the expectation |
| localhost | netif_eth0-packets | sent packets | ago | ago | 128.12 | 6.58 ± 1.12 | 90% | 108.7σ above the expectation |
| localhost | ip | sent | ago | ago | 180.63 | 21.04 ± 825.59m | 79% | 193.3σ above the expectation |
| localhost | nft_counters | inet/myfilter/cnt_output_estrel | ago | ago | 173.02 | 22.02 ± 1.48 | 77% | 102.1σ above the expectation |
| localhost | nft_counters | inet/myfilter/cnt_output | ago | ago | 177.74 | 23.84 ± 1.56 | 76% | 98.7σ above the expectation |
| localhost | journal_messages | notice/info | ago | ago | 7.99 | 1.10 ± 20.65m | 76% | 333.8σ above the expectation |
| localhost | numa-node0-locality | local | ago | ago | 8.67k | 1.29k ± 37.58 | 74% | 196.5σ above the expectation |
| localhost | numa-node0-locality | hit | ago | ago | 8.67k | 1.29k ± 37.58 | 74% | 196.5σ above the expectation |
| localhost | ip | received | ago | ago | 143.14 | 22.77 ± 562.86m | 73% | 213.9σ above the expectation |
| localhost | ip-ecn | Non-ECT | ago | ago | 144.61 | 23.72 ± 579.21m | 72% | 208.7σ above the expectation |
| localhost | vmstat-faults | all faults | ago | ago | 8.62k | 1.60k ± 42.46 | 69% | 165.3σ above the expectation |
| localhost | loadavg | 15 minutes | ago | ago | 886.75m | 181.87m ± 19.65m | 66% | 35.9σ above the expectation |
| localhost | cpu | user | ago | ago | 32.11 | 7.25 ± 220.63m | 63% | 112.7σ above the expectation |
| localhost | cpu-percore | core 1 | ago | ago | 26.10 | 6.49 ± 224.07m | 60% | 87.5σ above the expectation |
| localhost | nft_counters | inet/myfilter/base_checks/established - related | ago | ago | 87.27 | 21.95 ± 909.11m | 60% | 71.9σ above the expectation |
| localhost | nft_counters | inet/myfilter/cnt_input | ago | ago | 90.95 | 24.06 ± 876.62m | 58% | 76.3σ above the expectation |
| localhost | cpu-percore | core 0 | ago | ago | 22.86 | 6.33 ± 128.16m | 57% | 129.0σ above the expectation |
| localhost | cpu | system | ago | ago | 16.32 | 5.00 ± 126.73m | 53% | 89.4σ above the expectation |
| localhost | softnet | processed | ago | ago | 190.48 | 61.31 ± 2.47 | 51% | 52.3σ above the expectation |
| localhost | procs-scheduler | soft interrupts | ago | ago | 857.69 | 380.82 ± 6.52 | 39% | 73.2σ above the expectation |
| localhost | procs-scheduler | interrupts | ago | ago | 1.31k | 591.69 ± 11.53 | 38% | 61.9σ above the expectation |
| localhost | procs-scheduler | context switches | ago | ago | 1.47k | 703.46 ± 13.59 | 35% | 56.1σ above the expectation |
| localhost | netif_eth0-packets | received packets | ago | ago | 82.60 | 7.86 ± 1.26 | 83% | 59.1σ above the expectation |
| localhost | memory | available | ago | ago | 920.02M | 1.55G ± 25.85M | 27% | 25.9σ below the expectation |
| localhost | cpu | idle | ago | ago | 145.69 | 204.31 ± 869.99m | 17% | 67.4σ below the expectation |
| host — the host reporting the field | service — the graph the anomalous field is drawn on | field — the field whose values ran outside their expected range | first seen — when the excursion began — the link zooms to that window | last seen — the latest sample seen outside the expected range — the link zooms to that window | observed — the value at the excursion's peak | expected ± σ — what the seasonal model expected at that peak, ± its σ in the field's own units | impact — bounded symmetric impact: |observed − expected| ÷ (|observed| + |expected|), shown from 0% to 100%; unlike expected-relative effect it cannot become infinite at a zero or near-zero baseline; this is the default rank, with alternatives in the control above | peak σ — the peak's deviation from the pre-shift reference, in σ of the seasonal model; a row marked unconfirmed had no weekly model to be put to, and this table is newest-adapted first | level now — the level the reference has learned — the excursion-era mean at the hour the anomaly adapted; the expected column beside it is the pre-shift reference at the peak's hour, a different time of day, so the two are not a before/after pair for one row | adapted — when the reference finished learning the shifted level: the field never came back, the model went to meet it, and later verdicts are against the new level — a regime change, not a recovery |
|---|---|---|---|---|---|---|---|---|---|---|
| localhost | netif_veth1-saturation | outbound | ago | ago | 10.50µ | 0.00 ± 0.00 | 100% | ∞σ above the expectation | 12.83µ | ago |
| localhost | netif_veth1-packets | sent packets | ago | ago | 1.72 | 0.00 ± 0.00 | 100% | ∞σ above the expectation | 1.89 | ago |
| localhost | netif_veth1-packets | received packets | ago | ago | 1.90 | 324.07µ ± 199.61µ | 100% | 9529.1σ above the expectation | 2.00 | ago |
| localhost | netif_veth1 | up | ago | ago | 1.05k | 0.00 ± 0.00 | 100% | ∞σ above the expectation | 1.28k | ago |
| localhost | netif_podman0-saturation | outbound | ago | ago | 10.50µ | 0.00 ± 0.00 | 100% | ∞σ above the expectation | 12.83µ | ago |
| localhost | netif_podman0-packets | sent packets | ago | ago | 1.72 | 0.00 ± 0.00 | 100% | ∞σ above the expectation | 1.89 | ago |
| localhost | netif_podman0-packets | received packets | ago | ago | 1.90 | 324.07µ ± 199.61µ | 100% | 9529.1σ above the expectation | 2.00 | ago |
| localhost | netif_podman0 | up | ago | ago | 1.05k | 0.00 ± 0.00 | 100% | ∞σ above the expectation | 1.28k | ago |
| localhost | tcp-opens | estab resets | ago | ago | 3.28 | 168.64m ± 3.74m | 90% | 831.7σ above the expectation | 3.62 | ago |
| localhost | tcp-opens | passive opens | ago | ago | 1.99 | 961.34m ± 10.68m | 35% | 96.0σ above the expectation | 2.25 | ago |
| localhost | tcp-opens | active opens | ago | ago | 3.50 | 728.96m ± 6.82m | 66% | 406.7σ above the expectation | 3.73 | ago |
| localhost | nft_counters-bytes | inet/myfilter/output/loopback | ago | ago | 266.35 | 92.89 ± 1.87 | 48% | 92.9σ above the expectation | 278.15 | ago |
| localhost | nft_counters | inet/myfilter/output/loopback | ago | ago | 4.05 | 1.28 ± 18.89m | 52% | 146.7σ above the expectation | 4.33 | ago |
| localhost | tcp-opens | attempt fails | ago | ago | 695.57m | 188.17m ± 2.64m | 57% | 192.3σ above the expectation | 658.06m | ago |
| localhost | nft_counters-bytes | inet/myfilter/base_checks/loopback | ago | ago | 172.90 | 92.88 ± 1.87 | 30% | 42.8σ above the expectation | 187.74 | ago |
| localhost | nft_counters | inet/myfilter/base_checks/loopback | ago | ago | 2.61 | 1.27 ± 20.56m | 34% | 65.1σ above the expectation | 2.80 | ago |
| localhost | netif_veth3-saturation | inbound | ago | ago | 3.11n | 74.42µ ± 876.05n | 100% | 84.9σ below the expectation | 35.27µ | ago |
| localhost | netif_veth3 | down | ago | ago | 311.11m | 7.44k ± 87.60 | 100% | 84.9σ below the expectation | 3.53k | ago |
| host — the host reporting the field | service — the graph the anomalous field is drawn on | field — the field whose values ran outside their expected range | first seen — when the excursion began — the link zooms to that window | last seen — the latest sample seen outside the expected range — the link zooms to that window | observed — the value at the excursion's peak | expected ± σ — what the seasonal model expected at that peak, ± its σ in the field's own units | impact — bounded symmetric impact: |observed − expected| ÷ (|observed| + |expected|), shown from 0% to 100%; unlike expected-relative effect it cannot become infinite at a zero or near-zero baseline; this is the default rank, with alternatives in the control above | peak σ — the peak's deviation from expected, in σ of the seasonal model — a large multiple says that hour is normally very steady, not that the event was near-impossible; a row marked unconfirmed had no weekly model to be put to, and this table is newest-closed first; a row marked earlier rule was judged by a rule no longer in force, listed below | closed — when the field settled back inside the expected range |
|---|---|---|---|---|---|---|---|---|---|
| localhost | psi | CPU | ago | ago | 4.51 | 2.08 ± 89.17m | 37% | 27.3σ above the expectation | ago |
| localhost | netif_eth0-packets | received packets | ago | ago | 41.98 | 7.64 ± 580.26m | 69% | 59.2σ above the expectation | ago |
| localhost | diskstats_sda-iops | writes | ago | ago | 19.42 | 7.40 ± 638.59m | 45% | 18.8σ above the expectation | ago |
| localhost | cpu | softirq | ago | ago | 1.15 | 551.22m ± 24.03m | 35% | 24.8σ above the expectation | ago |
| localhost | selinux_avc | hits | ago | ago | 16.27k | 9.20k ± 112.92 | 28% | 62.6σ above the expectation | ago |
| localhost | softnet | processed | ago | ago | 181.34 | 59.92 ± 1.07 | 50% | 113.9σ above the expectation | ago |
| localhost | procs-scheduler | soft interrupts | ago | ago | 653.63 | 368.73 ± 10.47 | 28% | 27.2σ above the expectation | ago |
| localhost | procs-scheduler | interrupts | ago | ago | 992.56 | 577.04 ± 12.58 | 26% | 33.0σ above the expectation | ago |
| localhost | procs-scheduler | context switches | ago | ago | 1.10k | 691.10 ± 18.01 | 23% | 22.4σ above the expectation | ago |
| localhost | nft_counters-bytes | inet/myfilter/base_checks/established - related | ago | ago | 389.98k | 8.13k ± 520.95 | 96% | 733.0σ above the expectation | ago |
| localhost | nft_counters-bytes | inet/myfilter/cnt_input | ago | ago | 390.19k | 8.28k ± 524.56 | 96% | 728.1σ above the expectation | ago |
| localhost | nft_counters-bytes | inet/myfilter/cnt_output_estrel | ago | ago | 765.54k | 7.89k ± 767.32 | 98% | 987.4σ above the expectation | ago |
| localhost | nft_counters-bytes | inet/myfilter/cnt_output | ago | ago | 765.83k | 8.02k ± 770.95 | 98% | 983.0σ above the expectation | ago |
| localhost | nft_counters | inet/myfilter/base_checks/established - related | ago | ago | 138.19 | 21.76 ± 735.56m | 73% | 158.3σ above the expectation | ago |
| localhost | nft_counters | inet/myfilter/cnt_input | ago | ago | 141.57 | 24.06 ± 715.31m | 71% | 164.3σ above the expectation | ago |
| localhost | nft_counters | inet/myfilter/cnt_output_estrel | ago | ago | 156.93 | 21.85 ± 628.46m | 76% | 214.9σ above the expectation | ago |
| localhost | nft_counters | inet/myfilter/cnt_output | ago | ago | 161.38 | 23.58 ± 725.97m | 75% | 189.8σ above the expectation | ago |
| localhost | netif_eth0-packets | sent packets | ago | ago | 119.70 | 6.95 ± 499.39m | 89% | 225.8σ above the expectation | ago |
| localhost | netif_eth0-packets | received packets | ago | ago | 100.46 | 7.71 ± 346.89m | 86% | 267.4σ above the expectation | ago |
| localhost | netif_eth0 | up | ago | ago | 3.08M | 22.81k ± 3.05k | 99% | 1003.0σ above the expectation | ago |
| localhost | journal_messages | notice/info | ago | ago | 3.76 | 1.12 ± 29.60m | 54% | 89.1σ above the expectation | ago |
| localhost | ip-traffic | IPv4 outbound | ago | ago | 6.13M | 61.85k ± 3.92k | 98% | 1548.2σ above the expectation | ago |
| localhost | ip-traffic | IPv4 inbound | ago | ago | 3.12M | 62.20k ± 3.93k | 96% | 778.2σ above the expectation | ago |
| localhost | ip-ecn | Non-ECT | ago | ago | 142.19 | 23.48 ± 556.07m | 72% | 213.5σ above the expectation | ago |
| localhost | ip | sent | ago | ago | 159.47 | 21.43 ± 521.11m | 76% | 264.9σ above the expectation | ago |
| localhost | ip | received | ago | ago | 141.35 | 22.64 ± 489.73m | 72% | 242.4σ above the expectation | ago |
| localhost | cpu | softirq | ago | ago | 1.11 | 541.99m ± 26.91m | 34% | 21.1σ above the expectation | ago |
| localhost | netif_veth4-saturation | outbound | ago | ago | 3.11n | 84.66µ ± 1.81µ | 100% | 46.8σ below the expectation | ago |
| localhost | netif_veth4 | up | ago | ago | 311.11m | 8.47k ± 180.88 | 100% | 46.8σ below the expectation | ago |
| localhost | netif_veth1-saturation | inbound | ago | ago | 749.11µ | 6.82n ± 18.80n | 100% | 39850.2σ above the expectation | ago |
| localhost | netif_veth1 | down | ago | ago | 74.91k | 682.22m ± 1.88 | 100% | 39850.2σ above the expectation | ago |
| localhost | netif_podman0-saturation | inbound | ago | ago | 746.08µ | 5.99n ± 16.89n | 100% | 44184.5σ above the expectation | ago |
318 fields analyzed, 4 skipped in the latest detector pass — the detector's most recent pass over each host in scope — the pass that judged the rows the line above is dated by, not a running total. It works a fixed grid, so most poll cycles have no new row for it and leave this standing; the tracked count below is what the sidecars hold in all
213 fields awaiting a verdict — fields the detector could not judge in that same pass: too little history for the daily model, fewer than two same-time-of-day shifts, a non-finite model, or a model with no scale at all — expectation, σ and day spread every one of them zero, which leaves nothing for a departure to be material against; an otherwise flat model is answerable and detects material departures
current rule: an anomaly opens after 3 consecutive 30-minute rows more than 5.9σ out, more than 25% off the expectation and more than 50% of the field's own day spread (its 5th-to-95th-percentile range over the preceding day, applied once at least 21 rows of that day are known) — the σ comes from at most 6 same-time-of-day shifts over the previous 6 days, pooling the known rows around each shift, and the bar widens on a thinner pool because a σ estimated from that few carries its own error (a flat finite pool is deterministic: a material departure is outside, unless the expectation and the day spread are flat at zero too, which leaves no scale to be material against) — and is then put to the same hour of the same weekday, which discards it when that hour routinely looks like this; a field with under 3 weeks of history has no weekly model to be put to, so it opens marked unconfirmed rather than being held back; it closes after 6 rows each back inside 4.0σ or under either effect floor — and that closing run of 6 rows is then put to the two halves of its own reference, split at the excursion's first row: a run nearer the level the reference learned DURING the excursion than the level before it is filed as a regime change rather than a recovery (detector method 2)
529 fields tracked, 126 excursions suppressed, 50 regime changes adapted since the first scan — cumulative across every scan ever run, unlike the two per-cycle lines above: every field the detector has latched, every excursion the weekly gate has discarded as seasonally normal, and every anomaly closed because the reference learned its level