munin-rs

anomalies

last poll 2026-10-12 23:32:26 UTC · – ago · polls every 5m 0s

detector · rows through ago · 30m 0s grid — the poll above and this are different clocks

open anomalies
host — the host reporting the fieldservice — the graph the anomalous field is drawn onfield — the field whose values ran outside their expected rangefirst seen — when the excursion began — the link zooms to that windowlast seen — the latest sample seen outside the expected range — the link zooms to that windowobserved — the value at the excursion's peakexpected ± σ — what the seasonal model expected at that peak, ± its σ in the field's own unitsimpact — bounded symmetric impact: |observed − expected| ÷ (|observed| + |expected|), shown from 0% to 100%; unlike expected-relative effect it cannot become infinite at a zero or near-zero baseline; this is the default rank, with alternatives in the control abovepeak σ — the peak's deviation from expected, in σ of the seasonal model — a large multiple says that hour is normally very steady, not that the event was near-impossible (the σ pools few same-hour samples; the rule below has the numbers), which is why it is no longer what the table ranks by; a row marked unconfirmed had no weekly model to be put to; a row marked earlier rule was judged by a rule no longer in force, listed below
open (31)
28 fields began together — one host, one 30m 0s detector row, starting ago: read them as one event, ranked here by its worst
localhost netif_eth0 up ago ago 10.19M 21.74k ± 1.93k 100% 5278.3σ above the expectation
localhost ip-traffic IPv4 outbound ago ago 20.38M 60.40k ± 2.17k 99% 9359.2σ above the expectation
localhost nft_counters-bytes inet/myfilter/cnt_output_estrel ago ago 2.55M 7.64k ± 266.85 99% 9519.8σ above the expectation
localhost nft_counters-bytes inet/myfilter/cnt_output ago ago 2.55M 7.79k ± 274.60 99% 9251.8σ above the expectation
localhost nft_counters-bytes inet/myfilter/cnt_input ago ago 1.82M 8.42k ± 927.57 99% 1951.6σ above the expectation
localhost ip-traffic IPv4 inbound ago ago 9.30M 60.91k ± 1.23k 99% 7491.6σ above the expectation
localhost netif_eth0-packets sent packets ago ago 128.12 6.58 ± 1.12 90% 108.7σ above the expectation
localhost ip sent ago ago 180.63 21.04 ± 825.59m 79% 193.3σ above the expectation
localhost nft_counters inet/myfilter/cnt_output_estrel ago ago 173.02 22.02 ± 1.48 77% 102.1σ above the expectation
localhost nft_counters inet/myfilter/cnt_output ago ago 177.74 23.84 ± 1.56 76% 98.7σ above the expectation
localhost journal_messages notice/info ago ago 7.99 1.10 ± 20.65m 76% 333.8σ above the expectation
localhost numa-node0-locality local ago ago 8.67k 1.29k ± 37.58 74% 196.5σ above the expectation
localhost numa-node0-locality hit ago ago 8.67k 1.29k ± 37.58 74% 196.5σ above the expectation
localhost ip received ago ago 143.14 22.77 ± 562.86m 73% 213.9σ above the expectation
localhost ip-ecn Non-ECT ago ago 144.61 23.72 ± 579.21m 72% 208.7σ above the expectation
localhost vmstat-faults all faults ago ago 8.62k 1.60k ± 42.46 69% 165.3σ above the expectation
localhost loadavg 15 minutes ago ago 886.75m 181.87m ± 19.65m 66% 35.9σ above the expectation
localhost cpu user ago ago 32.11 7.25 ± 220.63m 63% 112.7σ above the expectation
localhost cpu-percore core 1 ago ago 26.10 6.49 ± 224.07m 60% 87.5σ above the expectation
localhost nft_counters inet/myfilter/cnt_input ago ago 90.95 24.06 ± 876.62m 58% 76.3σ above the expectation
localhost cpu-percore core 0 ago ago 22.86 6.33 ± 128.16m 57% 129.0σ above the expectation
localhost cpu system ago ago 16.32 5.00 ± 126.73m 53% 89.4σ above the expectation
localhost softnet processed ago ago 190.48 61.31 ± 2.47 51% 52.3σ above the expectation
localhost procs-scheduler soft interrupts ago ago 857.69 380.82 ± 6.52 39% 73.2σ above the expectation
localhost procs-scheduler interrupts ago ago 1.31k 591.69 ± 11.53 38% 61.9σ above the expectation
localhost procs-scheduler context switches ago ago 1.47k 703.46 ± 13.59 35% 56.1σ above the expectation
localhost netif_eth0-packets received packets ago ago 82.60 7.86 ± 1.26 83% 59.1σ above the expectation
localhost memory available ago ago 920.02M 1.55G ± 25.85M 27% 25.9σ below the expectation
localhost cpu idle ago ago 145.69 204.31 ± 869.99m 17% 67.4σ below the expectation
adapted — regime changes — each host keeps its last 50 closed and adapted together
host — the host reporting the fieldservice — the graph the anomalous field is drawn onfield — the field whose values ran outside their expected rangefirst seen — when the excursion began — the link zooms to that windowlast seen — the latest sample seen outside the expected range — the link zooms to that windowobserved — the value at the excursion's peakexpected ± σ — what the seasonal model expected at that peak, ± its σ in the field's own unitsimpact — bounded symmetric impact: |observed − expected| ÷ (|observed| + |expected|), shown from 0% to 100%; unlike expected-relative effect it cannot become infinite at a zero or near-zero baseline; this is the default rank, with alternatives in the control abovepeak σ — the peak's deviation from the pre-shift reference, in σ of the seasonal model; a row marked unconfirmed had no weekly model to be put to, and this table is newest-adapted firstlevel now — the level the reference has learned — the excursion-era mean at the hour the anomaly adapted; the expected column beside it is the pre-shift reference at the peak's hour, a different time of day, so the two are not a before/after pair for one rowadapted — when the reference finished learning the shifted level: the field never came back, the model went to meet it, and later verdicts are against the new level — a regime change, not a recovery
localhost netif_veth1-saturation outbound ago ago 10.50µ 0.00 ± 0.00 100% ∞σ above the expectation 12.83µ ago
localhost netif_veth1-packets sent packets ago ago 1.72 0.00 ± 0.00 100% ∞σ above the expectation 1.89 ago
localhost netif_veth1-packets received packets ago ago 1.90 324.07µ ± 199.61µ 100% 9529.1σ above the expectation 2.00 ago
localhost netif_veth1 up ago ago 1.05k 0.00 ± 0.00 100% ∞σ above the expectation 1.28k ago
localhost netif_podman0-saturation outbound ago ago 10.50µ 0.00 ± 0.00 100% ∞σ above the expectation 12.83µ ago
localhost netif_podman0-packets sent packets ago ago 1.72 0.00 ± 0.00 100% ∞σ above the expectation 1.89 ago
localhost netif_podman0-packets received packets ago ago 1.90 324.07µ ± 199.61µ 100% 9529.1σ above the expectation 2.00 ago
localhost netif_podman0 up ago ago 1.05k 0.00 ± 0.00 100% ∞σ above the expectation 1.28k ago
localhost tcp-opens estab resets ago ago 3.28 168.64m ± 3.74m 90% 831.7σ above the expectation 3.62 ago
localhost tcp-opens passive opens ago ago 1.99 961.34m ± 10.68m 35% 96.0σ above the expectation 2.25 ago
localhost tcp-opens active opens ago ago 3.50 728.96m ± 6.82m 66% 406.7σ above the expectation 3.73 ago
localhost nft_counters-bytes inet/myfilter/output/loopback ago ago 266.35 92.89 ± 1.87 48% 92.9σ above the expectation 278.15 ago
localhost nft_counters inet/myfilter/output/loopback ago ago 4.05 1.28 ± 18.89m 52% 146.7σ above the expectation 4.33 ago
localhost tcp-opens attempt fails ago ago 695.57m 188.17m ± 2.64m 57% 192.3σ above the expectation 658.06m ago
localhost nft_counters-bytes inet/myfilter/base_checks/loopback ago ago 172.90 92.88 ± 1.87 30% 42.8σ above the expectation 187.74 ago
localhost nft_counters inet/myfilter/base_checks/loopback ago ago 2.61 1.27 ± 20.56m 34% 65.1σ above the expectation 2.80 ago
localhost netif_veth3-saturation inbound ago ago 3.11n 74.42µ ± 876.05n 100% 84.9σ below the expectation 35.27µ ago
localhost netif_veth3 down ago ago 311.11m 7.44k ± 87.60 100% 84.9σ below the expectation 3.53k ago
closed — each host keeps its last 50 closed and adapted together
host — the host reporting the fieldservice — the graph the anomalous field is drawn onfield — the field whose values ran outside their expected rangefirst seen — when the excursion began — the link zooms to that windowlast seen — the latest sample seen outside the expected range — the link zooms to that windowobserved — the value at the excursion's peakexpected ± σ — what the seasonal model expected at that peak, ± its σ in the field's own unitsimpact — bounded symmetric impact: |observed − expected| ÷ (|observed| + |expected|), shown from 0% to 100%; unlike expected-relative effect it cannot become infinite at a zero or near-zero baseline; this is the default rank, with alternatives in the control abovepeak σ — the peak's deviation from expected, in σ of the seasonal model — a large multiple says that hour is normally very steady, not that the event was near-impossible; a row marked unconfirmed had no weekly model to be put to, and this table is newest-closed first; a row marked earlier rule was judged by a rule no longer in force, listed belowclosed — when the field settled back inside the expected range
localhost psi CPU ago ago 4.51 2.08 ± 89.17m 37% 27.3σ above the expectation ago
localhost netif_eth0-packets received packets ago ago 41.98 7.64 ± 580.26m 69% 59.2σ above the expectation ago
localhost diskstats_sda-iops writes ago ago 19.42 7.40 ± 638.59m 45% 18.8σ above the expectation ago
localhost cpu softirq ago ago 1.15 551.22m ± 24.03m 35% 24.8σ above the expectation ago
localhost selinux_avc hits ago ago 16.27k 9.20k ± 112.92 28% 62.6σ above the expectation ago
localhost softnet processed ago ago 181.34 59.92 ± 1.07 50% 113.9σ above the expectation ago
localhost procs-scheduler soft interrupts ago ago 653.63 368.73 ± 10.47 28% 27.2σ above the expectation ago
localhost procs-scheduler interrupts ago ago 992.56 577.04 ± 12.58 26% 33.0σ above the expectation ago
localhost procs-scheduler context switches ago ago 1.10k 691.10 ± 18.01 23% 22.4σ above the expectation ago
localhost nft_counters-bytes inet/myfilter/base_checks/established - related ago ago 389.98k 8.13k ± 520.95 96% 733.0σ above the expectation ago
localhost nft_counters-bytes inet/myfilter/cnt_input ago ago 390.19k 8.28k ± 524.56 96% 728.1σ above the expectation ago
localhost nft_counters-bytes inet/myfilter/cnt_output_estrel ago ago 765.54k 7.89k ± 767.32 98% 987.4σ above the expectation ago
localhost nft_counters-bytes inet/myfilter/cnt_output ago ago 765.83k 8.02k ± 770.95 98% 983.0σ above the expectation ago
localhost nft_counters inet/myfilter/base_checks/established - related ago ago 138.19 21.76 ± 735.56m 73% 158.3σ above the expectation ago
localhost nft_counters inet/myfilter/cnt_input ago ago 141.57 24.06 ± 715.31m 71% 164.3σ above the expectation ago
localhost nft_counters inet/myfilter/cnt_output_estrel ago ago 156.93 21.85 ± 628.46m 76% 214.9σ above the expectation ago
localhost nft_counters inet/myfilter/cnt_output ago ago 161.38 23.58 ± 725.97m 75% 189.8σ above the expectation ago
localhost netif_eth0-packets sent packets ago ago 119.70 6.95 ± 499.39m 89% 225.8σ above the expectation ago
localhost netif_eth0-packets received packets ago ago 100.46 7.71 ± 346.89m 86% 267.4σ above the expectation ago
localhost netif_eth0 up ago ago 3.08M 22.81k ± 3.05k 99% 1003.0σ above the expectation ago
localhost journal_messages notice/info ago ago 3.76 1.12 ± 29.60m 54% 89.1σ above the expectation ago
localhost ip-traffic IPv4 outbound ago ago 6.13M 61.85k ± 3.92k 98% 1548.2σ above the expectation ago
localhost ip-traffic IPv4 inbound ago ago 3.12M 62.20k ± 3.93k 96% 778.2σ above the expectation ago
localhost ip-ecn Non-ECT ago ago 142.19 23.48 ± 556.07m 72% 213.5σ above the expectation ago
localhost ip sent ago ago 159.47 21.43 ± 521.11m 76% 264.9σ above the expectation ago
localhost ip received ago ago 141.35 22.64 ± 489.73m 72% 242.4σ above the expectation ago
localhost cpu softirq ago ago 1.11 541.99m ± 26.91m 34% 21.1σ above the expectation ago
localhost netif_veth4-saturation outbound ago ago 3.11n 84.66µ ± 1.81µ 100% 46.8σ below the expectation ago
localhost netif_veth4 up ago ago 311.11m 8.47k ± 180.88 100% 46.8σ below the expectation ago
localhost netif_veth1-saturation inbound ago ago 749.11µ 6.82n ± 18.80n 100% 39850.2σ above the expectation ago
localhost netif_veth1 down ago ago 74.91k 682.22m ± 1.88 100% 39850.2σ above the expectation ago
localhost netif_podman0-saturation inbound ago ago 746.08µ 5.99n ± 16.89n 100% 44184.5σ above the expectation ago

318 fields analyzed, 4 skipped in the latest detector pass — the detector's most recent pass over each host in scope — the pass that judged the rows the line above is dated by, not a running total. It works a fixed grid, so most poll cycles have no new row for it and leave this standing; the tracked count below is what the sidecars hold in all

213 fields awaiting a verdict — fields the detector could not judge in that same pass: too little history for the daily model, fewer than two same-time-of-day shifts, a non-finite model, or a model with no scale at all — expectation, σ and day spread every one of them zero, which leaves nothing for a departure to be material against; an otherwise flat model is answerable and detects material departures

current rule: an anomaly opens after 3 consecutive 30-minute rows more than 5.9σ out, more than 25% off the expectation and more than 50% of the field's own day spread (its 5th-to-95th-percentile range over the preceding day, applied once at least 21 rows of that day are known) — the σ comes from at most 6 same-time-of-day shifts over the previous 6 days, pooling the known rows around each shift, and the bar widens on a thinner pool because a σ estimated from that few carries its own error (a flat finite pool is deterministic: a material departure is outside, unless the expectation and the day spread are flat at zero too, which leaves no scale to be material against) — and is then put to the same hour of the same weekday, which discards it when that hour routinely looks like this; a field with under 3 weeks of history has no weekly model to be put to, so it opens marked unconfirmed rather than being held back; it closes after 6 rows each back inside 4.0σ or under either effect floor — and that closing run of 6 rows is then put to the two halves of its own reference, split at the excursion's first row: a run nearer the level the reference learned DURING the excursion than the level before it is filed as a regime change rather than a recovery (detector method 2)

529 fields tracked, 126 excursions suppressed, 50 regime changes adapted since the first scan — cumulative across every scan ever run, unlike the two per-cycle lines above: every field the detector has latched, every excursion the weekly gate has discarded as seasonally normal, and every anomaly closed because the reference learned its level