# Files changed · PR \#109 · flambo/atlantis

[View on GitCafe](https://git.cafe/flambo/atlantis/pull/109/diffs)

Repository: [flambo/atlantis](https://git.cafe/flambo/atlantis)

Visibility: public

[feat\(netflow-db\): pin MAAD oracle to maad main @ b7bbb7d](https://git.cafe/flambo/atlantis/pull/109.md)

Version: 0

Head commit: b590ac31ce70a3734ec835b846055a8aea5799b3

Base commit: fd5f115fdc833e1706341d4846d1d9d7616dc93e

Comparison base: fd5f115fdc833e1706341d4846d1d9d7616dc93e

Showing up to 20 entries from this revision. If the pull request changes, reopen the first page before continuing.

## docs/code/maad-conformance.md

Status: modified; +50; −14

````diff
@@ -17,21 +17,23 @@
 ## Pinned oracle
 
 The Haskell oracle is pinned to
-`chris-misa/maad@3ae75363d44e08faacdee186d4ff8906c6ccd06a`. The only source
-change made to that oracle checkout is `deltaQ`: Atlantis uses `1/8` instead
-of the upstream `1/16`. The unpatched checkout is not comparable.
+`chris-misa/maad@b7bbb7dd119f9e050cb74239b89e472ad58ec5db` on `main` ("filter for critical region when
+estimating spectrum"). `main` carries the merged `atomic-full-sortout` branch.
+The only source change made to that oracle checkout is `deltaQ`: Atlantis uses
+`1/8` instead of the upstream `1/16`. The unpatched checkout is not comparable.
+The goldens are generated from this patched checkout.
 
 Build the oracle from that checkout by applying exactly that one-line change,
 then run its build script in the supplied Nix shell:
 
 ```sh
 git clone https://github.com/chris-misa/maad.git /tmp/maad
-git -C /tmp/maad checkout 3ae75363d44e08faacdee186d4ff8906c6ccd06a
+git -C /tmp/maad checkout b7bbb7dd119f9e050cb74239b89e472ad58ec5db
 (cd /tmp/maad && git apply --unidiff-zero <<'PATCH'
 diff --git a/MAAD.hs b/MAAD.hs
 --- a/MAAD.hs
 +++ b/MAAD.hs
-@@ -66 +66 @@ deltaQ :: Double
+@@ -93 +93 @@ deltaQ :: Double
 -deltaQ = 1.0 / 16.0
 +deltaQ = 1.0 / 8.0
 PATCH
@@ -39,9 +41,40 @@
 (cd /tmp/maad && nix-shell shell.nix --run './compile.sh')
 ```
 
-If the checkout's GHC dependencies are already installed, `./compile.sh` can
-be run directly instead.
+If the checkout's GHC dependencies (including `wide-word`) are already
+installed, `./compile.sh` can be run directly instead.
 
+### Changes from the previous pin
+
+The previous pin was `3ae75363d44e08faacdee186d4ff8906c6ccd06a`. For IPv4
+address sets, prefix selection and the estimator formulas are unchanged; only
+the last bits of some floating-point sums differ, from a different summation
+order. Spectrum filtering and dimension serialization changed, and Rust ports
+all three changes (the `mixed` fixture drops from 25 to 16 spectrum rows):
+
+- Dimension rows are ordered by `q` (`0`, `1`, `2`) instead of `1`, `0`, `2`.
+- Each dimension row carries `sd`. For `q = 0` and `q = 2` it is the structure
+  row's `sd` at that `q`; no rescaling is needed because `|q - 1| = 1`. For
+  `q = 1` upstream emits `0`, because `D1` comes from an entropy regression
+  with no variance estimate. Treat that `0` as "not estimated", not as a
+  certain value.
+- The spectrum keeps only rows from the critical region. With central
+  differences `alpha = (tau[i+1] - tau[i-1]) / (2 q_step)` and
+  `f = q alpha - tau`, the region runs from `q_min = min(0, min q <= 0 with
+f > 0)` to `q_max = max(1, max q >= 1 with f > 0)`. Inside it, the alpha run
+  must be non-increasing (upstream `a1 >= a2`) instead of strictly decreasing.
+  Structure rows and dimension values are unchanged.
+
+The new head also adds changes that Atlantis does not port here:
+
+- IPv6 input (`--ipv6`), with default prefix lengths `/23`--`/64`, 128-bit
+  prefixes, and a nearly-full test of `log2(count) / (128 - pl)`.
+- `--test` now runs a Hotelling T² test against a linear interpolation of the
+  structure function, and `--compare-structure` compares against a
+  precomputed structure CSV. Atlantis does not run hypothesis tests.
+- Partition-function `q` values and the `UniformSet` generator changed. Neither
+  is part of the compared output.
+
 Build the Rust release binary from this repository:
 
 ```sh
@@ -68,17 +101,20 @@
 ## Known edge cases
 
 The Haskell executable cannot produce JSON for an empty set, a singleton, or a
-set whose every prefix is filtered at the default `/8`--`/24` range: after
-filtering, it calls `foldl1` on an empty list and exits nonzero. Rust returns an
+set whose every prefix is filtered at the default `/8`--`/24` range. An empty
+set fails when the oracle reads its address family; the other cases call
+`foldl1` on an empty list after filtering. All exit nonzero. Rust returns an
 empty result for these inputs. The validator reports the Haskell command
 failure, so omit such cases from a passing conformance run or treat that
 failure as the expected oracle limitation.
 
-Rust also intentionally treats alpha decreases at or below its `1e-12` grid
-epsilon as non-decreasing when building the spectrum. Consequently, a
-sub-epsilon spectrum turn can produce fewer Rust rows than Haskell; row counts
-remain exact in the validator, so use cases with meaningful spectrum curvature
-when expecting a pass.
+Rust treats alphas within its `1e-12` grid epsilon as ties, so a sub-epsilon
+increase keeps the spectrum run going where upstream's exact `a1 >= a2` stops.
+Upstream's exact test reacts to last-bit noise in `tau`: on a perfectly linear
+structure curve, Haskell stops after 19 rows and Rust keeps all 30 tied rows.
+The goldens and 20 real windows match exactly with this tolerance, and so do
+tests with an exact comparison. Row counts remain exact in the validator, so
+use cases with meaningful spectrum curvature when expecting a pass.
 
 There is intentionally no synthetic-data generator, scheduled CI job, or
 persisted report format. Keep sensitive and private address sets local.
````

## scripts/local/validate_maad.py

Status: modified; +1; −1

```diff
@@ -230,7 +230,7 @@
     section_fields = {
         "structure": ("q", "tauTilde", "sd"),
         "spectrum": ("alpha", "f"),
-        "dimensions": ("q", "dim"),
+        "dimensions": ("q", "dim", "sd"),
     }
     for section, fields in section_fields.items():
         errors.extend(compare_rows(rust, haskell, section, fields, absolute, relative))
```

## tools/netflow-db/src/maad.rs

Status: modified; +150; −37

```diff
@@ -5,7 +5,7 @@
 use std::net::Ipv4Addr;
 
 const MIN_MAAD_ADDRESSES: usize = 2;
-const SCHEMA_VERSION: u32 = 2;
+const SCHEMA_VERSION: u32 = 3;
 const DEFAULT_FULL_THRESHOLD: f64 = 0.05;
 const DEFAULT_Q_STEP: f64 = 1.0 / 8.0;
 const DEFAULT_Q_MIN: f64 = -0.5;
@@ -110,6 +110,7 @@
 pub struct DimensionRow {
     pub q: f64,
     pub dim: f64,
+    pub sd: f64,
 }
 
 #[derive(Clone, Debug)]
@@ -458,26 +459,58 @@
         .collect()
 }
 
-fn compute_spectrum(structure: &[StructureRow], q_step: f64) -> Vec<SpectrumRow> {
+/// Central-difference `(q, alpha, f)` estimates for every interior structure row.
-    let alphas: Vec<_> = structure
+fn spectrum_estimates(structure: &[StructureRow], q_step: f64) -> Vec<(f64, SpectrumRow)> {
+    structure
         .windows(3)
         .map(|rows| {
             let row = rows[1];
             let alpha = (rows[2].tau_tilde - rows[0].tau_tilde) / (2.0 * q_step);
-            SpectrumRow {
-                alpha,
-                f: row.q * alpha - row.tau_tilde,
-            }
+            (
+                row.q,
+                SpectrumRow {
+                    alpha,
+                    f: row.q * alpha - row.tau_tilde,
+                },
+            )
         })
+        .collect()
+}
+
+/// The `(q_max, q_min)` critical region: the widest q range with a positive `f` estimate,
+/// always covering `0..=1`.
+fn compute_critical_region(structure: &[StructureRow], q_step: f64) -> (f64, f64) {
+    let estimates = spectrum_estimates(structure, q_step);
+    let positive = |keep: fn(f64) -> bool| {
+        estimates
+            .iter()
+            .filter(move |(q, row)| keep(*q) && row.f > 0.0)
+            .map(|(q, _)| *q)
+    };
+    (
+        positive(|q| q >= 1.0).fold(1.0, f64::max),
+        positive(|q| q <= 0.0).fold(0.0, f64::min),
+    )
+}
+
+/// Spectrum rows from the critical region where alpha is non-increasing.
+///
+/// Alphas closer than `GRID_EPSILON` count as ties, which upstream's `a1 >= a2` keeps.
+fn compute_spectrum(structure: &[StructureRow], q_step: f64) -> Vec<SpectrumRow> {
+    let (q_max, q_min) = compute_critical_region(structure, q_step);
+    let alphas: Vec<_> = spectrum_estimates(structure, q_step)
+        .into_iter()
+        .filter(|(q, _)| q_min <= *q && *q <= q_max)
+        .map(|(_, row)| row)
         .collect();
     let mut rows = Vec::new();
     let mut started = false;
     for pair in alphas.windows(2) {
-        let decreasing = pair[0].alpha - pair[1].alpha > GRID_EPSILON;
+        let non_increasing = pair[0].alpha - pair[1].alpha >= -GRID_EPSILON;
-        if !started && !decreasing {
+        if !started && !non_increasing {
             continue;
         }
+        if !non_increasing {
-        if !decreasing {
             break;
         }
         started = true;
@@ -495,20 +528,33 @@
     if prefix_lengths.is_empty() || structure.is_empty() {
         return Vec::new();
     }
-    let mut rows = vec![DimensionRow {
-        q: 1.0,
-        dim: info_dimension(counts, prefix_lengths, total_addresses),
-    }];
+    dimension_rows(
-    rows.extend(
+        structure,
+        info_dimension(counts, prefix_lengths, total_addresses),
+    )
+}
+
+fn dimension_rows(structure: &[StructureRow], information_dimension: f64) -> Vec<DimensionRow> {
+    let from_structure = |q: f64| {
         structure
             .iter()
+            .find(|row| row.q == q)
-            .filter(|row| row.q.abs() < 1e-12 || (row.q - 2.0).abs() < 1e-12)
             .map(|row| DimensionRow {
+                q,
+                dim: row.tau_tilde / (q - 1.0),
+                sd: row.sd,
+            })
+            .expect("validated q grids contain q=0 and q=2")
+    };
+    vec![
+        from_structure(0.0),
+        DimensionRow {
+            q: 1.0,
+            dim: information_dimension,
+            sd: 0.0,
+        },
+        from_structure(2.0),
+    ]
-                q: row.q,
-                dim: row.tau_tilde / (row.q - 1.0),
-            }),
-    );
-    rows
 }
 
 fn info_dimension(
@@ -770,17 +816,31 @@
                 .sum::<f64>()
                 / denominator
         };
+        let at = |q: f64| {
-        let mut rows = vec![DimensionRow { q: 1.0, dim: info }];
-        rows.extend(
             structure
                 .iter()
+                .find(|row| row.q == q)
+                .copied()
+                .expect("reference q grids contain q=0 and q=2")
-                .filter(|row| row.q.abs() < 1e-12 || (row.q - 2.0).abs() < 1e-12)
-                .map(|row| DimensionRow {
-                    q: row.q,
-                    dim: row.tau_tilde / (row.q - 1.0),
+        };
-                }),
+        let (q0, q2) = (at(0.0), at(2.0));
+        vec![
+            DimensionRow {
+                q: 0.0,
+                dim: -q0.tau_tilde,
+                sd: q0.sd,
+            },
+            DimensionRow {
+                q: 1.0,
+                dim: info,
+                sd: 0.0,
+            },
+            DimensionRow {
+                q: 2.0,
+                dim: q2.tau_tilde,
+                sd: q2.sd,
+            },
+        ]
-        );
-        rows
     }
 
     fn assert_matches_reference(addresses: Vec<Ipv4Addr>) {
@@ -803,6 +863,7 @@
         for (actual, expected) in result.dimensions.iter().zip(reference.dimensions) {
             close(actual.q, expected.q);
             close(actual.dim, expected.dim);
+            close(actual.sd, expected.sd);
         }
     }
 
@@ -869,26 +930,74 @@
         close(result.structure[0].tau_tilde, 0.0);
         close(result.structure[16].q, 1.5);
         close(result.structure[32].tau_tilde, 0.0);
+        assert_eq!(result.spectrum.len(), 8);
+        for row in &result.spectrum {
-        assert!(result.spectrum.is_empty());
+            close(row.alpha, 0.0);
+            close(row.f, 0.0);
+        }
         assert_eq!(result.dimensions.len(), 3);
-        close(result.dimensions[0].q, 1.0);
+        close(result.dimensions[0].q, 0.0);
         close(result.dimensions[0].dim, 0.0);
+        close(result.dimensions[1].q, 1.0);
-        close(result.dimensions[1].q, 0.0);
         close(result.dimensions[1].dim, 0.0);
+        close(result.dimensions[1].sd, 0.0);
         close(result.dimensions[2].q, 2.0);
         close(result.dimensions[2].dim, 0.0);
     }
 
     #[test]
-    fn linear_structure_curve_has_no_spectrum_rows() {
+    fn linear_structure_curve_collapses_to_one_spectrum_point() {
         let addresses = (0..1024).map(|index| Ipv4Addr::from(index << 22));
 
         let result = compute(addresses);
 
+        assert!(!result.spectrum.is_empty());
+        for row in &result.spectrum {
+            assert!((row.alpha - 1.0).abs() < 1e-9, "{row:?}");
+            assert!((row.f - 1.0).abs() < 1e-9, "{row:?}");
+        }
+    }
+
+    fn structure_curve(tau: impl Fn(f64) -> f64) -> Vec<StructureRow> {
+        (0..=32)
-        assert!(result.spectrum.is_empty());
+            .map(|index| {
+                let q = -0.5 + f64::from(index) * 0.125;
+                StructureRow {
+                    q,
+                    tau_tilde: tau(q),
+                    sd: 0.0,
+                }
+            })
+            .collect()
     }
 
     #[test]
+    fn exactly_tied_alphas_stay_in_the_spectrum() {
+        let structure = structure_curve(|q| q - 1.0);
+
+        let spectrum = compute_spectrum(&structure, 0.125);
+
+        assert_eq!(compute_critical_region(&structure, 0.125), (3.375, -0.375));
+        assert_eq!(spectrum.len(), 30);
+        assert!(
+            spectrum
+                .iter()
+                .all(|row| *row == SpectrumRow { alpha: 1.0, f: 1.0 })
+        );
+    }
+
+    #[test]
+    fn spectrum_stops_at_the_critical_region_q_max() {
+        let structure = structure_curve(|q| if q <= 2.0 { q - 1.0 } else { 1.0 });
+
+        let spectrum = compute_spectrum(&structure, 0.125);
+
+        assert_eq!(compute_critical_region(&structure, 0.125), (1.875, -0.375));
+        assert_eq!(spectrum.len(), 18);
+        assert_eq!(spectrum.last(), Some(&SpectrumRow { alpha: 1.0, f: 1.0 }));
+    }
+
+    #[test]
     fn sparse_two_address_set_is_safe_at_the_32_bit_boundary() {
         let result = compute([Ipv4Addr::new(192, 0, 2, 1), Ipv4Addr::new(192, 0, 2, 2)]);
 
@@ -912,7 +1021,7 @@
 
         assert_eq!(
             String::from_utf8(output).unwrap(),
-            "{\"schemaVersion\":2,\"metadata\":{\"input\":\"-\",\"prefixLengths\":[],\"minPrefixLength\":null,\"maxPrefixLength\":null,\"totalAddrs\":0},\"structure\":[],\"spectrum\":[],\"dimensions\":[]}\n"
+            "{\"schemaVersion\":3,\"metadata\":{\"input\":\"-\",\"prefixLengths\":[],\"minPrefixLength\":null,\"maxPrefixLength\":null,\"totalAddrs\":0},\"structure\":[],\"spectrum\":[],\"dimensions\":[]}\n"
         );
     }
 
@@ -952,8 +1061,12 @@
                 .iter()
                 .map(|row| row.q)
                 .collect::<Vec<_>>(),
+            vec![0.0, 1.0, 2.0]
-            vec![1.0, 0.0, 2.0]
         );
+        let structure_sd = |q: f64| result.structure.iter().find(|row| row.q == q).unwrap().sd;
+        assert_eq!(result.dimensions[0].sd, structure_sd(0.0));
+        assert_eq!(result.dimensions[1].sd, 0.0);
+        assert_eq!(result.dimensions[2].sd, structure_sd(2.0));
     }
 
     #[test]
```

## tools/netflow-db/src/pipeline.rs

Status: modified; +1; −1

```diff
@@ -1529,7 +1529,7 @@
         "maad": {
             "enabled": pipeline.run_maad,
             "backend": "in-process",
-            "contract_version": 2,
+            "contract_version": 3,
             "config": maad_config
         }
     });
```

## tools/netflow-db/tests/fixtures/maad-conformance/README.md

Status: modified; +1; −1

```diff
@@ -2,7 +2,7 @@
 
 Each `golden.json` is raw JSON emitted independently by the Haskell `MAAD`
 oracle from `chris-misa/maad` at commit
-`3ae75363d44e08faacdee186d4ff8906c6ccd06a`, with the sole source adjustment
+`b7bbb7dd119f9e050cb74239b89e472ad58ec5db` (`main`), with the sole source adjustment
 `deltaQ = 1.0 / 8.0` (instead of `1.0 / 16.0`). The oracle was invoked with
 `--input - --output - --format json --structure --spectrum --dimensions`,
 feeding the corresponding address list on standard input; therefore committed
```

## tools/netflow-db/tests/fixtures/maad-conformance/balanced-branches/golden.json

Status: modified; +20; −24

```diff
@@ -1,8 +1,8 @@
 {
   "dimensions": [
-    { "dim": 8.427644921917286e-2, "q": 1 },
-    { "dim": 4.749146600338848e-2, "q": 0 },
-    { "dim": 5.750094034403652e-2, "q": 2 }
+    { "dim": 4.749146600338848e-2, "q": 0, "sd": 7.277524401549904e-3 },
+    { "dim": 8.427644921917286e-2, "q": 1, "sd": 0 },
+    { "dim": 5.750094034403652e-2, "q": 2, "sd": 7.334906438556834e-4 }
   ],
   "metadata": {
     "didAutoStop": null,
@@ -61,23 +61,23 @@
     },
     {
       "q": 0.25,
-      "sd": 3.939700756730238e-3,
+      "sd": 3.939700756730237e-3,
       "tauTilde": -3.826793491517653e-2
     },
     {
       "q": 0.375,
+      "sd": 2.7249010009142566e-3,
-      "sd": 2.724901000914254e-3,
-      "tauTilde": -3.274338791462162e-2
+      "tauTilde": -3.274338791462165e-2
     },
     {
       "q": 0.5,
+      "sd": 1.7876273535263544e-3,
-      "sd": 1.7876273535263562e-3,
       "tauTilde": -2.6757173790453014e-2
     },
     {
       "q": 0.625,
       "sd": 1.0895807754644883e-3,
-      "tauTilde": -2.0409997245778744e-2
+      "tauTilde": -2.040999724577873e-2
     },
     {
       "q": 0.75,
@@ -92,12 +92,12 @@
     { "q": 1, "sd": 0, "tauTilde": 0 },
     {
       "q": 1.125,
+      "sd": 1.581222823749072e-4,
-      "sd": 1.581222823749028e-4,
+      "tauTilde": 7.066753568461228e-3
-      "tauTilde": 7.066753568461241e-3
     },
     {
       "q": 1.25,
-      "sd": 2.657434654985441e-4,
+      "sd": 2.657434654985486e-4,
       "tauTilde": 1.4200791334906737e-2
     },
     {
@@ -105,37 +105,33 @@
       "sd": 3.455953172019204e-4,
       "tauTilde": 2.137841260045105e-2
     },
-    { "q": 1.5, "sd": 4.1459115585856213e-4, "tauTilde": 2.858303993044291e-2 },
+    { "q": 1.5, "sd": 4.145911558585569e-4, "tauTilde": 2.8583039930442897e-2 },
     {
       "q": 1.625,
-      "sd": 4.838000151727363e-4,
+      "sd": 4.838000151727319e-4,
       "tauTilde": 3.5803488444396486e-2
     },
     { "q": 1.75, "sd": 5.590655492340367e-4, "tauTilde": 4.303254187899084e-2 },
     {
       "q": 1.875,
+      "sd": 6.423829221572997e-4,
-      "sd": 6.423829221573035e-4,
       "tauTilde": 5.026582537623958e-2
     },
     { "q": 2, "sd": 7.334906438556834e-4, "tauTilde": 5.750094034403652e-2 },
-    {
-      "q": 2.125,
-      "sd": 8.311060556725141e-4,
-      "tauTilde": 6.473681778901229e-2
-    },
+    { "q": 2.125, "sd": 8.311060556725194e-4, "tauTilde": 6.47368177890123e-2 },
+    { "q": 2.25, "sd": 9.336443733579639e-4, "tauTilde": 7.197324648447735e-2 },
-    { "q": 2.25, "sd": 9.336443733579606e-4, "tauTilde": 7.197324648447735e-2 },
     {
       "q": 2.375,
       "sd": 1.039549375793105e-3,
       "tauTilde": 7.921053674991818e-2
     },
-    { "q": 2.5, "sd": 1.1474105788588907e-3, "tauTilde": 8.644928676721365e-2 },
+    { "q": 2.5, "sd": 1.1474105788588937e-3, "tauTilde": 8.644928676721364e-2 },
     {
       "q": 2.625,
       "sd": 1.2559860182956041e-3,
+      "tauTilde": 9.369022472124042e-2
-      "tauTilde": 9.369022472124044e-2
     },
+    { "q": 2.75, "sd": 1.3641917380118928e-3, "tauTilde": 0.10093410586658329 },
-    { "q": 2.75, "sd": 1.3641917380118928e-3, "tauTilde": 0.10093410586658325 },
     { "q": 2.875, "sd": 1.471083858410273e-3, "tauTilde": 0.10818164857690803 },
     { "q": 3, "sd": 1.5758421038031495e-3, "tauTilde": 0.11543349746868865 },
     {
@@ -146,7 +142,7 @@
     { "q": 3.25, "sd": 1.7762181312777838e-3, "tauTilde": 0.129952224365334 },
     {
       "q": 3.375,
-      "sd": 1.8707089817726467e-3,
+      "sd": 1.8707089817726508e-3,
       "tauTilde": 0.13721991193638372
     },
     { "q": 3.5, "sd": 1.9607967625899243e-3, "tauTilde": 0.14449353168539514 }
```

## tools/netflow-db/tests/fixtures/maad-conformance/clustered/golden.json

Status: modified; +3; −3

```diff
@@ -1,8 +1,8 @@
 {
   "dimensions": [
-    { "dim": 1.4155452840928675e-2, "q": 1 },
-    { "dim": 5.8823529411764705e-2, "q": 0 },
-    { "dim": 3.272902050143441e-2, "q": 2 }
+    { "dim": 5.8823529411764705e-2, "q": 0, "sd": 0 },
+    { "dim": 1.4155452840928675e-2, "q": 1, "sd": 0 },
+    { "dim": 3.272902050143441e-2, "q": 2, "sd": 0 }
   ],
   "metadata": {
     "didAutoStop": null,
```

## tools/netflow-db/tests/fixtures/maad-conformance/mixed/golden.json

Status: modified; +55; −68

```diff
@@ -1,8 +1,8 @@
 {
   "dimensions": [
-    { "dim": 0.15848541574905534, "q": 1 },
-    { "dim": 0.48630818058102654, "q": 0 },
-    { "dim": 9.563138967007195e-2, "q": 2 }
+    { "dim": 0.48630818058102654, "q": 0, "sd": 2.451805340828242e-2 },
+    { "dim": 0.15848541574905534, "q": 1, "sd": 0 },
+    { "dim": 9.563138967007197e-2, "q": 2, "sd": 1.5013688311364334e-2 }
   ],
   "metadata": {
     "didAutoStop": null,
@@ -35,34 +35,25 @@
     { "alpha": 0.7151040096912356, "f": 0.4824890366161838 },
     { "alpha": 0.6998274338318646, "f": 0.48516537815620797 },
     { "alpha": 0.6772974801593303, "f": 0.48630818058102654 },
-    { "alpha": 0.6455726128772776, "f": 0.48401601395501814 },
-    { "alpha": 0.6040581066665642, "f": 0.4759295540283482 },
-    { "alpha": 0.5537099320218729, "f": 0.45994613518691974 },
+    { "alpha": 0.6455726128772772, "f": 0.48401601395501814 },
+    { "alpha": 0.6040581066665642, "f": 0.4759295540283483 },
+    { "alpha": 0.5537099320218735, "f": 0.45994613518691996 },
     { "alpha": 0.4966281828776834, "f": 0.4348016357950806 },
-    { "alpha": 0.43540034557717006, "f": 0.4002730809450279 },
-    { "alpha": 0.37259283272497956, "f": 0.357082082505681 },
-    { "alpha": 0.3105498318477856, "f": 0.30673075964486407 },
+    { "alpha": 0.4354003455771699, "f": 0.4002730809450278 },
-    { "alpha": 0.2514168402023277, "f": 0.2514168402023277 },
-    { "alpha": 0.19717329055224397, "f": 0.1939653985987442 },
-    { "alpha": 0.1495019677492229, "f": 0.13758413704846761 },
-    { "alpha": 0.10951898013906955, "f": 8.535855248138464e-2 },
-    { "alpha": 7.755435301208724e-2, "f": 3.965846184530247e-2 },
-    { "alpha": 5.3151073096857016e-2, "f": 1.7518603195348509e-3 },
-    { "alpha": 3.528589755176581e-2, "f": -2.8210515231452474e-2 },
-    { "alpha": 2.2682214892117247e-2, "f": -5.0910954928079406e-2 },
-    { "alpha": 1.4077917224721459e-2, "f": -6.747555522062904e-2 },
-    { "alpha": 8.384495618307064e-3, "f": -7.91425339680771e-2 },
-    { "alpha": 4.743841571771212e-3, "f": -8.705387003816349e-2 },
-    { "alpha": 2.5196640498434952e-3, "f": -9.216134543154411e-2 },
-    { "alpha": 1.2594329703956575e-3, "f": -9.520884716112045e-2 },
-    { "alpha": 6.500881751049237e-4, "f": -9.67539243328709e-2 },
-    { "alpha": 4.787254471624869e-4, "f": -9.720345665118899e-2 }
+    { "alpha": 0.37259283272497945, "f": 0.35708208250568096 },
+    { "alpha": 0.3105498318477857, "f": 0.3067307596448641 },
+    { "alpha": 0.2514168402023279, "f": 0.2514168402023279 },
+    { "alpha": 0.19717329055224406, "f": 0.1939653985987443 },
+    { "alpha": 0.14950196774922253, "f": 0.13758413704846714 },
+    { "alpha": 0.10951898013906958, "f": 8.535855248138477e-2 },
+    { "alpha": 7.755435301208763e-2, "f": 3.965846184530303e-2 },
+    { "alpha": 5.3151073096856793e-2, "f": 1.7518603195344623e-3 }
   ],
   "structure": [
-    { "q": -0.5, "sd": 3.1750082682512504e-2, "tauTilde": -0.8426837994684427 },
+    { "q": -0.5, "sd": 3.1750082682512504e-2, "tauTilde": -0.8426837994684426 },
     {
       "q": -0.375,
-      "sd": 2.9980992285519675e-2,
+      "sd": 2.998099228551968e-2,
       "tauTilde": -0.7514198098079999
     },
     {
@@ -72,7 +63,7 @@
     },
     {
       "q": -0.125,
-      "sd": 2.6660944071071255e-2,
+      "sd": 2.6660944071071252e-2,
       "tauTilde": -0.572643807385191
     },
     { "q": 0, "sd": 2.451805340828242e-2, "tauTilde": -0.48630818058102654 },
@@ -81,92 +72,88 @@
       "sd": 2.1713020013486028e-2,
       "tauTilde": -0.40331943734535847
     },
-    { "q": 0.25, "sd": 1.832313114421922e-2, "tauTilde": -0.32491502736170713 },
+    { "q": 0.25, "sd": 1.832313114421922e-2, "tauTilde": -0.32491502736170724 },
     {
       "q": 0.375,
       "sd": 1.4686874429319392e-2,
       "tauTilde": -0.2523049106787174
     },
-    { "q": 0.5, "sd": 1.1179905703881741e-2, "tauTilde": -0.1864875443562389 },
+    { "q": 0.5, "sd": 1.117990570388174e-2, "tauTilde": -0.18648754435623888 },
     {
       "q": 0.625,
       "sd": 8.017576091067168e-3,
       "tauTilde": -0.12814786495929656
     },
+    { "q": 0.75, "sd": 5.198594409935857e-3, "tauTilde": -7.76374579619464e-2 },
     {
-      "q": 0.75,
-      "sd": 5.198594409935857e-3,
+      "q": 0.875,
+      "sd": 2.576935719152976e-3,
+      "tauTilde": -3.499965677805169e-2
-      "tauTilde": -7.763745796194639e-2
     },
+    { "q": 1, "sd": 6.631981242474903e-18, "tauTilde": 9.42184265827917e-18 },
     {
+      "q": 1.125,
+      "sd": 2.584512082871893e-3,
-      "q": 0.875,
+      "tauTilde": 2.7854553272530284e-2
-      "sd": 2.576935719152974e-3,
-      "tauTilde": -3.499965677805167e-2
     },
-    { "q": 1, "sd": 3.729791574141121e-18, "tauTilde": 9.42184265827917e-18 },
-    { "q": 1.125, "sd": 2.58451208287189e-3, "tauTilde": 2.785455327253026e-2 },
-    { "q": 1.25, "sd": 5.116550994346554e-3, "tauTilde": 4.9293322638061e-2 },
+    { "q": 1.25, "sd": 5.116550994346557e-3, "tauTilde": 4.929332263806102e-2 },
     {
       "q": 1.375,
-      "sd": 7.483069153304027e-3,
-      "tauTilde": 6.523004520983598e-2
+      "sd": 7.483069153304022e-3,
+      "tauTilde": 6.523004520983591e-2
     },
-    { "q": 1.5, "sd": 9.586221403953814e-3, "tauTilde": 7.667306767282839e-2 },
+    { "q": 1.5, "sd": 9.586221403953816e-3, "tauTilde": 7.667306767282842e-2 },
     {
       "q": 1.625,
+      "sd": 1.1378375822747887e-2,
+      "tauTilde": 8.461863346285782e-2
+    },
+    {
+      "q": 1.75,
-      "sd": 1.1378375822747885e-2,
-      "tauTilde": 8.461863346285779e-2
+      "sd": 1.2859981842104167e-2,
+      "tauTilde": 8.996083594704261e-2
     },
-    { "q": 1.75, "sd": 1.285998184210417e-2, "tauTilde": 8.996083594704264e-2 },
     {
       "q": 1.875,
       "sd": 1.405941446360083e-2,
-      "tauTilde": 9.344010785079924e-2
+      "tauTilde": 9.344010785079926e-2
     },
+    { "q": 2, "sd": 1.5013688311364334e-2, "tauTilde": 9.563138967007197e-2 },
-    { "q": 2, "sd": 1.5013688311364334e-2, "tauTilde": 9.563138967007195e-2 },
     {
       "q": 2.125,
-      "sd": 1.575761960109601e-2,
+      "sd": 1.5757619601096014e-2,
       "tauTilde": 9.695958715697961e-2
     },
+    { "q": 2.25, "sd": 1.632031712927442e-2, "tauTilde": 9.772751357464864e-2 },
     {
-      "q": 2.25,
-      "sd": 1.6320317129274422e-2,
-      "tauTilde": 9.772751357464872e-2
-    },
-    {
       "q": 2.375,
-      "sd": 1.6725397355708053e-2,
+      "sd": 1.6725397355708056e-2,
       "tauTilde": 9.814554754992241e-2
     },
-    { "q": 2.5, "sd": 1.6992262847763635e-2, "tauTilde": 9.83574295871096e-2 },
+    { "q": 2.5, "sd": 1.6992262847763628e-2, "tauTilde": 9.835742958710962e-2 },
     {
       "q": 2.625,
+      "sd": 1.713727009691121e-2,
+      "tauTilde": 9.846040579252134e-2
-      "sd": 1.7137270096911208e-2,
-      "tauTilde": 9.846040579252133e-2
-    },
-    {
-      "q": 2.75,
-      "sd": 1.7174531016053783e-2,
-      "tauTilde": 9.851995163088582e-2
     },
+    { "q": 2.75, "sd": 1.7174531016053787e-2, "tauTilde": 9.85199516308858e-2 },
     {
       "q": 2.875,
       "sd": 1.711643429582498e-2,
-      "tauTilde": 9.858008715431195e-2
+      "tauTilde": 9.858008715431187e-2
     },
-    { "q": 3, "sd": 1.697401285087209e-2, "tauTilde": 9.867040853813017e-2 },
+    { "q": 3, "sd": 1.697401285087209e-2, "tauTilde": 9.867040853813007e-2 },
     {
       "q": 3.125,
       "sd": 1.6757231439643777e-2,
-      "tauTilde": 9.88108001685707e-2
+      "tauTilde": 9.881080016857072e-2
     },
-    { "q": 3.25, "sd": 1.647521942872769e-2, "tauTilde": 9.901456264568395e-2 },
     {
-      "q": 3.375,
+      "q": 3.25,
+      "sd": 1.6475219428727695e-2,
+      "tauTilde": 9.901456264568397e-2
-      "sd": 1.6136450539991503e-2,
-      "tauTilde": 9.929048214384534e-2
     },
+    { "q": 3.375, "sd": 1.61364505399915e-2, "tauTilde": 9.929048214384541e-2 },
     { "q": 3.5, "sd": 1.574886805827989e-2, "tauTilde": 9.96442019065783e-2 }
   ]
 }
```

## tools/netflow-db/tests/fixtures/maad-conformance/random/golden.json

Status: modified; +50; −46

```diff
@@ -1,8 +1,8 @@
 {
   "dimensions": [
-    { "dim": 0.5091568135164136, "q": 1 },
-    { "dim": 0.8115457541681342, "q": 0 },
-    { "dim": 0.6982893075100184, "q": 2 }
+    { "dim": 0.8115457541681342, "q": 0, "sd": 2.9623771521080443e-2 },
+    { "dim": 0.5091568135164136, "q": 1, "sd": 0 },
+    { "dim": 0.6982893075100185, "q": 2, "sd": 3.172496480854753e-2 }
   ],
   "metadata": {
     "didAutoStop": null,
@@ -25,36 +25,36 @@
   },
   "schemaVersion": 1,
   "spectrum": [
-    { "alpha": 0.9007621617148542, "f": 0.8078687779356823 },
-    { "alpha": 0.8860542567850467, "f": 0.8106218076253104 },
-    { "alpha": 0.8712160883010069, "f": 0.8115457541681342 },
+    { "alpha": 0.9007621617148547, "f": 0.8078687779356825 },
+    { "alpha": 0.8860542567850476, "f": 0.8106218076253102 },
+    { "alpha": 0.8712160883010065, "f": 0.8115457541681342 },
     { "alpha": 0.8562881136433309, "f": 0.8106105818536058 },
     { "alpha": 0.8413107959599657, "f": 0.8078014247472929 },
     { "alpha": 0.826324502430104, "f": 0.803118557069487 },
-    { "alpha": 0.8113693741090875, "f": 0.7965772872043193 },
-    { "alpha": 0.7964851594536251, "f": 0.7882077497894419 },
-    { "alpha": 0.7817110099051509, "f": 0.7780545677152324 },
-    { "alpha": 0.767085241145477, "f": 0.7661763586569309 },
-    { "alpha": 0.7526450674822465, "f": 0.7526450674822465 },
-    { "alpha": 0.7384263192164844, "f": 0.7375451149026219 },
-    { "alpha": 0.7244631539410838, "f": 0.7209723626222337 },
-    { "alpha": 0.7107877727763189, "f": 0.7030329048662444 },
-    { "alpha": 0.6974301518526542, "f": 0.6838417047807805 },
-    { "alpha": 0.6844177981605781, "f": 0.6635211013465818 },
-    { "alpha": 0.6717755374062504, "f": 0.6421992179225928 },
-    { "alpha": 0.659525339886692, "f": 0.6200083072716274 },
-    { "alpha": 0.6476861887270897, "f": 0.597083069944161 },
-    { "alpha": 0.6362739931706796, "f": 0.5735589832900013 },
-    { "alpha": 0.625301548024709, "f": 0.549570677252907 },
-    { "alpha": 0.614778538881759, "f": 0.5252503906403079 },
+    { "alpha": 0.8113693741090879, "f": 0.7965772872043195 },
-    { "alpha": 0.6047115913877095, "f": 0.5007265379461456 },
+    { "alpha": 0.796485159453625, "f": 0.7882077497894417 },
+    { "alpha": 0.7817110099051503, "f": 0.7780545677152321 },
+    { "alpha": 0.7670852411454772, "f": 0.766176358656931 },
+    { "alpha": 0.7526450674822461, "f": 0.7526450674822461 },
+    { "alpha": 0.7384263192164844, "f": 0.737545114902622 },
+    { "alpha": 0.7244631539410844, "f": 0.7209723626222344 },
+    { "alpha": 0.7107877727763191, "f": 0.7030329048662447 },
+    { "alpha": 0.697430151852654, "f": 0.6838417047807802 },
+    { "alpha": 0.6844177981605775, "f": 0.6635211013465807 },
+    { "alpha": 0.67177553740625, "f": 0.6421992179225923 },
+    { "alpha": 0.6595253398866929, "f": 0.620008307271629 },
+    { "alpha": 0.6476861887270906, "f": 0.5970830699441627 },
+    { "alpha": 0.6362739931706791, "f": 0.5735589832900003 },
+    { "alpha": 0.6253015480247095, "f": 0.5495706772529081 },
+    { "alpha": 0.614778538881759, "f": 0.5252503906403077 },
+    { "alpha": 0.6047115913877086, "f": 0.5007265379461434 },
     { "alpha": 0.5951043616465954, "f": 0.4761224122715155 },
-    { "alpha": 0.5859576638647361, "f": 0.4515550446932475 },
-    { "alpha": 0.5772696305610046, "f": 0.4271342348459066 },
-    { "alpha": 0.5690359001176484, "f": 0.4029617617779171 },
-    { "alpha": 0.5612498261167156, "f": 0.37913077856834265 },
-    { "alpha": 0.5539027027920147, "f": 0.35572538896984085 },
+    { "alpha": 0.5859576638647352, "f": 0.45155504469324503 },
+    { "alpha": 0.5772696305610046, "f": 0.42713423484590685 },
-    { "alpha": 0.5469840010072682, "f": 0.3328203996551331 }
+    { "alpha": 0.5690359001176493, "f": 0.40296176177791976 },
+    { "alpha": 0.5612498261167174, "f": 0.3791307785683484 },
+    { "alpha": 0.5539027027920147, "f": 0.3557253889698404 },
+    { "alpha": 0.5469840010072664, "f": 0.3328203996551269 }
   ],
   "structure": [
     { "q": -0.5, "sd": 4.262429053244891e-2, "tauTilde": -1.2618841781943309 },
@@ -63,11 +63,11 @@
       "sd": 3.948718532252456e-2,
       "tauTilde": -1.1465691301521548
     },
+    { "q": -0.25, "sd": 3.627794891293679e-2, "tauTilde": -1.0330593183643961 },
-    { "q": -0.25, "sd": 3.627794891293679e-2, "tauTilde": -1.033059318364396 },
     {
       "q": -0.125,
       "sd": 3.2991251280084774e-2,
-      "tauTilde": -0.9213785897234412
+      "tauTilde": -0.9213785897234411
     },
     { "q": 0, "sd": 2.9623771521080443e-2, "tauTilde": -0.8115457541681342 },
     {
@@ -85,37 +85,41 @@
     {
       "q": 0.625,
       "sd": 1.1596076968570646e-2,
-      "tauTilde": -0.2904045251309262
+      "tauTilde": -0.29040452513092607
     },
+    { "q": 0.75, "sd": 7.781261395379442e-3, "tauTilde": -0.19177131028636926 },
-    { "q": 0.75, "sd": 7.781261395379442e-3, "tauTilde": -0.19177131028636923 },
     {
       "q": 0.875,
       "sd": 3.9129024278057225e-3,
-      "tauTilde": -9.497677265463847e-2
+      "tauTilde": -9.497677265463848e-2
     },
-    { "q": 1, "sd": 1.377331113139258e-17, "tauTilde": 1.6017132519074587e-17 },
+    { "q": 1, "sd": 9.0472107655596e-18, "tauTilde": 1.6017132519074587e-17 },
     {
       "q": 1.125,
-      "sd": 3.947837333661058e-3,
-      "tauTilde": 9.318449421592315e-2
+      "sd": 3.947837333661057e-3,
+      "tauTilde": 9.318449421592305e-2
     },
     { "q": 1.25, "sd": 7.920619671509196e-3, "tauTilde": 0.18460657980412112 },
-    { "q": 1.375, "sd": 1.1908192692166784e-2, "tauTilde": 0.2743002827011941 },
+    {
+      "q": 1.375,
-    { "q": 1.5, "sd": 1.5900440304771484e-2, "tauTilde": 0.36230352299820084 },
+      "sd": 1.1908192692166784e-2,
+      "tauTilde": 0.27430028270119416
+    },
+    { "q": 1.5, "sd": 1.5900440304771484e-2, "tauTilde": 0.3623035229982009 },
     { "q": 1.625, "sd": 1.988746966289035e-2, "tauTilde": 0.44865782066435766 },
-    { "q": 1.75, "sd": 2.3859775751064495e-2, "tauTilde": 0.5334079725383454 },
-    { "q": 1.875, "sd": 2.7808383348594756e-2, "tauTilde": 0.6166017050159203 },
-    { "q": 2, "sd": 3.172496480854753e-2, "tauTilde": 0.6982893075100184 },
+    { "q": 1.75, "sd": 2.385977575106449e-2, "tauTilde": 0.5334079725383453 },
-    { "q": 2.125, "sd": 3.560193275307995e-2, "tauTilde": 0.7785232521976927 },
+    { "q": 1.875, "sd": 2.7808383348594756e-2, "tauTilde": 0.6166017050159202 },
+    { "q": 2, "sd": 3.172496480854753e-2, "tauTilde": 0.6982893075100185 },
+    { "q": 2.125, "sd": 3.560193275307995e-2, "tauTilde": 0.7785232521976928 },
     { "q": 2.25, "sd": 3.943250746001074e-2, "tauTilde": 0.8573578058026883 },
-    { "q": 2.375, "sd": 4.3210759376381566e-2, "tauTilde": 0.93484863920387 },
+    { "q": 2.375, "sd": 4.3210759376381566e-2, "tauTilde": 0.9348486392038702 },
     { "q": 2.5, "sd": 4.6931627819061e-2, "tauTilde": 1.011052440523128 },
     { "q": 2.625, "sd": 5.059091748983781e-2, "tauTilde": 1.0860265370507973 },
     { "q": 2.75, "sd": 5.418527492669052e-2, "tauTilde": 1.159828530934777 },
-    { "q": 2.875, "sd": 5.771214742173305e-2, "tauTilde": 1.2325159530169814 },
+    { "q": 2.875, "sd": 5.771214742173305e-2, "tauTilde": 1.2325159530169811 },
     { "q": 3, "sd": 6.116972725064629e-2, "tauTilde": 1.304145938575028 },
     { "q": 3.125, "sd": 6.455688427161593e-2, "tauTilde": 1.3747749280463935 },
+    { "q": 3.25, "sd": 6.787309005950386e-2, "tauTilde": 1.4444583951042074 },
-    { "q": 3.25, "sd": 6.787309005950386e-2, "tauTilde": 1.444458395104207 },
     { "q": 3.375, "sd": 7.11183367412341e-2, "tauTilde": 1.5132506037443971 },
     { "q": 3.5, "sd": 7.429305359238074e-2, "tauTilde": 1.581204395356024 }
   ]
```

## tools/netflow-db/tests/fixtures/maad-conformance/threshold-ancestor/golden.json

Status: modified; +3; −3

```diff
@@ -1,8 +1,8 @@
 {
   "dimensions": [
-    { "dim": 1.5712950457112795e-3, "q": 1 },
-    { "dim": 5.8823529411764705e-2, "q": 0 },
-    { "dim": 1.6803719801791572e-3, "q": 2 }
+    { "dim": 5.8823529411764705e-2, "q": 0, "sd": 0 },
+    { "dim": 1.5712950457112795e-3, "q": 1, "sd": 0 },
+    { "dim": 1.6803719801791572e-3, "q": 2, "sd": 0 }
   ],
   "metadata": {
     "didAutoStop": null,
```

## tools/netflow-db/tests/maad_conformance.rs

Status: modified; +1; −1

```diff
@@ -58,7 +58,7 @@
         &["q", "tauTilde", "sd"],
     );
     assert_numeric_rows(name, "spectrum", &actual, &expected, &["alpha", "f"]);
-    assert_numeric_rows(name, "dimensions", &actual, &expected, &["q", "dim"]);
+    assert_numeric_rows(name, "dimensions", &actual, &expected, &["q", "dim", "sd"]);
 }
 
 fn assert_metadata_matches(name: &str, actual: &Value, expected: &Value) {
```