edfd67a322
jank-lang/clojure-test-suite (per-core-fn clojure.test suites shared across Clojure dialects) joins the default gate as vendor/clojure-test-suite, run by host/chez/cts.sh: one joltc process per test namespace (a hang or crash is contained by a per-process timeout), through the test/chez/cts-app project and its cts-run runner, parallel workers. Gating is exact per namespace against test/chez/cts-known-failures.txt, like certify's allowlist: a namespace doing worse than the baseline fails, and one doing better also fails as stale until the baseline is updated in the same change. JOLT_CTS_WRITE_BASELINE=1 regenerates it; JOLT_CTS_NS runs a subset verbosely. Current standing: 243 namespaces, 5302 assertions pass, 340 fail + 236 error across 88 namespaces pinned in the baseline (dominant clusters: BigDecimal arithmetic operands, derive/ancestors hierarchy, transients, special-symbol?, clojure.string case fns, the accepted narrow-int and seq-type-model divergences). Two consecutive full runs produce identical counts. Wired into make ci; skips cleanly when the submodule isn't checked out.
91 lines
2.8 KiB
Text
91 lines
2.8 KiB
Text
# clojure-test-suite known failures: <namespace> <fail> <error>
|
|
# The gate fails on any per-namespace change, worse OR better; regenerate
|
|
# with: JOLT_CTS_WRITE_BASELINE=1 host/chez/cts.sh
|
|
clojure.core-test.abs 1 1
|
|
clojure.core-test.add-watch 0 3
|
|
clojure.core-test.ancestors 9 0
|
|
clojure.core-test.atom 14 0
|
|
clojure.core-test.bigint 6 0
|
|
clojure.core-test.bit-set 1 0
|
|
clojure.core-test.boolean-qmark 0 1
|
|
clojure.core-test.byte 7 5
|
|
clojure.core-test.char 1 0
|
|
clojure.core-test.compare 1 0
|
|
clojure.core-test.conj 1 0
|
|
clojure.core-test.contains-qmark 3 0
|
|
clojure.core-test.counted-qmark 1 0
|
|
clojure.core-test.dec 1 0
|
|
clojure.core-test.denominator 0 3
|
|
clojure.core-test.derive 21 0
|
|
clojure.core-test.descendants 4 0
|
|
clojure.core-test.double 0 4
|
|
clojure.core-test.double-qmark 3 0
|
|
clojure.core-test.empty 1 0
|
|
clojure.core-test.eq 2 0
|
|
clojure.core-test.eval 0 3
|
|
clojure.core-test.even-qmark 1 0
|
|
clojure.core-test.float 4 4
|
|
clojure.core-test.get 0 1
|
|
clojure.core-test.gt 0 2
|
|
clojure.core-test.gt-eq 0 2
|
|
clojure.core-test.ifn-qmark 2 0
|
|
clojure.core-test.inc 1 0
|
|
clojure.core-test.int 4 5
|
|
clojure.core-test.int-qmark 3 0
|
|
clojure.core-test.intern 2 0
|
|
clojure.core-test.keys 0 4
|
|
clojure.core-test.lazy-seq 3 0
|
|
clojure.core-test.long 2 5
|
|
clojure.core-test.lt 0 2
|
|
clojure.core-test.lt-eq 0 2
|
|
clojure.core-test.max 3 1
|
|
clojure.core-test.min 3 1
|
|
clojure.core-test.minus 2 0
|
|
clojure.core-test.mod 12 34
|
|
clojure.core-test.neg-int-qmark 1 0
|
|
clojure.core-test.not-eq 3 0
|
|
clojure.core-test.nth 0 1
|
|
clojure.core-test.num 2 1
|
|
clojure.core-test.number-qmark 3 0
|
|
clojure.core-test.numerator 0 3
|
|
clojure.core-test.odd-qmark 1 0
|
|
clojure.core-test.parents 10 0
|
|
clojure.core-test.parse-uuid 3 0
|
|
clojure.core-test.peek 2 0
|
|
clojure.core-test.plus 11 0
|
|
clojure.core-test.plus-squote 11 0
|
|
clojure.core-test.pop 0 1
|
|
clojure.core-test.pos-int-qmark 1 0
|
|
clojure.core-test.quot 12 31
|
|
clojure.core-test.rand-nth 0 1
|
|
clojure.core-test.rational-qmark 3 0
|
|
clojure.core-test.rationalize 10 0
|
|
clojure.core-test.realized-qmark 3 0
|
|
clojure.core-test.reduce 0 1
|
|
clojure.core-test.rem 12 30
|
|
clojure.core-test.remove-watch 0 1
|
|
clojure.core-test.run-bang 1 0
|
|
clojure.core-test.select-keys 2 0
|
|
clojure.core-test.seqable-qmark 1 0
|
|
clojure.core-test.short 7 5
|
|
clojure.core-test.shuffle 1 0
|
|
clojure.core-test.slash 2 48
|
|
clojure.core-test.some-fn 3 0
|
|
clojure.core-test.sort-by 2 0
|
|
clojure.core-test.special-symbol-qmark 20 0
|
|
clojure.core-test.star 22 0
|
|
clojure.core-test.star-squote 19 0
|
|
clojure.core-test.transient 23 0
|
|
clojure.core-test.underive 7 0
|
|
clojure.core-test.update 1 0
|
|
clojure.core-test.vals 0 3
|
|
clojure.core-test.vec 1 0
|
|
clojure.core-test.when-let 1 0
|
|
clojure.core-test.with-precision 17 0
|
|
clojure.edn-test.read-string 0 1
|
|
clojure.string-test.capitalize 0 4
|
|
clojure.string-test.ends-with-qmark 1 4
|
|
clojure.string-test.escape 1 0
|
|
clojure.string-test.lower-case 0 4
|
|
clojure.string-test.starts-with-qmark 1 10
|
|
clojure.string-test.upper-case 0 4
|