Janitor’s tests check whether the implementation behaves as documented: matching, cookies, database queries, failure handling and the privacy boundary between browser and server. They do not tell us how accurately Jev detects bots or how often browser recovery will be correct on your traffic.
What is verified
The v0.9.0 implementation passed these checks on September 23, 2026:
| Area | Evidence |
|---|---|
| TypeScript library | 277 tests across sixteen files, strict typechecking and lint. |
| Native Elixir | 66 tests against Postgres, plus two Phoenix endpoint tests. |
| Python | Six transport tests on Python 3.12 and 3.14; wheel installation and Flask route continuity/private-response checks. |
| Go | Four test groups with the race detector on Go 1.26, plus a compiled HTTP example. |
| Browser integration | Three Chromium tests, including installed PostHog, Mixpanel and Amplitude SDKs with analytics traffic intercepted locally. |
| Public site | Seventeen browser tests; language navigation/search, both themes, consent controls, all documentation links/anchors, mobile layout and motion fallbacks. 127 generated HTML pages. |
| Examples | Next.js production build, Cloudflare dry-run, native Phoenix migration/tests and Python/Go transports pass. |
| Installation | Seven compiled JavaScript packages, native Elixir archive, Python wheel/source archive and Go source module. Registry publication is separate. |
API activity checks use real Postgres and D1 SQL. They cover bounded aggregates, private outputs, leases/cache isolation, provider budgets, deadlines, malformed answers, deletion and repeated service instances. Analytics checks cover Amplitude/RudderStack protocols, account changes, logout and warehouse field projection. Warehouse SQL is documented against official provider contracts, but has not been executed in a paid Snowflake or BigQuery project. These are implementation checks, not evidence of bot-detection accuracy or API telemetry throughput at production scale.
The database contract tests execute real SQL using embedded Postgres (PGlite) and Cloudflare’s local D1 runtime (Miniflare). Core matching is not mocked. External Jev and analytics responses are mocked or intercepted so automated tests do not make paid inference calls or send test users to analytics projects.
Earlier checks cover bounded lookup planning, ten-candidate batch matching, cold-start/ambiguity abstention, private Jev learning on both databases, shared inference budgets and asynchronous analytics logout races. The real SDK integration also exercises createJanitorClient events and identity transitions. External inference remains mocked; these checks establish implementation behavior, not cross-device or bot-detection accuracy.
The earlier September 23 live playground follow-up passed 243 TypeScript tests and 15 site browser tests. New real-D1 checks cover session isolation, atomic call allowances, cache hits, erasure, billing rejection and the inference kill switch. The site produces 32 HTML pages, with navigation and internal documentation links checked in Chromium.
The deployed live playground passed real browser cookie continuity, controlled cookie-loss recovery, private-response, mobile-layout and erasure checks. After funding Cloudflare inference, a hosted flow received fresh Jev evaluations, reused a private cached answer without another model call, and restored its visitor cookie. The live response exposed a Cloudflare Completed envelope absent from the model documentation example; regression tests now cover it. The documented local HTTPS Worker flow also passed with local D1 and AI disabled.
See the latest CI runs and detailed dated records for exact commands, environments and historical counts.
What the benchmarks show
The benchmarks answer different questions. Keep their results separate:
| Benchmark | What it tests | What it does not establish |
|---|---|---|
| Controlled browser visits | Browser signals and recovery after controlled changes. | Accuracy across a representative real-user population. |
| Millions of stored observations | Indexed candidate lookup and history-read latency. | Sustained production throughput or correct person identity. |
| Connection and workload tests | Open connections, successful responses, overload and recovery. | 200,000 simultaneous successful identifications or real Jev capacity. |
The browser experiment includes a false match between indistinguishable profiles. The connection experiment includes controlled 503 overload responses. Those are part of the findings, not successes hidden inside headline counts.
What still needs real-world testing
Before using a score to trigger extra verification, test with independently labeled traffic from your application. Include ordinary browser updates, common identical device profiles, privacy browsers, accessibility tools, touch-only users, authorized agents and actual automation.
Measure at least:
- False matches: distinct browsers incorrectly assigned the same ID.
- Missed matches: a returning browser assigned a new ID.
- False risk alerts: ordinary activity flagged by your chosen threshold.
- Abstention and availability: how often Janitor cannot make a useful match or risk assessment.
- Latency and cost: the complete request path, including your database and actual AI provider.
Compare built-in matching with AI-assisted matching on the same held-out visits. Do not use Janitor’s own guessed IDs as the truth labels. Anonymous cross-device prediction needs a separate evaluation; see testing a learning model.
Run the checks yourself
From the repository root:
pnpm install --frozen-lockfile
pnpm typecheck
pnpm test
pnpm lint
pnpm exec playwright install chromium
pnpm test:e2e
pnpm site:check
pnpm site:testThe Elixir guide explains its Postgres test setup. Each framework example has its own build and run instructions.
What is published
The public source, documentation site and v0.8.1 GitHub archives are available. This patch includes the live Cloudflare response-envelope fix. All seven patch archives passed isolated npm, pnpm and Bun installation/import checks, including a completed-envelope evaluation; the native Hex-format archive also builds. npm publication requires release-account verification, and Hex publication requires an authenticated account; neither registry is claimed as published. Use the documented GitHub installation paths.
Installing the library does not create a hosted identity endpoint or configure your analytics project. Those run in your application. Live provider ingestion, real-user risk calibration and your production capacity need verification in that environment.