Reviewed September 23, 2026, before implementation. These are design choices for Janitor’s current pilot, not a claim that one model is best for every application.
Findings and decisions
| Primary source | Finding | Decision for Janitor |
|---|---|---|
| TypeSafe model customization | Customer fine-tuning and LoRA are not supported. Requests customize the shared model. | Train a separate Janitor classifier. Keep Jev weights unchanged. |
| TypeSafe feature-discovery cookbook | Typed Jev answers can be numeric inputs to a supervised model. Its examples compare baselines and use independent labels. | Compare telemetry-only and Jev-assisted candidates on identical rows. Never convert a Jev prediction into a training label. |
| Google’s Rules of ML | Start with a simple baseline, verify the pipeline, and test that training and serving agree. | Use an established logistic-regression solver and a small boosted-tree challenger offline. Ship a tiny TypeScript predictor and test cross-language score parity. |
| scikit-learn leakage guidance | Fitting preprocessing on test data leaks information. | Fit feature selection, imputation and scaling using training rows only. Exclude IDs, labels, feedback provenance and cohort tags from predictor inputs. |
| scikit-learn cross-validation and TESSERACT | Random splits can hide dependence between groups and unrealistic time order in security evaluations. | Use chronological fitting, calibration and validation windows; reserve later sessions from independently selected applications for final testing. Delayed feedback must have existed before its window’s cutoff. |
| Probability calibration | Calibration needs independent data; reliable probabilities depend on the population being evaluated. | Fit logistic recalibration on a separate window. Report Brier score, log loss, reliability bins, coverage and false positives. Scores describe the labeled pilot distribution, not a proven deployment-wide probability. |
| Uncertainty under dataset shift | Calibration can deteriorate when the input distribution changes. | Abstain on missing required evidence or a changed Jev version, expire artifacts, report cohort failures, and retain rollback. Good historic calibration is not a guarantee under drift. |
| NIST adversarial ML taxonomy | Poisoning, evasion and privacy attacks affect different stages of the lifecycle. | Preserve outcome provenance, exclude disputes and repeated evidence, cap contributors, keep operator credentials separate, and test mimicked human sessions. Enrollment reduces poisoning exposure; it cannot prove an implementer’s labels are honest. |
| scikit-learn model persistence and CatBoost JSON export | Model persistence formats have different compatibility and execution risks. | Export bounded numeric JSON. Production never loads pickle, Python code, arbitrary expressions or uploaded executable models. |
Chosen approach
Train offline with pinned Python dependencies. Compare logistic regression with a bounded numeric CatBoost classifier. Both use the same snapshots, splits and independent outcomes. Compare each telemetry-only candidate with its Jev-assisted counterpart when a complete, version-matched feature cache is available. Select the candidate and threshold using validation data before measuring the held-out applications. Extra model complexity or Jev calls must earn their place in that comparison.
The server exports only opted-in, unexpired, independently labeled rows and seals the export manifest. Models carry the dataset digest, revision, expiry, preprocessing, calibration, question-set identity, provider version and validation report. Import is an operator action. The importer validates the numeric artifact and report, but trusts the operator to have run the documented trainer; it is not a proof of arbitrary external training code.
Assistant activity and abuse are separate targets. The private classifier endpoint does not alter visitor IDs, account links, delegated permissions or the existing risk evaluator. Shadow scores support evaluation; canary mode makes a passing model available for explicit application use.
What the evidence does not establish
The selected papers concern other tasks and populations; their reported accuracy does not transfer to Janitor. Our generated fixtures verify pipeline behavior only. Before promotion on real traffic, independently review both positive and negative sessions, measure label-selection bias and the actual event base rate, and retain fresh test applications. Unlabeled sessions are unknown, not negative examples. A login or CAPTCHA result does not establish that a human operated the session.
Daily rotating session references prevent deduplication across every day or customer; no global person graph is added to solve that. The held-out-application test reduces shared-session leakage but does not establish generalization to every assistant family. Repeated experiments against one test set are not new independent evidence.
Question changes are reviewed and versioned. They create a new feature cache and require retraining and evaluation. An unconstrained agent rewriting prompts or promoting models from its own predictions is deliberately outside this pilot.