researchmoderationdata-searchOpen source

jev-spam-eval

A reproducible email-classification study comparing zero-shot Jev decisions with trained TF-IDF baselines.

What Jev decides

Whether an email is ham, spam, or phishing in the main experiment.

The main experiment uses Choice to classify 5,733 emails as ham, spam, or phishing, stores per-message predictions, and compares Jev with supervised baselines and a fresh out-of-distribution set.

CHOICENOUL

Jev role: Assign each email to a small, explicit set of message classes.
Input: Email subject, body, and selected metadata prepared by the evaluation script.
Output: A Choice distribution over ham, spam, and phishing.
Decision frequency: Once per email in the evaluation dataset.
Fallback: Compare against trained TF-IDF baselines and inspect confusion matrices and saved predictions.

How it works

Input

Email subject, body, and selected metadata prepared by the evaluation script.

Jev

CHOICE + NOUL

Output

A Choice distribution over ham, spam, and phishing.

Reported performance

Reported Jev accuracy

98.64 %

Three-way ham, spam, and phishing classification on 5,733 emails.

Author reported · Source reviewed

Captured Sep 21, 2026

Reported best TF-IDF accuracy

98.87 %

Same three-way 5,733-email classification dataset.

Author reported · Source reviewed

Captured Sep 21, 2026

Overview

jev-spam-eval is useful as a research artifact rather than a product demo. The repository includes dataset preparation, runners, cached predictions, evaluation scripts, confusion matrices, and cost analysis. Its main three-way task reports 98.64% accuracy for zero-shot Jev versus 98.87% for the best trained TF-IDF baseline.

These are author-run results, not an independent benchmark. Results on a newer email set are weaker and expose context mismatch, including newsletters that resemble spam. The current three-way experiment uses Choice; an earlier binary spam experiment used Noul.

Language: Python
License: MIT
Author: bitnovus