What I built
I built a Python evaluation harness for Sift's news-processing tasks. The first real categorization development run is published. A final held-out comparison across models and tasks remains pending.
First development result
On October 24, 2026, local Llama 3.1 8B Instruct Q4_K_M matched 319 of 400 stored category labels (79.8%) across ten reference categories. Macro-F1 was 0.7598, with a descriptive 95% bootstrap interval of 0.7100–0.8035. All 400 items completed without execution or parsing failures.
These references are Sift's existing model-assigned labels, not independently reviewed human answers. The score measures stored-label agreement. A disagreement may reflect a new model error, a noisy reference label or ambiguous category rules; it has not been adjudicated.
The 400 development articles came from a selected cohort spanning 55 outlets and publication dates from October 2024 to October 2026. This is not a random sample of recent production traffic. The top category had too few captured examples for reference sampling, though it remained a valid classifier output. Another 100 articles are reserved and unevaluated.
The run report records the selection, model digest, runtime and provenance checks. The aggregate artifact provides category counts and measurements. Article text and per-item outputs remain local; the public evidence contains aggregates and provenance hashes.
How a run works
Model adapters expose a shared interface. Task modules supply prompts, parse responses and calculate scores. The runner records inputs, model settings and outputs in JSONL files so interrupted runs can resume.
Categorization and summarization modules are implemented. Structured extraction and grounded question answering are planned. The study design covers nine models and keeps output quality, response time and cost separate for each task.
For pairwise summarization, judge selection avoids assigning Sonnet to pairs containing an Anthropic candidate. GPT-4o still judges pairs that include GPT-4o, so self-judging is not eliminated. The proposed overlap calibration has not been measured. Bradley-Terry ranking is implemented for pairwise results.
Keeping final scoring separate
The runner requires an explicit flag before using held-out data and checks its contents against a SHA-256 manifest. A sample lock and tamper-detection tests exercise that mechanism. The first real corpus now has a 400-item development set and a separate 100-item holdout manifest. The published run reports the holdout unchanged and disjoint from development.
A committed hash checks for changes to the dataset. It does not prevent prior inspection. The real holdout manifest was saved before inference and committed with publication, not in a pre-run commit.
Testing agent runs
A second track records an agent's tool calls, arguments, recovery from errors and final answer. Recorded runs support tests without API keys; checks that require a model judge use a separate live path. Fault and injection scenarios exercise failure handling.
These scripted tests cover the evaluation machinery. They do not measure a live agent's reliability or injection resistance. The categorization baseline is a separate model run.
What remains
The next steps are independent label review, judge calibration, the remaining task modules, broader model runs and final held-out scoring. DGX Spark timing and measured cost comparisons are also pending; the published baseline ran on local macOS ARM64, not that hardware.
One development run does not establish a model ranking or justify changing Sift's production routing. Its categorization-only prompt is also narrower than the production batch summarization operation.
The methodology site contains the detailed study plan; the repository contains the implementation.