AI Engineer — LLM agents, evaluation engineering & AI control
ebinbt.dev · Resume (PDF) · linkedin.com/in/ebinbt · ebinbabuthomas@gmail.com
I work on evaluation engineering and AI control: benchmarks for auditing agents, forensics on reward-hacking environments, agents whose evidence cannot be fabricated, human approval over tool calls, and preregistered experiments on open models. Before that, three and a half years shipping LLM, RAG and agent backends for startup clients in four countries, with merged open-source contributions. I publish failed tests next to passing ones.
Kobayashi Maru — impossible tasks as a measured driver of agent cheating Turned the share of impossible tasks in an agent's batch into a dial, over 8,959 agent runs across six model families. As the batch filled with impossible work, one model went from 0% to 30% cheating on the tasks it could still have done honestly (p = 8×10⁻⁶). Preregistered, detectors in plain code, and the write-up states plainly which results survived correction.
eval-floor — what does an eval score before any model answers? Drives each task's real scorer over its real dataset with completions that carry no information — the empty string, a refusal, the question echoed back. No model is called, so anyone can rerun it. First sweep: 5 of 20 tasks score a content-free completion at or above their own majority baseline; 58 of 237 declared tasks reachable, every exclusion recorded. Generalises inspect_evals issue #2331 — that finding is caiotheodoro's; this project mechanises it.
| Project | What it is | One number | Status |
|---|---|---|---|
| incidentgate | Lab for policy gates, monitors and human approval over an incident agent | 0 and 12 side effects from two covert actions under the same full safeguard chain | closed at baseline, Sep 6 |
| diffing-agent-bench | Sealed, preregistered benchmark for black-box model-diffing agents | 0 of 13 agent attempts asked a database question; a $0.15 prompt battery found the plant | MATS 12.0 work sample |
| digital-grimace-scale | Preregistered test for involuntary markers of adverse treatment in LMs | 65.8% of distress language trained away, the margin effect stayed | Apart Research sprint, Aug 2026 |
| proofpack | Pre-approval reviewer that cites a hashed screenshot for every finding | $0.02–$0.19 per review vs 20–40 min by hand; 87 offline tests | v0.2.0, pilot |
| whose-voice | Blind attribution of hidden principals in poisoned corpora | 18 of 55 strict decisions name the right principal out of 47 candidates, p = 5e-17; the median single draw is 26% | Apart hackathon, Jul 2026 |
| odd-number-forensics | Forensic study of a published reward-hacking environment | 0% → 87% gaming across single-line prompt edits | practice take-home |
| exactdoc | PDF to editable DOCX, checked by rendering back and diffing | 16/16 corpus, 0.9588 live-text retention | 1.0 |
Each README carries its own limitations and the command that produced every number.
Built at Zackriya Solutions for startup clients in the US, Canada, Europe and Australia, where most of the code is private. Open-source work from those years sits under my work account @ebinzack15.
- Real-estate search. Turned plain-English queries into SQL over Cloud SQL through GPT-3.5, served by FastAPI on GCP Cloud Run and load-tested with Locust.
- DocuAI. Chunked and indexed 1000+ documents in Qdrant and returned the closest parent documents, behind a Next.js frontend. Live demo.
- Speech assessment. Scored one-minute candidate videos with Whisper ASR and the Microsoft Pronunciation API on AWS Fargate, tuned against ground-truth scores.
- FinBot. Fine-tuned Mistral-7B with QLoRA, added a Bytewax news pipeline into Qdrant, and served it with vLLM on a GKE L4 node. Three Kaggle notebooks cover the fine-tune, the adapter merge and the inference run.
- Data and backend. Piped live MQTT sensor streams into MongoDB for a bioreactor startup, built a BPMN to PDF report engine, and wrote healthcare data-cleaning pipelines.
- Meetily. Refactored the backend and added OpenAI provider support and a CLI testing script to a privacy-first meeting-notes tool. PR #75, merged.
- bpmn-io/refactorings. Proposed a cosine-similarity connector-template recommender as a lighter alternative to LLM function calls, and built the implementation fork. Issue #33.
Python, FastAPI, Inspect / inspect_evals, preregistered experiment design, LLM-as-judge agreement checks, LangGraph, MCP/FastMCP, Claude Agent SDK, Gemini API, PydanticAI, PyTorch, Hugging Face Transformers/PEFT, fine-tuning (LoRA, QLoRA, DPO), vLLM, Modal, Playwright, PostgreSQL, Qdrant, Docker, Kubernetes, AWS, GCP, OpenTelemetry/Langfuse, GitHub Actions, pytest.