Skip to the entries
Vishal Khan
Vishal Khan, head and shoulders.

Vishal Khan

AI and ML engineer, London

Facts checked

I build AI systems, and I check the numbers before I believe them.

I started in finance and tax, filing GST and income tax returns where every figure had to survive someone checking it months later. After that came over two years of sales forecasting and reporting, then a year and a half building Databricks pipelines over telecom records at Teleperformance.

Now I am finishing an MSc in AI and Computer Science at Northeastern University London, and most of what I build asks whether an AI system's numbers can be trusted.

Looking for an AI or ML internship now and full-time roles from early 2027. No sponsorship needed for the internship, or for 18 months of full-time work.

vishalkhan251@gmail.com

CV, PDFGitHub, opens in a new tabLinkedIn, opens in a new tabAsk my AI twin, opens in a new tab

A robot head with wires for hair, rendered in grey metal.
Plate 1. Robot girl with hair wires, by GR-video. CC BY 4.0. Retinted for this page, and here because I like it.

A program that cannot read scores higher than every model in a published benchmark's own results table.

Expected no better than 0.183; measured 0.973. Joint accuracy on GAIA through TRAIL's own scorer. The hatched span is where a program that reads nothing should have landed, and the red mark is where it did. 0.974, the thin tick, is the most anything can score.
Written down first
At 10:16 on 27 August, before the repository existed: a predictor that emits every span with every error category should score no better than the best real judge.
Measured
0.973 joint accuracy on GAIA through TRAIL's own unmodified scorer, against 0.183 for the best row in the paper's Table 1. It gets there by emitting 129.1x as many errors as the answer key holds.

Both headline metrics divide by the number of errors in the answer key and never by the number the judge reported, so flagging everything is close to optimal play. Of the 9 properties I wrote down, 7 came back violated and 2 latent: real defects in the code that move no number on this data.

How it was checked, and what it does not claim

TRAIL is Patronus AI's benchmark of human-annotated agent traces, published under MIT. Its scorer is fetched at a pinned commit, checked against a SHA-256 and run exactly as published. The predictor's whole input is a list of span identifiers, so it never sees a trace's contents or the answer key.

It does not re-score the paper's table, because TRAIL publishes no raw model outputs to re-score, and it says nothing about whether the human annotators marked the right errors. There is no replacement metric and no fork: auditing one benchmark is the project.

The audit is only possible because the authors published the scorer, the gold labels and the traces. A benchmark that publishes a table and keeps its scoring code cannot be checked from outside at all.

uv run trailaudit report --check
Repository, trail-scorer-audit, opens in a new tabThe pre-registration, trail-scorer-audit, opens in a new tab

Figures read from the trail-scorer-audit README and its pre-registration.

Rerunning a pipeline five times caught four planted bugs, flagged one correct float step, and missed a bug that held still.

Each step of the reference pipeline: whether a bug was planted in it, and how many of 4 comparisons diverged.
StepPlantedDiverged
generate_inputsNo0 of 4
daily_revenueYes4 of 4
customer_keysYes4 of 4
apply_price_updatesYes2 of 4
append_audit_logYes4 of 4
mean_basketcorrect code, float noiseNo4 of 4
sparse_customer_keysreal bug, held stillYes0 of 4
roll_up_keysNo0 of 4
One run of five on pipelines/reference.py, so four comparisons against the first run per step. The counts in red are where the result disagreed with what was planted.
Planted beforehand
Real bugs in five of the eight steps, four distinct ones, with the other three steps left correct.
Measured
5 of 8 steps diverged in 13.4 s. mean_basket is correct code whose float average moves in the last few bits. sparse_customer_keys carries a real bug and agreed with itself on all 4 comparisons.

DuckDB adds the terms of a parallel sum in whatever order its threads finish, and float addition is not associative, so a plain two-run diff reports hundreds of differences on correct code. Running the pipeline is the easy half. Deciding whether two outputs are the same answer is the project.

How it decides, and where it is blind

Each divergent step is re-run at one thread. Four of the five stop diverging, which names thread order as the cause; the audit-log append keeps diverging, because its answer depends on having run before.

The sharpest limit: a backfill that silently drops a column the same way on every run gets a clean zero. A rerun checker is blind to a deterministic wrong answer by construction, and the README shows the case on real NYC taxi data.

The README also says plainly that bit-exact multiset equality matched the oracle on every step it was scored on, so those tables hold no evidence for the oracle over the cheap comparison. What the oracle adds is the divergence class, its size and its cause.

uv run twicerun run pipelines/reference.py
Repository, twicerun, opens in a new tab

Figures read from the twicerun README.

Splitting HAM10000 by image leaks 38% of the test set, and fixing it cost this model about one and a half points.

accuracy: +1.54, interval +0.44 to +2.64. balanced accuracy: +1.94, interval -0.84 to +4.73. macro F1: +1.31, interval -0.97 to +3.59. Image split minus lesion split, paired within seed over 8 seeds, with 95% intervals. Only accuracy clears zero.
Expected, from published audits
The correction costs several points of accuracy.
Measured
+1.54 points of accuracy, 95% interval +0.44 to +2.64, paired over 8 seeds. On balanced accuracy and macro F1 the interval crosses zero.

The leak is real and reproducible from a 690 KB metadata file: 4,501 of the 10,015 images repeat a lesion already in the set. What it is worth to a model is a separate question, and for this one the answer was smaller than I expected.

Why the gap might be small, and what was not tried

The same small CNN is trained twice and only the split changes: 390,695 parameters at 64 pixels, ten epochs, on a CPU. Both arms land within a handful of images of the same train and test sizes.

Three explanations the evidence cannot separate: the model is too small to memorise a lesion, 64 pixels destroys the cue, or ten epochs is not enough to overfit. All three predict a gap that grows with capacity, and that run is not done here because each arm at 224 pixels is hours on a CPU.

Accuracy is the least useful column. Answering the majority class every time scores 0.667, so balanced accuracy, against a chance floor near 0.14, is the one to read. No transfer learning, no tuning, and no clinical claim.

uv run lesion-split leakage
Repository, lesion-split, opens in a new tab

Figures read from the lesion-split README.

A live assistant that answers questions about me from 89 documents, cites what it used, and says so when they do not cover the question.

  1. Questionplus the last 2 user turns
  2. Embedtext-embedding-3-small
  3. Search20 candidates from 89 documents in ChromaDB
  4. Cutanything past 0.75 cosine distance
  5. Boostpreferred sources 0.12 closer, by intent
  6. Groundbest 5 chunks, delimited as untrusted data
  7. Answergpt-5-mini, streamed over SSE with its sources
How a question becomes an answer, with the settings the repository ships. If every match scores badly the single best one is kept, so the refusal comes from the prompt rules rather than from an empty context.
Built
FastAPI and ChromaDB with Azure OpenAI, intent-boosted retrieval, and answers streamed over SSE to a React 19 front end. 390 tests run in CI, 276 backend and 114 frontend.
Running
Front end on Vercel, backend on Azure Container Apps, behind per-IP rate limits and a daily spend cap.
What I would change
Its defence against instructions planted in a retrieved README is a paragraph of the system prompt, and the test for it only asserts the paragraph is present. Nothing measures whether it works. untrusted-rows is where I am measuring that kind of defence.

The 89 documents are YAML knowledge files, the CV and live READMEs from public GitHub, read from the running deployment on 12 Oct 2026. The number moves as repositories are added.

Why those settings, and what the README admits

Follow-up questions such as "what tools did he use?" embed to almost nothing on their own, so the last two user turns are folded into the retrieval query. That is cheaper and more predictable than an extra model call to rewrite the question.

Intent never filters the search. It nudges preferred sources 0.12 closer, so a question about skills still finds the project where the skill was used.

Ingestion runs in the background and keys every document on a content hash, so a cold start answers health checks at once instead of restart-looping while it embeds.

Ask the twin, ai-professional-twin, opens in a new tabRepository, ai-professional-twin, opens in a new tab

Figures read from the live twin's readiness endpoint, its repository description, the twin's README and the untrusted-rows README.

White matter tracts of a human brain, rendered in pale grey.
Plate 2. Neural Networks of the Brain, by Universal Design for Learning. CC BY 4.0. Tractography, not a metaphor. The twin is a retriever over 89 documents and looks nothing like this.

In progress

  • untrusted-rows, opens in a new tab

    Started

    A DuckDB MCP server that treats the rows it returns as untrusted input, with its false-positive rate on real database text still unmeasured.

    Started21 Sep 2026

  • cannot-read, opens in a new tab

    Started

    Whether recall-only benchmark metrics have a floor their papers do not report, written down before any code and not yet measured.

    Started25 Sep 2026

Earlier work

A chess set mid-game, rendered in three dimensions.
Plate 3. A Beautiful Game, by Moeen Sayed and Mujtaba Sayed, glTF conversion by Ed Mackey. CC BY 4.0.

Background

to now

Where the work moved, 2018 to 2027: filings, then reporting, then pipelines, then the MSc in red. Study ran alongside the work, the outlined bars underneath. Every span is drawn from the dates listed below.

Work

  1. Teleperformance Global Business, Data Engineer

    Jun 2024 to Jan 2026

    Databricks pipelines over millions of telecom records, feeding a data engineering team of 50 or more. Migrated on-prem SQL Server schemas to Databricks, with query performance up by about 20% and infrastructure cost down by about 20%.

    Jun 2024 to Jan 2026

  2. Avant Garde Clean Room and Engg. Solutions, Executive

    Aug 2021 to Dec 2023

    ML sales forecasts reaching 80 to 90% accuracy on monthly revenue, used daily for planning. Rebuilt revenue and collections reporting in Excel and VBA, from about two hours a month to about thirty minutes.

    Aug 2021 to Dec 2023

  3. Aryan Care, E-commerce and finance executive

    Jan 2021 to Aug 2021

    GST and income tax filings and audits, beside an e-commerce operation.

    Jan 2021 to Aug 2021

  4. PJRJ Associates, Finance, accounts and taxation

    Jul 2018 to Sep 2020

    Two years of finance and taxation work: GST filings, income tax returns and audits.

    Jul 2018 to Sep 2020

Study

  1. MSc, AI and Computer Science, Northeastern University London

    Jan 2026 to Jan 2027

    Dissertation in progress, on agent reliability. Nothing to read yet.

    Jan 2026 to Jan 2027

  2. MSc, Data Science, Liverpool John Moores University

    Mar 2023 to Mar 2024

    Dissertation on reinforcement learning for dynamic pricing: DQN, A2C and PPO in a Gym-style environment built on real e-commerce data, where DQN gave the most stable profit curve. It is not published online, so that result cannot be checked from here. I will send the document on request.

    Mar 2023 to Mar 2024

  3. Executive PG Programme, Data Science, IIIT Bangalore

    Jan 2022 to Jan 2023

    Jan 2022 to Jan 2023

  4. Bachelor of Commerce, University of Delhi

    Nov 2018 to Nov 2021

    Nov 2018 to Nov 2021

Skills, and where each was used

Evaluation

  • Pre-registered checks: trail-scorer-audit, twicerun
  • Baselines and intervals: lesion-split, twicerun
  • Leakage and split design: lesion-split

LLM systems

  • RAG over ChromaDB: the twin
  • Azure OpenAI: the twin
  • MCP: untrusted-rows
  • Prompt-injection handling: the twin, untrusted-rows

Data

  • Databricks: Teleperformance
  • SQL Server to Databricks: Teleperformance
  • DuckDB: twicerun, untrusted-rows
  • pandas: women in chess

Machine learning

  • PyTorch: lesion-split
  • scikit-learn, XGBoost: UK HE completion risk
  • TensorFlow, Keras: CIFAR-10
  • DQN, A2C, PPO: LJMU dissertation

Engineering

  • Python, pytest, offline CI: every entry here
  • FastAPI, SSE: the twin
  • React 19, TypeScript: the twin, this site
  • Docker: the twin's backend image

Contact

Email gets read the same day. A call is fifteen minutes, and I will have the repository open while we talk.

vishalkhan251@gmail.com

Location
London, United Kingdom
Sponsorship
Not needed for an internship now, or for 18 months of full-time work from early 2027. The Student visa and the Graduate route both carry their own permission to work.
Until January 2027
An AI or ML internship alongside the MSc: twenty hours a week in term time and full time in the vacations. No notice period.
From early 2027
Full time, on the Graduate route, which gives 18 months of work with no sponsor.