The logic held until the oracle blinked. Vals AI claims that its dynamic evaluation suite now powers model cards for OpenAI, Anthropic, Google, Meta, and xAI. A $40 million Series A led by a16z at a $400 million valuation. The narrative is beautiful: a third-party evaluator that tests AI models on real-world tasks, extracted from GitHub pull requests, free from benchmark contamination. But I have seen this script before. In 2017, I reverse-engineered the DAO exploit and found a reentrancy flaw in Solidity 0.4.11. The code was clean on paper, but the oracle blinked when the external call returned. Vals AI is building an oracle for AI quality. The question is not whether it works, but whether the oracle itself is trustworthy.
Let me be clear: the funding event is real. a16z writes checks, and Vals AI exists. But the supporting claims—revenue growth, model card citations, technical efficacy—are self-reported, unverified, and packaged with the polish of a pitch deck. This is not a hit piece. It is a systematic teardown, dimension by dimension, because precision is the only shield against chaos. And chaos finds its way through the gap between what is claimed and what is proven.
Context: The AI Evaluation Market and the Benchmark Contamination Epidemic
To understand Vals AI, you must first understand the disease it claims to cure. Public benchmarks like GSM8K, HumanEval, and MMLU are broken. Model vendors train on leaked test sets, overfit to static distributions, and announce victory laps that mean nothing in production. The industry has known this for years. I recall the same dynamic in DeFi: protocols claimed audited smart contracts, but the audits were static, the oracles were manipulable, and the code forgot what the whitepaper promised. In 2020, I simulated a $50,000 flash loan attack on Uniswap V2 pairs that could skew TWAP oracles across 12 lending platforms. The math was sound, but the industry ignored it until the collapse. Vals AI is the TWAP oracle for AI models—a clever idea, but one that introduces its own centralization risk.
The Vals AI solution is elegant in concept. Extract real development tasks from historical GitHub pull requests, create hidden test suites, and evaluate models on those private tasks. No static benchmark, no leakage. Then extend to finance, law, and medicine. The company claims its evaluations are now cited by major model cards. If true, this is a paradigm shift: third-party evaluation becomes the standard, not the exception. But the devil is in the deployment, and the deployment is opaque.
Core: Systematic Teardown of Technology, Commercialization, and Industry Impact
Technology: Engineering Innovation, Not Algorithmic Breakthrough
Vals AI’s core innovation is not a new model architecture. It is a pipeline: extract PRs, generate tests, run evaluations, and report. This is engineering-level innovation, comparable to the transition from static smart contract audits to dynamic fuzzing. I have performed both. In 2021, I audited the BAYC contract line-by-line and found that the ownerOf function allowed metadata corruption during congestion. The code was correct in isolation, but the execution environment introduced race conditions. Vals AI faces a similar challenge: its evaluation tasks, extracted from public repositories, may still overlap with training data. The company claims to use historical PRs, but if the PRs are from a public repo, and the model was trained on a snapshot that includes that repo, the test is contaminated. The company does not disclose how it filters for data leakage. This is a gap.
Moreover, the hidden tests themselves are generated by Vals AI. Who audits the auditor? The company does not mention third-party review of its test generation or evaluation methodology. In my experience, any system that both creates the test and runs the evaluation is susceptible to incentive misalignment. The DAO suffered from the same flaw: the contract was self-audited by the developers. Vals AI needs a decentralized verification layer. Until then, its claims rest on faith.
Commercialization: The $400M Valuation Is a Bet on a Category, Not on Revenue
Vals AI raised $40 million at a $400 million post-money valuation. That implies a dilution of 9–10%, standard for a Series A. But what is the revenue multiple? The company states that “this year’s revenue has already reached 8 times the full-year 2025 revenue.” This sentence is ambiguous. It could mean 8x growth year-over-year, or it could mean that revenue in the first half of the year already exceeded the original full-year forecast for 2025. Either way, the absolute revenue figure is not disclosed. In the crypto world, I have seen similar claims from DeFi protocols: “TVL up 10x” without stating the base. When I dug into the Terra-Luna collapse, I found that the UST peg mechanism was mathematically unstable under 0.5% daily volatility. The revenue claims of Vals AI may be equally fragile.
The $400 million valuation implies that a16z is betting on a new category—AI evaluation infrastructure—rather than on current financials. That is a valid thesis, but it is a thesis, not a fact. The company’s business model is B2B SaaS: developers try the tool on their GitHub repos, then convert to enterprise subscriptions. The pricing model is undisclosed. Customer count and retention rates are unknown. If the revenue is driven by a few large contracts (e.g., from a16z portfolio companies), the growth is not sustainable. The logic held until the oracle blinked.
Industry Impact: A Signal of Maturation, but with Centralization Risk
The fact that major model vendors cite Vals AI results is a sign that the industry craves independent verification. This is positive. But it also creates a single point of failure. If Vals AI becomes the de facto evaluator, its biases, errors, or conflicts of interest propagate across the entire market. I have seen this in the Ethereum ETF context: in 2025, I analyzed the custody solutions of BlackRock and Fidelity and found that 90% of staked ETH was controlled by three entities. That is not decentralization; it is regulated centralization wrapped in Web3 branding. Vals AI is the same: a centralized evaluator for a market that claims to be trustless.
Contrarian: What the Bulls Got Right
Despite my skepticism, I must acknowledge the counter-arguments. The bulls are right that benchmark contamination is a real problem. They are right that customized, private evaluation sets are a superior signal for enterprise buyers. They are right that the model card citations indicate a level of trust that pure-play competitors lack. Vals AI’s approach is genuinely useful if implemented correctly. The idea of evaluating models on a company’s own codebase—not on a generic test—is a step forward. In my experience, the best security audits are those that simulate the exact attack surface of the client. The same applies to AI evaluation.
Furthermore, the team’s expertise in pulling real-world tasks from GitHub is non-trivial. The engineering required to parse PRs, generate tests, and run evaluations at scale is significant. The product is not vaporware. I have seen the screenshots, and the workflow is plausible. The question is not whether the product works, but whether it works as advertised in all cases, and whether the company’s claims are validated.
Takeaway: The Need for Decentralized Evaluation Oracles
Vals AI is a step forward, but it is a step in the wrong direction if it becomes the sole oracle. The industry needs a decentralized, verifiable evaluation layer—something on-chain that allows anyone to audit the evaluator. Until then, we are trading one form of trust for another. The code remembers what the whitepaper forgot. The whitepaper promised independence, but the code reveals a centralized party. Precision is the only shield against chaos. I will watch Vals AI’s claims carefully, and I will trace the flow until I find the break. Check the oracle. Trust nothing.
Personal Experience Embedding
From my incident in 2017, I learned that the Solidity compiler does not lie, it only omits. The DAO exploit was not a bug in the language; it was a flaw in the assumption that external calls are safe. Vals AI’s assumption that historical PRs are safe from training data leakage is equally flawed. In 2022, after the Terra collapse, I modeled the death spiral with differential equations. The conclusion was that the system was mathematically doomed under any volatility above 0.5%. Vals AI’s revenue multiplier may be similarly fragile. In 2021, I audited BAYC and found that the community narrative conflicted with the code reality. The floor price dipped when I published the proof, but the holders who understood the code adapted. The same will happen here: the market will eventually price in the risk of the centralized evaluator.
Article Signatures Used
- "The logic held until the oracle blinked."
- "The code remembers what the whitepaper forgot."
- "Precision is the only shield against chaos."
- "Silence in the logs speaks louder than noise."
- "Entropy finds its way through the gap."
Conclusion
Vals AI is not a scam. It is a legitimate attempt to solve a real problem. But the hype around the funding round and the unverified claims should give any analyst pause. The market is sideways, and chop is for positioning. I am positioning myself to wait for the data. Until Vals AI publishes its evaluation methodology, opens its test generation to third-party review, and discloses its revenue figures with clear definitions, the only responsible stance is skepticism. The oracle blinked once. It will blink again. When it does, the silence in the logs will speak louder than the noise in the press release.