The frontier is not the model. It's the system.

As a vertical AI company, one of our goals is easy to define: How do we ensure our AI systems stay on the Pareto frontier across accuracy, cost & speed?

‍Over the last few years of my research career and now at FurtherAI, the world moved from ML → AI → agents but the meta lesson hasn’t really changed. Models keep getting better. The abstractions around them keep changing. But the core job is the same: build the measurement, experimentation, and production loops that tell you what actually works.‍

The frontier is not just the model. It’s the system and product around it.

‍Here’s the high-level blueprint I’d use to build something like that:‍

Build the measurement layer

Start with canonical datasets and evals. You need to know what the system is good at, where it fails, whether a change actually improved it, and whether that improvement holds across important slices and distribution shifts.

A few years ago, synthetic data was a hard problem. It’s crazy how much that has changed.

Today, if a model can solve a task, it can increasingly help generate realistic examples of that task too. A few years ago Magpie showed how far this can go with instruction data; CoEvolve goes further and uses an agent’s own uncertainty and failures to decide what data to generate next.

That changes the bottleneck: Generating data is becoming easy. Generating the right distribution is hard.‍

This matters even more in regulated verticals, where production data is sensitive and often hard to reuse across customers. Synthetic data lets us expand around the real distribution - especially rare failures and cases near the capability boundary.

The hard questions become: What parts of the distribution are we missing? What should mirror production vs deliberately stress the edges? Can failures and uncertainty drive what we generate next without introducing correlated evaluator bias?‍

Build the experimentation loop

Once you can measure the system, every model, prompt, harness, tool, retrieval strategy, agent architecture should become a reproducible experiment - pinned dataset, scorer, config and baseline.

The unit of evaluation is not just the model. It’s the whole system.

For each candidate, measure more than average accuracy:

• Quality overall and on important slices / failure modes
• Cost per task and tool / token usage
• Variance across repeated runs - and enough statistical confidence to know a 2% win is real
  

Then compare systems on the actual Pareto surface rather than collapsing everything into one score. A small model + strong harness may dominate a frontier model; an agent may only be worth it on the hardest slice. Winning configs should graduate through offline evals → shadow traffic → production monitoring, with regressions feeding back into the eval set.

So when a new model ships, the question becomes, “Which tasks does it move the frontier on, and by enough to justify changing the system?”

Build the intelligence layer

The best system probably varies by task and increasingly, by input. A clean three-page PDF and a 200-page scanned loss run shouldn’t take the same path.

Eventually, the system should get better at answering - What is the cheapest and fastest approach to clear the bar?

In practice, that becomes a calibration and routing problem: estimate expected quality from a cheap first pass, then route based on confidence / complexity / historical performance on similar cases. This is where a lot of the interesting technical questions live.

For example:

• What models should we make available? Where do OSS models fit?
• Small model vs expensive frontier model?
• Can we dynamically route based on the input?
• What tools do the models need today? What tools need to phase out?
• What should memory actually contain - everything from facts, previous outputs, corrections, or successful procedures?
  

Instead of having strong ideological answers, evals should decide what wins.

And the answer should be allowed to change over time. That’s the Bitter Lesson playing out in product form.

One primitive I’m especially interested in right now is continual learning. I remember Voyager was a wildly popular paper. The idea that an agent could accumulate reusable skills from experience & use it for later felt very different then. So far skills are the most practical mechanism for it. However true proponents of Continual learning will argue skills won't cut it (And I agree!) There will be many mechanisms here: memory, retrieval over previous trajectories, test-time training, etc.

Build the production flywheel

Once the measurement and experimentation system exists, production should continuously make it better.

• A model fails on a weird edge case → that becomes an eval.
• A human corrects an extraction → that becomes signal.
• A low-confidence run fails → that improves escalation.
• An expensive model keeps solving cases a smaller model can handle → that becomes a routing opportunity.
• An agent figures out a robust way to do something → maybe that becomes a skill.
  

The loop becomes:

Production → Signal → Datasets & Evals → Experiments → Better Intelligence → Production

If you get this right, the system starts compounding. Every meaningful failure can become something you test forever. Every new model can immediately be evaluated against everything you’ve learned so far.

And every improvement in the underlying models gives you another opportunity to simplify the system around them.

We’re hiring for this

None of this is particularly hard to explain. But each step becomes extremely deep when you try to make it actually work against real-world distributions, with real latency and cost constraints, for real customers.

That’s why I think this is some of the coolest AI engineering work possible right now. We’re hiring Staff-level engineers at FurtherAI to own these systems end-to-end.

If these are the kinds of problems you want to work on, join us.