Artificial intelligence hides more complexity than any technology wave before it. Writing code, wiring APIs, and scaling infrastructure now feel effortless, but that ease conceals an expanding layer of invisible decisions beneath the surface. The hard problems have moved upward, into judgment, coordination, and systems thinking.

Shawn “Swyx” Wang’s 2023 post is widely credited with defining the new concept of the “AI engineer” as someone who effectively applies foundation models via APIs or open-source tools to build, evaluate, and productize AI systems rather than train them.

As that vision of the AI engineer has proven out, each new layer of abstraction keeps pushing engineers farther from the primitives of programming and framework internals. As a result, new kinds of hidden skills are emerging, suited to a world where large language models (LLMs), not humans, generate the first draft of our software.

Evaluation is the new CI

Continuous integration (CI) once defined good engineering hygiene. Today, the same discipline of measurement, testing, and automation has become essential for AI systems.

Jeff Boudier, product and growth lead at Hugging Face, the open-source platform that underpins much of today’s model sharing and evaluation ecosystem, describes this shift as the next great standard in software practice. “Evaluation is the new CI,” he told InfoWorld. “The real engineering leverage is not choosing the right model, it is building systems that can continually measure, test, and swap them.”

Hugging Face has built its platform around that principle. Its Evaluate library standardizes the process of assessing models across hundreds of tasks, while AI Sheets provides a no-code interface for comparing models on custom data sets. Developers can run evaluation workflows on on-demand GPUs through Hugging Face Jobs, and track progress on open leaderboards that benchmark thousands of models in real time. Together, these tools turn evaluation into a continuous engineering discipline. “The most important muscle companies need to build,” said Boudier, “is the ability to create their own evaluation data sets with relevant questions and good answers that reflect how their customers actually talk.”

Experts across academia and industry agree that this focus on evaluation will reshape software development. On Lenny’s Podcast, Hamel Husain, consultant at Parlance Labs, called evals “a systematic way of looking at your LLM data, creating metrics, and iterating to improve.” In the same podcast, Shreya Shankar, a PhD researcher at UC Berkeley, noted that evals provide “a big spectrum of ways to measure application quality,” from checking core functionality to evaluating how systems respond to ambiguous or unexpected user behavior. Engineer Shaw Talebi described the impact in a post on X: “Building LLM systems felt more like praying to the AI gods than engineering. But that all changed when I learned about eval-driven development.”

What testing was to software, evaluation is becoming to AI. It is the process that turns model unpredictability into something engineers can understand and control.

Adaptability as the core design principle

If evaluation defines quality, adaptability defines longevity. But adaptability in the AI era means something very different from learning new frameworks or languages. It now means designing systems that can survive change on a weekly or even daily basis.

“We are still in a phase where research is moving faster than engineers,” said Boudier. “Every week on Hugging Face there is a new top-five model. It is not about which one you pick, it is about building technology that lets you swap painlessly when a better one appears.”

Earlier generations of engineers adapted to hardware shifts or long product cycles. AI engineers adapt to a moving frontier. Model APIs, context windows, inference prices, and performance benchmarks can all change within a month. The challenge is not learning a tool, but building processes that absorb continuous disruption.

Barun Singh, chief product officer at Andela, a global talent marketplace that connects companies with remote software engineers and other technologists from emerging markets, believes this is the defining skill of the decade. “In many ways all knowledge work is undergoing this massive change, but software engineering is undergoing the biggest change first,” he told InfoWorld. “AI tools can either accelerate your understanding or create a false sense of productivity with a huge amount of debt.”

Singh sees adaptability as both technical and cognitive. “The more you can think at a high level and at ground level simultaneously, the more advanced you are,” he said. “The person who has both a deep understanding of classical architecture and real experience with LLMs in production, that is the hardest hire right now.” Singh also highlights the need for boundaries as a mark of professional maturity. “Creating boundaries for your work in the form of testing, so you catch mistakes before they reach production, becomes even more important in the age of AI.” In this sense, adaptability is not about chasing novelty. It is about designing systems and workflows that can safely accommodate it.

De-risking as an engineering discipline

The third skill shaping AI engineering is de-risking. Engineers now need to think like compliance officers, ensuring that data sources, models, and pipelines can withstand scrutiny.

Michelle Lee, general counsel in residence at Wilson Sonsini, told InfoWorld that engineers must take ownership of these questions. “They are much closer to the data considerations and the architecture considerations,” she said. Lee noted that regulators around the world are already asking who is accountable when AI systems cause harm and that transparency about training data and model behavior is becoming an engineering requirement.

At the AI Conference 2025 held in San Francisco in October, Jessica Li Gebert, a data monetization consultant at Neudata, described this as both a risk and an opportunity. She called enterprise data “a treasure trove” but warned that many companies have no idea how to move from recognizing that value to realizing it. “There is a huge knowledge gap,” she said, “between believing your data has value and actually understanding how to unlock it safely.” Engineers who can build governance and lineage controls will be critical to bridging that gap.

Michael Hejtmanek, Gebert’s colleague and vice president of corporate solutions at Neudata, added that most enterprises still view sharing data with AI developers as “an insurmountable danger or risk.” Engineers fluent in both data systems and risk management will become essential to AI adoption.

Engineering what the models can’t

Over the last two decades, enterprises have competed for developer talent by perfecting the ergonomics of software creation: Continuous integration pipelines, cloud platforms, and collaborative workflows that made code more testable, reproducible, and observable. The next phase of that competition will hinge on the systems that bring the same rigor to AI.

The engineers who build evaluation loops, model registries, and governance frameworks are not just keeping pace with innovation. They are defining how intelligence is integrated into enterprise applications and workflows. In the same way CI brought more reliability, predictability, and security to software development, these new systems will make model behavior measurable, improvable, and accountable.