15 min read3 days ago

–

(This article was inspired by Stephan Schwab’s excellent piece “Why We’ve Tried to Replace Developers Every Decade Since 1969” which traces the recurring dream of eliminating software developers from COBOL through to AI. Reading it, I recognised the same pattern playing out in my own field of data warehousing, data analytics and business intelligence; a fifty-year cycle of tools promising to democratise data work, each delivering genuine value while leaving the fundamental need for specialists stubbornly intact.)

The Pattern That Frustrates Everyone

Every decade brings new promises: this time, we’ll finally make building analytics platforms simple enough that we won’t need so many specialists. From SQL to OLAP to AI, the pattern repeats. Business leaders grow frustrated waiting months for a data warehouse that should take weeks, or weeks for a dashboard that should take days. Data teams feel overwhelmed by request backlogs they can never clear. Understanding why this cycle persists for fifty years reveals what both sides need to know about the nature of data analytics work.

Press enter or click to view image in full size

The Dream Was Born When Databases Learned English

When IBM researchers Donald Chamberlin and Raymond Boyce created SEQUEL (later SQL) in 1974, they had an explicit goal: create a query language that non-programmers could use. The vision was revolutionary; structured data could be retrieved using something approaching plain English. SELECT, FROM, WHERE, GROUP BY. The syntax was designed to be readable by business analysts, not just computer scientists.

“If we make the language readable enough, anyone who understands the data can query it themselves.”

This vision had genuine appeal. Businesses were accumulating data faster than data analytics developers could write COBOL report programs. SQL promised to democratise data access, letting the people who understood the business retrieve the information they needed without submitting requests to the data processing department.

What happened instead? SQL became another technical skill requiring specialised training. Business analysts who tried to write their own queries quickly discovered that readable syntax didn’t eliminate the complexity of joins, subqueries, aggregations or understanding why the results didn’t match last month’s report. A new class of “report writers” and “database analysts” emerged. The dream of eliminating specialised data analytics developers remained unfulfilled.

The 1980s: OLAP Servers and Fourth-Generation Languages Will Let Business Users Model Their Own Data

The 1980s brought a fundamentally different approach to the data access problem. Rather than making query languages more readable, pioneers like John D.C. Little at MIT asked a different question: what if we stored data in the format that business users actually think about it?

Little’s work on decision support systems had identified four criteria for computers that would support decision-making: robustness, ease of control, simplicity and completeness of relevant detail. His research project BrandAid, developed in 1975, evolved into a commercial product that would become known as Express; a fourth-generation language and modelling system that spawned an entire industry.

If you imagine a financial system, you tend to store your data by year, by line item, and possibly by branch; so what you have is a three-dimensional set of data. What the products were trying to do in those days was to put the data into the format and store it in the format that users understood it. The idea was: if you could put the data in the form they understood it, surely then it would be easier for them to get at it and do the analysis they wanted.

Express Server, along with competitors like TM/1 and IBM’s System W, represented a genuinely different philosophy. These multidimensional OLAP (MOLAP) servers didn’t just make querying easier; they restructured how data was stored. Dimensions, hierarchies, aggregations and calculations were pre-computed and held in memory or specialised file structures optimised for analytical access patterns.

The promise was compelling: business users, particularly in finance and planning, could build their own analytical applications. Essbase in particular was sold to power users, not to IT departments. An Essbase server could run under a finance analyst’s desk. The calculation language resembled spreadsheet formulas. Users who understood Excel could, in theory, build sophisticated planning and analysis applications without waiting for IT.

And to a remarkable degree, it worked. These tools created genuine productivity for their target users. The number of true believers and evangelists amongst customers, amongst FP&A guys, was huge. They loved the product. It wasn’t waiting for IT to build a table and build a bunch of SQL queries. They owned it.

Yet the dream of eliminating specialised data analytics developers still didn’t materialise. MOLAP applications required someone to design the dimensional structures. Someone had to understand how to handle sparse data, manage calculation order dependencies and optimise aggregation strategies. Complex business rules still required programming, even if the language looked like spreadsheet formulas. Performance tuning was an art form. Integration with source systems demanded ETL expertise.

A new class of specialists emerged: the Express developer, the Cognos PowerPlay consultant, the TM/1 expert. The tools had changed; the need for people who deeply understood both the technology and the business had not.

The MOLAP era taught the industry something important: you could restructure data to match how users thought about it, but the intellectual work of understanding the business well enough to structure it correctly; that remained irreducibly human.

The 1990s: Data Warehouses and Dimensional Modelling Will Solve Everything

The 1990s brought Ralph Kimball’s dimensional modelling methodology and the enterprise data warehouse movement. Building on the conceptual foundations that MOLAP had established, Kimball codified the approach for relational databases. The promise was compelling: build a properly modelled data warehouse with star schemas, conformed dimensions and consistent facts, and the complexity would be hidden from end users.

“Structure the data correctly once, and anyone can analyse it.”

This was meant to democratise the benefits that MOLAP had delivered to finance departments, but for the entire enterprise. Facts and dimensions. Slowly changing dimensions. Conformed hierarchies. The complexity would live in the ETL layer, maintained by specialists, while business users enjoyed a simplified, denormalised view of their world through OLAP cubes and query tools.

Organisations invested millions. Multi-year programmes launched to build enterprise data warehouses that would finally break the reporting bottleneck.

Yet most data warehouse initiatives struggled or failed outright. Projects took years and cost fortunes. By the time the warehouse was “complete,” source systems had changed and requirements had evolved. The ETL pipelines became unmaintainable spaghetti. Most critically, building an accurate dimensional model required understanding the same business complexity that MOLAP development and report writing demanded. Where does a sale belong when the customer’s region changed mid-quarter? How do you handle late-arriving facts? What’s the grain of the transaction?

The methodology was sound. The tools were capable. But the intellectual work of understanding business processes well enough to model them correctly; that couldn’t be automated away. Data analytics and data warehouse developers remained essential, and a new specialisation emerged: the ETL developer.

The 2000s: Semantic Layers and Self-Service Reporting Will Free Business Users

The 2000s saw two parallel movements, both promising to finally break the IT bottleneck.

First, enterprise BI platforms matured with a powerful concept: the semantic layer. Business Objects’ “universe,” Oracle BI EE’s’ “semantic layer” and similar abstractions promised to hide technical complexity behind business-friendly names. Data warehouse developers would build the semantic layer once, mapping cryptic table and column names to business terms. Then business users could build their own reports using familiar vocabulary (“Revenue,” “Customer,” “Product Category”) without knowing anything about the underlying SQL.

Second, tools like Tableau and later Power BI began democratising visualisation itself. Drag a field onto a canvas, see a chart appear. No semantic layer configuration required. No IT involvement needed. “Self-service BI” became the goal of every MIS team lead.

Both movements delivered genuine value. The semantic layer concept proved durable and important. Self-service tools genuinely lowered the barrier to creating visualisations. More people could explore data and build reports than ever before.

Yet neither eliminated the need for specialists. Semantic layers required constant maintenance as source systems changed. Self-service created new problems: dozens of conflicting dashboards with different definitions of the same metrics. Performance issues. Security concerns. “Data chaos” became the term for organisations drowning in ungoverned self-service content.

Meanwhile, something had been lost; the tools had democratised reporting, but in doing so, they’d lost something that the MOLAP era had provided: genuine analytical capability, the ability to model, calculate and explore. And the response to self-service chaos? A new wave of centralisation. Governed data sources. Certified datasets. Published data models. In other words: the work that data analytics developers had always done, now rebranded for the self-service era.

The 2010’s: Analytics Engineering and the Promise That Analysts Could Do Their Own Data Engineering

The mid-2010s brought a more fundamental reimagining of who should build data pipelines. Cloud data warehouses like Amazon Redshift (2012), Google BigQuery (2010, broadly adopted mid-decade) and Snowflake (2014) changed the economics and architecture of data warehousing. With virtually unlimited compute and storage available on demand, a new philosophy emerged: ELT instead of ETL.

The insight was simple but powerful. Traditional ETL required specialised tools (Informatica, DataStage, SSIS) and specialised developers who understood their arcane interfaces and proprietary transformation languages. But if you loaded raw data into a cloud warehouse first, you could transform it using SQL. And analysts already knew SQL.

“If analysts can write SQL, why can’t they build their own data pipelines?”

dbt (data build tool), created by Fishtown Analytics in 2016, crystallised this vision into a product. Write your transformations as SELECT statements. Organise them into a directed acyclic graph of dependencies. Add tests. Generate documentation. Apply software engineering practices (version control, code review, CI/CD) to data transformation. No proprietary ETL tool required. No Java or Scala. Just SQL.

The promise was intoxicating: the same analysts who understood the business well enough to write reports could now build the data models those reports consumed. The bottleneck of waiting for data engineers to build pipelines would dissolve. A new role emerged with a new title: the “analytics engineer”; someone who combined the data engineer’s technical rigour with the analyst’s business understanding.

And to a remarkable degree, it worked. dbt and the “modern data stack” genuinely lowered the barrier to building maintainable data transformations. Analysts who had been frustrated waiting for IT could now build their own staging tables, their own dimension tables, their own metrics. Data transformation became more accessible, more testable, more documented than it had ever been under the old ETL regime.

Yet the dream of eliminating specialists proved, once again, elusive.

Building a correct dimensional model still required understanding the business deeply. Deciding how to handle slowly changing dimensions still required judgment. Performance optimisation still required expertise. Managing dependencies across hundreds of models still required architectural thinking. And as dbt projects grew from dozens of models to hundreds to thousands, new problems emerged: sprawling DAGs, unclear ownership, conflicting definitions, the same “data chaos” that had plagued self-service BI, now manifesting in the transformation layer.

The analytics engineer role didn’t eliminate the need for specialists; it created a new specialism. A very valuable one, to be sure. But still a role requiring deep expertise in both business logic and technical implementation. The tools had democratised access to building data pipelines, but the intellectual work of understanding what to build and why; that remained as demanding as ever.

Get Mark Rittman’s stories in your inbox

Join Medium for free to get updates from this writer.

Semantic layers returned with renewed vigour in this era. Looker’s LookML. The dbt metrics layer. The promise: define business logic once, in code, and expose it consistently across all consumption tools. But building and maintaining a semantic layer still required someone who understood both the business definitions and the technical implementation. The tools were better. The work remained.

AI and the Latest Chapter: What’s Actually Different This Time

Today’s AI tools can do things that would have seemed like science fiction even five years ago. Claude Code can scaffold a complete dbt package (staging models, dimension tables, fact tables, schema tests, documentation) in minutes rather than days. Feed it a DDL file and some requirements, and it will generate internally-consistent SQL that actually runs. Point a Looker MCP Server at your warehouse and an LLM can interrogate the semantic layer, discover available fields and generate dashboard definitions on demand.

This is genuinely impressive. And it’s genuinely useful. But it’s worth being precise about what it changes and what it doesn’t.

What AI handles well is the mechanical transcription of intent into code. Once you know you need a dimension table with SCD Type 2 handling, Claude Code can write the SQL and the dbt configuration faster than any human. Once you’ve decided which metrics belong on a dashboard and how they should be visualised, an AI can generate the LookML or the Tableau workbook. The typing, the syntax, the boilerplate; that’s where AI excels.

What AI doesn’t handle is the thinking that precedes the typing. Should this be SCD Type 2 or Type 1? The AI doesn’t know your business well enough to decide. What’s the grain of this fact table? The AI can’t attend your requirements workshop. How should “active customer” be defined when sales, marketing and finance all have different definitions? The AI will happily implement whichever definition you give it, but it can’t navigate the political and analytical trade-offs of choosing one.

I’ve been working on projects recently where AI tools handle much of the code generation. The dbt models get written faster. The LookML views get produced more quickly. But the project still takes two weeks to deliver an MVP, not two hours. The time hasn’t disappeared; it’s been reallocated.

Instead of spending those two weeks with developers heads-down writing SQL while stakeholders wait, then reviewing theoretical designs documented in Confluence, the time gets spent differently. More workshops with users. More iterations on working prototypes. More conversations about whether the dashboard actually answers the question they need to ask. The AI generates a first draft of the data model in an afternoon; the next ten days are spent refining it based on what users see when they actually interact with real data.

This is a genuine improvement. But it’s not the improvement the “replace developers” dream imagines. The human expertise hasn’t been eliminated; it’s been concentrated on the parts of the work that were always the hardest and most valuable: understanding the business, making modelling decisions, translating vague requirements into precise definitions and iterating based on feedback.

The bottleneck was never really typing speed. It was thinking speed. And thinking still takes time.

What’s genuinely different this time is that the feedback loop can be faster. When generating code takes minutes instead of days, you can show stakeholders a working prototype sooner. When they say “that’s not quite what I meant,” you can regenerate and try again. The conversation shifts from “sign off on this specification document and we’ll build it over the next quarter” to “here’s a working version based on our conversation yesterday; what would you change?”

That’s valuable. It might even be transformative for how analytics projects get delivered. But it’s transformation through faster iteration, not transformation through eliminating the need for people who understand data modelling.

The AI generates the code. Humans still decide what code to generate. And that decision (what to build, how to model it, what questions it needs to answer) remains the hard part.

Why the Dream Persists

The recurring pattern reveals something important about how we think about complexity. Data warehousing and BI look like they should be simple because we can describe what we want in plain language. “Show me sales by region compared to last year.” That description sounds straightforward.

The complexity emerges in the details. Which date defines when a sale happened: order date, ship date, invoice date or payment date? How do you handle returns processed in a different period than the original sale? What about currency conversion for international transactions? Should you use the exchange rate at transaction time or reporting time? How do you attribute revenue when a sales rep changes territory mid-quarter? What happens when organisational hierarchies are restructured; do you restate history or not?

Each answer leads to more questions. The accumulated decisions, edge cases and business rules create genuine complexity that no tool or language can eliminate. Someone must think through these scenarios. That thinking is data warehouse and BI development, regardless of whether it’s expressed in SQL, Express 4GL, LookML, dbt or a prompt to an AI.

The Constraint That Never Changes

This fifty-year pattern teaches us something fundamental about data work itself. If the problem were primarily mechanical (too much SQL, too complex transformations, too many steps) we would have solved it by now. SQL made querying readable. MOLAP eliminated the need to think in tables and joins. Semantic layers hid technical complexity. Self-service tools eliminated IT gatekeepers. Analytics engineering democratised pipeline development. AI can now generate entire data models from descriptions.

Each advancement addressed a real friction point. Yet the fundamental challenge persists because it’s not mechanical. It’s intellectual.

A data warehouse is a model of a business. Building an accurate model requires understanding the business deeply; its processes, its edge cases, its exceptions, its evolving definitions. A BI solution is a lens through which people make decisions. Building an effective lens requires understanding what decisions matter, what questions lead to action and what context is needed to interpret the numbers correctly.

You can’t shortcut that understanding any more than you can shortcut the understanding required to design a building or diagnose a medical condition. Better tools help. Experience helps. But someone must still think it through.

What This Means for Data Leaders

Understanding this pattern changes how you evaluate new tools and approaches. When someone promises that their platform will let business users build data solutions without DW/BI specialists, you can appreciate the aspiration while maintaining realistic expectations.

The right question isn’t “Will this eliminate our need for data warehouse and BI developers?” The right questions are:

• Will this help our data team work more effectively on complex problems?
• Will this enable us to deliver certain types of solutions faster?
• Does this reduce time spent on repetitive work so developers can focus on high-value modelling challenges?
• Will our team need to develop new skills to use this effectively?
• Does this tool require a well-governed semantic layer to function correctly; and do we have one?

These questions acknowledge that DW/BI work involves irreducible complexity while remaining open to tools that provide genuine leverage.

The Dream Serves a Purpose

Perhaps the recurring dream of replacing data analytics developers isn’t a mistake. Perhaps it’s a necessary optimism that drives tool creation. Each attempt to make data work more accessible produces tools that genuinely help. The dream doesn’t come true as imagined, but pursuing it creates value.

SQL didn’t let executives query databases themselves, but it did create a more accessible foundation than COBOL. MOLAP tools like Express and Essbase didn’t eliminate the need for data expertise, but they gave finance teams genuine analytical power they’d never had before; and created products so beloved that customers were still running unsupported versions decades after decommissioning. Semantic layers didn’t make metric definition trivial, but they made it more maintainable. Self-service BI didn’t remove IT from the equation, but it expanded who could explore data. Analytics engineering didn’t let analysts replace data engineers, but it gave them tools to contribute more directly to the data platform.

AI won’t replace data analytics developers either. But it might finally let them spend most of their time on the work that actually requires human judgment (understanding businesses, making modelling decisions, having conversations with stakeholders) instead of the mechanical transcription of those decisions into code.

That’s not the fulfilment of the fifty-year dream. It’s something more realistic and possibly more valuable: tools that let experts be more expert, faster.

Moving Forward with Clear Eyes

The next wave of data development tools will arrive. Some will provide genuine value. Some will repeat familiar promises with new technology. Having perspective on this recurring pattern helps you engage with new tools productively.

Use AI assistants. Evaluate semantic layer platforms. Experiment with new transformation frameworks. But invest primarily in your people’s ability to think clearly about business complexity and data modelling. That capability remains the constraining factor, just as it was when Chamberlin and Boyce designed SQL to be readable by business users, and when John D.C. Little built BrandAid to let marketers analyse their own data.

The organisations that thrive aren’t those waiting for tools to eliminate the need for data platform development expertise. They’re the ones investing in people who can think clearly about business complexity; and giving them the best tools available to express that thinking efficiently.

Understanding this doesn’t mean rejecting new tools. It means using them with clear expectations about what they can provide and what will always require human judgment.

Fifty years of tool evolution. Fifty years of promises. Fifty years of the same fundamental truth: data warehouses and BI solutions are business understanding made queryable. The dashboards and reports we create are the visible outcome of invisible reasoning about what questions matter, how to measure the answers accurately and what the numbers actually mean. Better tools help us express that reasoning more efficiently. They don’t eliminate the need to do it.

INTERESTED? FIND OUT MORE

Rittman Analytics is a boutique data analytics consultancy that helps ambitious, digital-native businesses scale-up their approach to data, analytics and generative AI.

We’re authorised delivery partners for Google Cloud along with Oracle, Segment, Cube, Dagster, Preset, dbt Labs and Fivetran and are experts at helping you design an analytics solution that’s right for your organisation’s needs, use-cases and budget and working with you and your data team to successfully implement it.

If you’re looking for some help and assistance with connecting Claude to Looker or would just like to talk shop and share ideas and thoughts on what’s going on in your organisation and the wider data analytics world, contact us now to organise a 100%-free, no-obligation call – we’d love to hear from you!