Simplicity Won
An Informatica veteran catches up on the 2026 data stack: lakehouses, DuckDB, Kubernetes, GPUs, and the quiet reabsorption of NoSQL. The boxes changed. The discipline didn't.
This week, out of curiosity, I went looking for the Scala I thought I remembered writing. What I found was someone else's tutorial repo I had cloned in 2020, back when we were evaluating whether to port our batch processes to Spark DataFrames. Zero commits from me. That settled the Scala question (I don't write Scala).
But it opened a bigger one. My mental model of large data architecture froze somewhere around 2020, and parts of it are older than that. I spent twelve years as a Development Architect at Informatica, mostly around Master Data Management for Fortune 500 customers. ETL, staging areas, star schemas, nightly batch windows. That world. So I spent the morning with Claude asking basic questions. Is Spark still a thing? Is Kubernetes still popular, or did it go the way of MapReduce? Are we back to monoliths, just mass parallelization against GPUs? What is a "lakehouse"? Does anyone still need MongoDB?
The answers surprised me. Nearly every specialized system the 2010s produced has been folded back into something general-purpose. Simplicity won.
Where I left off
The stack I knew: PowerCenter pulled from source systems, transformed in flight, and loaded a warehouse (Teradata, Oracle). The warehouse was one expensive, tightly governed box that owned both the storage and the compute. Analytics meant SQL against star schemas. Batch ran at night, and you hoped it finished before morning.
The lake, the swamp, and the lakehouse
In the 2010s the industry tried data lakes: dump everything as files onto cheap object storage, figure out the schema later. The storage was cheap, but there were no rules. No transactions, no schema enforcement, two jobs writing the same data could corrupt each other. Practitioners started calling them data swamps.
The fix turned out to be a thin metadata layer called an open table format. Apache Iceberg won that war (Delta Lake and Hudi were the rivals). Your data is still just Parquet files (a columnar file format) sitting in object storage, but the table format adds back what the warehouse used to provide: transactions, schema evolution, snapshots, concurrent writers. You get the warehouse's guarantees at the lake's cost. That combination is the lakehouse, and the name means exactly what it sounds like: data lake plus data warehouse.
The part that struck me is what open tables did to the database itself. Because a table is now open files plus open metadata, any engine can read and write it. Spark, DuckDB, Trino, Snowflake, all speaking to the same Iceberg tables. The database is no longer one product. It's a set of files and metadata that every engine agrees on.
Spark matured, and the laptop caught up
RDDs, the thing I cloned that repo to learn, are effectively legacy. They still exist under the hood, but "using Spark" in 2026 means DataFrames and Spark SQL, mostly driven from Python. DataFrames won because they run through a query optimizer that can reorder and prune your work, where RDDs just ran your code blindly. Our 2020 evaluation picked the right side of that transition, for whatever that's worth now.
Spark itself is mature and stable. It still runs the heavy enterprise pipelines, but the new work is happening below it. Most datasets, it turns out, fit on one big machine. Engines like DuckDB and Polars process in a single node what used to need a cluster, and a laptop today outruns the Hadoop cluster I would have specced in 2015. These single-node engines read and write Iceberg natively, so they sit inside the same lakehouse as equals. Most times, something basic will do.
Kubernetes won and became invisible
My guess was that Kubernetes had faded like MapReduce. I had it backwards. Kubernetes won so completely it became invisible. It's the default layer everything runs on now (Spark itself largely moved off YARN onto it), and nobody gives conference talks about it for the same reason nobody gives conference talks about Linux.
Part of my guess was right, though. The big GPU training clusters often bypass Kubernetes for Slurm-style HPC schedulers, because gang-scheduling thousands of GPUs across a fast interconnect wasn't what Kubernetes was built for. Even that gap is closing: SchedMD, the Slurm company, is now part of NVIDIA and ships tooling that runs Slurm on top of Kubernetes.
The parallelism moved
So are we back to monoliths, just hammering GPUs with CUDA? Half right. The heavy compute really did move from CPU clusters to GPUs. But large-scale training is the opposite of monolithic. It's the most distributed computing anyone has ever done: data, tensor, and pipeline parallelism coordinated across thousands of GPUs. The parallelism didn't go away. It moved to the GPU fabric, and the interesting distribution problems moved with it. What shrank to a single box is the small end, where one big machine now covers what used to justify a cluster.
Postgres ate NoSQL
MongoDB's 2012 pitch was that your data is JSON and relational databases can't handle it. Then Postgres added JSONB (binary JSON columns with real indexes), and by Postgres 17 the full SQL/JSON standard. That's roughly 90% of the document flexibility inside the relational database you already run, so "just use Postgres" became the default advice for new projects. People leave MongoDB for a simple reason: it's one more system to operate, and the system they already run does the job now.
I can confirm this one from my own stack. I build on Supabase, which is Postgres, with pgvector for embeddings. Documents, auth, and vector search in one database. I only reach for a document store when a repo or project I'm building on demands it. And now, with AI-assisted coding, it's easier to just take what you need from that project and tailor it to use pgvector instead.
What didn't change
The vocabulary changed more than the ideas. You'll hear about "medallion architecture" with bronze, silver, and gold layers. I smiled when I learned this. It's staging, conformed, and presentation with new paint.
And governance is back. Once every engine can touch the same tables, lineage, catalogs, and one clean version of the truth become the hard problems again. That is master data management thinking. The years I spent on "who owns the golden record, and who is allowed to change it" turn out to be more relevant to the lakehouse than they ever were to the swamp.
The boxes changed. Storage and compute got separated, the specialized systems folded back into the general ones, and a laptop replaced the cluster. The discipline underneath, as far as I can tell, is the one I already knew.