We need data engineering benchmarks for LLMs(structuredlabs.substack.com)
structuredlabs.substack.com
We need data engineering benchmarks for LLMs
https://structuredlabs.substack.com/p/why-we-need-data-engineering-benchmarks
4 comments
AI can do some amazing things inferring from past behavior, but it certainly is a fair distance away from anything approaching understanding what the end goal is or what all the constraints are or might be in the future. Sure, people can set up some boilerplate problems and get custom boilerplate solutions but I really see AI as another level of tooling/abstraction that will only empower engineers to work less on the easy stuff and spend more time at a higher level of concern.
In other words, I think the near term gain is that we will actually ACTUALLY not have to worry about centering a div anymore.
In other words, I think the near term gain is that we will actually ACTUALLY not have to worry about centering a div anymore.
have you really pasted the contents of your own substack post as the first comment on your own submission of your own substack post?
They sure did. Bold, not gonna lie.
Why SWE-Bench Falls Short SWE-bench evaluates LLMs on real-world software engineering tasks by using GitHub issue–pull request pairs from popular repositories.
Success is measured by the quality, reliability, and scalability of data workflows, not just correctness of code.
Data engineers solve complex systems-level problems involving constant data motion, quality maintenance, and continuous evolution. Treating these disciplines as equivalent is doing data engineering a disservice.
DE deals with raw, messy, and large-scale data that must be cleaned, transformed, and made accessible.
i.e. handling data quality, schema evolution, and compliance, not just code correctness.
DE focuses on automating and orchestrating multi-step workflows, which requires understanding task dependencies, scheduling, and retries.
Edge-cases! DE must handle schema drift, missing values, malformed records, and outliers, which are rarely part of SWE workflows.
DE is less about writing application logic and more about making data usable, accessible, and reliable. Data engineers don’t just write code. They manage workflows that operate across multiple systems, handle changing requirements, and scale with data growth. A DE benchmark needs to reflect these realities.
One might argue that text-to-SQL benchmarks are a step toward evaluating LLMs for data engineering tasks. While useful, text-to-SQL falls far short of the needs of DE benchmarks for several reasons:
Text-to-SQL only handles querying structured data. Data engineering is so much more: it involves transforming, orchestrating, and making sense of chaotic, mixed-format data.
Lack of Pipeline Context. DE isn’t more than single queries. it’s about creating end-to-end workflows that deliver business value.
Handling real-world problems like schema drift or malformed records
Data engineers work with Spark, Airflow, dbt—not just databases. Generating a SQL query is child’s play compared to orchestrating a complex Spark job across petabytes of data.
Reliability, scalability, and optimization are make-or-break factors for DE. These are entirely missed by simple SQL benchmarks.
In short, text-to-SQL is just one piece of the DE puzzle. Evaluating DE copilots requires a broader, pipeline-focused benchmark.
A DE-bench would simulate real-world DE workflows, evaluating LLMs on their ability to solve practical, pipeline-oriented problems. Here’s how it might look:
Dataset Sources:
Use public datasets (e.g., NYC Taxi data, Kaggle datasets).
Generate synthetic data to simulate edge cases (e.g., missing values, schema drift).
Data Ingestion: Load raw data from APIs or files into a database or data lake.
Data Transformation: Normalize data formats, remove duplicates, and compute aggregates.
Pipeline Orchestration: Create an Airflow DAG for a multi-step ETL pipeline.
Schema Management: Migrate data between schemas safely.
A DE-bench would provide a structured, objective framework for assessing LLMs on real-world DE tasks, ensuring that these tools are reliable, efficient, and robust. It’s time we hold DE copilots to the same high standards as their SWE counterparts.