Data Engineering Projects for 2026: 6 Portfolio Builds With Stack, Dataset and Real Costs

You finish the tutorial, push the repo, and it sits there. The one hiring manager who opens it finds a notebook called Untitled1.ipynb, a hard-coded path from your Downloads folder, and no way to tell whether it ever ran twice in a row. That is the gap between doing a project and having one, and it has little to do with which tools you picked.

What Makes Data Engineering Projects Worth Building in 2026

Every serious pipeline, whether it moves 40 rows or 40 billion, has the same six stages. A portfolio project earns attention when a reviewer sees all six in one repo, in order, with the boring parts done.

Take an illustrative case: Ravi, a support analyst in Pune with eight free hours a week and working SQL. His instinct is five projects, so the CV looks full. But reviewers spend four to ten minutes on a repo, and in that window depth shows where breadth does not. Build one project touching all six boxes, then two that each add one capability.

That last row surprises people. A dashboard is worth less than a schema.yml with six tests in it: the dashboard proves taste, the tests prove you have been burned before. To make the visual layer count, tie it to a credential employers recognise, which is what 360DT's Power BI and PL-300 data analyst course practises live each weekend.

6 Data Engineering Project Ideas, With Stack and Dataset

Also read: dbt Tutorial for Beginners 2026 if you have never run dbt build. Project 1 assumes you have.

Project 1, Walked End to End in About 40 Minutes

The whole first project, from empty folder to tested table. Any public CSV with a date and a category column works.

Step 1: land the raw file without touching it

Raw data is written once and never modified. Land a dated copy and read everything downstream from it, so a bug in your transform costs a re-run, not the data.

python -m venv .venv && source .venv/bin/activate
pip install duckdb requests dbt-duckdb

import pathlib, requests, duckdb

RAW = pathlib.Path("data/raw")
RAW.mkdir(parents=True, exist_ok=True)

stamp = "2026-09-23"
(RAW / f"rainfall_{stamp}.csv").write_bytes(
    requests.get(SOURCE_URL, timeout=30).content)

con = duckdb.connect("warehouse.duckdb")
con.execute("""
    CREATE OR REPLACE TABLE raw_rainfall AS
    SELECT *, filename AS source_file
    FROM read_csv_auto('data/raw/rainfall_*.csv', filename = true)
""")
print(con.execute("SELECT count(*) FROM raw_rainfall").fetchone())

Two details do the real work. CREATE OR REPLACE TABLE over a glob of every landed file makes the load idempotent by construction: run it ten times, get the same table. And filename = true keeps each row's provenance, so a wrong number traces back to the file it came in. DuckDB's stable release is 1.5.2 from April 2026, with a v2.0 alpha announced on 2 September 2026 for October, so pin your version.

Step 2: model it, then test it harder than you want to

-- models/marts/monthly_rainfall.sql
{{ config(materialized='table') }}

select
    subdivision,
    date_trunc('month', measured_on) as month,
    round(sum(rainfall_mm), 1)       as rainfall_mm,
    count(*)                         as readings
from {{ ref('stg_rainfall') }}
where rainfall_mm >= 0
group by 1, 2

# models/marts/schema.yml
models:
  - name: monthly_rainfall
    tests:
      - unique:
          column_name: "subdivision || '-' || month"
    columns:
      - name: subdivision
        tests: [not_null]
      - name: rainfall_mm
        tests: [not_null]

Run dbt build and the summary should read like PASS=4 WARN=0 ERROR=0 SKIP=0 TOTAL=4. Now the step most people skip: break it deliberately. Edit one staging row to duplicate a month, re-run, watch the unique test turn the run red, and put that screenshot in your README. A reviewer who sees a test genuinely catch something trusts the whole repo.

Step 3: put it on a schedule, so it is a pipeline and not a script

name: nightly-elt
on:
  schedule:
    - cron: "30 20 * * *"   # 02:00 IST
jobs:
  run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python ingest.py && dbt build

Actions cron runs on UTC, the commonest reason an Indian learner's nightly job fires at lunchtime. 20:30 UTC is 02:00 IST next day. Leave it running two weeks before you apply, then link the run history from your README. Twenty green runs beats any CV bullet.

Projects 2 to 6, and What Will Bite You in Each

Project 2, incremental loading. Store the maximum timestamp you loaded, then request only rows after it. The bite: if the source emits events late, a strict > comparison drops them forever. Use an hour of lookback and deduplicate on a natural key.

Project 3, slowly changing dimensions. Snapshot a price list daily and keep validity ranges. The bite: people build it, never query it historically, and never notice their valid_to boundaries overlap by a day. Write the query proving no product has two active rows on a date.

Project 4, the lakehouse. The only build with a real bill, worth doing if your target listings say Fabric or DP-700. Delta and medallion patterns are the exam territory 360DT's live Microsoft Fabric data engineering program works through in its lab weekends, DP-900 included.

Project 5, micro-batch instead of streaming. The bite is the interesting part: out-of-order events, duplicate deliveries and a backlog after you shut the laptop. Those are the hard parts of streaming; Kafka only adds broker operations on top.

Project 6, quality and freshness. Record row count, maximum source timestamp and duration per run, then alert when the count drifts from the trailing median. This is where data engineering meets production ML operations, covered by the MLOps engineer course built around AI-300; containerising the jobs is core DevOps practice from the AWS track, with the Microsoft equivalent in the Azure AZ-305 and AZ-400 program.

What usually goes wrong here

Three warnings a colleague would give you over a desk, none about tool choice. You will spend week two fighting the source API's pagination and decide you are bad at this. You are not; ingestion is the most tedious part of the job, which is why people who can do it get paid. You will then want to rebuild everything after learning a better pattern in week six. Do not; ship the next project instead.

Also read: Docker Tutorial for Beginners 2026, the fastest way to make these reproducible on a reviewer's machine.

What These Data Engineering Projects Actually Cost to Run

GitHub's billing documentation states that standard hosted runners are free and unmetered on public repositories, so project 1's nightly job costs nothing if the repo is public; private repos get 2,000 minutes a month on the Free plan, with extra Linux two-core minutes at $0.006 after GitHub cut runner rates on 1 January 2026. Fabric is the outlier. Pricing summaries published in 2026 put entry-level F2 capacity near $263 a month pay as you go, about $156 on a one-year reservation, with OneLake storage around $0.023 per GB per month and Power BI Pro near $14 per user per month below F64.

So pause capacity when you are not using it: leave an F2 running a month and you have spent more than an exam fee on idle compute. Build 1, 2, 3, 5 and 6 free on your laptop, and start Fabric only once a DP-700 attempt is booked. The full certifications overview maps exams to stacks if you are still deciding.

What Interviewers Ask About Your Data Engineering Projects

They are predictable, because the interviewer is checking whether you built the thing or followed a video.

"What happens if this job runs twice?" The usual opener. Answer with the mechanism, not reassurance: name the replace or merge semantics and the key.

"Where does this break at a hundred times the volume?" Name a specific limit, such as DuckDB exhausting memory on one machine, then what you would move to.

"Which test caught a real bug?" This is why you screenshot the red run. If no test has ever failed, your tests are decorative.

"Why not Kafka, Spark or Snowflake?" Say the trade-off out loud: the volume did not justify the operational cost, and you preferred something you could run free every night over something you could run once.

"How would you serve this to an application or an LLM?" Increasingly common, since retrieval layers now sit on warehouse marts. Exposing a clean semantic layer is enough of an answer; the deeper retrieval work belongs to the AI engineer course covering RAG and AI agents.

Also read: SQL Query Optimization in 2026, because that second question almost always becomes a query plan discussion.

Related guides

• Data Engineer Roadmap 2026 the wider path these projects slot into.
• Data Engineer Salary in India 2026 what these roles actually advertise.
• What Is Apache Iceberg in 2026? read before attempting the lakehouse build.
• Microsoft Fabric vs Databricks in 2026 how to pick the platform for project 4.
• Data Engineer Jobs in Hyderabad 2026 which stacks employers name in listings.

Frequently asked questions

What I Would Do in Your Position

Build project 1 this weekend and let it run on a schedule for a fortnight before starting anything else. It is the cheapest test of whether you enjoy this work, because whatever annoys you in a ten-hour project will annoy you for ten years. Then go 1, 2, 6: twelve weeks at eight hours a week.

Let your target listings decide project 4. If they name Fabric and DP-700, book the exam and run capacity for those weeks only, so certification and project finish together. A free webinar costs an hour and tells you whether a live cohort suits how you learn, and a demo class shows the lab format first. Otherwise the Microsoft Fabric Data Engineer course runs the DP-700 syllabus live across eight weekends, with someone reviewing your pipeline.

About this guide. 360 Digital Transformation is an Anthropic Partner and a Microsoft Authorized Training Partner. Other certification bodies, vendors and employers named here are not affiliated with us. Tools and versions change quickly; commands and figures cited were checked on 23 September 2026.