Designing and Deciding Like a Staff Data Engineer

Dan Sullivan

Principal data engineer & author

Practice the staff-level skills that AI can't do for you

AI now handles much of the work that used to show someone was a strong senior data engineer. It writes the pipeline, drafts the dbt models, and debugs the Spark job. What gets you to staff is how you handle the problems AI can't settle for you: ambiguous requests, competing designs, uncertain costs, and stakeholders who need a decision.

Over two live 2-hour sessions, a few days apart, you'll work through Cairnfield Logistics, a fictional third-party logistics company whose VP asks for "real-time inventory visibility across all our warehouses." The case unfolds in six stages, one for each staff dimension, and each stage changes what a good answer looks like. After every stage, you apply the same idea to a real problem from your own work.

Between the sessions, you draft the first half of a staff-level decision memo on your problem. You finish it within a week of the second session and get written feedback from me, plus a 30-day check-in on your 90-day plan.

What you’ll learn

Expand the context. Improve the decisions. Help others grow. These three shifts are what move a strong senior engineer to staff.

Map the boundaries and buffers of a pipeline and find where rate, schema, permission and freshness mismatches hide.
Describe how a design fails, how far the damage spreads, and how it recovers.

Set baselines for freshness, volume, schema and distribution so real anomalies stand out from normal variation.
Reconcile record counts across boundaries, and pair every alert with a runbook entry.

Trace a requested solution back to the business decision it's meant to improve.
Separate data, decision and action latency so you pay for freshness only where it matters.

Compare alternatives out loud and eliminate options with stated constraints.
Work through decisions with other engineers so they build judgment, not just follow yours.

Translate one technical fact for engineers, product managers and executives.
Write egoless status updates: state, impact, risk, response and the decision needed.

Estimate the full workflow, including approvals, dependencies and unknowns, not just the build.
Weigh the cost to build and run against what the decision is worth to the business.

Learn directly from Dan

Dan Sullivan

Contact

Dan Sullivan is a data engineer and architect, O’Reilly author, and educator.

Hydrolix, New Relic, Virginia Bioinformatics Institute

See all products from Dan

Who this course is for

Senior data engineers preparing for a staff promotion who want practice with the skills that don't show up in a code review.
Engineers already doing staff-level work without the title who want a clear way to explain and defend their decisions.
Tech leads who keep getting vague platform requests and want a repeatable way to turn them into sound decisions.

What's included

Live sessions

Learn directly from Dan Sullivan in a real-time, interactive format.

Lifetime access

Go back to course content and recordings whenever you need to.

Community of peers

Stay accountable and share insights with like-minded professionals.

Certificate of completion

Share your new skills with your employer or on LinkedIn.

Maven Guarantee

Your purchase is backed by the Maven Guarantee.

Course syllabus

2 live sessions • 8 lessons

Week 1

Oct 27—Oct 29

Day 1: Architecture, Observability and Translating Needs

3 items

Oct

Day 1: Architecture, Observability and Translating Needs

Tue 10/273:00 PM—5:00 PM (UTC)

Between sessions: Draft the first half of your memo

2 items

Day 2: Leading Engineers, Stakeholder Communication and Cost Judgment

4 items

Oct

Day 2: Leading Engineers, Stakeholder Communication and Cost Judgment

Thu 10/293:00 PM—5:00 PM (UTC)

Schedule

Live sessions

4 hrs

Tue, Oct 27
3:00 PM—5:00 PM (UTC)
Thu, Oct 29
3:00 PM—5:00 PM (UTC)

Pre-work and between-session memo draft

1-2 hrs

About 20 minutes before Day 1 to choose a problem from your own work and describe it in one paragraph. Between Day 1 and Day 2, 60 to 90 minutes to draft the first half of your decision memo from your Day 1 worksheets.

Finish your decision memo

2-3 hrs

In the week after Day 2, finish your 2-page staff-level decision memo using your Day 2 worksheets, then submit it for written feedback.