Workshop

Designing and Deciding Like a Staff Data Engineer

Dan Sullivan

Principal data engineer & author

Practice the staff-level skills that AI can't do for you

AI now handles much of the work that used to show someone was a strong senior data engineer. It writes the pipeline, drafts the dbt models, and debugs the Spark job. What gets you to staff is how you handle the problems AI can't settle for you: ambiguous requests, competing designs, uncertain costs, and stakeholders who need a decision.

In this 4-hour live workshop, you'll work through Cairnfield Logistics, a fictional third-party logistics company whose VP asks for "real-time inventory visibility across all our warehouses." The case unfolds in six stages, one for each staff dimension, and each stage changes what a good answer looks like. After every stage, you apply the same idea to a real problem from your own work.

You'll leave with the raw material for a staff-level decision memo. Finish it within a week and get written feedback from me, plus a 30-day check-in on your 90-day plan.

What you’ll learn

Expand the context. Improve the decisions. Help others grow. These three shifts are what move a strong senior engineer to staff.

  • Map the boundaries and buffers of a pipeline and find where rate, schema, permission and freshness mismatches hide.

  • Describe how a design fails, how far the damage spreads, and how it recovers.

  • Set baselines for freshness, volume, schema and distribution so real anomalies stand out from normal variation.

  • Reconcile record counts across boundaries, and pair every alert with a runbook entry.

  • Trace a requested solution back to the business decision it's meant to improve.

  • Separate data, decision and action latency so you pay for freshness only where it matters.

  • Compare alternatives out loud and eliminate options with stated constraints.

  • Work through decisions with other engineers so they build judgment, not just follow yours.

  • Translate one technical fact for engineers, product managers and executives.

  • Write egoless status updates: state, impact, risk, response and the decision needed.

  • Estimate the full workflow, including approvals, dependencies and unknowns, not just the build.

  • Weigh the cost to build and run against what the decision is worth to the business.

Workshop agenda

  • Architectural judgment: find risk at the boundaries

    Map where components meet and where buffers fill. Describe how a design scales, fails and recovers, and where one client's data could leak to another.

  • Observability and operations: know what normal looks like

    Healthy infrastructure can still carry unhealthy data. Set baselines for freshness, volume and distribution, reconcile lost records, and pair each alert with a response.

  • Translating needs: diagnose before you prescribe

    Trace a request for "real-time data" back to the business decision it supports. Separate data, decision and action latency, and write the requirement that decision needs.

  • Leading engineers: make your reasoning visible

    Settle a design disagreement by comparing alternatives and eliminating options with stated constraints, so the engineers involved build judgment, not just follow a decision.

  • Stakeholder communication: don't surprise the boss

    Translate one technical fact for an engineer, a product manager and an executive. Write an egoless status: state, impact, risk, response and the decision you need.

  • Cost judgment: estimate the workflow, not just the work

    Rebuild estimates to include approvals, dependencies and unknowns. Weigh the cost to build and run against what the decision is worth, then make a recommendation.

  • Your self-assessment, 90-day plan and decision memo

    Rate yourself on all six dimensions, choose one to build over 90 days, and turn your worksheets into a staff-level decision memo that gets written feedback.

Learn directly from Dan

Dan Sullivan

Dan Sullivan

Contact

Dan Sullivan is a data engineer and architect, O’Reilly author, and educator.

New Relic
Hydrolix
Virginia Tech
See all products from Dan

Who this workshop is for

  • Senior data engineers preparing for a staff promotion who want practice with the skills that don't show up in a code review.

  • Engineers already doing staff-level work without the title who want a clear way to explain and defend their decisions.

  • Tech leads who keep getting vague platform requests and want a repeatable way to turn them into sound decisions.

What's included

Dan Sullivan

Live sessions

Learn directly from Dan Sullivan in a real-time, interactive format.

Lifetime access

Go back to course content and recordings whenever you need to.

Community of peers

Stay accountable and share insights with like-minded professionals.

Certificate of completion

Share your new skills with your employer or on LinkedIn.

Maven Guarantee

Your purchase is backed by the Maven Guarantee.

Frequently asked questions

Maven for Teams

Reimbursement

Get your company to pay

Everything L&D needs: email template, receipts, and certificate of completion.

Get reimbursed

Private cohort

Run a cohort for your org

A dedicated cohort with a custom schedule and curriculum, tailored to your team.

Book a private cohort