Pipeline Cost and Throughput Baseline (Onboarding)

Pipeline Cost and Throughput Baseline (Onboarding)

1. 01

   Inventory every source system the client expects in scope and tag each one by update cadence

   Separate daily-refresh sources from hourly or streaming ones. A 600-connector marketing integration layer such as Adverity behaves very differently at hourly cadence than a nightly batch pull, and the baseline has to reflect which is which before any quote goes out.

2. 02

   Record current row volumes and growth rate per source over the trailing 90 days

   Ask the client for warehouse query logs or export counts. If they cannot produce them, estimate from a two-week sample and label the number as an estimate in the baseline document.

3. 03

   Measure the manual hours the client's team currently spends on extraction and reconciliation each week

   This is the number the retainer conversation rests on. Capture it as a range (for example 12 to 18 hours weekly) rather than a single figure, and name who performs the work.

4. 04

   Price the target architecture against a managed orchestration option and a self-hosted option

   A managed Airflow service such as Astronomer removes infrastructure headcount but adds a per-task fee; a self-managed scheduler inverts that trade. Put both columns in front of the client rather than choosing for them.

5. 05

   Flag any source that requires scraping or anti-bot handling as a separate cost line

   Web extraction carries proxy and rendering overhead that database connectors do not. Services like Zyte or Bright Data bill on request volume, so a source behind bot protection can double the ingestion budget on its own.

6. 06

   Set a throughput ceiling and an alert threshold for each pipeline before build starts

   Define the row-per-minute rate the pipeline must sustain and the point at which the client gets notified. Without a written ceiling, scope creep arrives disguised as a performance complaint.

7. 07

   Get the client to sign off on the baseline document before engineering hours are booked

   The signed baseline becomes the reference point for every later change request. Agencies that skip this step absorb rework into the retainer instead of billing it.