Pipeline Source Intake and Connector Vetting (Onboarding)

Pipeline Source Intake and Connector Vetting (Onboarding)

1. 01

   Inventory every source system the client will hand over before quoting the engagement

   List each SaaS app, database, ad platform, and flat-file drop by name and owner. A retainer scoped on three sources that turns out to be eleven is a margin problem discovered too late.

2. 02

   Score each source on access method, refresh cadence, and schema stability

   REST API with documented rate limits is a different build than a screen-scraped portal. Zyte and WebScrapingAPI exist precisely because some sources have no API at all, and that changes both cost and fragility.

3. 03

   Confirm whether the client will grant read-only credentials or expects the agency to hold write access

   Write access to a CRM or ad account raises liability. Classify every workflow by autonomy level and add a human checkpoint to anything touching client-facing records.

4. 04

   Test connector coverage against the actual source list, not the vendor's headline count

   A platform advertising 300+ connectors (Peliqan) or 600+ marketing sources (Adverity) still needs a gap check against this client's specific stack. Run a two-source pilot before committing the full build.

5. 05

   Decide the open-source versus managed split and write it into the statement of work

   Airflow, dbt, and Spark give the client portability; managed orchestration (Astronomer, Databricks) cuts build time. Name which layers are proprietary so a future migration is a priced decision, not a surprise.

6. 06

   Document the data contract: fields, types, nullability, and expected volume per run

   This becomes the acceptance test. Without it, every downstream quality dispute is a matter of opinion.

7. 07

   Set the credential rotation and offboarding path on day one

   Record who holds keys, where secrets live, and how access is revoked when the retainer ends. Clients ask this during procurement more often than agencies expect.