Storage Tier and Query Engine Selection (Onboarding)

Storage Tier and Query Engine Selection (Onboarding)

1. 01

   Map each client data source to its query pattern before naming any platform

   Separate append-only event streams, nightly batch loads, and ad hoc client reporting into distinct buckets. A dashboard refreshing every 30 seconds and a monthly board pack do not belong on the same engine.

2. 02

   Size the working set in terabytes and project 12-month growth from the client's own numbers

   Pull row counts and average row width from the source systems rather than estimating. Storage decisions made on guessed volumes are the most common cause of a mid-engagement migration.

3. 03

   Split hot analytical data from cold archive and price each tier separately

   Column-oriented engines such as ClickHouse handle petabyte-scale aggregation at millisecond latency, while flat-rate object storage tiers carry the archive. Mixing the two budgets hides where the retainer margin actually sits.

4. 04

   Model egress and API request volume against the client's access pattern

   A client pulling daily extracts for their own BI team can generate egress charges that dwarf storage. Flat-rate providers remove that variable; hyperscaler-style billing reintroduces it.

5. 05

   Decide whether the client needs geo-distribution or single-region is sufficient

   Distributed SQL options such as CockroachDB earn their operational overhead when a client has contractual data residency or failover requirements. Absent those, a single-region managed warehouse is cheaper to run and easier to hand to a junior engineer.

6. 06

   Confirm white-label and resale terms if the warehouse is billed through the agency

   Some object storage providers permit full white-label resale, which lets an agency present the platform as its own product. Verify the terms in writing before quoting a client a bundled infrastructure line.

7. 07

   Write the selection decision into the statement of work with the rejected alternatives

   Record why the chosen architecture won and what was ruled out. This protects the agency when a client later asks for a capability the selected platform does not offer.

8. 08

   Run a two-week pilot load with production-shaped data before committing the retainer

   Load a representative slice, run the client's three most important queries, and time them. A pilot that surfaces a latency problem costs two weeks; discovering it in month four costs the account.