Data platform architecture: from raw data to trusted dashboards
A data platform turns the data your business already produces into numbers people can trust. Here is how the pieces fit, which service plays each part on AWS, Azure and Google Cloud, and what to decide before you build.
Quick answer
A data platform architecture copies data from business systems into a data lake, cleans it with ETL or ELT jobs, and loads modelled tables into a data warehouse. Dashboards and analysts query the warehouse, while a catalog records what each table means and who may see it. All three clouds offer a managed service for each layer.
| Best for | Reporting and analysis across several systems, such as sales, finance and operations data |
|---|---|
| Less suited when | One system's built-in reports already answer the questions, or the data fits comfortably in a spreadsheet |
| Start with | One or two sources, a lake, a small warehouse and five dashboards that answer questions people already ask |
| Main services on AWS | AWS Step Functions, Amazon Athena, AWS Glue Data Catalog, AWS Glue, Amazon S3 + AWS Lake Formation, Amazon Redshift, Amazon Quick Sight |
| Main services on Azure | Azure Logic Apps, Microsoft Fabric SQL analytics endpoint, Microsoft Purview, Azure Data Factory, Microsoft Fabric OneLake, Microsoft Fabric, Power BI |
| Main services on Google Cloud | Workflows, BigQuery external tables, Knowledge Catalog, Cloud Data Fusion, BigQuery lakehouse, BigQuery, Looker |
The idea in plain words
Most companies already hold the data they need, spread across billing, customer records, apps, spreadsheets and logs. Each system reports on itself, and nobody can say with confidence what last quarter's revenue was across all of them. A data platform fixes that by copying the data to one place, tidying it and giving everyone the same definitions.
The usual shape has layers. A data lake keeps raw and cleaned files in cheap object storage. Jobs reshape that data into business tables in a data warehouse that answers SQL quickly. Dashboards and analysts sit on top, and a catalog records what each table means, who owns it and who may see it. A lakehouse blends the first two layers by adding table features to files in the lake.
Build it around questions, not tools. List the ten decisions the business wants to make with data, then build only the pipelines those decisions need. A platform that starts with every source and no clear users tends to become expensive and unused.
The architecture
Boxes show the job each part does. Pick a cloud to see the service for each job.
Step by step
- Pick the questionsList the decisions people want data for, such as which customers leave or which products earn most. These choose the sources, the tables and the dashboards, so the platform stays small and useful.
- Connect the sourcesCopy data from business systems on a schedule. For databases, read only the changes since the last run instead of copying everything again, so loads stay fast and the source system is not slowed down.
- Land it unchangedWrite each copy to the raw area of the data lake exactly as it arrived. Keeping the original lets you rebuild any later table when a business rule changes or a job turns out to be wrong.
- Clean and standardiseFix data types and dates, remove duplicates, check required fields and hide personal details that analysts do not need. Save the result in a columnar file format in a cleaned area of the lake.
- Model it for the businessBuild tables that match how people think, such as customers, orders and daily sales, and load them into the warehouse. Give each measure one definition, for example what counts as revenue.
- Describe and protectRegister every table in the catalog with an owner, a plain description and a sensitivity label. Set who may see which tables, rows and columns before anyone is invited.
- Serve people and watch the platformGive most people dashboards, analysts a SQL workspace and other apps an extract or an interface. Alert on late or failed jobs, and track what each query and dashboard costs.
Services side by side
The main service for each job on each cloud, with a second option where one exists. Names checked on 23 October 2026.
| Job | AWS | Azure | Google Cloud |
|---|---|---|---|
| Workflows and orchestration |
|
|
|
| Query files in storage with SQL (serverless) |
|
|
|
| Data catalog and governance |
|
|
|
| ETL and data pipelines |
|
|
|
| Data lake and lakehouse |
|
|
|
| Data warehouse |
|
|
|
| Dashboards and business intelligence |
|
|
|
The lake and the warehouse overlap. Several services query lake files directly and several warehouses read lake files, so a small platform can start with one of the two and add the other when the need is clear.
Common variations
- Data arrives all day, not overnight. Add a streaming path for the sources that need fresh numbers and keep nightly loads for the rest. See the real-time data pipeline pattern.
- Teams want to build their own models. Give data scientists read access to the cleaned area of the lake and a machine learning workspace. See the machine learning project pattern.
- Many departments, each owning its data. Let each domain publish tested tables under shared standards, and use the catalog so others can find them. This approach is often called a data mesh.
- Data must be shared with partners. Use the sharing features of your warehouse or lake to give read-only access to chosen tables, instead of sending files by email.
Serverless services or managed Spark clusters?
Most of the platform can run on serverless services that you do not size or patch. When transformations are heavy or need custom code, many teams add managed Spark clusters, shown in the table.
| AWS | Azure | Google Cloud | |
|---|---|---|---|
| Spark, Hadoop and big-data processing |
|
|
|
Serverless data services
Good
- Nothing to size, patch or switch off
- Costs follow use, which suits uneven workloads
- Fastest route to a first dashboard
Watch out
- Per-query pricing can surprise heavy users
- Less control over how jobs run
- Features differ between clouds
Managed Spark clusters
Good
- Full control of custom code and libraries
- Handles heavy transformations and data preparation for machine learning
- Open-source engine that runs on every cloud
Watch out
- You choose sizes, scaling and shutdown
- Idle clusters keep costing money
- Needs people who know Spark
Keep in mind
- Questions before tools. A platform with no clear users becomes a costly archive. Start with a handful of dashboards that replace real manual reports, show them to users early and add sources only when someone asks.
- One definition for each number. If sales and finance each calculate revenue their own way, a new platform only makes the argument faster. Agree the definitions, write them in the catalog and calculate each one once.
- Data quality is a daily job. Check that loads arrive on time, row counts look normal and key fields are filled in. Stop a bad load before it reaches a dashboard, and tell the owner of the data when something breaks.
- Personal data and the law. Know which tables hold personal data and why you keep it. Laws such as the GDPR in Europe and the UK, the CCPA in California and the DPDP Act in India set what you may collect, share and keep, and some give people the right to have their data deleted.
- Cost. Warehouses charge by data scanned or by compute time, and storage by volume. Partition big tables by date, avoid scanning everything on each dashboard refresh, move old data to cheaper storage and set budget alerts.
- Access and sharing. Give people the least they need, with row and column rules for sensitive fields. Review who can see what every quarter, and remove access when people change roles or leave.
- Open formats keep options open. Files in open formats such as Parquet, with an open table format such as Apache Iceberg or Delta Lake on top, can be read by many engines. That makes a later move, or a second tool, much easier.
Words used on this page
Services in this pattern that have changed
Cloud providers rename and retire services often. These affect the parts above. Checked on 23 October 2026.
- AWS Renamed Amazon QuickSight is now Quick Sight inside Amazon Quick Suite (October 2025). Existing dashboards and APIs keep working. Details
- Azure Direction changed Azure Synapse Analytics is not being discontinued, but Microsoft steers new analytics projects to Microsoft Fabric. Details
- Google Cloud Renamed Dataproc is now Managed Service for Apache Spark, and Cloud Composer is now Managed Service for Apache Airflow. Details
- Google Cloud Renamed Looker Studio is now called Data Studio. Details
- Google Cloud Renamed Dataplex is now Knowledge Catalog. Details
Sources and further reading
Check details such as features, limits, prices and regions in the providers' own documentation.
- AWS documentation (opens in a new tab)
- Microsoft Azure documentation (opens in a new tab)
- Google Cloud documentation (opens in a new tab)
Announcements about the changed services above
- Amazon QuickSight: the provider's announcement (opens in a new tab)
- Azure Synapse Analytics: the provider's announcement (opens in a new tab)
- Dataproc and Cloud Composer: the provider's announcement (opens in a new tab)
- Looker Studio: the provider's announcement (opens in a new tab)
- Dataplex: the provider's announcement (opens in a new tab)
Frequently asked questions
What is the difference between a data lake and a data warehouse?
A data lake stores raw and cleaned data as files in cheap storage, in any format. A data warehouse stores modelled tables built for fast SQL and reporting. Most platforms use both: the lake keeps everything, and the warehouse serves the tables people query every day.
What is a lakehouse?
A lakehouse adds warehouse features, such as tables, transactions and fast SQL, to files kept in a data lake, using open table formats. One copy of the data can then serve both reporting and data science, which reduces how much you copy between systems.
Should I use ETL or ELT?
ETL transforms data before loading it into the target, while ELT loads it first and transforms it inside the lake or warehouse. ELT is common in the cloud, because warehouses scale well and raw data stays available. Use ETL when you must remove or mask data before it lands.
Do I need a data platform, or is a spreadsheet or database enough?
If one system's reports answer your questions, or the data fits in a spreadsheet, you probably do not need a platform. It pays off when data sits in several systems, reports disagree, files are too big for spreadsheets, or many people need the same numbers.
What drives the cost of a data platform?
Four things: storage volume, compute for loading and transforming data, compute for queries and dashboards, and the people who run it. Scan-based query pricing can climb quickly with careless queries, so partition tables, limit refresh rates and set budget alerts from the start.
Want this reviewed for your case?
CloudBrum architects can adapt this pattern to your documents, users, rules and budget, and estimate the cost. Nothing is sent until you choose.
Building something else?
Describe it in plain words and get matching services on all three clouds.
Service suggester → Cloud consulting →