Generative AI app architecture: build on a language model
Most generative AI apps are an ordinary app with a language model behind an API. Here is how the pieces fit, which service plays each part on AWS, Azure and Google Cloud, and what to settle before you build.
Quick answer
A generative AI app is an ordinary app with a language model behind an API: users call your API, your code builds the prompt, a managed model returns the answer, and safety checks screen the input and output. Optional extras add your own documents through retrieval, or tools the model may call through agents.
| Best for | Chat, summaries, drafting, code help and other text features inside your product |
|---|---|
| Less suited when | A rule or a lookup already gives the exact answer, or the output must always be identical |
| Start with | Call a managed model API from a small backend, with a test set of real prompts |
| Main services on AWS | Amazon S3, Amazon OpenSearch Service, Amazon Bedrock AgentCore, Amazon API Gateway, Amazon Bedrock, Amazon Bedrock Guardrails |
| Main services on Azure | Azure Blob Storage, Azure AI Search, Microsoft Foundry Agent Service, Azure API Management, Azure OpenAI in Microsoft Foundry Models, Azure AI Content Safety |
| Main services on Google Cloud | Cloud Storage, Vector Search, Gemini Enterprise Agent Platform, API Gateway, Model Armor |
The idea in plain words
You do not train a model to build a generative AI app. You call one that a cloud provider already hosts, send it a prompt (your instructions plus the user's request), and use the text, code or image that comes back. The rest of the app is familiar: sign-in, an API, storage and monitoring.
What is new is what surrounds the model. A prompt has to be built carefully, answers need safety checks, usage needs limits because you pay for every word, and the model's output must never be trusted blindly.
Two optional additions are common. Give the model your own content when answers must come from your documents (see the RAG pattern), or give it tools to call when it must do things, not just talk (an AI agent).
The architecture
Boxes show the job each part does. Pick a cloud to see the service for each job.
Step by step
- A user makes a requestA person uses your web, mobile or chat app, which sends the request to your API.
- The API front door checks and limitsAn API gateway checks who is calling, applies rate limits and keeps your model credentials out of the app.
- Your app builds the promptYour code combines the user's request with your instructions (the system prompt), the conversation so far and any data the answer needs.
- Optionally add your own contentFor answers from your documents, look up the matching passages in a vector index and add them to the prompt. The RAG pattern covers this in detail.
- Optionally let the model use toolsFor tasks, give the model a defined set of tools or APIs it may call, run by an agent service, with limits and approvals for anything that changes data.
- Call the language modelSend the prompt to a model through a managed API and stream the reply back as it is written.
- Check and returnRun safety checks on what goes into the model and what comes out, log only what you need, and send the answer back to the app.
Services side by side
The main service for each job on each cloud, with a second option where one exists. Names checked on 23 October 2026.
| Job | AWS | Azure | Google Cloud |
|---|---|---|---|
| Object storage (files, images, backups) |
|
|
|
| Vector database (embeddings and similarity search) |
|
|
|
| AI agents (tools, reasoning, multi-step tasks) |
|
|
|
| Publish and secure APIs |
|
|
|
| Generative AI models (LLMs) through an API |
|
|
|
| Content moderation and AI safety guardrails |
|
|
|
| Chatbot or Q&A over your own documents (RAG)Optional |
|
|
|
The model service that writes the answers usually also provides the embedding models. The index can be a dedicated vector database, or a search or database service with vector support.
Common variations
- Answers must come from your documents. Add retrieval over your own content. See the RAG architecture pattern.
- The app must take actions, not just answer. Add an agent with a short list of tools, limits on what each can do, and human approval for anything that changes data.
- Images or files as input. Many models accept images or documents. Check which model supports what, the size limits, and the cost of large inputs.
- Large batches, such as summarising thousands of files overnight. Use queued, asynchronous calls or the provider's batch option instead of live requests, to control cost and rate limits.
Managed model API or host a model yourself?
The managed model services above are the simplest route: you call a model and pay for what you use. Hosting a model yourself needs GPU servers, shown below, and a team to run them.
| AWS | Azure | Google Cloud | |
|---|---|---|---|
| GPU servers for AI and graphics |
|
|
|
Managed model API
Good
- No servers to run or patch
- Pay for what you use, with several models behind one service
- Built-in security, logging and safety features
Watch out
- Limits on rate, quota and context size
- Which models are offered varies by region and changes over time
- Prompts go to the provider, so read its data-use terms
Host a model yourself
Good
- Full control of the model, data path and updates
- Open models you can adapt
- Cost can be steadier at high, constant volume
Watch out
- GPU capacity is costly and sometimes scarce
- You run scaling, patching and monitoring
- Needs people who know model serving
Keep in mind
- Keys and access. Keep model credentials on the server, never inside a web page or mobile app. Use roles and short-lived credentials rather than shared keys.
- Untrusted text. Anything the model reads, whether a user typed it or your code fetched it, may try to steer the model (prompt injection). Limit the tools the model can use, and check its output before acting on it.
- Cost control. Cost grows with the amount of text sent and received. Set limits per user, cache repeated answers, choose the smallest model that does the job, and watch usage every day at first.
- Quality and testing. Keep a set of real prompts with the answers you expect, and run it again after every change to the prompt, the model or its settings. Models are updated and retired, so pin versions where offered and plan upgrades.
- Safety and misuse. Filter inputs and outputs, rate-limit users, log abuse, and decide in advance what the app must never do or say.
- Privacy. Do not send personal or confidential data unless the provider's terms and your own rules allow it. Check where prompts are processed and how long they are kept. Privacy laws such as the GDPR, the CCPA and India's DPDP Act can apply to prompts and answers that contain personal data.
- Availability. Model services have rate limits and can fail. Add timeouts, retries with back-off, and a clear fallback message or a second model.
Words used on this page
Services in this pattern that have changed
Cloud providers rename and retire services often. These affect the parts above. Checked on 23 October 2026.
- AWS Closed to new customers Amazon Kendra stopped accepting new customers on 30 July 2026. AWS recommends Amazon Bedrock Managed Knowledge Base. Details
- AWS Closed to new customers Amazon Q Business is closed to new customers. Amazon Quick is its successor. Details
- AWS Closed to new customers Amazon Bedrock Agents Classic closed to new customers on 30 July 2026. AWS recommends Amazon Bedrock AgentCore. Details
- Azure Renamed Azure AI Foundry is now Microsoft Foundry, and Azure AI services are now called Foundry Tools. Resource types and APIs are unchanged. Details
- Google Cloud Renamed Vertex AI is now Gemini Enterprise Agent Platform (22 April 2026). Vertex AI Search is now Agent Search, and Vertex AI Agent Engine is now Agent Runtime. Google says the Vertex AI documentation is no longer updated. Details
Sources and further reading
Check details such as features, limits, prices and regions in the providers' own documentation.
- AWS documentation (opens in a new tab)
- Microsoft Azure documentation (opens in a new tab)
- Google Cloud documentation (opens in a new tab)
Announcements about the changed services above
- Amazon Kendra: the provider's announcement (opens in a new tab)
- Amazon Q Business: the provider's announcement (opens in a new tab)
- Amazon Bedrock Agents Classic: the provider's announcement (opens in a new tab)
- Azure AI Foundry: the provider's announcement (opens in a new tab)
- Vertex AI: the provider's announcement (opens in a new tab)
Frequently asked questions
What is a generative AI application?
It is an application that uses a generative model, usually a large language model, to produce text, code, images or answers. Most are built by calling a hosted model through an API and adding your own logic, data and safety checks around it.
Which cloud has the best models?
All three offer several models from different makers through one managed service, and the choice, regions and prices change often. Compare on your own task with your own test prompts rather than on headlines.
Do I need RAG or an agent?
Only if the app needs them. Use retrieval when answers must come from your documents, and an agent when the app must use tools or take actions. A plain prompt to a model is enough for many tasks.
How do I keep the cost under control?
Cost depends on how much text goes in and out. Limit usage per user, trim prompts, cache repeats, use the smallest model that meets your quality bar, and set budget alerts on the account.
Can I use an open-source or my own model?
Yes. Managed services offer some open models, and you can host any model on GPU servers if you are ready to run them. Start with a managed API and move only if you have a clear reason.
Want this reviewed for your case?
CloudBrum architects can adapt this pattern to your documents, users, rules and budget, and estimate the cost. Nothing is sent until you choose.
Building something else?
Describe it in plain words and get matching services on all three clouds.
Service suggester → Cloud consulting →