RAG architecture: answer questions from your own documents
Retrieval-augmented generation (RAG) lets a language model answer from your own documents, with sources, instead of from memory alone. Here is how the pieces fit, which service plays each part on AWS, Azure and Google Cloud, and what to decide first.
Quick answer
A RAG (retrieval-augmented generation) architecture stores your documents in object storage and turns them into searchable passages in a vector index. For each question it retrieves the best-matching passages and sends them to a language model with safety checks, so answers come from your own content and can cite sources.
| Best for | Answering questions from many internal documents, with sources |
|---|---|
| Less suited when | People ask only a fixed set of questions, where search or an FAQ page is enough |
| Start with | A managed RAG service on a small set of real documents, then measure answer quality |
| Main services on AWS | Amazon S3, Amazon Bedrock, Amazon OpenSearch Service, Amazon Bedrock Guardrails |
| Main services on Azure | Azure Blob Storage, Azure OpenAI in Microsoft Foundry Models, Azure AI Search, Azure AI Content Safety |
| Main services on Google Cloud | Cloud Storage, Gemini Enterprise Agent Platform, Vector Search, Model Armor |
The idea in plain words
A language model knows what it was trained on. It does not know your policies, contracts, manuals or last month's tickets. RAG fixes that without retraining the model: you keep your documents in a searchable index, find the few passages that match each question, and give them to the model together with the question.
That has three benefits. Answers come from your content, they can show their sources, and keeping answers current means updating a document, not training a model.
Use it when you have many documents and want grounded answers. If people only ever ask a fixed set of questions, a good search box or an FAQ page may be enough.
The architecture
Boxes show the job each part does. Pick a cloud to see the service for each job.
Step by step
- Store your documentsKeep the source files (PDFs, web pages, wiki exports, tickets) in object storage, so there is one place to read from and to rebuild from.
- Split and embedCut each document into passages of a few paragraphs, and use an embedding model to turn each passage into a list of numbers that captures its meaning.
- IndexSave the passages and their numbers in a vector index (a vector database, or a search service with vector support), with the document name, page and who may read it stored alongside.
- Receive the questionA user asks in your app or through an API. Your code checks who the user is.
- RetrieveTurn the question into numbers the same way and ask the index for the closest passages, limited to the documents this user may see.
- GenerateSend the question and those passages to a language model, with an instruction to answer only from them and to say so when the answer is not there.
- Check and returnRun safety checks on the question and the answer, then return the answer together with its sources.
Services side by side
The main service for each job on each cloud, with a second option where one exists. Names checked on 23 October 2026.
| Job | AWS | Azure | Google Cloud |
|---|---|---|---|
| Object storage (files, images, backups) |
|
|
|
| Generative AI models (LLMs) through an API |
|
|
|
| Vector database (embeddings and similarity search) |
|
|
|
| Content moderation and AI safety guardrails |
|
|
|
| Full-text searchOptional |
|
|
|
The model service that writes the answers usually also provides the embedding models. The index can be a dedicated vector database, or a search or database service with vector support.
Common variations
- Scanned files, tables or images. Add a document-parsing or OCR step before splitting, so the text and tables are read correctly.
- Many users with different rights. Store who may read each passage in the index, and filter on it at query time.
- Questions with exact names, codes or part numbers. Combine vector search with keyword search (hybrid search), which a full-text search service provides.
- The assistant must take actions, not just answer. Look at AI agents, which let a model call your tools and APIs. Add approvals for anything that changes data.
Managed service or build it yourself?
Each cloud has a managed service that does the preparing, indexing and retrieving steps for you: you point it at your documents and call it. Or you assemble the parts above yourself.
| AWS | Azure | Google Cloud | |
|---|---|---|---|
| Chatbot or Q&A over your own documents (RAG) |
|
|
|
Managed RAG service
Good
- Fastest way to a working prototype
- Less to build, run and patch
- Sensible defaults for splitting and retrieval
Watch out
- Fewer controls over splitting, ranking and filtering
- Limits on file types, size and access rules
- Cost and behaviour follow the service's pricing and changes
Assemble the parts
Good
- Full control of splitting, ranking, filters and models
- Per-document access rules where you need them
- Swap one part without rebuilding the rest
Watch out
- More to build, test and operate
- You own quality and security of every step
- Needs people who know search and model behaviour
Keep in mind
- Access control. Retrieval must respect who may see which document. Keep the access list with each passage and filter on it when searching. Do not rely on the model to hide things.
- Untrusted text. Text inside documents is untrusted input: a document can contain instructions aimed at your model (prompt injection). Give the model as few tools as possible, and run safety checks on retrieved text as well as on questions.
- Quality comes from retrieval. Most bad answers come from retrieving the wrong passages. Build a small test set of real questions with the expected source, and re-run it whenever you change the splitting, the model or the index.
- Sources and honesty. Show the sources with every answer, and tell the model to say it does not know when the passages do not contain the answer.
- Freshness and deletion. Plan how new and changed documents reach the index and how deleted ones leave it. Stale or deleted content in an index is a common surprise.
- Cost. Cost has three parts: embedding your documents, storing and searching the index, and model usage on every question (more retrieved text costs more). Estimate with each provider's pricing calculator and measure on a pilot.
- Data location and privacy. Check where documents, embeddings and prompts are stored and processed, and whether the models you want run in a Region you are allowed to use. If the documents hold personal data, laws such as the GDPR, the CCPA or India's DPDP Act may apply, so ask your compliance team before sending them to any model.
Words used on this page
Services in this pattern that have changed
Cloud providers rename and retire services often. These affect the parts above. Checked on 23 October 2026.
- AWS Closed to new customers Amazon Kendra stopped accepting new customers on 30 July 2026. AWS recommends Amazon Bedrock Managed Knowledge Base. Details
- AWS Closed to new customers Amazon Q Business is closed to new customers. Amazon Quick is its successor. Details
- Azure Renamed Azure AI Foundry is now Microsoft Foundry, and Azure AI services are now called Foundry Tools. Resource types and APIs are unchanged. Details
- Google Cloud Renamed Vertex AI is now Gemini Enterprise Agent Platform (22 April 2026). Vertex AI Search is now Agent Search, and Vertex AI Agent Engine is now Agent Runtime. Google says the Vertex AI documentation is no longer updated. Details
Sources and further reading
Check details such as features, limits, prices and regions in the providers' own documentation.
- AWS documentation (opens in a new tab)
- Microsoft Azure documentation (opens in a new tab)
- Google Cloud documentation (opens in a new tab)
Announcements about the changed services above
Frequently asked questions
What is RAG in simple words?
RAG stands for retrieval-augmented generation. The system first looks up the most relevant passages from your own documents, then gives them to a language model to write the answer. The model answers from your content instead of guessing.
Should I use RAG or fine-tune a model?
Use RAG when the knowledge changes or answers must show sources. Fine-tuning changes how a model behaves or writes, and is a poor way to store facts. Many systems use RAG alone.
Do I need a vector database?
Not always. A managed RAG service hides it, some databases and search services you may already run support vector search, and very small collections can use simpler search. A dedicated vector index matters when collections are large or you need fast similarity search with filters.
Are my documents kept private?
Your documents stay in your cloud account, but each question and the passages retrieved for it are sent to the model service. Check each provider's data-use and retention terms, choose the region carefully, and do not send data you must not share.
How much will it cost?
It depends on how many documents you have, how often they change, how many questions people ask and which model you use. Build a small pilot, measure real usage, and price it with each provider's calculator.
Want this reviewed for your case?
CloudBrum architects can adapt this pattern to your documents, users, rules and budget, and estimate the cost. Nothing is sent until you choose.
Building something else?
Describe it in plain words and get matching services on all three clouds.
Service suggester → Cloud consulting →