README.md
RAG Document Intelligence
A full-stack Retrieval-Augmented Generation (RAG) application that lets you upload documents and web pages, then ask questions about them using any major LLM provider.
Features
- Multi-source ingestion — upload PDFs, DOCX, Markdown, and plain text files, or paste a URL to scrape and index web pages
- Semantic search — FAISS-powered vector search using
sentence-transformersembeddings - Cross-encoder reranking — improves retrieval precision by reranking top candidates before generation
- Multi-provider LLM support — works with Google Gemini, OpenAI, Anthropic, Mistral, xAI, Qwen, MiniMax, Meta Llama, Groq, and more
- Per-user isolation — documents, indexes, and API keys are scoped to individual accounts
- Chat history — persisted conversation history with source citations and feedback
- Document preview — rendered Markdown with KaTeX math typesetting in the document viewer
- Admin dashboard — usage statistics and vector index inspection for staff users
- Collections — organise documents into named collections
Tech Stack
| Layer | Technology |
|---|---|
| Backend | Django 6, Django REST Framework |
| Auth | JWT (SimpleJWT) with token blacklisting |
| Vector search | FAISS + sentence-transformers |
| Reranking | cross-encoder/ms-marco-MiniLM-L-6-v2 |
| LLM routing | OpenAI-compatible SDK + Google GenAI SDK |
| Web scraping | BeautifulSoup4 (Trafilatura pinned in requirements, migration pending) |
| Database | PostgreSQL (SQLite fallback for dev) |
| Frontend | React 19 + Vite + TypeScript 7 |
| Math rendering | KaTeX |
Getting Started
Prerequisites
- Python 3.12+
- Node 20.19+
- PostgreSQL (or use SQLite for local dev)
Backend
| 1 | # 1. Clone and create virtual environment |
| 2 | git clone https://github.com/Narayan201120/rag_web_app.git |
| 3 | cd rag_web_app/backend |
| 4 | python -m venv venv |
| 5 | venv\Scripts\activate # Windows |
| 6 | # source venv/bin/activate # macOS/Linux |
| 7 | |
| 8 | # 2. Install dependencies |
| 9 | pip install -r requirements.txt |
| 10 | |
| 11 | # 3. Configure environment variables |
| 12 | cp .env.example .env # edit with your values (see below) |
| 13 | |
| 14 | # 4. Run migrations |
| 15 | python manage.py migrate |
| 16 | |
| 17 | # 5. Start the dev server |
| 18 | python manage.py runserver |
Frontend
| 1 | cd frontend |
| 2 | cp .env.example .env # edit if your backend URL differs |
| 3 | npm install |
| 4 | npm run dev |
The app is available at http://localhost:3000.
npm run dev with VITE_MOCK=1 serves the UI from built-in fixtures, so no backend is needed. npm run build type-checks with tsc --noEmit and writes dist/. npm test runs Vitest.
Environment Variables
Create a .env file in the backend/ directory:
| 1 | # Django |
| 2 | DJANGO_SECRET_KEY=your-secret-key |
| 3 | DJANGO_DEBUG=true |
| 4 | DJANGO_ALLOWED_HOSTS=localhost,127.0.0.1 |
| 5 | |
| 6 | # Database (omit to use SQLite) |
| 7 | DATABASE_URL=postgresql://user:password@localhost:5432/rag_db |
| 8 | DJANGO_DB_SSL_REQUIRE=false |
| 9 | |
| 10 | # JWT |
| 11 | JWT_SECRET_KEY=your-jwt-secret |
| 12 | |
| 13 | # CORS (comma-separated) |
| 14 | DJANGO_CORS_ALLOWED_ORIGINS=http://localhost:3000 |
| 15 | |
| 16 | # Optional: encrypt stored provider API keys |
| 17 | FIELD_ENCRYPTION_KEY= |
| 18 | |
| 19 | # Optional: Redis/Celery background tasks |
| 20 | CELERY_BROKER_URL=redis://localhost:6379/0 |
| 21 | CELERY_RESULT_BACKEND=redis://localhost:6379/0 |
For the frontend, create frontend/.env from frontend/.env.example:
| 1 | VITE_API_URL=http://127.0.0.1:8000/api |
| 2 | VITE_GOOGLE_CLIENT_ID= |
| 3 | VITE_MOCK=0 |
Vite inlines VITE_* variables at build time. Social sign-in buttons stay hidden until their client ID is set.
When DJANGO_DEBUG=false, the backend requires production values for
DJANGO_SECRET_KEY, JWT_SECRET_KEY, FIELD_ENCRYPTION_KEY, and
DATABASE_URL. DJANGO_ALLOWED_HOSTS defaults to Render's
RENDER_EXTERNAL_HOSTNAME when deployed on Render; set it explicitly only for
custom domains or non-Render hosting. DJANGO_CORS_ALLOWED_ORIGINS defaults to
Render's backend URL, but should be set to your frontend URL when the frontend
is hosted separately.
Generate FIELD_ENCRYPTION_KEY with:
| 1 | python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())" |
Render
If you deploy from render.yaml, Render creates the Postgres database, sets
DATABASE_URL, and generates DJANGO_SECRET_KEY and JWT_SECRET_KEY. You still
need to provide FIELD_ENCRYPTION_KEY and your frontend origin for
DJANGO_CORS_ALLOWED_ORIGINS when Render prompts for sync: false values.
If you created the Render backend manually as a Docker service, render.yaml
does not populate the service environment. Add these variables in the Render
dashboard: DATABASE_URL, DJANGO_SECRET_KEY, JWT_SECRET_KEY,
FIELD_ENCRYPTION_KEY, DJANGO_DEBUG=false, DJANGO_DB_SSL_REQUIRE=true, and
DJANGO_CORS_ALLOWED_ORIGINS=<your frontend URL>.
API Overview
| Method | Endpoint | Description |
|---|---|---|
| POST | /api/sign-up/ | Register a new user |
| POST | /api/sign-in/ | Obtain JWT access token and refresh cookie |
| POST | /api/token/refresh/ | Refresh access token from the refresh cookie |
| POST | /api/logout/ | Blacklist refresh token and clear refresh cookie |
| POST | /api/auth/social/ | Sign in with a supported social provider |
| GET | /api/account/ | Get current account details |
| DELETE | /api/account/delete/ | Delete the current account |
| POST | /api/forgot-password/ | Request a password reset |
| POST | /api/reset-password/ | Reset password with a valid token |
| GET/POST | /api/settings/api-key/ | Get or save LLM provider + API key |
| POST | /api/settings/api-key/test/ | Test LLM provider connectivity |
| POST | /api/upload/ | Upload a file |
| POST | /api/upload-url/ | Ingest a web page or YouTube URL |
| GET | /api/documents/ | List user documents |
| GET/DELETE | /api/documents/<filename>/ | Preview or delete a document |
| PUT | /api/documents/<filename>/move/ | Move a document to a collection |
| PATCH | /api/documents/<filename>/rename/ | Rename a document |
| GET/POST | /api/collections/ | List or create document collections |
| GET | /api/collections/<collection_id>/ | Get a collection with its documents |
| POST | /api/ingest/ | Trigger indexing |
| POST | /api/ask/ | Ask a single RAG question |
| POST | /api/chat/ | Ask a question in a persisted conversation |
| POST | /api/chat/stream/ | Stream answer tokens over SSE |
| GET | /api/chat/history/ | List conversations |
| GET/PATCH/DELETE | /api/chat/conversations/<conversation_id>/ | Get, rename/pin, or delete a conversation |
| POST | /api/chat/<chat_id>/feedback/ | Submit answer feedback |
| GET | /api/feedback/hard-queries/ | List downvoted queries for review |
| GET | /api/chat/<chat_id>/citations/ | Get citations for a chat answer |
| GET | /api/chat/<chat_id>/export/ | Export a chat answer |
| POST | /api/search/ | Semantic search |
| POST | /api/search/rerank/ | Semantic search with cross-encoder reranking |
| GET | /api/search/suggest/?q=<query> | Search suggestions |
| GET | /api/tasks/<task_id>/ | Get background task status |
| POST | /api/tasks/<task_id>/cancel/ | Cancel a background task |
| GET | /api/status/ | Authenticated system/vector status |
| GET | /api/admin/usage/ | Staff-only API usage stats |
| GET | /api/admin/vectors/ | Staff-only vector index stats |
| POST | /api/evaluate/generate/ | Generate evaluation questions |
| POST | /api/evaluate/run/ | Run RAG evaluation |
| GET | /api/evaluate/results/ | List evaluation results |
| GET | /api/health/ | Health check |
Project Structure
| 1 | rag_web_app/ |
| 2 | ├── backend/ # Django backend |
| 3 | │ ├── api/ # Django app |
| 4 | │ │ ├── views.py # All API endpoints + web scraping + RAG pipeline |
| 5 | │ │ ├── generator.py # LLM provider routing |
| 6 | │ │ ├── retriever.py # FAISS + BM25 hybrid search + reranking |
| 7 | │ │ ├── chunker.py # Semantic chunking |
| 8 | │ │ ├── compressor.py # Context compression |
| 9 | │ │ ├── llm_catalog.py # Provider -> model lists |
| 10 | │ │ ├── tasks.py # Celery background tasks |
| 11 | │ │ ├── throttles.py # Rate limiting |
| 12 | │ │ ├── encryption.py # API key encryption |
| 13 | │ │ ├── middleware.py # Request tracing |
| 14 | │ │ └── models.py # UserProfile, Conversation, ChatMessage, Collection, Task |
| 15 | │ ├── config/ # Settings, URLs, Celery app |
| 16 | │ ├── manage.py |
| 17 | │ ├── Dockerfile |
| 18 | │ └── requirements.txt |
| 19 | ├── frontend/ # React application |
| 20 | │ └── src/ |
| 21 | │ ├── apiClient.js # Axios client + SSE streaming |
| 22 | │ └── components/ # Login, Signup, Chat, Documents, Search, Settings, Admin |
| 23 | ├── docker-compose.yml # Postgres + Redis + backend + celery + frontend |
| 24 | ├── .github/workflows/ci.yml |
| 25 | └── render.yaml # Render deployment config |
Uploaded and scraped documents are stored locally under backend/documents/.
That directory is ignored by Git because it may contain private user data.
Running Tests
| 1 | # Backend (from backend/ directory) |
| 2 | venv\Scripts\python.exe manage.py test |
| 3 | |
| 4 | # Frontend (from frontend/ directory) |
| 5 | npm.cmd run test:ci |
License
MIT