AskDocs — Chat with your files

Ask a question about a document you already have, and get an answer that stays tied to the page it came from. AskDocs is a document chat app for reports, contracts, notes, and spreadsheets — not another library to organize.

🔗 Live at askdocs.ayushdixit.me

---

What Is AskDocs?

AskDocs is a document chat application that turns uploaded files into a searchable conversation. Ask questions in plain English, receive streamed answers, and inspect the document excerpts behind numbered citations.

The project uses retrieval-augmented generation (RAG): it extracts text from files, stores searchable chunks and embeddings in PostgreSQL, retrieves relevant passages for a question, and provides those passages to a Gemini model as answer context.

Contents

• ▸Product overview
• ▸Tech stack
• ▸Architecture
• ▸Repository structure
• ▸Prerequisites
• ▸Local setup
• ▸Using AskDocs
• ▸Environment variables
• ▸Development commands
• ▸Database and background jobs
• ▸API overview
• ▸Testing and code quality
• ▸Production build and deployment
• ▸Troubleshooting
• ▸Current limitations
• ▸Contributing

Product overview

AskDocs is built for finding information inside reports, research, policies, contracts, notes, and spreadsheet exports. A conversation can use multiple attached documents and retain context across follow-up questions.

Features

Supported files

Current limits: 5 files per message/upload batch, 10 MiB per file, and 8,000 characters per message. Extensions, MIME types, and file contents are validated. Scanned PDFs need OCR, which is not implemented. Excel formulas are not recalculated.

Tech stack

Versions below describe the package manifests; pnpm-lock.yaml records the exact dependency graph.

Pinecone is used only for optional reranking. Document vectors live in PostgreSQL with pgvector; a Pinecone index is not required.

Architecture

flowchart TD
    Browser[Browser] --> Web[Next.js web app :3000]
    Web -->|/api/* rewrite| API[Express API :5001]
    Browser -->|Presigned PUT| R2[Private Cloudflare R2 bucket]
    API -->|Presign and read files| R2
    API --> DB[(PostgreSQL + pgvector)]
    API --> Auth[Better Auth + Resend]
    API --> Gemini[Gemini models]
    API -->|Optional reranking| Pinecone[Pinecone inference]
    API --> Workers[pg-boss workers in API process]
    Workers --> DB
    Workers --> R2
    Workers --> Gemini

Upload and indexing

1. ▸The authenticated client requests upload URLs. The API creates document records and returns short-lived presigned URLs.
2. ▸The browser uploads bytes directly to R2, then calls the completion endpoint.
3. ▸The API verifies the uploaded size and queues ingestion with pg-boss.
4. ▸A worker validates file contents, extracts text, and splits it into passages. The chunk target is 1,024 characters with up to 128 characters of overlap.
5. ▸Gemini generates 768-dimensional embeddings. Chunks, embeddings, and source metadata are saved in PostgreSQL, and the document becomes ready.

Document states are uploading → pending → processing → ready, with failed used when processing cannot complete. Progress can report embedding, waiting for capacity, and saving stages.

Question and answer

1. ▸The API checks the session, ownership, and per-user AI limits, then saves the user message.
2. ▸Routing selects general chat or document retrieval using the question and conversation context. Newly attached documents are allowed to finish indexing before answering.
3. ▸Retrieval searches the user's ready documents associated with that conversation. The default hybrid_rerank mode combines vector similarity and PostgreSQL full-text results using reciprocal rank fusion.
4. ▸Optional Pinecone inference reranks candidates. Missing credentials, unavailable budget, timeouts, or provider errors fall back to local BM25 reranking.
5. ▸Up to 8 evidence passages are selected within the evidence budget and sent to the answer model with recent history, the stored summary, and personalization preferences.
6. ▸Tokens stream to the client. The completed assistant message and valid citations are saved, and a summary job is queued when history crosses the configured threshold.

Embedding caches reduce repeated document embedding work. Retrieval cache keys include the user and document versions, and access is rechecked before cached evidence is returned.

Repository structure

.
├── apps/
│   ├── web/                         # Next.js frontend
│   │   ├── app/                     # Landing, auth, and protected chat routes
│   │   ├── components/              # Landing sections, chat UI, app components
│   │   ├── lib/                     # API clients, sessions, stores, streaming
│   │   ├── hooks/                   # App hooks
│   │   ├── tests/                   # Frontend logic tests
│   │   ├── proxy.ts                 # Optimistic route guard
│   │   ├── next.config.js           # /api rewrite and shared UI transpilation
│   │   └── DESIGN.md                # Visual design reference
│   └── api/                         # Express backend
│       ├── src/
│       │   ├── config/              # Environment and middleware configuration
│       │   ├── controllers/         # HTTP handlers
│       │   ├── core/                # Errors, responses, validation schemas
│       │   ├── db/                  # Drizzle client and application schemas
│       │   ├── docs/                # OpenAPI specification
│       │   ├── emails/              # Verification and reset email templates
│       │   ├── lib/                 # Auth, R2 storage, pg-boss setup
│       │   ├── middlewares/         # Auth, limits, validation, logging, errors
│       │   ├── routes/              # Route definitions
│       │   ├── services/            # Chats, documents, AI admission, RAG
│       │   │   └── rag/             # Ingestion, retrieval, streaming, summaries
│       │   ├── scripts/             # Database reset helpers
│       │   ├── app.ts               # Express app factory
│       │   └── index.ts             # Server and worker startup
│       ├── drizzle/                 # SQL migrations and metadata
│       └── env.example              # API environment template
├── packages/
│   ├── ui/                          # @repo/ui: components, hooks, theme CSS
│   ├── eslint-config/               # @repo/eslint-config: shared lint presets
│   ├── prettier-config/             # @repo/prettier-config: shared formatting
│   └── typescript-config/           # @repo/typescript-config: TS presets
├── .husky/                          # Commit and push hooks
├── pnpm-workspace.yaml              # Workspaces and package-age configuration
└── turbo.json                       # Task dependencies and cache outputs

Prerequisites

• ▸Node.js 22.12 or newer. The root manifest still declares >=18.12, but current dependencies require newer Node: pg-boss requires >=22.12.0 and file-type requires >=22.
• ▸pnpm 10.34.5, matching the root packageManager field.
• ▸PostgreSQL with pgvector available, and a role that can apply migrations and initialize pg-boss database objects. Use a managed database such as Neon or a local pgvector-enabled instance.
• ▸A Google Gemini API key with access to the configured chat and embedding models.
• ▸A private Cloudflare R2 bucket and S3 API credentials for uploads and processing.
• ▸A Resend API key and permitted sender address for verification and reset emails.
• ▸Optional: Google OAuth credentials and a Pinecone API key for reranking.

Redis and a separate vector database are not required. Background jobs and vector search both use PostgreSQL.

Local setup

Run commands from the repository root unless another directory is specified.

1. Clone and install

git clone https://github.com/25f3005585/ai-document-rag-app.git
cd ai-document-rag-app

# If the matching pnpm version is not installed:
npm install --global pnpm@10.34.5

pnpm install

Installation runs the root prepare script to install Husky hooks. The workspace enforces a seven-day minimum package release age with minimumReleaseAge: 10080.

2. Configure the API

cp apps/api/env.example apps/api/.env

# Generate a value for BETTER_AUTH_SECRET:
openssl rand -base64 32

Edit apps/api/.env and replace the template values for:

• ▸DATABASE_URL and DATABASE_URL_DIRECT.
• ▸BETTER_AUTH_SECRET and GEMINI_API_KEY.
• ▸RESEND_API_KEY and EMAIL_FROM.
• ▸R2_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY, and R2_BUCKET_NAME.

Keep the local URLs aligned:

PORT=5001
NODE_ENV=development
BETTER_AUTH_URL=http://localhost:3000
WEB_URL=http://localhost:3000
ALLOWED_ORIGINS=http://localhost:3000,http://localhost:3001

For Neon, use the pooled connection for DATABASE_URL and the direct, non-pooler connection to the same database for DATABASE_URL_DIRECT. pg-boss uses the direct connection for polling and maintenance. Without a pooler, both can use the same connection string. If DATABASE_URL_DIRECT is omitted, it falls back to DATABASE_URL.

3. Configure the web app

There is currently no checked-in web .env.example. Create apps/web/.env.local:

cat > apps/web/.env.local <<'EOF'
NEXT_PUBLIC_API_URL=http://localhost:5001
NEXT_PUBLIC_WEB_URL=http://localhost:3000
NEXT_PUBLIC_GOOGLE_AUTH_ENABLED=false
EOF

NEXT_PUBLIC_API_URL is the Express upstream for Next.js rewrites and server-side session lookups. Browser application requests use the web origin's /api/* paths. Auth runs on Express but is reached through Next.js so cookies are set on the web host.

4. Prepare storage and email

Create a private R2 bucket and configure CORS to permit the frontend origin. This local policy matches the storage helper's intended configuration:

[
  {
    "AllowedOrigins": ["http://localhost:3000", "http://localhost:3001"],
    "AllowedMethods": ["PUT", "GET", "HEAD"],
    "AllowedHeaders": ["*", "content-type", "content-length"],
    "ExposeHeaders": ["ETag", "Content-Length"],
    "MaxAgeSeconds": 3600
  }
]

R2 credentials need access to upload, read, inspect, and delete objects in the selected bucket. Bucket verification and CORS helpers exist in the API, but startup does not call them; configure the bucket explicitly.

For Resend, set EMAIL_FROM to a sender your account can use, typically on a verified domain. Replace the template sender. Email/password login requires email verification, so working email delivery is part of signup.

5. Apply database migrations

On a fresh database:

pnpm --filter api db:migrate

The migrations create tables and indexes, enable vector, and install SQL functions for AI request limits and reranking budgets. If the application role cannot create extensions, have the database owner enable pgvector first:

CREATE EXTENSION IF NOT EXISTS vector;

Use migrations for a complete bootstrap. db:push synchronizes the Drizzle schema but does not install custom SQL functions. For a database previously created with db:push, reconcile migration history before applying the migration sequence; use a fresh development database if no data needs to be retained.

6. Start the application

pnpm dev

The API starts pg-boss workers after connecting to PostgreSQL. No separate worker command is needed.

Check the API in another terminal:

curl http://localhost:5001/health/live
curl http://localhost:5001/health
curl http://localhost:5001/health/ready

Liveness confirms the HTTP process responds. Health checks PostgreSQL and reports process metrics; readiness also applies a memory threshold. These endpoints do not validate Gemini, R2, or email delivery.

Optional: Google sign-in

1. ▸Set GOOGLE_CLIENT_ID and GOOGLE_CLIENT_SECRET in apps/api/.env.
2. ▸Register http://localhost:3000/api/auth/callback/google as an authorized OAuth redirect URI.
3. ▸Set NEXT_PUBLIC_GOOGLE_AUTH_ENABLED=true in apps/web/.env.local.
4. ▸Restart both apps.

The backend enables Google only when both credentials exist; the web flag controls the sign-in button.

Using AskDocs

1. ▸Open the web app, create an account, and follow the verification email link. Google sign-in is also available when configured.
2. ▸Open /chats and start a conversation.
3. ▸Attach a supported file and watch upload/indexing progress. A question submitted during indexing can wait for the attachment.
4. ▸Ask a specific question, such as “What are the main findings?” or “Which section describes the renewal period?”
5. ▸Inspect source chips for excerpts and available page information.
6. ▸Ask follow-up questions in the same conversation to reuse its document context.
7. ▸Use account settings to personalize answers, edit your profile, or change the theme.

Environment variables

Keep API secrets in apps/api/.env and web configuration in apps/web/.env.local. NEXT_PUBLIC_ values are public and must not contain secrets.

API: services and authentication

DATABASE_URL, GEMINI_API_KEY, BETTER_AUTH_SECRET, and RESEND_API_KEY are required when the API environment module loads. R2 values are checked when storage is used.

API: AI and retrieval

Set GEMINI_EMBED_MODEL explicitly: the example file uses gemini-embedding-001, while the code fallback is gemini-embedding-2. Ingestion and query embeddings must use the same model and dimensionality. Changing the model requires re-embedding existing documents so vectors remain comparable.

The application's optional Pinecone reranking budget allows up to 500 calls per sliding 31 days and 60 per minute per scope. These are application limits implemented in SQL, not provider plan or pricing guarantees.

API: request and health controls

Web

Restart development servers after changing environment values. Set web public variables before a production build because they are used in the built app.

Development commands

validate does not include tests or the build. It does not auto-fix source; tooling may still write caches or generated types. Root formatting tasks delegate to package scripts, so check a root README edit explicitly with pnpm exec prettier --check README.md.

Run one workspace or task:

pnpm exec turbo run dev --filter=web
pnpm exec turbo run dev --filter=api

pnpm --filter web build
pnpm --filter api build
pnpm --filter api clean

pnpm --filter web test
pnpm --filter api test

Turbo caches build outputs and ESLint caches. Server tasks are persistent and uncached. Build, lint, and type-check tasks respect upstream workspace dependencies.

Database and background jobs

Data model

PostgreSQL stores auth users/sessions/accounts, chats, messages, rolling summaries, document metadata and associations, chunks and vectors, citations, personalization, embedding caches, reranking usage, and AI request reservations. Search uses HNSW vector indexes and GIN full-text indexes. File bytes live in R2, and pg-boss maintains its own queue objects in PostgreSQL.

Database commands

pnpm --filter api db:generate  # Generate SQL after editing Drizzle schemas
pnpm --filter api db:migrate   # Apply checked-in migrations
pnpm --filter api db:push      # Schema sync; excludes custom SQL functions
pnpm --filter api db:studio    # Open the Drizzle database browser

Review generated SQL before applying it. Changes needing functions or other SQL-only behavior must include a custom migration.

Two destructive development reset helpers also exist:

These helpers delete database rows, not R2 objects or pg-boss jobs. Stop the API before a development reset and handle storage/queue cleanup separately if needed. Both refuse to run with NODE_ENV=production unless explicitly forced.

Queues

Workers run inside Express. Keep the API process running while files are indexing or summaries are queued.

API overview

Browser clients use the web origin's /api/* paths. Express registers the following routes. Chat, document, and profile routes require a session and enforce resource ownership where applicable.

Message pages contain up to 15 messages and pagination metadata. Resource IDs and the before cursor are UUIDs.

Request examples

Request upload URLs with file metadata:

{
  "files": [
    {
      "name": "report.pdf",
      "size": 245760,
      "mimeType": "application/pdf"
    }
  ]
}

Send a message with optional attachments:

{
  "content": "What are the main findings in this report?",
  "documentIds": ["11111111-1111-4111-8111-111111111111"]
}

Replace the ID with a document owned by the signed-in user. A message requires text or at least one document; duplicate document IDs are rejected.

Responses and streaming

Normal app endpoints return success, message, data, and metadata containing a timestamp and trace ID. Failures return success: false, a message, an error object with code and trace ID, and statusCode. Better Auth handles its own response formats.

Message submission uses text/event-stream after successful preparation:

Streams send heartbeat comments every 15 seconds and have a five-minute timeout. Clients must handle HTTP failures before streaming and error events after streaming starts.

Swagger UI is available outside production, but the current OpenAPI file documents only health endpoints. Route files and validation schemas describe the complete implemented API.

Testing and code quality

Tests use Node's built-in runner through tsx. API tests live alongside code as *.test.ts; web logic tests live in apps/web/tests. Coverage includes ingestion validation, chunking, retrieval, reranking, citations, summaries, streaming, progress, chat state, and pagination.

pnpm validate
pnpm test
pnpm exec prettier --check README.md
pnpm build

The repo uses strict TypeScript with noUncheckedIndexedAccess, ESLint typed rules, sorted imports, and complexity/size limits. Shared Prettier settings use semicolons, single quotes, trailing commas, two spaces, and a 100-character print width.

No CI workflow is checked in. Run the quality gate, tests, and build locally before submitting changes.

Production build and deployment

pnpm build
pnpm start

Web uses next build / next start; the API compiles into apps/api/dist and runs node dist/index.js. The root start task declares a build dependency, so it can trigger builds before starting servers.

1. ▸Provision PostgreSQL with pgvector, a private R2 bucket, Gemini access, and email delivery.
2. ▸Configure each app's environment in its deployment service. Set NODE_ENV=production for the API and use HTTPS public URLs.
3. ▸Set NEXT_PUBLIC_API_URL to the API address reachable by the Next.js server. Set NEXT_PUBLIC_WEB_URL, WEB_URL, and BETTER_AUTH_URL to the public frontend origin.
4. ▸Apply migrations as a release step before serving the updated app.
5. ▸Build and start each app in its workspace, or use the root scripts on one host.
6. ▸Update API CORS, R2 CORS, and any Google OAuth redirect URI for the public frontend origin.
7. ▸Keep Express running for pg-boss workers. Hosting and proxies must support long-lived SSE requests without response buffering.

Production mode enables secure auth cookies, one-hop proxy trust, security monitoring, and JSON logs. Swagger UI is disabled. Health/readiness cover PostgreSQL and process state rather than all integrations.

Troubleshooting

Current limitations

• ▸Scanned/image-only PDFs, image uploads, legacy .doc / .xls files, and OCR are unsupported.
• ▸Spreadsheet extraction reads cached values; it does not execute formulas or provide a calculation engine.
• ▸Retrieval selects passages within a context budget rather than including every document in every request.
• ▸Upload counts, file sizes, message lengths, output tokens, and provider quotas bound requests.
• ▸API and workers share one process; a separate worker deployment entry point is not provided.
• ▸OpenAPI documentation covers health routes only.
• ▸No Docker/Compose configuration or CI workflow is included.
• ▸No root license file is included. The API package's ISC metadata does not establish a repository-wide license.

Contributing

1. ▸Read AGENTS.md and relevant app instructions before changing code.
2. ▸For web UI, read apps/web/DESIGN.md and apps/web/AGENTS.md. The visual reference is a warm, document-focused workspace.
3. ▸Put reusable primitives in packages/ui and app features in apps/web. Import shared components through @repo/ui exports.
4. ▸For API work, follow apps/api/AGENTS.md. Separate routes, controllers, validation, and business logic; API code uses ESM with .js import extensions.
5. ▸Include migrations for schema changes and custom SQL behavior.
6. ▸Run pnpm validate, relevant tests, and pnpm build, and update docs when setup or behavior changes.

Add shared UI components from the root with:

pnpm dlx shadcn@latest add <component-name> -c apps/web

The CLI routes primitives to packages/ui and app blocks to apps/web. Keep both components.json files aligned for style, base color, and icon library.