ai-pr-reviewer

A self-hosted GitHub App that reviews your pull requests with your own LLM — like CodeRabbit or Copilot code review, but open source (MIT) and running against any OpenAI-compatible provider: OpenAI, Ollama, vLLM, LM Studio, Groq, OpenRouter, and anything else that speaks the chat-completions API.

• 🔔 Triggered automatically on every PR (opened / new pushes / ready for review)
• 💬 Posts one review per push: inline comments on the exact lines + a markdown summary
• 🔌 Bring your own model — one base URL + key + model name
• 🏠 Single small Node service, no database, deploys anywhere (Docker or plain Node)
• ⚙️ Per-repo config file (.aireview.yml) for ignores, extra instructions, and severity filters
• 🛡️ Non-blocking: reviews are advisory (COMMENT), never REQUEST_CHANGES

How it works

GitHub ── pull_request webhook ──▶ POST /webhook (signature-verified, 202 fast-ack)
                                      │
                                      ▼
                          fetch PR files + .aireview.yml
                                      │
                                      ▼
                    filter (ignores, caps) ─▶ build prompt (diff hunks)
                                      │
                                      ▼
                       your LLM (OpenAI-compatible endpoint)
                                      │
                                      ▼
              parse JSON findings ─▶ clamp lines to real diff hunks
                                      │
                                      ▼
              POST PR review: summary + inline comments

Hallucinated line numbers are clamped to lines that actually exist in the diff, so GitHub never rejects a comment. Duplicate webhook deliveries are deduped per head SHA. If the LLM returns unparseable output, it gets one corrective retry and then falls back to posting the raw response.

Quick start

1. Create the GitHub App

On GitHub: Settings → Developer settings → GitHub Apps → New GitHub App (or the equivalent org-level page).

Field	Value
GitHub App name	anything, e.g. ai-pr-reviewer
Webhook URL	your service's public HTTPS URL, e.g. https://reviewer.example.com/webhook
Webhook secret	a random string (→ WEBHOOK_SECRET)
Permissions	Pull requests: Read & write, Contents: Read-only, Checks: Read & write (for status checks)
Subscribe to events	Pull request, Issue comment (/review command)
Where can this app be installed	Only this account / organization (for an internal app)

After creating it: note the App ID (→ APP_ID), and under Private keys generate a .pem (→ PRIVATE_KEY). Install the app on the repos you want reviewed.

2. Configure and run

cp .env.example .env   # fill in APP_ID, PRIVATE_KEY, WEBHOOK_SECRET, LLM_*
docker compose up -d   # or: npm ci && npm run build && npm start
curl http://localhost:3000/healthz

Open a non-draft PR in a repo where the app is installed — the review lands within a minute or two.

Local development without a public URL

Use smee.io (or ngrok) to forward webhooks to your laptop:

npx smee-client --target http://localhost:3000/webhook --url https://smee.io/XXXX
npm run dev

Point the app's Webhook URL at the smee.io URL while developing. You can also test the engine without any webhook:

npm run review:pr -- owner/repo#123

(prints the review to stdout instead of posting it — handy for prompt iteration.)

Configuration

Service environment (.env)

Variable	Required	Default	Notes
APP_ID	✅		GitHub App ID
PRIVATE_KEY	✅		App private key; literal \n escapes are fine
WEBHOOK_SECRET	✅		Must match the app's webhook secret
LLM_BASE_URL	✅		e.g. https://api.openai.com/v1, http://localhost:11434/v1
LLM_MODEL	✅		e.g. gpt-4o-mini, qwen2.5-coder:7b
LLM_API_KEY	➖		Omit for keyless local servers (Ollama, vLLM)
GITHUB_API_URL	➖		GitHub Enterprise override
PORT	➖	3000
LOG_LEVEL	➖	info	pino level
LLM_TIMEOUT_MS / LLM_MAX_TOKENS / LLM_JSON_MODE	➖	120000 / 4096 / on
MAX_FILES / MAX_DIFF_CHARS	➖	30 / 120000	Review-size caps
CHECKS_ENABLED	➖	on	Set false to disable check-runs (needs Checks: Read & write)

Per-repo config (.aireview.yml at repo root, optional)

# Extra glob patterns to skip, merged with the built-in list
# (lockfiles, dist/, node_modules/, minified files, ...)
ignore:
  - "src/generated/**"

# Override the service's MAX_FILES for this repo
max_files: 15

# Extra context injected into the prompt
instructions: "We use Fastify and Vitest. Flag missing await and missing tests."

# Only post findings at these severities (default: all)
severities: [critical, warning, suggestion]

# Check-runs: set false to skip status checks for this repo
checks: true

# Fail the check (red X) when these severities appear; default [] = never fail
fail_on: [critical]

Severities: 🔴 critical (bug/security/data loss), 🟠 warning (likely bug/risky), 🔵 suggestion (meaningful improvement), ⚪ nit (polish).

Development

npm ci
npm test        # 102 unit/integration tests, all external calls mocked
npm run lint
npm run build   # emits dist/

Layout: src/github/ (webhooks + API client), src/review/ (engine, prompt, findings clamping, filters, report rendering), src/llm/ (fetch-based chat client), src/server.ts (node:http). Design doc: docs/superpowers/specs/.

Manual re-review

Comment /review on any PR to force a fresh review (bypasses the duplicate-SHA skip, works on drafts too). Setup: GitHub App → Permissions (Pull requests: Read & write, Contents: Read-only) → Subscribe to Issue comment events.

Roadmap

• /review comment command to re-request a review
• Check-run status (optional gating via fail_on)
• Queue mode (BullMQ) for high-volume orgs
• Packaged CLI + GitHub Action wrappers around the same engine
• GitLab support

License

MIT © Thrishank Chintham