Real-time status monitoring dashboard for SaaS services used across an enterprise IT environment (10 in the committed example registry). Polls vendor status pages every 60 seconds, detects changes, generates impact statements using a service dependency graph, posts Slack alerts, and displays a unified dark-themed operations dashboard.
- v1 (demo-ready) — SHIPPED. All original spec delivered: polling, normalization, change detection, Slack alerting, React UI, dependency graph, timeline, SLA tracking, incident clustering, auto reports.
- v2 (production-ready) — SHIPPED. Phases 0–6 of the production roadmap complete: bearer-token auth, vendor resilience (stamina + purgatory), alert quality (flap suppression, dedup, tier routing, dependency correlation, maintenance windows, flapping-badge UI), observability (structlog, Prometheus
/metrics, Sentry, Healthchecks.io dead-man's switch), data lifecycle (production pragmas, retention, Litestream streaming + dailyVACUUM INTOsnapshot), UX productionization (severity-sorted grid, distinct poller-broken state, a11y + keyboard nav, Executive/Engineer view toggle, PWA,rechartsSLA trend), and platform polish (CI, pre-commit, service supervision, reverse-proxy posture, OS-backed secret storage). 378 tests passing. - v2 Phase 2B + Phase 7 — in tree, gated off. Statuspage inbound webhook receiver (
WEBHOOKS_ENABLED), Slack ack flow (SLACK_ACK_ENABLED), postmortem drafts (POSTMORTEMS_ENABLED), scheduled multi-burn-rate alerting (SLO_BURN_RATE_ENABLED; SLO view/API always available), and Slack/itstatusslash command (SLACK_SLASH_ENABLED) all shipped with tests but default off. Webhook and Slack flags require signed callback reachability; postmortems need a writablePOSTMORTEMS_DIR, and SLO burn-rate alerting runs locally. - v2 Phase 7 remainder — optional. LLM-layer impact statements; log-aggregation / ITSM / synthetic-monitoring integrations. Not on a fixed schedule; add as demand emerges.
Active roadmap: PRODUCTION-ROADMAP.md — exit-criteria detail for every phase. Historical spec: IMPLEMENTATION-ROADMAP.md — archived; v1 is complete.
[Vendor Status Pages]
|-- Statuspage.io JSON API (8 example services)
|-- Custom JSON adapters (available; unused by the example registry)
|-- Manual updates via POST /api/admin/status (2 example services)
| (async poll every 60s)
[Poll Orchestrator]
|
[Status Normalizer] --> 5-state enum: operational|degraded|partial_outage|major_outage|unknown
|
[Change Detector] --> diff against DB, write status_events
|
[Impact Statement Engine] --> dependency graph + templates
[Slack Alerter] --> Block Kit message to the ops-alert channel
[SQLite Writer] --> update services, insert events
|
[FastAPI REST API] --> /api/services, /api/timeline, /api/summary
|
[React Dashboard] <-- auto-refresh 30s
# 1. Clone and enter project
git clone <repo-url> && cd ITServiceHealth
# 2. Set up Python environment
python3.13 -m venv .venv && source .venv/bin/activate
pip install -r backend/requirements.txt
# 3. Build frontend
cd frontend && npm ci && npm run build && cd ..
# 4. (Optional) Seed demo data for a populated timeline
cd backend && PYTHONPATH=.. python -m scripts.seed_demo_data && cd ..
# 5. Run (serves dashboard + API on port 8000)
cd backend && python run.pyOpen http://localhost:8000 in your browser.
For local development, open http://localhost:8000 after starting the backend.
For a private deployment, serve the read dashboard behind your organization's
normal access controls and keep admin writes protected by bearer-token auth. The
public repo intentionally describes the deployment shape, not a real host,
machine, or network boundary.
Services are organized into ten categories. The committed example registry
(backend/config/services.yaml) ships a generic, runnable set that monitors
public developer-tool status pages, so the dashboard works immediately after
clone:
| Category | Example services |
|---|---|
| Identity & Access | Identity provider (SSO) |
| Engineering | GitHub, npm, PyPI, Sentry |
| Productivity | Dropbox |
| Collaboration | Discord |
| Network & VPN | Cloudflare |
| Support | Ticketing / ITSM |
| Other | Datadog |
To monitor your own organization's services, copy the example to a gitignored
backend/config/services.local.yaml (the loader prefers it when present) and
list your real registry there — see that file's header for the schema and the
full category list (identity, productivity, collaboration, engineering, HR,
finance, sales, marketing, networking, support).
For services without automated polling (e.g. an identity provider, an HR system, or any service with no public status API), update status via curl. Admin endpoints require a bearer token (set ADMIN_API_TOKEN in your env).
export TOKEN="<demo-admin-token>"
# Set a service to degraded
curl -X POST http://localhost:8000/api/admin/status \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"service_id": "ticketing", "new_status": "degraded", "detail": "Slow login page", "reason": "Reported by user in the help channel"}'
# Set to major outage
curl -X POST http://localhost:8000/api/admin/status \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"service_id": "identity-provider", "new_status": "major_outage", "detail": "SSO completely unavailable", "reason": "Confirmed with vendor"}'
# Resolve (set back to operational)
curl -X POST http://localhost:8000/api/admin/status \
-H "Authorization: Bearer $TOKEN" \
-H 'Content-Type: application/json' \
-d '{"service_id": "identity-provider", "new_status": "operational", "reason": "Vendor posted recovery"}'Valid statuses: operational, degraded, partial_outage, major_outage, unknown. The reason field is required for audit trail.
| Variable | Default | Description |
|---|---|---|
SLACK_WEBHOOK_URL |
(none) | Slack incoming webhook URL for ops-alert channel notifications |
DATABASE_PATH |
data.db |
SQLite database file path |
POLL_INTERVAL_SECONDS |
60 |
How often to poll vendor status pages (1–3600) |
HOST |
127.0.0.1 |
Server bind address; override only for a controlled private deployment |
PORT |
8000 |
Server port |
LOG_LEVEL |
INFO |
Logging level |
ADMIN_API_TOKEN |
(none) | Bearer token required for /api/admin/* endpoints. If unset, admin endpoints refuse all requests. |
CORS_ORIGINS |
http://localhost:5173,http://127.0.0.1:5173 |
Comma-separated list of allowed CORS origins |
SEED_DEMO_DATA |
false |
Dev-only: auto-populate the DB with synthetic data on boot |
POLLER_HEALTH_SLACK_WEBHOOK_URL |
(none) | Separate webhook for poller-health alerts. Falls back to SLACK_WEBHOOK_URL when unset. |
ALERT_CONFIRM_THRESHOLD_POLLS |
3 |
Consecutive polls required before firing a worsening alert (flap suppression) |
ALERT_RECOVERY_THRESHOLD_POLLS |
2 |
Consecutive successes required before firing a recovery alert |
ALERT_MIN_STATE_DURATION_SECONDS |
600 |
Minimum dwell time (seconds) for worsening transitions |
ALERT_DEDUP_WINDOW_SECONDS |
86400 |
Dedup window for repeat alerts on the same dedup key |
DEPENDENCY_CORRELATION_THRESHOLD |
3 |
Min affected dependents before emitting one aggregated upstream alert |
BREAKER_THRESHOLD |
3 |
Consecutive failures before the per-host circuit breaker opens |
BREAKER_TTL_SECONDS |
300 |
How long an open breaker stays open before half-opening |
POLLER_FAILURE_THRESHOLD |
3 |
Consecutive failures before a service's poller_health flips to broken |
LOG_JSON |
true |
JSON structured logging vs pretty console |
LOG_FILE |
(none) | Optional path for Python-side file logging (uses WatchedFileHandler). Default: stderr |
SENTRY_DSN |
(none) | Enable Sentry error tracking when set |
SENTRY_ENVIRONMENT |
production |
Environment tag reported to Sentry |
SENTRY_TRACES_SAMPLE_RATE |
0.0 |
0.0–1.0 sample rate for Sentry performance traces |
HEALTHCHECK_PING_URL |
(none) | Healthchecks.io (or similar) URL pinged by the heartbeat job |
HEARTBEAT_INTERVAL_SECONDS |
30 |
How often the heartbeat job marks itself alive |
HEARTBEAT_STALE_AFTER_SECONDS |
120 |
/healthz returns 503 past this threshold |
RETENTION_DAYS_STATUS_EVENTS |
90 |
Auto-purge status_events rows older than this (0 = disable) |
RETENTION_DAYS_ALERT_SENT_LOG |
90 |
Auto-purge alert_sent_log rows older than this (0 = disable) |
RETENTION_INTERVAL_HOURS |
168 |
How often the retention job runs |
WAL_CHECKPOINT_INTERVAL_HOURS |
24 |
How often the truncating WAL checkpoint runs |
BACKUP_DIR |
backups |
Directory for the daily VACUUM INTO snapshot |
BACKUP_TIME_HOUR |
2 |
UTC hour for the daily snapshot (independent of Litestream) |
BACKUP_RETENTION_DAYS |
7 |
How many daily snapshots to keep |
WEBHOOKS_ENABLED |
false |
Enable inbound Statuspage subscriber webhooks. Requires public reachability and STATUSPAGE_WEBHOOK_SECRET. |
STATUSPAGE_WEBHOOK_SECRET |
(none) | HMAC-SHA256 shared secret configured in Statuspage → Subscribers → Webhook settings. Required when WEBHOOKS_ENABLED=true. |
SLACK_ACK_ENABLED |
false |
Enable the Slack ack-button flow. Requires public reachability and SLACK_SIGNING_SECRET. |
SLACK_SIGNING_SECRET |
(none) | Signing secret from your Slack app's "Basic Information → App Credentials" page. Required when SLACK_ACK_ENABLED=true. |
SLACK_SLASH_ENABLED |
false |
Enable the /itstatus slash-command endpoint. Requires public reachability and SLACK_SIGNING_SECRET. |
POSTMORTEMS_ENABLED |
false |
Write Google-SRE-style Markdown postmortem drafts on service recovery. |
POSTMORTEMS_DIR |
docs/postmortems |
Directory where postmortem drafts are written (created if absent). |
SLO_BURN_RATE_ENABLED |
false |
Enable the multi-burn-rate SLO alerting scheduler job. |
SLO_TARGET_PERCENT |
99.9 |
SLO uptime target used for error-budget calculations (90.0–99.99). |
SLO_BURN_RATE_CHECK_INTERVAL_SECONDS |
300 |
How often the burn-rate cycle runs (1–3600). |
SLO_BURN_RATE_FAST_THRESHOLD |
14.4 |
Fast-burn multiplier — triggers page-worthy alert (e.g. 14.4× SLO error rate). |
SLO_BURN_RATE_SLOW_THRESHOLD |
6.0 |
Slow-burn multiplier — triggers warning-level alert. |
SLO_BURN_RATE_TICKET_THRESHOLD |
1.0 |
Ticket-severity burn multiplier — low-urgency notification only. |
Copy .env.example to backend/.env and configure (settings load .env from the working directory used to run the backend):
cp .env.example backend/.env
# Edit backend/.env with your valuesRun frontend and backend separately with hot reload:
# Terminal 1: Backend (auto-reload on Python changes)
cd backend && python run.py --dev
# Terminal 2: Frontend (Vite dev server with HMR)
cd frontend && npm run devFrontend dev server at localhost:5173 proxies /api/* to localhost:8000.
Use a fresh checkout without a copied .env, real webhook credentials, or an
existing database. python run.py starts pollers and other configured jobs;
seeding, retention, backup/restore, and live alert callbacks are not verification
fixtures. The unit tests use temporary/in-memory SQLite and mocked vendor input.
From backend, match the Python 3.12 CI environment:
uv venv --python 3.12
uv pip install -r requirements.txt
uv pip install ruff mypy
uv run pytest tests/test_normalizer.pyBroader checks from that same directory are uv run pytest, uv run ruff check .,
uv run ruff format --check ., and uv run mypy app. Dependencies come from
requirements.txt; the tool commands/configuration are in
backend/pyproject.toml and CI.
CI currently treats mypy failures as non-blocking; report local typing failures
rather than interpreting a green CI run as a complete typing pass.
From frontend, use Node 20.19+ (20.x), 22.13+ (22.x), or 24+ (Vite 8 and ESLint 10), npm ci, npm run lint, and
npm run build, using its committed package-lock.json. CI currently tolerates
frontend ESLint failures. The frontend has no test runner, formatter, or separate
typecheck script configured; do not claim those checks from Vite's build.
For changed dashboard, timeline, or report behavior, serve the frontend on a
separate local port and intercept /api/ responses with synthetic fixtures, or
use an explicitly isolated demo backend with vendor/alert boundaries stubbed.
Check changed views, empty/error states, severity ordering, and keyboard use.
Do not point a browser verification session at an existing deployment or enable
Slack/webhook/poller integrations merely to verify docs. Browser checks are
conditional on user-facing changes; this fixture test lane starts no service.
The production path is intentionally self-hosted and private-network oriented:
- Run the FastAPI process under an OS service manager.
- Put a reverse proxy in front for TLS and request headers.
- Store tokens and webhook secrets in the host secret manager, not in git.
- Keep read access behind the organization access controls.
- Require bearer-token auth for admin writes and signature verification for webhook and Slack callbacks.
- Monitor
/api/health,/healthz,/metrics, and the heartbeat job.
Exact host paths, service-manager commands, firewall posture, and log locations belong in a private runbook, not in the public README.
SQLite is the primary store; Litestream streams WAL frames to an external replica so the dashboard can recover from host failure.
Use the checked-in config template as a starting point, keep the real replica destination out of git, validate the config before enabling the sidecar, and monitor snapshots as part of routine operations.
Litestream RPO is ~1 second — after the initial snapshot, every WAL frame ships as it is written.
Restore procedure: stop writers, restore the latest snapshot plus WAL frames into the configured database location, restart the service, and let startup migrations run. Keep the exact command sequence in a private runbook because it depends on the host service manager, paths, and replica destination.
The dashboard auto-prunes old rows to keep the DB from growing without bound:
| Table | Default retention | Env var |
|---|---|---|
status_events |
90 days | RETENTION_DAYS_STATUS_EVENTS |
alert_sent_log |
90 days | RETENTION_DAYS_ALERT_SENT_LOG |
The retention job runs every RETENTION_INTERVAL_HOURS (default 168 = weekly) and a truncating WAL checkpoint runs every WAL_CHECKPOINT_INTERVAL_HOURS (default 24) to truncate the WAL; deleted database pages remain available for reuse. Set any retention window to 0 to keep data forever.
| Endpoint | Method | Description |
|---|---|---|
/api/health |
GET | Backend health check |
/api/services |
GET | All services with status counts |
/api/services/{id} |
GET | Service detail with dependencies |
/api/timeline |
GET | Recent status change events |
/api/summary |
GET | Overall health + active incidents |
/api/maintenance |
GET | Upcoming scheduled maintenances |
/api/services/uptime |
GET | Per-service per-day worst status over the past 7 days |
/api/services/sla |
GET | Per-service uptime % for 24h, 7d, and 30d windows |
/api/services/sla/history |
GET | Daily uptime % per service (1–90 days, default 30d) |
/api/services/graph |
GET | Service dependency graph (nodes + links) for visualization |
/api/services/slo |
GET | Per-service SLO snapshot: error-budget remaining + active burn-rate breaches |
/api/admin/status |
POST | Manual status update (requires Authorization: Bearer $ADMIN_API_TOKEN) |
/healthz |
GET | Dead-man's switch — 200 fresh / 503 stale. Can be monitored by a service supervisor; the heartbeat job separately pings HEALTHCHECK_PING_URL. |
/metrics |
GET | Prometheus text exposition. |
/api/webhooks/statuspage/{id} |
POST | Inbound Statuspage subscriber webhook, HMAC-verified. 404 unless WEBHOOKS_ENABLED=true. |
/api/slack/interactivity |
POST | Slack block-actions receiver (ack button). 404 unless SLACK_ACK_ENABLED=true. |
/api/slack/slash |
POST | Slack /itstatus slash-command handler. 503 unless SLACK_SLASH_ENABLED=true. |
All production phases (0–6) and the primary Phase 7 reach features are complete. Full exit-criteria history is in PRODUCTION-ROADMAP.md. Remaining optional work:
- Phase 7 — LLM layer: Natural-language impact statements; deferred post-Phase-7.
- Phase 7 — Integrations: log aggregation, synthetic monitoring, metrics, and ITSM platforms — deferred to demand.