Database Schema

clusters

Top-level GPU cluster resource. Groups nodes and GPUs. Each cluster has a unique API key used by cluster agents to register nodes and push metrics.

nodes

Compute nodes (physical or VM) within a cluster. Each node hosts one or more GPUs and reports health via periodic heartbeat registration.

Indexes: nodes_cluster_id_idx on (cluster_id)

gpus

Individual GPU devices on a node. The job_id column enables the GPU-to-job allocation map. status is updated by both the scheduler (allocation) and metrics ingestion (utilization-based).

Indexes: gpus_node_id_idx, gpus_job_id_idx

gpu_metrics

Time-series utilization data from cluster agents. Written by POST /api/clusters/metrics. Queried by dashboard and the alert evaluation loop. Partitioned by recorded_at — prune old rows via a scheduled cleanup job.

Indexes: gpu_metrics_gpu_id_idx on (gpu_id), gpu_metrics_recorded_at_idx on (recorded_at DESC)

jobs

GPU workload queue. Lifecycle managed by both the scheduler daemon (queued→running, queued→blocked) and the caller via PATCH /api/jobs/:id (running→completed/failed). block_reason carries the specific quota or capacity constraint that caused a block.

Indexes: jobs_cluster_id_idx, jobs_status_idx, jobs_tenant_id_idx

tenants

Multi-tenant orgs. Each tenant has a GPU quota enforced by the scheduler at both submission time (inline check in routes/jobs.js) and scheduling time (inside the per-job transaction in scheduler.js). Quota is the ceiling on concurrent running-job GPUs.

gpu_rates

Rate card: GPU model → $/hr. Both gpu_type (NOT NULL, UNIQUE) and gpu_model (nullable) are present. The gpu_model column was added via migration and backfilled from gpu_type to support flexible rate lookups. Rates are used to calculate cost_estimate in usage_events.

Seed data: H100 $4.76/hr, A100 $2.21/hr, A6000 $1.28/hr

usage_events

Immutable billing ledger. One row per billing unit (a job run or a time-bucketed sample). Cost calculated as gpu_hours × rate_per_hour using the gpu_rates table. This is the source of truth for revenue reporting — never mutate after insert.

Indexes: usage_events_tenant_id_idx, usage_events_started_at_idx

invoices

Stripe-linked billing records. One invoice per tenant per billing period. stripe_payment_url is generated via the Polsia Stripe proxy. The invoice is created only after confirming total usage ≥ $1 (Stripe payment link minimum).

Indexes: invoices_tenant_id_idx, invoices_status_idx

alert_rules

Configurable fleet alert thresholds. Three rule types defined at startup via seedDefaultAlertRules() in routes/scheduler.js. Operators can toggle enabled via PATCH /api/alerts/rules/:id. Severity and alert_message are derived from rule_type (not stored, computed in queries).

Indexes: alert_rules_type_idx on (rule_type), alert_rules_enabled_idx partial on (enabled) WHERE enabled = TRUE

alert_events

Fired alert instances. One row created each time a rule transitions from unmet→met. Auto-resolved when the condition clears (no-op if already fired). details is JSONB containing a snapshot of the state that triggered the alert (e.g., tenant_name, fleet_utilization_pct).

api_keys

Hashed API keys. Raw key is returned once at creation time and never stored. Lookup uses SHA-256 hash + revoked_at IS NULL. A Demo Admin Key is auto-seeded on server startup if no keys exist.

Indexes: api_keys_key_hash_idx on (key_hash), api_keys_tenant_id_idx on (tenant_id)

scheduler_logs

Audit log for each scheduler cycle. Written regardless of success or failure — even a crashed cycle leaves a record. details is JSONB with per-job outcomes; error is set only on catastrophic cycle errors.

early_access_signups

Landing page early access signups. email has an implicit UNIQUE constraint via the application (POST fails with 409 on duplicate insert; the DB has no explicit UNIQUE index — enforced by application-level 23505 catch-and-return).

demo_requests

Enterprise demo requests from the landing page Request Demo modal. email has an index for deduplication lookups. Duplicate submissions return {success: true, existing: true} — idempotent UX.

Indexes: demo_requests_email_idx on (email)

page_views

Server-side page view log. Written by trackServerPageView() in routes/analytics.js — called as Express middleware on page-rendered routes. Also augmented by GA4 client-side events (GA4 is primary for real-time; this table is for backup/reporting).

Indexes: page_views_path_created_idx on (path, created_at DESC)

Arrows show foreign key direction. "1→N" = one-to-many. Nullable FKs are dashed. CASCADE/SET NULL defined on delete.

Job Lifecycle

Managed by both the scheduler daemon (assigns GPUs, blocks on quota/capacity) and callers (PATCH endpoint for completion signals). Priority ordering: 1 (highest) → higher numbers = lower priority.

Transitions:

• queued → running — scheduler assigns GPUs via FOR UPDATE SKIP LOCKED
• queued → blocked — inline check in POST /api/jobs (quota breach or cluster capacity)
• blocked → queued — automatically by scheduler when conditions clear (next cycle re-evaluates)
• running → completed or failed — caller sets via PATCH /api/jobs/:id
• running → cancelled — caller sets via PATCH /api/jobs/:id (scheduler releases GPUs)
• running → blocked — scheduler detects quota exceeded mid-cycle (retry-safe)

Key fields per transition: started_at set on →running. completed_at set on →completed/failed/cancelled. block_reason set on →blocked and cleared on →queued/running.

GPU Status

Updated by two sources: the scheduler (allocation/release) and metrics ingestion (utilization-based). The scheduler takes precedence — it holds a transaction lock while setting job_id and status.

Transitions:

• idle → active — scheduler assigns GPU to a running job (sets job_id)
• active → idle — job completes/fails/cancels; scheduler clears job_id
• idle → active — metrics ingestion marks active when utilization > 10%
• active → idle — metrics ingestion marks idle when utilization ≤ 10%
• * → error — node goes offline or GPU health check fails (future expansion)

Tenant Status

Operators set suspended manually via PUT /api/tenants/:id. Suspended tenants cannot have new jobs scheduled (their running jobs are unaffected; quota still enforced). Used for billing non-payment or policy violations.

Invoice Lifecycle

Transitions: pending → paid (operator marks via PATCH when payment received). pending → overdue (operator marks manually). overdue → paid (resolved). No auto-transitions — operator controls all state changes.

Alert Lifecycle

State is implicit via resolved_at IS NULL (triggered) vs resolved_at IS NOT NULL (resolved). Transitions happen inside runAlertEvaluation(): conditionMet + noOpenEvents → INSERT. !conditionMet + openEvents → UPDATE resolved_at = NOW().

GET  /api/clusters
Auth: optional 200

List all clusters with computed counts. Uses correlated COUNT subqueries to avoid GROUP BY on the main result set.

Response:

[
  {
    "id": 1,
    "name": "us-east-prod",
    "region": "us-east-1",
    "api_key": "gf_demo_us_east",
    "status": "active",
    "created_at": "2026-06-01T00:00:00Z",
    "updated_at": "2026-06-01T00:00:00Z",
    "node_count": 3,
    "gpu_count": 12,
    "active_jobs": 2,
    "queued_jobs": 1
  }
]

GET  /api/clusters/metrics
Auth: optional 200

Fleet-wide aggregate stats. 8 × COUNT subqueries across gpus/jobs tables. NOTE: must be registered BEFORE /:id or Express matches "metrics" as a cluster ID.

{
  "total_clusters": 3,
  "total_nodes": 9,
  "total_gpus": 36,
  "active_gpus": 8,
  "idle_gpus": 28,
  "running_jobs": 3,
  "queued_jobs": 2
}

GET  /api/clusters/:id
Auth: optional 200 404

Single cluster with its nodes and last 50 jobs. Three queries: cluster, nodes (with GPU count per node), jobs.

{
  "id": 1,
  "name": "us-east-prod",
  "region": "us-east-1",
  "status": "active",
  "nodes": [
    {
      "id": 1,
      "hostname": "node-1-1",
      "ip_address": "10.1.1.1",
      "status": "online",
      "gpu_count": 4
    }
  ],
  "jobs": [
    { "id": 1, "name": "LLaMA-70B Training", "status": "running", "gpu_count": 4 }
  ]
}

POST /api/clusters
Auth: optional name, region 201 400

Create a new GPU cluster. Generates a cluster-scoped API key (gf_ + 48 hex chars) returned once at creation time.

// Request
{ "name": "us-east-staging", "region": "us-east-1" }

// Response (201)
{
  "id": 4,
  "name": "us-east-staging",
  "region": "us-east-1",
  "api_key": "gf_a3f8b1c2d4e5f6789012345678...",  // shown once only
  "status": "active",
  "created_at": "2026-06-13T00:00:00Z",
  "updated_at": "2026-06-13T00:00:00Z"
}

POST /api/clusters/nodes
Auth: required cluster_id, hostname 201 400

Register or update a compute node. ON CONFLICT DO NOTHING on hostname makes it idempotent — if hostname exists, no-op (falls back to UPDATE if 0 rows returned). Sets status = 'online' on update.

// Request
{
  "cluster_id": 1,
  "hostname": "node-1-4",
  "ip_address": "10.1.4.1",
  "total_gpus": 8,
  "cpu_cores": 128,
  "ram_gb": 1024
}

// Response (201)
{ "id": 7, "cluster_id": 1, "hostname": "node-1-4", "status": "online", ... }

POST /api/clusters/gpus
Auth: required node_id, gpus[] 201 400

Register one or more GPUs on a node. Each GPU uses ON CONFLICT DO NOTHING — idempotent. gpus[] items: {gpu_index, model, vram_mb}.

// Request
{
  "node_id": 5,
  "gpus": [
    { "gpu_index": 0, "model": "H100", "vram_mb": 80000 },
    { "gpu_index": 1, "model": "H100", "vram_mb": 80000 }
  ]
}

// Response (201) — only newly created GPUs returned
[{ "id": 37, "node_id": 5, "gpu_index": 0, "model": "H100", "vram_mb": 80000, "status": "idle" }]

POST /api/clusters/metrics
Auth: required metrics[] 200 400

Bulk GPU metrics ingestion from cluster agents. For each metric row: INSERT into gpu_metrics, then UPDATE GPU status (util > 10% → 'active', else 'idle').

// Request
{
  "metrics": [
    {
      "gpu_id": 1,
      "utilization_pct": 85.2,
      "memory_used_mb": 72000,
      "memory_total_mb": 80000,
      "temperature_c": 72.5,
      "power_draw_w": 340.0
    }
  ]
}

// Response (200)
{ "accepted": 8 }

POST /api/jobs
Auth: optional cluster_id, name 201 400

Submit a new workload. Inline pre-scheduling checks:

• If tenant_id provided: verify current running GPU count + requested ≤ quota → blocked with block_reason if exceeded
• If cluster has fewer idle GPUs than gpu_count → blocked with block_reason
• Otherwise → queued

// Request
{
  "cluster_id": 1,
  "name": "BERT Fine-tuning",
  "gpu_count": 4,
  "gpu_type": "A100",
  "priority": 2,
  "submitted_by": "ci-pipeline",
  "estimated_duration_min": 120,
  "tenant_id": 1
}

// Response (201)
{
  "id": 15,
  "cluster_id": 1,
  "name": "BERT Fine-tuning",
  "gpu_count": 4,
  "gpu_type": "A100",
  "priority": 2,
  "status": "queued",
  "block_reason": null,
  "submitted_by": "ci-pipeline",
  "submitted_at": "2026-06-13T18:30:00Z"
}

PATCH /api/jobs/:id
Auth: required status 200 400 404

Update job status. Valid statuses: queued, running, completed, failed, cancelled, blocked. Side effects: started_at on →running; completed_at on →completed/failed/cancelled; block_reason cleared on non-blocked states.

// Request (mark job completed)
{ "status": "completed" }

// Response (200)
{
  "id": 15,
  "status": "completed",
  "started_at": "2026-06-13T18:31:00Z",
  "completed_at": "2026-06-13T19:31:00Z",
  "block_reason": null
}

GET  /api/tenants
Auth: optional 200

List all tenants with live GPU usage. Uses FILTER (WHERE j.status = 'running') aggregate syntax so tenants with zero jobs are not excluded.

[
  {
    "id": 1,
    "name": "Acme ML",
    "contact_email": "billing@acme.ai",
    "plan_tier": "enterprise",
    "gpu_quota": 12,
    "status": "active",
    "gpus_in_use": 4,
    "running_jobs": 1,
    "queued_jobs": 2
  }
]

POST /api/tenants
Auth: required name, gpu_quota 201 400

Create a tenant. gpu_quota must be ≥ 1. plan_tier defaults to starter.

// Request
{ "name": "GammaTech", "contact_email": "ops@gamma.tech", "plan_tier": "pro", "gpu_quota": 8 }

// Response (201)
{ "id": 5, "name": "GammaTech", "plan_tier": "pro", "gpu_quota": 8, "status": "active", ... }

PUT  /api/tenants/:id
Auth: required field to update 200 404

Update tenant fields. All fields optional — only provided fields are updated (COALESCE pattern). Can change status to suspended.

// Request
{ "gpu_quota": 16, "status": "suspended" }

// Response (200)
{ "id": 1, "gpu_quota": 16, "status": "suspended", ... }

DELETE /api/tenants/:id
Auth: required 200 404

Delete tenant. Jobs are first unlinked (tenant_id → NULL) to preserve audit trail. Then tenant is deleted (CASCADE would affect invoices — handled separately).

// Response (200)
{ "deleted": true }

GET  /api/tenants/:id/usage
Auth: optional ?days=N 200

Usage events and cost summary for a tenant. Query param days clamped [1, 90], default 30. Reads usage_events (immutable billing log), not invoices.

// Response (200)
{
  "summary": [
    { "gpu_type": "H100", "total_gpu_hours": "1840.00", "total_cost": "8758.40", "event_count": 30 }
  ],
  "daily": [
    { "day": "2026-06-01", "gpu_hours": "96.00", "cost": "456.96" }
  ],
  "totals": { "total_gpu_hours": "1840.00", "total_cost": "8758.40" },
  "days": 30
}

GET  /api/rates
Auth: optional 200

GPU rate card — all models with $/hr pricing, ordered DESC by rate.

[
  { "id": 1, "gpu_type": "H100", "gpu_model": "H100", "rate_per_hour": "4.7600", "created_at": "..." },
  { "id": 2, "gpu_type": "A100", "gpu_model": "A100", "rate_per_hour": "2.2100", "created_at": "..." },
  { "id": 3, "gpu_type": "A6000", "gpu_model": "A6000", "rate_per_hour": "1.2800", "created_at": "..." }
]

POST /api/billing/invoice/:tenantId
Auth: required ?days=N 200 400 404 502

Calculate usage cost and generate a Stripe payment link via Polsia Stripe proxy. Minimum amount $1 (Stripe requirement). Idempotent — returns existing pending invoice if one exists for the period.

// Response (200)
{
  "invoice": {
    "id": 3,
    "tenant_id": 1,
    "total_gpu_hours": "1840.0000",
    "total_amount": "8758.40",
    "stripe_payment_url": "https://checkout.stripe.com/...",
    "status": "pending"
  },
  "tenant": { "id": 1, "name": "Acme ML", "contact_email": "billing@acme.ai" },
  "total_gpu_hours": "1840.0000",
  "total_amount": 8758.40,
  "payment_url": "https://checkout.stripe.com/..."
}

GET  /api/billing/summary
Auth: optional 200

Billing totals and per-tenant breakdown. Three parallel queries: overall totals, per-tenant invoice history, recent 20 invoices.

{
  "totals": {
    "total_invoices": 12,
    "total_invoiced": "24830.50",
    "total_paid": "16342.00",
    "total_pending": "8758.40",
    "total_overdue": "0.00"
  },
  "per_tenant": [...],
  "recent_invoices": [...]
}

GET  /api/billing/tenants/:id/invoices
Auth: optional 200

Invoice history for a tenant, ordered newest first, limit 50.

PATCH /api/billing/invoices/:id/status
Auth: required status 200 400 404

Mark invoice paid/pending/overdue. Valid: paid, pending, overdue.

GET  /api/alerts/rules
Auth: optional 200

List all alert rules with computed severity (derived from rule_type) and event counts. Severity: idle_gpu/quota_breach → warning; health_degraded → critical.

[
  {
    "id": 1,
    "rule_type": "idle_gpu",
    "threshold": "60.00",
    "tenant_id": null,
    "enabled": true,
    "severity": "warning",
    "active_events": 0,
    "fired_count": 0
  }
]

PATCH /api/alerts/rules/:id
Auth: required enabled 200 400 404

Toggle an alert rule's enabled state. enabled must be a boolean.

GET  /api/alerts/events
Auth: optional ?limit=N 200

Recent alert events, active (unresolved) first, then resolved. limit clamped [1, 100], default 50. tenant_name extracted from details JSONB via ->> with fallback subquery.

POST /api/alerts/notify
Auth: required rule_type, threshold, details 200

Manual alert notification log (internal/test). Logs to console and returns success. Used for testing notification pipeline.

POST /api/api-keys
Auth: required label 201 400

Generate a new API key. Raw key format: gfk_ + 64 hex chars (32 random bytes). The raw key is returned ONLY HERE — never retrievable afterward.

// Request
{ "label": "CI/CD Pipeline", "tenant_id": 1, "permissions": ["read", "write"] }

// Response (201) — raw key is the ONLY time it's visible
{
  "id": 3,
  "key_prefix": "gfk_a3f8b1c2",
  "label": "CI/CD Pipeline",
  "tenant_id": 1,
  "permissions": ["read", "write"],
  "key": "gfk_a3f8b1c2d4e5f6789012345678901234567890123456789012345678901234",
  "created_at": "2026-06-13T00:00:00Z"
}

GET  /api/api-keys
Auth: required 200

List all API keys (no raw keys ever returned). Includes tenant name via LEFT JOIN.

DELETE /api/api-keys/:id
Auth: required 200 404

Soft-revoke an API key. Sets revoked_at = NOW(). Record preserved for audit trail. Returns 404 if already revoked or not found.

GET  /api/dashboard
Auth: optional 200

Aggregate stats, node details with latest metrics (LATERAL join), and recent 20 jobs. 8 × COUNT subqueries for stats; LATERAL on gpu_metrics for per-GPU latest reading.

{
  "stats": {
    "total_clusters": 3, "online_nodes": 9, "offline_nodes": 0,
    "total_gpus": 36, "active_gpus": 8,
    "running_jobs": 3, "queued_jobs": 2, "blocked_jobs": 1
  },
  "avg_utilization": 72.5,
  "nodes": [{ "node_id": 1, "hostname": "node-1-1", "gpu_id": 1, "utilization_pct": 85.2, ... }],
  "recent_jobs": [...]
}

GET  /api/dashboard/kpis
Auth: optional 200

Executive KPI summary: fleet utilization, revenue leakage, quota utilization, fleet health, active job counts. Revenue leakage = idle allocated GPUs / total quota × 100.

{
  "fleet_utilization": { "percentage": 72.4, "active_gpus": 8, "total_gpus": 11, "status": "good" },
  "revenue_leakage": { "percentage": 27.6, "idle_allocated_gpus": 3, "total_quota": 40, "direction": "down" },
  "quota_utilization": { "percentage": 72.4, "gpus_in_use": 29, "total_quota": 40, "per_tenant": [...] },
  "fleet_health": { "healthy": 11, "total": 11, "status": "all_healthy" },
  "active_jobs": { "running": 3, "queued": 2, "blocked": 1 }
}

GET  /api/dashboard/revenue
GET  /api/dashboard/utilization-history
GET  /api/dashboard/scheduling-metrics
GET  /api/dashboard/gpu-allocation
GET  /api/dashboard/leakage
Auth: optional 200

Specialized dashboard views: revenue (per-tenant, per-GPU-type, daily trend), utilization-history (1-min buckets, last 1h), scheduling-metrics (wait times, throughput, active queue), gpu-allocation (full GPU-to-job map with metrics), leakage (idle-allocated, billing gap, underutilized analysis).

GET  /api/scheduler/logs
Auth: optional ?limit=N 200

Scheduler cycle audit log, newest first. limit clamped [1, 100], default 20.

[
  {
    "id": 100,
    "ran_at": "2026-06-13T18:30:00Z",
    "jobs_processed": 3,
    "jobs_started": 2,
    "jobs_blocked": 1,
    "duration_ms": 42,
    "error": null
  }
]

GET  /api/health
Auth: none 200

Health check endpoint. Returns {"status": "healthy"}. Render health probe target.

POST /api/demo/seed
POST /api/demo/reset
POST /api/early-access
GET  /api/analytics
Auth: varies 200

/api/demo/seed: idempotent — no-op if clusters exist. /api/demo/reset: truncates operational tables and re-seeds. /api/early-access: landing page signup — validates name/email/company, 23505 → {success: true, existing: true}. /api/analytics: returns signup count, demo request count, top pages (14d), daily page views (14d).

server.js — Entry Point (172 lines)

Wiring only. Owns: middleware chain, route mounts, API key auth middleware, scheduler daemon, app.listen. Does NOT own: query functions, scheduler daemon logic, alert evaluation.

Middleware chain (in order)

1. express.json() — JSON body parsing
2. WWW redirect: www.gpuforge.com → gpuforge.com
3. /health and /api/health routes (no auth)
4. apiKeyAuth() — sets req.apiKey (null for demo, row for authenticated)
5. express.static() — serves public/

Route mounts

All route groups use express.Router(). Legacy billing paths are rewritten in-place before handing off to the billing router.

app.use('/api/clusters',    require('./routes/clusters'))
app.use('/api/jobs',        require('./routes/jobs'))
app.use('/api/tenants',     require('./routes/tenants'))
app.use('/api/billing',     require('./routes/billing'))
app.use('/api/alerts',      require('./routes/alerts'))
app.use('/api/api-keys',    require('./routes/api-keys'))
app.use('/api/dashboard',   require('./routes/dashboard'))
app.use('/api/demo',        require('./routes/demo'))
app.use('/api/demo-requests', require('./routes/demo-requests'))
app.use('/api/scheduler',   schedulerRoutes)
app.use('/api/analytics',   require('./routes/analytics'))
app.use('/api',             require('./routes/misc'))   // /api/rates, /api/early-access

Scheduler daemon

Guarded by POLSIA_IN_PROCESS_CRONS_ENABLED === 'true'. Render sets this; Blaxel shadow sets it to false. On Render: setTimeout(5s delay) then setInterval(30s). On Blaxel: polsia.toml [[crons]] runs jobs/scheduler-cycle.js.

if (process.env.POLSIA_IN_PROCESS_CRONS_ENABLED === 'true') {
  setTimeout(async () => { await seedDefaultAlertRules(); runSchedulerCycle(); }, 5000);
  setInterval(runSchedulerCycle, 30000);
} else {
  // Blaxel handles scheduling via polsia.toml [[crons]]
}

routes/clusters.js

Owns all cluster-related endpoints. Key design note: GET /api/clusters/metrics must be registered BEFORE GET /:id — Express matches paths in order, and "/metrics" would be caught as a cluster ID of 'metrics' (invalid integer → 500).

Endpoints

• GET / — correlated COUNT subqueries avoid GROUP BY on main result set
• POST / — generates cluster API key as gf_ + 24 random bytes hex
• GET /metrics — fleet-wide aggregate counts (before /:id)
• GET /:id — cluster + nodes + last 50 jobs
• POST /nodes — ON CONFLICT DO NOTHING for idempotent registration; fallback UPDATE
• POST /gpus — batch GPU registration; each with ON CONFLICT DO NOTHING
• POST /metrics — bulk INSERT to gpu_metrics + status update per GPU

routes/jobs.js

Owns: job submission and status update. Does NOT own: GPU assignment — handled by scheduler daemon. Pre-scheduling checks run inline at submission time (not deferred to scheduler) so the caller gets immediate feedback.

POST / — inline pre-scheduling checks

// 1. Tenant quota check (if tenant_id provided)
const usageRes = await pool.query(
  `SELECT COALESCE(SUM(gpu_count), 0)::int AS current_usage
   FROM jobs WHERE tenant_id = $1 AND status = 'running'`, [tenant_id]
);
if (current + requested > quota) → status = 'blocked', block_reason = quota message

// 2. Cluster capacity check (idle GPUs on target cluster)
const idleGpus = await pool.query(
  `SELECT COUNT(*) FILTER (WHERE g.status = 'idle')::int AS idle_gpus
   FROM gpus g JOIN nodes n ON g.node_id = n.id WHERE n.cluster_id = $1`, [cluster_id]
);
if (idleGpus < requested) → status = 'blocked', block_reason = capacity message

PATCH /:id — status update with side effects

Setting →running sets started_at = NOW(). Setting →completed/failed/cancelled sets completed_at = NOW(). Setting →blocked accepts optional block_reason; setting to any non-blocked status clears block_reason.

routes/tenants.js

Owns: tenant CRUD, usage events query, rate card. Key query patterns: LEFT JOIN with FILTER aggregation for running/queued job counts per tenant.

SELECT t.*,
  COALESCE(SUM(j.gpu_count) FILTER (WHERE j.status = 'running'), 0)::int AS gpus_in_use,
  COUNT(j.id) FILTER (WHERE j.status = 'running')::int AS running_jobs,
  COUNT(j.id) FILTER (WHERE j.status = 'queued')::int AS queued_jobs
FROM tenants t LEFT JOIN jobs j ON j.tenant_id = t.id
GROUP BY t.id

routes/billing.js

Owns: invoice generation, billing summary, invoice status. Integration: Polsia Stripe proxy for payment link creation.

POST /invoice/:tenantId — flow

1. Sum usage_events for period → total_amount
2. Reject if amount < $1 (Stripe minimum)
3. Check for existing pending invoice in this period → return if found (idempotent)
4. POST to https://polsia.com/api/proxy/stripe/payment-link
5. INSERT into invoices with status pending
6. Return invoice + payment_url

routes/scheduler.js — Scheduler Daemon

Owns: runSchedulerCycle(), runAlertEvaluation(), seedDefaultAlertRules(), GET /api/scheduler/logs. Does NOT own: HTTP route mounts or the POLSIA_IN_PROCESS_CRONS_ENABLED guard (those live in server.js).

runSchedulerCycle() — per-job transaction

BEGIN
  → Check tenant quota: running GPU count + requested ≤ quota
     Exceeded → UPDATE job to 'blocked', COMMIT, continue

  → Acquire GPUs: SELECT ... FOR UPDATE SKIP LOCKED
     Fewer available than requested → UPDATE job to 'blocked', COMMIT, continue

  → Mark GPUs: UPDATE gpus SET job_id = job.id, status = 'active'
  → Mark job running: UPDATE jobs SET status = 'running', started_at = NOW()
  → COMMIT

  → Write scheduler_log (always — even on error)
  → Run alert evaluation
ROLLBACK on any exception

FOR UPDATE SKIP LOCKED: two scheduler instances (Render + Blaxel shadow during migration) can run concurrently without double-assigning GPUs. A job being processed by one instance appears "locked" to the other, which skips it.

runAlertEvaluation() — state machine per rule

conditionMet && noOpenEvents  → INSERT alert_events, send email
!conditionMet && openEvents   → UPDATE resolved_at = NOW()  (auto-resolve)
conditionMet && openEvents    → no-op (already alerted)
!conditionMet && noOpenEvents → no-op (healthy)

seedDefaultAlertRules() — idempotent on startup

Seeds 3 rules only if SELECT COUNT(*)::int FROM alert_rules returns 0. No-op on subsequent restarts.

idle_gpu       threshold: 60  (alert when < 60% of fleet GPUs are active)
quota_breach   threshold: 90  (alert when any tenant exceeds 90% of GPU quota)
health_degraded threshold: 80 (alert when < 80% of fleet GPUs are healthy)

routes/alerts.js

Owns: alert rules list/update, alert events list, manual notify. Does NOT own: alert evaluation — that lives in runAlertEvaluation() (routes/scheduler.js), triggered at the end of each scheduler cycle.

Severity is derived from rule_type, not stored. Query uses CASE WHEN to compute it inline. Active events counted with: COUNT(*)::int FROM alert_events WHERE rule_id = ar.id AND resolved_at IS NULL.

routes/api-keys.js

Owns: key generation, listing, revocation. Does NOT own: API key authentication middleware (server.js), key validation queries (db/auth.js).

Key generation: gfk_ + 32 random bytes hex → SHA-256 hash stored in DB. Raw key returned once at creation. key_prefix (first 12 chars) stored in plaintext for human identification.

Revocation: soft-delete — sets revoked_at = NOW(). Lookup in db/auth.js excludes keys where revoked_at IS NOT NULL.

routes/dashboard.js

Owns: aggregate stats, KPIs, revenue, utilization history, scheduling metrics, GPU allocation, leakage. Eight sub-endpoints — all read-only, no auth required.

Key SQL patterns

• COUNT(*) FILTER (WHERE status = 'running')::int — counts with conditional filter
• LEFT JOIN LATERAL (SELECT ... ORDER BY ... LIMIT 1) ON true — latest reading per GPU without N+1
• date_trunc('minute', recorded_at) — time-bucket for utilization history
• EXTRACT(EPOCH FROM (started_at - submitted_at)) / 60 — wait time in minutes
• ILIKE '%' || gpu_model || '%' — flexible rate lookup in leakage analysis

routes/demo.js — Demo Data Management

Owns: demo seed, demo reset, early access signup. Demo data structure:

• Clusters: 3 (us-east-prod, eu-west-prod, ap-south-dev)
• Tenants: 4 (Acme ML/enterprise/12, BetaCorp/pro/8, DataSci Inc/starter/4, NeuralWorks/enterprise/16)
• GPU rates: H100 $4.76/hr, A100 $2.21/hr, A6000 $1.28/hr
• Nodes: 3 per cluster × 4 GPUs = 36 total
• Jobs: 3 running, 2 queued, 1 blocked, 2 completed
• GPU metrics: last 60 min at 5-min intervals for first 12 GPUs
• Usage events: 30 days of historical billing data (1 event per tenant per day)

Reset truncates in order: usage_events → gpu_rates → gpu_metrics → gpus → jobs → nodes → clusters → tenants. Tenants and api_keys intentionally preserved — they represent real accounts.

routes/demo-requests.js

Owns: POST (create), GET (list), PATCH (status update). On POST: stores to DB, fires HTML email to founders@gpuforge.ai via Polsia email proxy. Email send is non-blocking (fire-and-forget with .catch()) — failure doesn't affect API response.

XSS prevention: message field rendered as HTML (newlines → <br>) is escaped with escapeHtml() before interpolation. Covers: & < > " '

db/index.js — Pool Constructor

The only file that constructs new Pool(). Does NOT own: query logic (that's db/auth.js, db/demo-requests.js, db/analytics.js).

const pool = new Pool({
  connectionString: process.env.DATABASE_URL,
  ssl: process.env.DATABASE_URL?.includes('localhost') ? false : { rejectUnauthorized: false }
});
module.exports = pool;

db/auth.js — API Key Validation

Owns: lookupApiKey(), touchLastUsed(), seedDemoApiKey(). Does NOT own: key generation (routes/api-keys.js) or auth middleware wiring (server.js).

async function lookupApiKey(rawKey) {
  const keyHash = crypto.createHash('sha256').update(rawKey).digest('hex');
  const result = await pool.query(
    `SELECT * FROM api_keys WHERE key_hash = $1 AND revoked_at IS NULL`,
    [keyHash]
  );
  return result.rows[0] || null;
}

function touchLastUsed(id) {
  pool.query(`UPDATE api_keys SET last_used_at = NOW() WHERE id = $1`, [id]).catch(() => {});
}

db/demo-requests.js & db/analytics.js

Domain-specific query functions. Each owns only the queries for its entity. db/analytics.js exports: logPageView(), getSignupCount(), getDemoRequestCount(), getPageViewsByPath(), getDailyPageViews(), getAnalyticsSummary() (parallel query for dashboard summary).

Current Auth Model (Mock)

API key authentication via Authorization: Bearer <key> header. No real RBAC enforcement — the system uses a demo/mock auth model appropriate for internal tooling. The Bearer key validation flow:

1. Missing header → req.apiKey = null, request passes through (demo mode)
2. Invalid format (Authorization: Bearer <key>) → 401 "Invalid Authorization header format"
3. Key not found or revoked → 401 "Invalid or revoked API key"
4. Valid key → req.apiKey = keyRow, touchLastUsed() fire-and-forget update
5. Exception during lookup → 500 "Authentication check failed"

async function apiKeyAuth(req, res, next) {
  const authHeader = req.headers['authorization'];
  if (!authHeader) { req.apiKey = null; return next(); }

  const match = authHeader.match(/^Bearer\\s+(.+)$/i);
  if (!match) return res.status(401).json({ error: 'Invalid Authorization header format' });

  const apiKey = await lookupApiKey(match[1]);
  if (!apiKey) return res.status(401).json({ error: 'Invalid or revoked API key' });
  touchLastUsed(apiKey.id);
  req.apiKey = apiKey;
  next();
}

API Key Security

• Hashing: Raw key → SHA-256 hash stored in DB. Hash is irreversible — raw key cannot be recovered from the DB.
• Key prefix: First 12 chars stored in plaintext for human identification. Insufficient for auth — only the full raw key authenticates.
• Revocation: Soft delete — sets revoked_at = NOW(). Record preserved. Lookup explicitly excludes revoked keys.
• Demo key: One gfk_... key is auto-seeded on startup if no keys exist. Raw key is logged to console.
• Rate limiting: Not currently implemented. Consider adding for production.

Session Management (localStorage)

The frontend stores session data in localStorage under gpuforge_session. Client-side code checks this key before showing authenticated UI. Server does not issue session tokens — the API is stateless and key-based.

// Client-side (Vanilla JS)
localStorage.setItem('gpuforge_session', JSON.stringify({ tenantId: 1, role: 'admin' }));
const session = JSON.parse(localStorage.getItem('gpuforge_session') || '{}');
// Show/hide admin UI elements based on session.role

Role-Based Access Control

Currently enforced client-side via localStorage session flags. No server-side role enforcement. In the demo/admin dashboard: role: 'admin' in the session unlocks admin-only UI elements. All API endpoints are accessible to any valid API key or demo mode.

Future improvement: add permissions column to api_keys (already in schema as JSONB) and enforce at the route level.

SQL Injection Prevention

All queries use parameterized statements ($1, $2, ...). No string concatenation of user input into SQL. All DB access goes through named functions in db/ or inline parameterized queries in routes — no raw pool.query() with template literals outside db/index.js and routes files.

XSS Prevention (Demo Requests)

The message field in demo requests is rendered as HTML in the sales email (newlines → <br>). The escapeHtml() function encodes & < > " ' before interpolation, preventing script injection in the email body.

function escapeHtml(str) {
  return String(str || '').replace(/&/g, '&')
                          .replace(//g, '>')
                          .replace(/"/g, '"')
                          .replace(/'/g, ''');
}

WWW Redirect

www.gpuforge.com → gpuforge.com redirect via Express middleware checking req.hostname === 'www.gpuforge.com'. Prevents cookie-sharing or SEO split between www and root.

HTTP Status Code Patterns

Error Response Shape

All errors return JSON with an error key. Success responses match the domain — no wrapping envelope.

// Error
{ "error": "Cluster not found" }

// Success (varies by endpoint — no wrapper)
[ { "id": 1, "name": "us-east-prod", ... } ]

Per-Route Pattern

Every route wraps DB calls in try/catch with structured logging:

try {
  // DB query or external call
} catch (err) {
  console.error('Error description:', err);   // structured log
  res.status(500).json({ error: 'Human-readable message' });
}

Never expose raw DB errors (err.message, err.code) to clients — those may contain table names, constraint names, or internal paths. Always return a generic message.

Non-Critical Failures

Some failures are non-critical and must not block the response:

• touchLastUsed(): Fire-and-forget — .catch(() => {}) so a slow UPDATE doesn't delay the response
• Email send: .catch(err => console.error(...)) — demo request is stored before email attempt; API returns 201 even if email fails
• Page view logging: Wrapped in try/catch with next() always called — page view failure never blocks a page render
• Scheduler log write: Inner try/catch — failed log write is swallowed silently (cycle already completed or errored)

Scheduler Error Handling

Per-job errors: caught, ROLLBACK issued, client released, logged to details JSONB. Cycle-level errors: caught, duration_ms logged, error message stored in scheduler_logs.error field. In both cases, the scheduler loop continues to the next job rather than crashing.

// Per-job
} catch (jobErr) {
  await client.query('ROLLBACK');
  console.error(`[scheduler] Error processing job ${job.id}:`, jobErr.message);
  details.push({ job_id: job.id, name: job.name, result: 'error', error: jobErr.message });
} finally {
  client.release();
}

// Cycle-level
} catch (cycleErr) {
  await pool.query(`INSERT INTO scheduler_logs (...)`, [processed, started, blocked, durationMs, details, cycleErr.message]);
}

PostgreSQL Error Codes

The codebase checks specific PostgreSQL error codes in a few places:

• err.code === '23505' — unique constraint violation (duplicate email in early_access_signups or demo_requests). Treated as success with {existing: true} for idempotent UX.
• err.message?.includes('does not exist') — used to suppress expected error when seeding demo API key on fresh DB (table may not exist yet during migration).