Setup Guide

This guide walks you through installing PowerTesting, configuring its SQLite database, seeding test cases from JSON, and adding new products / validators. It is the operator-facing companion to the Concepts and User Guides — if you are bringing up a fresh environment or onboarding a new product, start here.

Architecture

Components

PowerTesting ships as two cooperating services and a JSON seed corpus:

Repository Layout

PowerTesting/
├── testing-api/
│   ├── powertesting_api/        # FastAPI app (config, db, models, routers, seed)
│   ├── seed/
│   │   ├── products.json
│   │   ├── validator_types.json
│   │   ├── product_categories.json
│   │   └── test_cases/
│   │       ├── powersolver/*.json
│   │       ├── powerforecast/*.json
│   │       ├── powerdecision/*.json
│   │       ├── powerai/*.json
│   │       └── powerorchestrator/*.json
│   ├── scripts/
│   │   ├── seed_db.py           # CLI re-seed
│   │   ├── run_dev.sh           # Local dev runner
│   │   └── smoke_test.sh        # Post-deploy smoke
│   ├── data/
│   │   └── powertesting.db      # SQLite file (created on first start)
│   ├── docker-compose.yml
│   ├── Dockerfile
│   └── pyproject.toml
└── testing-ui/                  # Static SPA (deployed independently)

Quickstart (Docker)

The supported way to bring up the API is Docker Compose. From the repo root:

cd testing-api
docker compose up -d --build

# Verify
curl http://localhost:8090/health
# → {"status":"ok"}

# Confirm the seed loaded
curl http://localhost:8090/products | jq '.[].name'
# → "PowerSolver" "PowerForecast" "PowerDecision" "PowerAI" "PowerOrchestrator"

On first start the container creates /data/powertesting.db in the powertesting-data named volume, runs the SQLAlchemy schema migrations, and loads every JSON file under seed/.

Quickstart (Local Dev, no Docker)

For iterating on the API itself:

cd testing-api
python -m venv .venv
source .venv/bin/activate     # Windows: .venv\Scripts\Activate.ps1
pip install -e .

# Run the dev server (port 8090, hot-reload)
./scripts/run_dev.sh           # Windows: .\scripts\run_dev.bat

# Or directly
uvicorn powertesting_api.main:app --reload --port 8090

Configuration

Environment Variables

Every setting is read by powertesting_api/config.py with the POWERTESTING_ prefix. Defaults work for local dev; override in docker-compose.yml or a .env file for everything else.

CORS

POWERTESTING_CORS_ORIGINS is a comma-separated string — never a JSON array. pydantic-settings would try to json.loads() a JSON default and crash the container.

# dev: allow everything
POWERTESTING_CORS_ORIGINS=*

# prod: lock to the SPA origin
POWERTESTING_CORS_ORIGINS=https://testing.planningpowertools.com

Database

Where the DB Lives

• Local dev: testing-api/data/powertesting.db (created on first start).
• Docker: /data/powertesting.db inside the container, mounted from the named volume powertesting-data.

Move the volume between hosts to migrate data; copy the .db file out for backup. SQLite is single-file — no pg_dump or extra tooling needed.

Schema Overview

Every table is declared in powertesting_api/models.py:

Seed-on-Startup

With POWERTESTING_SEED_ON_STARTUP=true (the default), main.py calls load_all_seed() on app startup. This walks seed/ and upserts every JSON file. The startup log prints a summary:

Seeded: {'products': 5, 'validator_types': 28,
         'product_categories': 14, 'test_cases': 47}

Upserts are keyed: products by name, validator_types by name, test_cases by uuid. Test-case files without a uuid field are silently skipped — this is deliberate, so cases created at runtime via POST /test-cases aren't overwritten by an unrelated file.

Manual Re-seed

To re-seed without restarting (e.g. after editing a JSON file):

# From inside the container
docker exec -it powertesting-api python -m scripts.seed_db

# Or, in local dev
cd testing-api
python scripts/seed_db.py

Backup & Reset

# Backup (Docker)
docker run --rm -v powertesting-data:/data -v "$PWD":/backup \
  alpine sh -c 'cp /data/powertesting.db /backup/powertesting-$(date +%F).db'

# Wipe and re-seed from scratch
docker compose down -v          # removes the volume
docker compose up -d --build    # fresh DB, fresh seed

Seed Files

Four kinds of file live under testing-api/seed/:

products.json

One object per product. Determines what shows up in the sidebar and where runs send their HTTP calls.

[
  {
    "name": "PowerSolver",
    "display_name": "PowerSolver",
    "color_hex": "#4E79A7",
    "icon": "cpu",
    "local_base_url": "http://localhost:8001",
    "prod_base_url":  "https://api.planningpowertools.com",
    "default_timeout_ms": 60000,
    "supports_levels": ["L1", "L2"],
    "display_order": 1,
    "enabled": true
  }
]

validator_types.json

Every validator the UI may reference must have a row here. product_scope is null for generic validators (StatusValidator, FieldMatchValidator, SchemaValidator, ResponseTimeValidator) and a product name otherwise.

{
  "name": "ScoreValidator",
  "description": "Validates Timefold solver score, e.g. '0hard/-5soft'.",
  "product_scope": "PowerSolver",
  "arg_schema": {
    "type": "object",
    "properties": {
      "pattern":  { "type": "string" },
      "feasible": { "type": "boolean" }
    },
    "required": ["pattern"]
  },
  "implementation_module": "products/powersolver.js"
}

product_categories.json

Per-product category buckets used to group cases in the UI (e.g. feasibility, regression, performance). Optional — cases also work without a declared category, though the UI groups them under "Uncategorised".

test_cases/<product>/*.json

One JSON file per test case, one folder per product. The seed loader walks the tree and upserts each file by its uuid.

Test Case Schema

Field Reference

Worked Example

A real PowerSolver smoke case — this is what seed/test_cases/powersolver/mock-feasible.json looks like:

{
  "uuid": "11111111-1111-1111-1111-111111111111",
  "name": "mock-feasible",
  "product": "PowerSolver",
  "level": "L1",
  "category": "feasibility",
  "endpoint_template": "{base}/api/solve",
  "http_method": "POST",
  "request_payload": { "problem": "mock" },
  "validator_refs": [
    { "name": "StatusValidator", "args": { "expected": "success" } },
    { "name": "ScoreValidator",  "args": { "maxSoftPenalty": 100 } }
  ],
  "tags": ["smoke"],
  "environments": ["local"]
}

Add a Test Case

You have two options:

1. File-based (recommended for cases you want under version control).

# Create the file
seed/test_cases/powersolver/my-new-case.json

# Generate a UUID and paste it into the "uuid" field
python -c "import uuid; print(uuid.uuid4())"

# Re-seed
docker exec -it powertesting-api python -m scripts.seed_db
# or restart the container — POWERTESTING_SEED_ON_STARTUP=true picks it up

2. Runtime (recommended for ad-hoc experimentation).

curl -X POST https://testing-api.planningpowertools.com/test-cases \
  -H "Content-Type: application/json" \
  -d @my-new-case.json

Runtime-created cases live only in the DB — they are not written back to disk. Promote them to the seed corpus when you want them version-controlled.

Add a Product

1. Add a row to seed/products.json with name, base URLs, supported levels, and display order.
2. (Optional) Add product-scoped validators to seed/validator_types.json.
3. (Optional) Add the categories you'll use to seed/product_categories.json.
4. Re-seed the DB (restart or run scripts/seed_db.py).
5. On the UI side, create js/products/<new-product>.js exporting a module with the product's adapter and any product-specific validators, then register it in js/main.js.
6. Add at least one test case in seed/test_cases/<new-product>/ so the product tab isn't empty.

Add a Validator Type

A validator has two halves: the metadata row and the JS implementation.

Metadata — add to seed/validator_types.json:

{
  "name": "MyCustomValidator",
  "description": "Asserts foo equals bar in the response payload.",
  "product_scope": "PowerSolver",
  "arg_schema": {
    "type": "object",
    "properties": { "expected": { "type": "string" } },
    "required": ["expected"]
  },
  "implementation_module": "products/powersolver.js"
}

Implementation — in the referenced JS module:

export const myCustomValidator = {
  name: 'MyCustomValidator',
  scope: 'PowerSolver',
  validate(envelope, args) {
    const actual = envelope.result?.foo;
    const ok = actual === args.expected;
    return {
      ok,
      message: ok
        ? `OK (foo == ${args.expected})`
        : `Expected foo=${args.expected}, got ${actual}`,
    };
  },
};

Then re-seed the DB (so the metadata row is registered) and reload the UI (so the JS implementation is picked up). Coverage view will now show the validator under its product scope.

Auth Bootstrap

The UI itself is open access, but operator-level write actions (run prod, set prod prefs) require a token. To create the first admin:

# In docker-compose.yml or a .env file
POWERTESTING_BOOTSTRAP_ADMIN_USERNAME=admin
POWERTESTING_BOOTSTRAP_ADMIN_PASSWORD=<a strong password>

# Then start the container — main.py calls bootstrap_admin_if_empty()
docker compose up -d

The password is hashed (stdlib scrypt) before being written. The bootstrap is a no-op if any user already exists or if the password env var is empty — this is deliberate so a default like change-me never ships.

Verifying Setup

The repository ships a smoke script that exercises the major endpoints:

cd testing-api
./scripts/smoke_test.sh http://localhost:8090

# Expected:
# ✓ /health
# ✓ /products       (5 rows)
# ✓ /validator-types (28 rows)
# ✓ /test-cases     (≥1 row)
# ✓ /test-runs      (POST, then GET)

If any line fails, jump to Troubleshooting.

Production Deploy

The cloud setup running at testing-api.planningpowertools.com is:

• Hetzner host running Docker.
• docker-compose.yml from testing-api/, with prod env overrides.
• Caddy / nginx in front terminating TLS and forwarding to :8090.
• powertesting-data named volume mounted at /data — this is the only stateful dependency.

# prod env overrides (in deploy .env)
POWERTESTING_ENVIRONMENT=prod
POWERTESTING_SEED_ON_STARTUP=true                         # safe: upserts only
POWERTESTING_CORS_ORIGINS=https://testing.planningpowertools.com
POWERTESTING_BOOTSTRAP_ADMIN_PASSWORD=<set once, then unset>

The static testing-ui is published separately to testing.planningpowertools.com. It reads the API base from window.POWERTESTING_API_URL in index.html — update that line if the API hostname ever changes.

Troubleshooting