Tutorials

Hands-on walkthroughs that take you from opening the UI for the first time to shipping a custom validator. Each tutorial assumes the previous ones — if you are starting fresh, work through them in order.

Prerequisites

• The PowerTesting UI is reachable. In production: testing.planningpowertools.com. Locally: http://localhost:8091 after ./run-local.ps1.
• The testing-api is up — the env switcher should show product names rather than UNKNOWN cards.
• For tutorials 3+ you'll want curl (or your favourite HTTP client) and a code editor.

1. Run Your First Test

Goal: confirm PowerTesting can drive a real product end-to-end.

1. Open the UI. The Dashboard loads by default.
2. Confirm the environment switcher in the top-right says local (or prod if that's what you intend).
3. Wait for the product health cards to settle — you want at least one card showing UP.
4. Click that product in the sidebar (e.g. PowerSolver).
5. Pick a small case — tag smoke if available — and tick its checkbox.
6. Click Run selected.
7. Watch the progress panel. When it finishes, the results table appears on the right.

Click the row to expand it: you should see the captured response, every validator's pass/fail pill, and the elapsed milliseconds. That is the canonical envelope after the adapter has done its job.

2. Compare Two Runs

Goal: spot regressions between two points in time.

1. Run a small suite (5-10 cases) twice in a row, with no code changes between them.
2. Click History in the sidebar.
3. Open the most recent run.
4. In the run detail header, use the Diff against... picker and select the run before it.
5. Read the buckets:
   • regressions should be empty — if it isn't, something is non-deterministic.
   • stablePass should hold every case that passed both times.

Now make a deliberate, breaking change in one product (e.g. tighten a constraint that the case relies on). Run the suite again, diff against the prior good run — that broken case should land in regressions.

3. Add a Test Case via API

Goal: ship a new test case end-to-end.

Cases live in the testing-api's SQLite database. Add one with a POST:

curl -X POST https://testing-api.planningpowertools.com/api/test-cases \
  -H "Content-Type: application/json" \
  -d '{
    "id": "solver-smoke-tiny",
    "product": "PowerSolver",
    "level": "L1",
    "endpoint": "/api/solve",
    "method": "POST",
    "payload": {
      "tasks": [{ "id": "t1", "duration": 30 }],
      "resources": [{ "id": "r1", "capacity": 60 }]
    },
    "validators": [
      { "name": "status",        "config": { "expected": 200 } },
      { "name": "response-time", "config": { "maxMs": 2000 } },
      { "name": "field-match",   "config": { "path": "score.hardScore", "expected": 0 } }
    ],
    "tags": ["smoke", "regression"]
  }'

Refresh the PowerSolver tab — solver-smoke-tiny should appear in the case list. Tick it, run it, and confirm it passes. If it doesn't, expand the row to see which validator complained.

4. Write a Custom Validator

Goal: add a product-specific assertion that the generic validators can't express. Example: assert that PowerSolver's solution leaves no resource over-allocated.

1. Open the product's module file:

// js/products/powersolver.js
import { registerValidator } from '../core/registry.js';

export const noOverallocationValidator = {
  name: 'solver-no-overallocation',
  scope: 'PowerSolver',
  validate(response, _config) {
    const tasks = response.result?.tasks ?? [];
    const usage = new Map();
    for (const t of tasks) {
      usage.set(t.resourceId, (usage.get(t.resourceId) ?? 0) + t.duration);
    }
    for (const r of response.result?.resources ?? []) {
      const used = usage.get(r.id) ?? 0;
      if (used > r.capacity) {
        return { ok: false, message: `Resource ${r.id} overallocated: ${used} > ${r.capacity}` };
      }
    }
    return { ok: true, message: 'No resource overallocated' };
  },
};

2. Register it in the product module's setup:

export const powerSolverModule = {
  name: 'PowerSolver',
  validators: [
    noOverallocationValidator,
    // ...other product validators
  ],
  // ...
};

3. Reference it in any test case:

"validators": [
  { "name": "status", "config": { "expected": 200 } },
  { "name": "solver-no-overallocation", "config": {} }
]

4. Run the case. The new validator pill appears alongside the rest, green or red.

5. Set Up a Regression Baseline

Goal: lock in today's pass-set as the bar for tomorrow's deploys.

1. Pick the suite that should be green — usually the regression tag across all products.
2. Run it against prod. Confirm zero failures.
3. Note the run id (visible in the URL on the run detail page).
4. After the next deploy, re-run the same suite against prod.
5. Diff the new run against the baseline run id. regressions > 0 means do not roll forward.

Repeat after every successful deploy: yesterday's known-good run is today's baseline.

6. Test the L2 AI Pathway

L1 cases hit a product directly. L2 cases go through PowerOrchestrator's /ai/generate route, which means they exercise the AI router's choice of product and prompt assembly. Add an L2 case for a feature you care about:

{
  "id": "ai-route-to-forecast",
  "product": "PowerOrchestrator",
  "level": "L2",
  "endpoint": "/ai/generate",
  "method": "POST",
  "payload": {
    "prompt": "Forecast next 12 weeks of demand for SKU A using the last 2 years of weekly sales."
  },
  "validators": [
    { "name": "status",      "config": { "expected": 200 } },
    { "name": "field-match", "config": { "path": "result.routedTo", "expected": "PowerForecast" } },
    { "name": "schema",      "config": { "path": "result.forecast", "shape": "array" } }
  ],
  "tags": ["regression", "l2", "routing"]
}

Now any change to prompt templates, the router, or PowerForecast itself can flip this case — and you'll see the regression in the diff bucket immediately.

Troubleshooting