Andrew Simonson
76f460499a
safety and availability don't reduce to physics formulas the way power_density/range_fuel/cost_efficiency do -- they're judgment calls (risk assessment, infrastructure prevalence), and running them through the same log-normalize() built for physical quantities produced incoherent results: safety's raw value is already a "0-1" score, and normalizing it again turned 0.6 into an unexplainable 0.678 that even the LLM reviewing it could only cite, never justify (see combo 1540). Removed both from domain_metric_weights (safety from 4 domains, availability from urban_commuting) and renormalized the remaining weights to sum to 1.0. Pass 4 now produces one holistic RATING (LOW/MEDIUM/HIGH) alongside the existing VERDICT, with safety and accessibility folded in as qualitative considerations feeding that single judgment rather than scored separately -- not a checklist of independent numbers. New qualitative_rating column, filterable in the results UI. Also added domain name/description to the review prompt so the LLM judges a metric like range against what the domain actually needs (urban_commuting: 1-50km) instead of generic real-world expectations for the platform category -- confirmed live on a combo where phi4 had called a 396km range "limited" by comparing to typical aircraft rather than a domain that needs 1-50km. Pass 2 is estimator-only now -- self.llm is never consulted there, reserved entirely for pass 4. Restructured Pipeline.run() from combo-first to phase-parallel: each pass now runs to completion across every combo before the next pass starts, rather than walking each combo through all four passes before the next combo. This surfaced a real bug: domain- blocked combos (status stays "valid" by design, not "_fail") were slipping past a naive status-based skip guard and getting silently re-processed by pass 2. Fixed with a shared dead-combo check that catches both generic failures and domain blocks correctly. Also fixes a results-page display bug found while reviewing a live combo: the per-metric "position" bar showed raw distance from norm_min without inverting for lower_is_better metrics, so an excellent cost score (near the good end) rendered as a ~0%, near-empty bar -- looked bad next to its own 0.99 normalized score. Validated live against phi4 (real Ollama calls, not mocked): full-domain phase-parallel run (2,970 combos, estimator-only p2, 1.6s) followed by a real pass-4 run (111 reviewed, 11m, 0 crashes, 0 null ratings). Two tests that relied on the old LLM-driven pass 2 to force deterministic outcomes were updated to test pass 4's verdict-wiring directly instead. All 100 tests pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
38 lines
1.4 KiB
Python
38 lines
1.4 KiB
Python
"""Shared response-parsing helpers for LLM providers."""
|
|
|
|
from __future__ import annotations
|
|
|
|
import json
|
|
import re
|
|
|
|
from physcom.models.domain import MetricBound
|
|
|
|
|
|
def parse_verdict(text: str) -> bool:
|
|
"""Extract VERDICT: PLAUSIBLE/IMPLAUSIBLE from response; default to True."""
|
|
m = re.search(r"VERDICT:\s*(PLAUSIBLE|IMPLAUSIBLE)", text, re.IGNORECASE)
|
|
if m:
|
|
return m.group(1).upper() == "PLAUSIBLE"
|
|
return True
|
|
|
|
|
|
def parse_rating(text: str) -> str | None:
|
|
"""Extract RATING: LOW/MEDIUM/HIGH from response; None if absent (older
|
|
reviews saved before this existed, or a malformed response)."""
|
|
m = re.search(r"RATING:\s*(LOW|MEDIUM|HIGH)", text, re.IGNORECASE)
|
|
return m.group(1).upper() if m else None
|
|
|
|
|
|
def parse_metric_json(text: str, metrics: list[MetricBound]) -> dict[str, float]:
|
|
"""Strip markdown fences and parse JSON; fall back to each metric's own
|
|
norm_min/norm_max midpoint on error — a flat constant like 0.5 is
|
|
guaranteed wrong-magnitude for at least some metrics regardless of unit.
|
|
"""
|
|
names = {mb.metric_name for mb in metrics}
|
|
text = re.sub(r"```(?:json)?\s*", "", text).strip().rstrip("`").strip()
|
|
try:
|
|
data = json.loads(text)
|
|
return {k: float(v) for k, v in data.items() if k in names}
|
|
except (json.JSONDecodeError, ValueError, TypeError):
|
|
return {mb.metric_name: (mb.norm_min + mb.norm_max) / 2 for mb in metrics}
|