close guardrail gaps and fix the scoring pipeline top to bottom

Constraint resolver: aggregate mass/footprint across a combo instead of
pairwise-only checks, treat medium/atmosphere as agreement not supply/demand,
reduce multi-provider checks by best/sum instead of AND-ing every provider,
fail closed on unrecognized mutex values, add a propulsion-viability
(thrust-to-weight) rule. Seed data updated to match (nuclear/solar-sail
footprint floors, water-medium exclusions, explicit ground/gravity providers).

Domain metric units were stored globally per metric name instead of
per-domain, silently corrupting cost_efficiency for every domain but the
first one seeded — fixed with a schema migration.

Stub estimator's cost_efficiency/safety/availability/reliability were a
backwards formula and flat constants; replaced with heuristics grounded in
each entity's thrust_profile/energy_form/infrastructure.

LLM estimate_physics() now receives each metric's unit and expected range
instead of a bare name, fixing wildly miscalibrated estimates traced back to
the prompt's own hardcoded example anchoring the model to the wrong order of
magnitude. Sharpened the safety-estimation and plausibility-review prompts.
Deduped provider parsing logic into llm/parsing.py.

Web pipeline form can now pick an LLM provider per run instead of only via
server env var.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-26 00:13:16 -05:00
parent 63295ab80e
commit 434df718d7
18 changed files with 836 additions and 175 deletions

View File

@@ -1,5 +1,25 @@
"""Prompt templates for LLM-assisted passes."""
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from physcom.models.domain import MetricBound
def format_metrics_for_prompt(metrics: list["MetricBound"]) -> str:
"""Render each metric with its unit and expected range, so the model
anchors on the right order of magnitude instead of a generic decimal."""
lines = []
for mb in metrics:
unit = mb.unit or "dimensionless"
lines.append(
f"- {mb.metric_name} ({unit}): typical range {mb.norm_min:g} to {mb.norm_max:g}"
)
return "\n".join(lines)
PHYSICS_ESTIMATION_PROMPT = """\
You are a physics estimation assistant. Given the following transportation concept, \
estimate the requested metrics using order-of-magnitude physics reasoning.
@@ -8,15 +28,35 @@ estimate the requested metrics using order-of-magnitude physics reasoning.
{description}
## Metrics to estimate
Each metric's unit and the typical range values fall in for this domain are given —
match that magnitude, don't guess a generically "reasonable-looking" decimal.
{metrics}
## Instructions
- Use real-world physics to estimate each metric.
- Use real-world physics to estimate each metric, in the exact unit given.
- For "safety" specifically: consider hazards that arise from THIS combination's
specific interactions — a fuel that's safe in an open vehicle can be far more
dangerous inside a sealed tube or enclosed structure, a stable actuator on a
fragile platform can be a real risk even if neither is risky alone. Don't just
rate how safe the platform or actuator would be in isolation.
- If the concept is implausible, still provide your best estimate.
- Return ONLY valid JSON mapping metric names to numeric values.
- Example: {{"power_density": 500.0, "cost_efficiency": 0.15, "safety": 0.7}}
- Return ONLY valid JSON mapping metric names to numeric values, e.g.
{{"some_metric": <number>, "another_metric": <number>}} — no explanatory text.
"""
# ponytail: pass 4 only sees pass 2's raw numbers, not its reasoning. Sharpened
# prompts on both sides closed most of the gap (a bad safety estimate went from
# 0.95 to 0.80 on the same combo once pass 2 was told to consider combination-
# specific hazards), but pass 4 still doesn't reliably call out a contradiction
# by name when one remains — qwen2.5:7b doesn't follow that meta-instruction
# consistently. Upgrade path if this isn't good enough in practice: have
# estimate_physics() also return a short per-metric reason, persist it
# alongside raw_value (new nullable column), and feed it into this prompt so
# pass 4 has something concrete to agree or disagree with. Deferred because it
# needs a schema/interface change across LLMProvider + both providers +
# pipeline + scorer + repository, and more generated tokens per combo.
#
# If we plan to LLM-review every p2 pass then maybe p2 and p4 should be combined.
PLAUSIBILITY_REVIEW_PROMPT = """\
You are reviewing a novel transportation concept for social and practical viability.
@@ -32,6 +72,10 @@ Review this concept for:
2. Practical barriers — what engineering or regulatory obstacles exist?
3. Novelty — does anything similar already exist?
4. Overall plausibility — is this a genuinely interesting innovation or nonsense?
5. Consistency — if your assessment conflicts with any score above (e.g. you
consider this hazardous but its safety score is high), say so explicitly by
naming the metric and the discrepancy. Don't silently contradict a given
score in your reasoning without calling out that you're doing so.
Provide a concise 2-4 sentence assessment, then on a final line write exactly:
VERDICT: PLAUSIBLE