close guardrail gaps and fix the scoring pipeline top to bottom
Constraint resolver: aggregate mass/footprint across a combo instead of pairwise-only checks, treat medium/atmosphere as agreement not supply/demand, reduce multi-provider checks by best/sum instead of AND-ing every provider, fail closed on unrecognized mutex values, add a propulsion-viability (thrust-to-weight) rule. Seed data updated to match (nuclear/solar-sail footprint floors, water-medium exclusions, explicit ground/gravity providers). Domain metric units were stored globally per metric name instead of per-domain, silently corrupting cost_efficiency for every domain but the first one seeded — fixed with a schema migration. Stub estimator's cost_efficiency/safety/availability/reliability were a backwards formula and flat constants; replaced with heuristics grounded in each entity's thrust_profile/energy_form/infrastructure. LLM estimate_physics() now receives each metric's unit and expected range instead of a bare name, fixing wildly miscalibrated estimates traced back to the prompt's own hardcoded example anchoring the model to the wrong order of magnitude. Sharpened the safety-estimation and plausibility-review prompts. Deduped provider parsing logic into llm/parsing.py. Web pipeline form can now pick an LLM provider per run instead of only via server env var. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -1,5 +1,25 @@
|
||||
"""Prompt templates for LLM-assisted passes."""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from typing import TYPE_CHECKING
|
||||
|
||||
if TYPE_CHECKING:
|
||||
from physcom.models.domain import MetricBound
|
||||
|
||||
|
||||
def format_metrics_for_prompt(metrics: list["MetricBound"]) -> str:
|
||||
"""Render each metric with its unit and expected range, so the model
|
||||
anchors on the right order of magnitude instead of a generic decimal."""
|
||||
lines = []
|
||||
for mb in metrics:
|
||||
unit = mb.unit or "dimensionless"
|
||||
lines.append(
|
||||
f"- {mb.metric_name} ({unit}): typical range {mb.norm_min:g} to {mb.norm_max:g}"
|
||||
)
|
||||
return "\n".join(lines)
|
||||
|
||||
|
||||
PHYSICS_ESTIMATION_PROMPT = """\
|
||||
You are a physics estimation assistant. Given the following transportation concept, \
|
||||
estimate the requested metrics using order-of-magnitude physics reasoning.
|
||||
@@ -8,15 +28,35 @@ estimate the requested metrics using order-of-magnitude physics reasoning.
|
||||
{description}
|
||||
|
||||
## Metrics to estimate
|
||||
Each metric's unit and the typical range values fall in for this domain are given —
|
||||
match that magnitude, don't guess a generically "reasonable-looking" decimal.
|
||||
{metrics}
|
||||
|
||||
## Instructions
|
||||
- Use real-world physics to estimate each metric.
|
||||
- Use real-world physics to estimate each metric, in the exact unit given.
|
||||
- For "safety" specifically: consider hazards that arise from THIS combination's
|
||||
specific interactions — a fuel that's safe in an open vehicle can be far more
|
||||
dangerous inside a sealed tube or enclosed structure, a stable actuator on a
|
||||
fragile platform can be a real risk even if neither is risky alone. Don't just
|
||||
rate how safe the platform or actuator would be in isolation.
|
||||
- If the concept is implausible, still provide your best estimate.
|
||||
- Return ONLY valid JSON mapping metric names to numeric values.
|
||||
- Example: {{"power_density": 500.0, "cost_efficiency": 0.15, "safety": 0.7}}
|
||||
- Return ONLY valid JSON mapping metric names to numeric values, e.g.
|
||||
{{"some_metric": <number>, "another_metric": <number>}} — no explanatory text.
|
||||
"""
|
||||
|
||||
# ponytail: pass 4 only sees pass 2's raw numbers, not its reasoning. Sharpened
|
||||
# prompts on both sides closed most of the gap (a bad safety estimate went from
|
||||
# 0.95 to 0.80 on the same combo once pass 2 was told to consider combination-
|
||||
# specific hazards), but pass 4 still doesn't reliably call out a contradiction
|
||||
# by name when one remains — qwen2.5:7b doesn't follow that meta-instruction
|
||||
# consistently. Upgrade path if this isn't good enough in practice: have
|
||||
# estimate_physics() also return a short per-metric reason, persist it
|
||||
# alongside raw_value (new nullable column), and feed it into this prompt so
|
||||
# pass 4 has something concrete to agree or disagree with. Deferred because it
|
||||
# needs a schema/interface change across LLMProvider + both providers +
|
||||
# pipeline + scorer + repository, and more generated tokens per combo.
|
||||
#
|
||||
# If we plan to LLM-review every p2 pass then maybe p2 and p4 should be combined.
|
||||
PLAUSIBILITY_REVIEW_PROMPT = """\
|
||||
You are reviewing a novel transportation concept for social and practical viability.
|
||||
|
||||
@@ -32,6 +72,10 @@ Review this concept for:
|
||||
2. Practical barriers — what engineering or regulatory obstacles exist?
|
||||
3. Novelty — does anything similar already exist?
|
||||
4. Overall plausibility — is this a genuinely interesting innovation or nonsense?
|
||||
5. Consistency — if your assessment conflicts with any score above (e.g. you
|
||||
consider this hazardous but its safety score is high), say so explicitly by
|
||||
naming the metric and the discrepancy. Don't silently contradict a given
|
||||
score in your reasoning without calling out that you're doing so.
|
||||
|
||||
Provide a concise 2-4 sentence assessment, then on a final line write exactly:
|
||||
VERDICT: PLAUSIBLE
|
||||
|
||||
Reference in New Issue
Block a user