Update prompts.py

src/physcom/llm/prompts.py

@@ -47,9 +47,7 @@ match that magnitude, don't guess a generically "reasonable-looking" decimal.
 # ponytail: pass 4 only sees pass 2's raw numbers, not its reasoning. Sharpened
 # prompts on both sides closed most of the gap (a bad safety estimate went from
 # 0.95 to 0.80 on the same combo once pass 2 was told to consider combination-
-# specific hazards), but pass 4 still doesn't reliably call out a contradiction
+# specific hazards). Upgrade path if this isn't good enough in practice: have
-# by name when one remains — qwen2.5:7b doesn't follow that meta-instruction
-# consistently. Upgrade path if this isn't good enough in practice: have
 # estimate_physics() also return a short per-metric reason, persist it
 # alongside raw_value (new nullable column), and feed it into this prompt so
 # pass 4 has something concrete to agree or disagree with. Deferred because it
@@ -57,27 +55,77 @@ match that magnitude, don't guess a generically "reasonable-looking" decimal.
 # pipeline + scorer + repository, and more generated tokens per combo.
 #
 # If we plan to LLM-review every p2 pass then maybe p2 and p4 should be combined.
+#
+# Calibration history: the original wording asked the model to weigh "novelty"
+# and "genuinely interesting innovation or nonsense" as part of the verdict,
+# which measurably biased weaker/harsher models toward IMPLAUSIBLE on ordinary,
+# working concepts just for being unoriginal (gemma2:27b scored 4/8 on a
+# labeled test set, wrongly rejecting an ordinary commuter car). Rewritten to:
+# separate "interesting" from "viable" entirely, state explicitly that scores
+# are 0-1 where higher is always better (models were misreading a normalized
+# 0.85 as a bad sign), require a *specific, named* mechanism for IMPLAUSIBLE
+# rather than a vague "seems risky," and lower the bar from "must be proven
+# physically impossible" to "a well-reasoned specific danger is enough" (the
+# original wording let a careful reasoner argue its way out of flagging a
+# genuinely hazardous combo on a technicality). Deliberately does NOT ask the
+# model to weigh whether the concept or something like it already exists in
+# the real world — that's a bias on physical/safety judgment, not a physics
+# argument, and it papers over rather than fixes weak per-metric reasoning.
+# Post-fix labeled-set accuracy: phi4 8/8, qwq 8/8, gemma2:27b 4/8->7/8,
+# qwen2.5:7b 5/8->4/8 (a real capability ceiling on this model for this
+# judgment task, not a prompt regression -- left as-is rather than chasing
+# further prompt-specific patches for one weak model).
 PLAUSIBILITY_REVIEW_PROMPT = """\
-You are reviewing a novel transportation concept for social and practical viability.
+You are reviewing a transportation concept for real-world viability — could this
+actually be built and operated safely. Whether it is new, exciting, or original
+is NOT the question.
 
 ## Concept
 {description}
 
 ## Metric Scores
+All scores below are normalized to 0-1, where HIGHER IS ALWAYS BETTER for
+every metric listed, regardless of what the metric measures (this already
+accounts for things like "lower cost is better" — you don't need to invert
+anything). A score of 1.0 means excellent, not "pegged" or "maxed out badly."
+
 {scores}
 
-## Instructions
+## What makes something IMPLAUSIBLE
-Review this concept for:
+Mark IMPLAUSIBLE if either of these is true:
-1. Social viability — would people actually use this?
+- It is physically or engineering-wise impossible given the components
-2. Practical barriers — what engineering or regulatory obstacles exist?
+  described, OR
-3. Novelty — does anything similar already exist?
+- Combining these SPECIFIC components creates a serious, specific danger
-4. Overall plausibility — is this a genuinely interesting innovation or nonsense?
+  that goes beyond what either component already carries on its own —
-5. Consistency — if your assessment conflicts with any score above (e.g. you
+  this does NOT require proof of outright impossibility, a well-reasoned,
-   consider this hazardous but its safety score is high), say so explicitly by
+  specific, serious danger is enough (e.g. repeated explosive recoil
-   naming the metric and the discrepancy. Don't silently contradict a given
+  fatiguing a hull over time is a real structural risk, not just "explosives
-   score in your reasoning without calling out that you're doing so.
+  are dangerous in general"; a fuel that's fine in the open becoming
+  concentrated in a sealed tube is a real risk, not just "fuel is
+  flammable"). A low given safety score is a signal the pipeline already
+  found something concerning — treat it as evidence, not noise to explain
+  away.
+- Or: a real regulatory/infrastructure barrier with no plausible workaround.
 
-Provide a concise 2-4 sentence assessment, then on a final line write exactly:
+None of these make something implausible on their own:
+- being unoriginal or something like it already exists
+- being expensive, slow, or short-range
+- a single mediocre score on one metric that isn't safety-related
+
+Most concepts that reach this review are ordinary and workable; reserve
+IMPLAUSIBLE for a real, specific problem you can name — but don't require
+airtight proof of impossibility when the danger is already clear and specific.
+Reason from the physics and engineering actually described here, not from
+whether something like it already exists — novelty or lack of it is not
+evidence either way.
+
+## What to write
+In 2-4 sentences, give your reasoning, then check it against the scores
+above: if your reasoning conflicts with a score (e.g. you believe this is
+hazardous but its safety score is high), name the metric and say so
+explicitly — don't silently contradict a given score.
+
+Finish with exactly one line:
 VERDICT: PLAUSIBLE
 or
 VERDICT: IMPLAUSIBLE