diff --git a/src/physcom/llm/prompts.py b/src/physcom/llm/prompts.py index e82d34d..521cfb1 100644 --- a/src/physcom/llm/prompts.py +++ b/src/physcom/llm/prompts.py @@ -47,9 +47,7 @@ match that magnitude, don't guess a generically "reasonable-looking" decimal. # ponytail: pass 4 only sees pass 2's raw numbers, not its reasoning. Sharpened # prompts on both sides closed most of the gap (a bad safety estimate went from # 0.95 to 0.80 on the same combo once pass 2 was told to consider combination- -# specific hazards), but pass 4 still doesn't reliably call out a contradiction -# by name when one remains — qwen2.5:7b doesn't follow that meta-instruction -# consistently. Upgrade path if this isn't good enough in practice: have +# specific hazards). Upgrade path if this isn't good enough in practice: have # estimate_physics() also return a short per-metric reason, persist it # alongside raw_value (new nullable column), and feed it into this prompt so # pass 4 has something concrete to agree or disagree with. Deferred because it @@ -57,27 +55,77 @@ match that magnitude, don't guess a generically "reasonable-looking" decimal. # pipeline + scorer + repository, and more generated tokens per combo. # # If we plan to LLM-review every p2 pass then maybe p2 and p4 should be combined. +# +# Calibration history: the original wording asked the model to weigh "novelty" +# and "genuinely interesting innovation or nonsense" as part of the verdict, +# which measurably biased weaker/harsher models toward IMPLAUSIBLE on ordinary, +# working concepts just for being unoriginal (gemma2:27b scored 4/8 on a +# labeled test set, wrongly rejecting an ordinary commuter car). Rewritten to: +# separate "interesting" from "viable" entirely, state explicitly that scores +# are 0-1 where higher is always better (models were misreading a normalized +# 0.85 as a bad sign), require a *specific, named* mechanism for IMPLAUSIBLE +# rather than a vague "seems risky," and lower the bar from "must be proven +# physically impossible" to "a well-reasoned specific danger is enough" (the +# original wording let a careful reasoner argue its way out of flagging a +# genuinely hazardous combo on a technicality). Deliberately does NOT ask the +# model to weigh whether the concept or something like it already exists in +# the real world — that's a bias on physical/safety judgment, not a physics +# argument, and it papers over rather than fixes weak per-metric reasoning. +# Post-fix labeled-set accuracy: phi4 8/8, qwq 8/8, gemma2:27b 4/8->7/8, +# qwen2.5:7b 5/8->4/8 (a real capability ceiling on this model for this +# judgment task, not a prompt regression -- left as-is rather than chasing +# further prompt-specific patches for one weak model). PLAUSIBILITY_REVIEW_PROMPT = """\ -You are reviewing a novel transportation concept for social and practical viability. +You are reviewing a transportation concept for real-world viability — could this +actually be built and operated safely. Whether it is new, exciting, or original +is NOT the question. ## Concept {description} ## Metric Scores +All scores below are normalized to 0-1, where HIGHER IS ALWAYS BETTER for +every metric listed, regardless of what the metric measures (this already +accounts for things like "lower cost is better" — you don't need to invert +anything). A score of 1.0 means excellent, not "pegged" or "maxed out badly." + {scores} -## Instructions -Review this concept for: -1. Social viability — would people actually use this? -2. Practical barriers — what engineering or regulatory obstacles exist? -3. Novelty — does anything similar already exist? -4. Overall plausibility — is this a genuinely interesting innovation or nonsense? -5. Consistency — if your assessment conflicts with any score above (e.g. you - consider this hazardous but its safety score is high), say so explicitly by - naming the metric and the discrepancy. Don't silently contradict a given - score in your reasoning without calling out that you're doing so. +## What makes something IMPLAUSIBLE +Mark IMPLAUSIBLE if either of these is true: +- It is physically or engineering-wise impossible given the components + described, OR +- Combining these SPECIFIC components creates a serious, specific danger + that goes beyond what either component already carries on its own — + this does NOT require proof of outright impossibility, a well-reasoned, + specific, serious danger is enough (e.g. repeated explosive recoil + fatiguing a hull over time is a real structural risk, not just "explosives + are dangerous in general"; a fuel that's fine in the open becoming + concentrated in a sealed tube is a real risk, not just "fuel is + flammable"). A low given safety score is a signal the pipeline already + found something concerning — treat it as evidence, not noise to explain + away. +- Or: a real regulatory/infrastructure barrier with no plausible workaround. -Provide a concise 2-4 sentence assessment, then on a final line write exactly: +None of these make something implausible on their own: +- being unoriginal or something like it already exists +- being expensive, slow, or short-range +- a single mediocre score on one metric that isn't safety-related + +Most concepts that reach this review are ordinary and workable; reserve +IMPLAUSIBLE for a real, specific problem you can name — but don't require +airtight proof of impossibility when the danger is already clear and specific. +Reason from the physics and engineering actually described here, not from +whether something like it already exists — novelty or lack of it is not +evidence either way. + +## What to write +In 2-4 sentences, give your reasoning, then check it against the scores +above: if your reasoning conflicts with a score (e.g. you believe this is +hazardous but its safety score is high), name the metric and say so +explicitly — don't silently contradict a given score. + +Finish with exactly one line: VERDICT: PLAUSIBLE or VERDICT: IMPLAUSIBLE