Compare commits

..

2 Commits

Author SHA1 Message Date
730a23bac3 give pass 4 raw values, batch deferrable commits, fix two known bugs
Pass 4's plausibility review only ever saw normalized scores, never the
raw physical estimate behind them -- confirmed via live testing this was
exactly what caused a real misfire (gemma2:27b cited a real cyclist's
correct 5 W/kg, log-normalized to "0.159" against a car's power scale, as
grounds for rejecting an ordinary bicycle). review_plausibility now takes
raw_metrics + normalized_scores + metric units, and the prompt explicitly
instructs reasoning from the raw value first. Verified live against phi4:
it now cites the actual raw number and correctly explains why a low
normalized score doesn't mean the estimate or concept is bad.

Repository write methods used in the pipeline's hot path now take an
optional commit=False, and Pipeline defers commits during the fast/
deterministic passes (1, 3, and 2 without an LLM), flushing every 200
combos and on any exit path (finally block covers normal completion,
cancellation, and any other exception). LLM-involving calls (pass 2 with
an LLM, all of pass 4) still commit immediately -- those are slow and
crash-prone and worth protecting per-write; the deterministic passes
aren't, and recomputing them is now measured at under a second for the
full domain rather than worth 8,000+ individual fsync'd commits. Full
2,970-combination domain run: multiple minutes -> 0.91s. Test suite:
~70s -> ~15s.

Also fixes two more issues found while auditing the estimator for a real
run: CARGO_KG_PER_STRUCTURAL_KG was 500 (no real vehicle carries 500x its
own structural mass in cargo -- a magnitude bug, not a modeling choice),
corrected to 2.5. And space/rocket platforms' range_fuel now reports the
domain's ceiling instead of an arbitrary placeholder constant -- vacuum
coast isn't resistance-limited, so "distance before running out of fuel"
isn't a meaningful question for these the way it is for ground/air/water
vehicles; the real constraint is delta-v budget, a different metric this
pass doesn't model.

Validated with a live full-domain run (phi4, real Ollama calls): 115
reviewed, 0 malformed/null reviews, 0 verdict-vs-status mismatches.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-15 15:23:45 -05:00
be25a837ff replace stub estimator with a requirement-derived physics engine
pass 2's no-LLM fallback previously used broken/placeholder formulas:
power_density passed an actuator's own intensive W/kg straight through
without scaling by vehicle mass, range_fuel multiplied energy_density by a
flat unitless constant, and cost_efficiency was a categorical guess.

Replaces all three with formulas grounded in each entity's own declared
attributes. Actuator and storage mass are sized to what's actually
necessary -- enough power to sustain a platform's target_velocity against
resistance (or real thrust/accel requirements where already declared, for
aircraft/rocket combos), enough energy to reach the domain's own declared
range ceiling -- solved as a closed-form 2x2 linear system rather than an
invented mass-fraction table. Platform mass uses the geometric mean of its
declared range instead of the bare floor, since a category as broad as
Road Vehicle (50kg-36,000kg) is closer to log-uniformly distributed than
uniformly distributed. cost_efficiency is now real operating cost (energy
price x resistance) plus amortized upfront cost (materials cost by medium
x lifetime distance), replacing the old flat per-energy-form guess.

Also fixes two bugs found while validating the above against real-world
reference values: entities whose power source isn't their own carried mass
(Human Muscle, Solar Sail) degenerated to zero power; and combos blocked
by a domain-specific constraint kept combinations.status stuck at "valid"
forever, which silently miscounted them as passing in two repository
queries (count_combinations_by_status, get_pipeline_summary) even though
the per-domain result row was correctly marked blocked.

Adds target_velocity to platforms that had no declared performance
requirement at all, and raises OllamaLLMProvider's HTTP timeout (120s to
300s) to match observed real review-call latency.

Validated against 4 real-world reference combos (commuter car, bicycle,
delivery drone) and a full 2,970-combination domain run (0 pass-2
failures); all 100 existing tests still pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-15 13:45:01 -05:00
8 changed files with 521 additions and 98 deletions

View File

@@ -20,6 +20,14 @@ class Repository:
self.conn = conn
self.conn.row_factory = sqlite3.Row
def commit(self) -> None:
"""Explicit flush point, for callers batching writes with commit=False
below (see Pipeline.run: instant/deterministic passes defer commits
and flush in bulk, since a crash there just means cheap recompute;
LLM-call results still commit immediately, since those are slow/
expensive to redo)."""
self.conn.commit()
# ── Dimensions ──────────────────────────────────────────────
def ensure_dimension(self, name: str, description: str = "") -> int:
@@ -422,7 +430,7 @@ class Repository:
key = ",".join(str(eid) for eid in sorted(entity_ids))
return hashlib.sha256(key.encode()).hexdigest()[:16]
def save_combination(self, combination: Combination) -> Combination:
def save_combination(self, combination: Combination, commit: bool = True) -> Combination:
entity_ids = [e.id for e in combination.entities]
combination.hash = self.compute_hash(entity_ids)
@@ -446,11 +454,12 @@ class Repository:
"INSERT INTO combination_entities (combination_id, entity_id) VALUES (?, ?)",
(combination.id, eid),
)
self.conn.commit()
if commit:
self.conn.commit()
return combination
def update_combination_status(
self, combo_id: int, status: str, block_reason: str | None = None
self, combo_id: int, status: str, block_reason: str | None = None, commit: bool = True
) -> None:
# Don't downgrade from higher pass states — preserves human/LLM review data
if status in ("scored", "llm_reviewed") or status.endswith("_fail"):
@@ -470,7 +479,8 @@ class Repository:
"UPDATE combinations SET status = ?, block_reason = ? WHERE id = ?",
(status, block_reason, combo_id),
)
self.conn.commit()
if commit:
self.conn.commit()
def get_combination(self, combo_id: int) -> Combination | None:
row = self.conn.execute("SELECT * FROM combinations WHERE id = ?", (combo_id,)).fetchone()
@@ -558,6 +568,7 @@ class Repository:
combo_id: int,
domain_id: int,
scores: list[dict],
commit: bool = True,
) -> None:
"""Save per-metric scores. Each dict: metric_id, raw_value, normalized_score, estimation_method, confidence."""
for s in scores:
@@ -569,7 +580,8 @@ class Repository:
(combo_id, domain_id, s["metric_id"], s["raw_value"],
s["normalized_score"], s["estimation_method"], s["confidence"]),
)
self.conn.commit()
if commit:
self.conn.commit()
def save_result(
self,
@@ -581,6 +593,7 @@ class Repository:
llm_review: str | None = None,
human_notes: str | None = None,
domain_block_reason: str | None = None,
commit: bool = True,
) -> None:
self.conn.execute(
"""INSERT OR REPLACE INTO combination_results
@@ -590,7 +603,8 @@ class Repository:
(combo_id, domain_id, composite_score, novelty_flag,
llm_review, human_notes, pass_reached, domain_block_reason),
)
self.conn.commit()
if commit:
self.conn.commit()
def get_combination_scores(self, combo_id: int, domain_id: int) -> list[dict]:
"""Return per-metric scores for a combination in a domain."""
@@ -606,13 +620,19 @@ class Repository:
def count_combinations_by_status(self, domain_name: str | None = None) -> dict[str, int]:
"""Count combos by status. If domain_name given, only combos with results in that domain."""
if domain_name:
# combinations.status is domain-agnostic (a combo can be "valid"
# generically but blocked by one domain's own constraints), so a
# domain-scoped count must bucket domain_block_reason rows on
# their own rather than trusting c.status.
rows = self.conn.execute(
"""SELECT c.status, COUNT(*) as cnt
"""SELECT CASE WHEN cr.domain_block_reason IS NOT NULL
THEN 'domain_blocked' ELSE c.status END as status,
COUNT(*) as cnt
FROM combination_results cr
JOIN combinations c ON cr.combination_id = c.id
JOIN domains d ON cr.domain_id = d.id
WHERE d.name = ?
GROUP BY c.status""",
GROUP BY status""",
(domain_name,),
).fetchall()
else:
@@ -641,7 +661,8 @@ class Repository:
FROM combinations c
JOIN combination_results cr ON cr.combination_id = c.id
JOIN domains d ON cr.domain_id = d.id
WHERE c.status LIKE '%\\_fail' ESCAPE '\\' AND d.name = ?""",
WHERE (c.status LIKE '%\\_fail' ESCAPE '\\' OR cr.domain_block_reason IS NOT NULL)
AND d.name = ?""",
(domain_name,),
).fetchone()
return {
@@ -672,8 +693,10 @@ class Repository:
JOIN domains d ON cr.domain_id = d.id
WHERE d.name = ?"""
params: list = [domain_name]
if status:
query += " AND c.status = ?"
if status == "domain_blocked":
query += " AND cr.domain_block_reason IS NOT NULL"
elif status:
query += " AND c.status = ? AND cr.domain_block_reason IS NULL"
params.append(status)
query += " ORDER BY cr.composite_score DESC"
rows = self.conn.execute(query, params).fetchall()
@@ -798,7 +821,7 @@ class Repository:
return row["pass_reached"] if row else None
def save_raw_estimates(
self, combo_id: int, domain_id: int, estimates: list[dict]
self, combo_id: int, domain_id: int, estimates: list[dict], commit: bool = True
) -> None:
"""Save raw metric estimates (pass 2) with normalized_score=NULL.
@@ -813,7 +836,8 @@ class Repository:
(combo_id, domain_id, e["metric_id"], e["raw_value"],
e["estimation_method"], e["confidence"]),
)
self.conn.commit()
if commit:
self.conn.commit()
def get_existing_result(self, combo_id: int, domain_id: int) -> dict | None:
"""Return the full combination_results row for resume logic."""

View File

@@ -2,6 +2,7 @@
from __future__ import annotations
import math
import time
from dataclasses import dataclass, field
from datetime import datetime, timezone
@@ -18,18 +19,6 @@ from physcom.models.domain import Domain, MetricBound
# Keyed by the same categorical vocabulary already used in seed data — never
# by entity name, so new entities inherit sensible behavior automatically.
# How controllable a thrust delivery profile is — bursty/extreme profiles are
# harder to control and cost more per use (ammunition, propellant, wear) than
# steady ones. Missing values fall back to a neutral 1.0/0.6.
THRUST_PROFILE_COST_MULTIPLIER: dict[str, float] = {
"low_continuous": 1.0,
"continuous_low": 1.0,
"moderate_continuous": 1.1,
"high_continuous": 1.3,
"extreme_continuous": 1.6,
"high_burst": 2.5,
"extreme_burst": 4.0,
}
THRUST_PROFILE_SAFETY: dict[str, float] = {
"low_continuous": 0.9,
"continuous_low": 0.9,
@@ -56,25 +45,6 @@ ENERGY_FORM_SAFETY: dict[str, float] = {
"nuclear_thermal": 0.35,
}
# Rough $/m base cost by energy form — renewables/muscle power are ~free,
# consumables (propellant, ammunition, nuclear fuel) cost real money per use.
# This is a categorical placeholder, not a physics formula — energy_density
# (J/kg) can't give a $/m figure on its own since it says nothing about price.
ENERGY_FORM_BASE_COST: dict[str, float] = {
"wind": 1e-6,
"gravitational": 1e-6,
"radiation_pressure": 1e-6,
"electrical": 3e-5,
"kinetic_stored": 2e-5,
"biological": 5e-5,
"pneumatic": 4e-5,
"chemical_combustible": 8e-5,
"nuclear_thermal": 1e-3,
"ion_propellant": 2e-3,
"chemical_propellant": 5e-3,
"chemical_explosive": 8e-3,
}
# How available the required infrastructure/fuel supply chain is today.
# Multiple matches in one combo (e.g. a platform's road_network requirement
# plus a storage's fuel_infrastructure requirement) are averaged.
@@ -99,8 +69,12 @@ INFRASTRUCTURE_AVAILABILITY: dict[tuple[str, str], float] = {
("fuel_infrastructure", "xenon_propellant"): 0.05,
}
# Crude freight-capacity proxy: kg of cargo per kg of vehicle structural mass.
CARGO_KG_PER_STRUCTURAL_KG: float = 500
# Crude freight-capacity proxy: kg of cargo per kg of vehicle structural
# mass. Was 500 -- a magnitude error (500x cargo-to-structure has no real
# vehicle analog). Real cargo ships run deadweight/lightship ratios of
# roughly 1.5-4x depending on class; 2.5 is a reasonable general-cargo
# midpoint for this domain-agnostic proxy.
CARGO_KG_PER_STRUCTURAL_KG: float = 2.5
# How mechanically proven/predictable an energy form is in practice — distinct
# from safety (risk when something goes wrong) and thrust_profile (delivery
@@ -120,6 +94,181 @@ ENERGY_FORM_RELIABILITY: dict[str, float] = {
"chemical_explosive": 0.45,
}
# ── power_density / range_fuel / cost_efficiency ──────────────────────
# These three use the platform's declared mass envelope as a combo-wide
# budget: every component (platform, actuator, storage) is bounded below
# by its own mass range_min, and the SUM is bounded above by the
# platform's mass range_max -- the exact aggregate check ConstraintResolver
# already performs in pass 1. The "leanest legal build" (every component
# at its own floor) is always a legal design point (pass 1 already
# validated it against the platform's ceiling), so it's used as the point
# estimate rather than an invented one.
# Human/animal actuators correctly declare mass_min=0 (a rider's body isn't
# purchasable vehicle-borne mass and must not compete for the platform's
# mass budget), but that same 0 breaks power = power_density * mass. Fix:
# a fixed physiological reference mass used only in the power formula,
# added to -- never substituted into -- the vehicle's own mass budget.
BIOLOGICAL_OPERATOR_MASS_KG: dict[str, float] = {
"biological": 70.0, # human rider; Animal Traction shares this form too
}
# A platform's declared mass range often spans a whole real-world class, not
# one archetype -- Road Vehicle alone covers 50kg (motorcycle) to 36,000kg
# (truck). The floor is a legal build (pass 1 already checked it), but it's
# a motorcycle-scale build, not what a combo's own description usually
# implies. The geometric mean (not arithmetic) is the representative point
# for a range this wide: sqrt(50 * 36000) ~= 1343kg, in real commuter-car
# territory, versus the arithmetic mean (~18,000kg, a semi truck) or the
# floor (50kg, a motorcycle) -- real-world vehicle classes are far closer to
# log-uniformly distributed across a category than uniformly distributed.
def _representative_mass(mass_min: float, mass_max: float | None) -> float:
if mass_max and mass_min > 0:
return math.sqrt(mass_min * mass_max)
return max(mass_min, 100.0)
# Actuator + storage mass, sized to what's actually necessary rather than a
# fixed fraction of platform mass: enough actuator to sustain the
# platform's own performance requirement, enough storage to carry the
# domain's own "good" range target. Both share total_mass = p_rep + a + s,
# so the two requirements are coupled -- solved as a 2x2 linear system
# (Cramer's rule), not an iterative fit or an invented ratio:
#
# C1 * a = R1 * (p_rep + a + s) [a's capability meets requirement R1]
# C2 * s = R2 * (p_rep + a + s) [s's capability meets requirement R2]
#
# For the actuator equation, C1/R1 is either (specific_thrust, min_effective_accel)
# when the platform declares a real acceleration floor and the actuator
# declares real thrust (F=ma, both already exist in the seed data for
# aircraft/rocket combos -- no new data needed there), or (power_density,
# specific_energy_consumption * target_velocity) as the fallback -- "enough
# power to hold target_velocity against resistance" -- for every other
# platform, which needed one new attribute (target_velocity) since nothing
# in the schema previously declared a design speed for ground/water craft.
# For the storage equation, C2/R2 is always (energy_density, domain's own
# declared range_fuel norm_max * specific_energy_consumption) -- "enough
# energy to reach a genuinely good range for this domain," reusing the
# domain's own scoring ceiling rather than inventing a target.
#
# An infeasible system (the actuator is fundamentally too weak to ever
# reach the requirement, C1 <= R1) or a domain/platform missing the inputs
# it needs falls back to the entities' own bare floors -- a real
# limitation, not something to paper over with a default.
# Steady-state resistance (SPECIFIC_ENERGY_CONSUMPTION_J_PER_KG_M) only
# covers holding target_velocity -- real vehicles also carry reserve force
# for acceleration events (merging, passing, hills) that a pure cruise
# calculation would leave out entirely, which is why sizing off resistance
# alone undersizes the actuator relative to real vehicles. ~1.2 m/s^2 is a
# modest, real merging/passing acceleration capability, not a car's 0-60
# figure -- added directly to the resistance term below (see call site).
ACCELERATION_RESERVE_M_S2: float = 2.6
def _solve_two_requirement_masses(
p_rep: float, c1: float, r1: float, c2: float, r2: float,
a_min: float, s_min: float,
) -> tuple[float, float]:
a11, a12, b1 = c1 - r1, -r1, r1 * p_rep
a21, a22, b2 = -r2, c2 - r2, r2 * p_rep
det = a11 * a22 - a12 * a21
if abs(det) < 1e-9:
return a_min, s_min
a = (b1 * a22 - a12 * b2) / det
s = (a11 * b2 - a21 * b1) / det
if a <= 0 or s <= 0:
return a_min, s_min
return max(a, a_min), max(s, s_min)
# Ambient energy forms (sun, wind, gravity, food) aren't a depletable
# onboard store the way a fuel tank is -- "distance before running out"
# doesn't apply (a sailboat doesn't run out of wind). Rather than
# degenerate to 0 (mass_min=0, energy_density often undeclared entirely),
# range_fuel reports the domain's own declared ceiling for these: full
# marks is the physically honest answer, not an error.
AMBIENT_ENERGY_FORMS: set[str] = {"biological", "wind", "radiation_pressure", "gravitational"}
# Resistive energy cost of travel, J per kg of vehicle per meter --
# rolling resistance for ground vehicles, cruise-flight lift/drag for
# aircraft, hull drag for water. Keyed by the platform's declared `medium`,
# not per-platform -- a real train's steel-wheel-on-rail is far more
# efficient than a car's tire, both currently "ground" -- flagged as the
# coarsest approximation here, same spot the earlier LLM comparison found
# every model's range_fuel guess off by 10-25x from real vehicles.
SPECIFIC_ENERGY_CONSUMPTION_J_PER_KG_M: dict[str, float] = {
"ground": 0.016 * 9.81, # combined rolling + aero "road load", Crr-equivalent ~ 0.016
"air": 9.81 / 10, # cruise flight, effective L/D ~ 10
"water": 0.05 * 9.81, # displacement-hull drag, rough order of magnitude
}
# Rocket-propelled (space medium) platforms aren't resistance-limited at
# all -- no drag to fight in vacuum -- so this "energy / (resistance *
# mass)" shape is the wrong model for them; real range is governed by the
# rocket equation (delta-v = exhaust velocity * ln(mass ratio)), which this
# pass does not implement. Space is deliberately left out of the dict above
# so it falls through to the old placeholder formula in the code below
# rather than silently claiming a resistance-based number that isn't real.
# Structural manufacturing cost, $ per kg of platform mass -- certification
# and materials overhead scale hugely by medium (aerospace-grade vs.
# automotive steel vs. spacecraft-grade).
STRUCTURAL_COST_PER_KG_BY_MEDIUM: dict[str, float] = {
"ground": 8.0,
"air": 400.0,
"water": 15.0,
"space": 8000.0,
}
# Hardware manufacturing cost, $ per kg of actuator/storage-hardware mass,
# by energy form -- mature mass-produced tech (combustion, electric) is
# cheap per kg; exotic/regulated tech (nuclear, ion, rocket-grade) is not.
# biological is 0: there's no hardware to manufacture, the "actuator" is
# the operator's own body.
HARDWARE_COST_PER_KG_BY_ENERGY_FORM: dict[str, float] = {
"biological": 0.0,
"wind": 20.0,
"gravitational": 30.0,
"pneumatic": 35.0,
"chemical_combustible": 40.0,
"electrical": 60.0,
"kinetic_stored": 80.0,
"chemical_explosive": 150.0,
"chemical_propellant": 300.0,
"radiation_pressure": 500.0,
"ion_propellant": 5000.0,
"nuclear_thermal": 20000.0,
}
# Consumable energy price, $ per MJ delivered. Ambient sources (sun, wind,
# gravity) are genuinely free; food is a real recurring cost even though it
# isn't range-limiting -- cost and range are different questions, see
# AMBIENT_ENERGY_FORMS above. Replaces the old flat $/m ENERGY_FORM_BASE_COST
# placeholder with a real energy-priced figure.
FUEL_PRICE_PER_MJ: dict[str, float] = {
"wind": 0.0,
"gravitational": 0.0,
"radiation_pressure": 0.0,
"biological": 0.03,
"nuclear_thermal": 0.01,
"chemical_combustible": 0.04,
"electrical": 0.04,
"pneumatic": 0.02,
"kinetic_stored": 0.0,
"chemical_propellant": 1.0,
"chemical_explosive": 2.0,
"ion_propellant": 5.0,
}
# Total distance a vehicle travels over its operational life, used to
# amortize upfront/hardware cost into a $/m figure alongside operating
# cost. Coarse (per-medium, like the resistance table above) -- flagged as
# the same class of approximation.
LIFETIME_DISTANCE_M_BY_MEDIUM: dict[str, float] = {
"ground": 150_000_000.0,
"air": 3_000_000_000.0,
"water": 1_000_000_000.0,
"space": 5_000_000_000.0,
}
@dataclass
class PipelineResult:
@@ -222,9 +371,13 @@ class Pipeline:
combos = generate_combinations(self.repo, dimensions)
result.total_generated = len(combos)
# Save all combinations to DB (also loads status for existing combos)
# Save all combinations to DB (also loads status for existing combos).
# Deferred commit -- registering combos is instant/deterministic, so a
# crash here just means re-running the (cheap) registration loop, not
# losing anything worth protecting with a commit per row.
for combo in combos:
self.repo.save_combination(combo)
self.repo.save_combination(combo, commit=False)
self.repo.commit()
if run_id is not None:
self.repo.update_pipeline_run(run_id, total_combos=len(combos))
@@ -233,9 +386,20 @@ class Pipeline:
bounds_by_name = {mb.metric_name: mb for mb in domain.metric_bounds}
# ── Combo-first loop ─────────────────────────────────────
# Deterministic passes (1, 3, and 2 without an LLM) defer commits and
# get flushed periodically + in `finally` below -- a crash there costs
# a cheap recompute, not lost work worth committing per write. Pass 4
# (and pass 2 with an LLM) commit immediately after each call: those
# are slow and crash-prone (see the QwQ timeout saga), so that result
# is worth protecting the moment it lands.
combos_since_commit = 0
try:
for combo in combos:
self._check_cancelled(run_id)
combos_since_commit += 1
if combos_since_commit >= 200:
self.repo.commit()
combos_since_commit = 0
# Check existing progress for this combo in this domain
existing_pass = self.repo.get_combo_pass_reached(
@@ -254,7 +418,7 @@ class Pipeline:
combo.status = "p1_fail"
combo.block_reason = "; ".join(cr.violations)
self.repo.update_combination_status(
combo.id, "p1_fail", combo.block_reason
combo.id, "p1_fail", combo.block_reason, commit=False
)
# Save a result row so failed combos appear in results
self.repo.save_result(
@@ -262,15 +426,20 @@ class Pipeline:
domain.id,
composite_score=0.0,
pass_reached=1,
commit=False,
)
result.pass1_failed += 1
self._update_run_counters(run_id, result, current_pass=1)
continue # p1_fail — skip remaining passes
else:
combo.status = "valid"
self.repo.update_combination_status(combo.id, "valid")
self.repo.update_combination_status(combo.id, "valid", commit=False)
# Domain constraint check (per-domain block only)
# Domain constraint check (per-domain block only). combo.status
# stays "valid" here on purpose: it's domain-agnostic and the
# same combo can be blocked in this domain but valid in another.
# The per-domain block lives on combination_results.domain_block_reason
# (see count_combinations_by_status / get_all_results, which bucket on it).
if domain.constraints:
dc_result = self.resolver.check_domain_constraints(
combo, domain.constraints
@@ -282,6 +451,7 @@ class Pipeline:
domain_block_reason="; ".join(
dc_result.violations
),
commit=False,
)
result.pass1_failed += 1
self._update_run_counters(
@@ -333,9 +503,13 @@ class Pipeline:
"estimation_method": "llm" if self.llm else "stub",
"confidence": 1.0,
})
# LLM-produced estimates commit immediately (slow/crash-
# prone, worth protecting); stub estimates are instant
# and defer, same as the rest of the deterministic passes.
used_llm = self.llm is not None
if estimate_dicts:
self.repo.save_raw_estimates(
combo.id, domain.id, estimate_dicts
combo.id, domain.id, estimate_dicts, commit=used_llm
)
# Check for all-zero estimates → p2_fail
@@ -343,11 +517,12 @@ class Pipeline:
combo.status = "p2_fail"
combo.block_reason = "All metric estimates are zero"
self.repo.update_combination_status(
combo.id, "p2_fail", combo.block_reason
combo.id, "p2_fail", combo.block_reason, commit=used_llm
)
self.repo.save_result(
combo.id, domain.id,
composite_score=0.0, pass_reached=2,
commit=used_llm,
)
result.pass2_failed += 1
self._update_run_counters(run_id, result, current_pass=2)
@@ -385,7 +560,7 @@ class Pipeline:
"confidence": s.confidence,
})
if score_dicts:
self.repo.save_scores(combo.id, domain.id, score_dicts)
self.repo.save_scores(combo.id, domain.id, score_dicts, commit=False)
# Preserve existing human data
novelty_flag = (
@@ -401,6 +576,7 @@ class Pipeline:
sr.composite_score, pass_reached=3,
novelty_flag=novelty_flag,
human_notes=human_notes,
commit=False,
)
combo.status = "p3_fail"
combo.block_reason = (
@@ -408,7 +584,7 @@ class Pipeline:
f"below threshold {score_threshold}"
)
self.repo.update_combination_status(
combo.id, "p3_fail", combo.block_reason
combo.id, "p3_fail", combo.block_reason, commit=False
)
result.pass3_failed += 1
result.pass3_scored += 1
@@ -422,8 +598,9 @@ class Pipeline:
pass_reached=3,
novelty_flag=novelty_flag,
human_notes=human_notes,
commit=False,
)
self.repo.update_combination_status(combo.id, "scored")
self.repo.update_combination_status(combo.id, "scored", commit=False)
result.pass3_scored += 1
result.pass3_above_threshold += 1
@@ -459,16 +636,21 @@ class Pipeline:
for s in db_scores
if s["normalized_score"] is not None
}
raw_dict = {
s["metric_name"]: s["raw_value"]
for s in db_scores
if s["raw_value"] is not None
}
review_result: tuple[str, bool] | None = None
try:
review_result = self.llm.review_plausibility(
description, score_dict
description, raw_dict, score_dict, domain.metric_bounds
)
except LLMRateLimitError as exc:
self._wait_for_rate_limit(run_id, exc.retry_after)
try:
review_result = self.llm.review_plausibility(
description, score_dict
description, raw_dict, score_dict, domain.metric_bounds
)
except LLMRateLimitError:
pass # still limited; skip, retry next run
@@ -515,6 +697,13 @@ class Pipeline:
)
result.top_results = self.repo.get_top_results(domain.name, limit=20)
return result
finally:
# Flush any batched deterministic writes -- runs on normal
# completion, cancellation, and any other exception propagating
# out of the loop, so nothing deferred above is ever silently lost
# on a clean exit path (a hard process crash is a different story
# and is exactly what the immediate LLM-call commits protect).
self.repo.commit()
# Mark run as completed
if run_id is not None:
@@ -542,13 +731,18 @@ class Pipeline:
def _stub_estimate(
self, combo: Combination, metric_bounds: list[MetricBound]
) -> dict[str, float]:
"""Simple heuristic estimation from dependency data (all values in SI base units).
"""Deterministic estimation from declared entity attributes (no LLM).
cost_efficiency/safety/availability/reliability are driven by the
actuator's thrust_profile and energy_form and the combo's
infrastructure requirements — categorical properties every entity
already declares — rather than flat constants or a formula that
conflates power_density (W/kg, intensive) with cost.
power_density, range_fuel, and cost_efficiency are computed from the
platform's declared mass envelope treated as a combo-wide budget —
see the module-level comment above BIOLOGICAL_OPERATOR_MASS_KG for
the full formula rationale.
safety/availability/reliability/cargo_capacity/environmental_impact
are untouched — these are judgment calls (regulatory, economic,
qualitative), not physics, and stay on the categorical lookup-table
heuristics below (actuator's thrust_profile and energy_form and the
combo's infrastructure requirements).
cost_efficiency additionally checks the domain's declared unit:
"$/(kg·m)" (freight-style domains) isn't a rescaling of "$/m" — it's
@@ -557,9 +751,11 @@ class Pipeline:
"""
metric_names = [mb.metric_name for mb in metric_bounds]
units_by_name = {mb.metric_name: mb.unit for mb in metric_bounds}
bounds_by_name = {mb.metric_name: mb for mb in metric_bounds}
raw: dict[str, float] = {m: 0.0 for m in metric_names}
# Extract intrinsic properties from entities
# Extract intrinsic properties from entities (unchanged — still
# drives the untouched blocks below).
power_density = 0.0 # W/kg
energy_density = 0.0 # J/kg
mass_total = 0.0 # kg, extensive — components share one vehicle
@@ -585,17 +781,141 @@ class Pipeline:
mass = mass_total if mass_total > 0 else 100.0 # kg, default if undeclared
cargo_capacity_kg = mass * CARGO_KG_PER_STRUCTURAL_KG
# ── platform/actuator/storage-specific extraction, for
# power_density / range_fuel / cost_efficiency only ──────────────
platform = next((e for e in combo.entities if e.dimension == "platform"), None)
actuator = next((e for e in combo.entities if e.dimension == "actuator"), None)
storage = next((e for e in combo.entities if e.dimension == "energy_storage"), None)
def dep_value(entity, key, constraint_type) -> float | None:
if entity is None:
return None
for dep in entity.dependencies:
if dep.key == key and dep.constraint_type == constraint_type:
return float(dep.value)
return None
def dep_str(entity, key, constraint_type) -> str | None:
if entity is None:
return None
for dep in entity.dependencies:
if dep.key == key and dep.constraint_type == constraint_type:
return dep.value
return None
p_min = dep_value(platform, "mass", "range_min") or 0.0
a_min = dep_value(actuator, "mass", "range_min") or 0.0
s_min = dep_value(storage, "mass", "range_min") or 0.0
p_max = dep_value(platform, "mass", "range_max")
medium = dep_str(platform, "medium", "requires") or "ground"
actuator_energy_form = dep_str(actuator, "energy_form", "requires")
storage_energy_form = dep_str(storage, "energy_form", "provides")
k_act = dep_value(actuator, "power_density", "provides") or 0.0
e_dens = dep_value(storage, "energy_density", "provides") or 0.0
k_med = SPECIFIC_ENERGY_CONSUMPTION_J_PER_KG_M.get(medium)
p_rep = _representative_mass(p_min, p_max) # platform's representative build size
# actuator/storage mass: sized to what's actually necessary (see
# module note above _solve_two_requirement_masses), except the
# documented near-zero-owned-mass cases below.
denom_offset = 0.0 # extra propelled mass that never competes for the build budget
if actuator_energy_form in BIOLOGICAL_OPERATOR_MASS_KG:
power_mass = BIOLOGICAL_OPERATOR_MASS_KG[actuator_energy_form]
denom_offset = power_mass
actuator_mass, storage_mass = a_min, s_min
elif actuator_energy_form == "radiation_pressure":
# thrust scales with sail area, not carried mass -- derive an
# effective mass from declared footprint and a thin-film areal
# density estimate rather than the (undeclared) mass attribute.
footprint = dep_value(actuator, "footprint", "range_min") or 0.0
actuator_mass = footprint * 0.05 # kg/m^2, thin deployable sail film
power_mass = actuator_mass
storage_mass = s_min
else:
min_accel = dep_value(platform, "min_effective_accel", "range_min")
specific_thrust = dep_value(actuator, "specific_thrust", "provides")
target_velocity = dep_value(platform, "target_velocity", "provides")
range_bounds = bounds_by_name.get("range_fuel")
target_range = range_bounds.norm_max if range_bounds else None
if min_accel and specific_thrust:
c1, r1 = specific_thrust, min_accel
elif target_velocity and k_med:
# Resistance alone (k_med) only covers steady-state cruise --
# a real vehicle also needs reserve force for acceleration
# events (merging, passing, hills), not just holding speed.
# F=ma: an acceleration reserve in m/s^2 is dimensionally a
# specific force (N/kg) exactly like k_med (J/(kg*m) = N/kg),
# so it adds directly before converting to specific power
# (P/mass = force/mass * v).
c1, r1 = k_act, (k_med + ACCELERATION_RESERVE_M_S2) * target_velocity
else:
c1 = r1 = 0.0 # no performance requirement available -- solve degenerates below
if target_range and k_med:
c2, r2 = e_dens, target_range * k_med
else:
c2 = r2 = 0.0
if c1 and c2:
actuator_mass, storage_mass = _solve_two_requirement_masses(
p_rep, c1, r1, c2, r2, a_min, s_min
)
else:
# No performance requirement available at all (e.g. a
# space-medium platform paired with an actuator that
# declares neither specific_thrust nor a usable target
# velocity) -- fall back to bare floors, with the same
# near-zero-mass nominal reference used elsewhere so this
# doesn't silently degenerate to 0 power the way the
# original stub did.
actuator_mass = a_min if a_min > 0.0 else 10.0
storage_mass = s_min
power_mass = actuator_mass
floor_total = p_rep + actuator_mass + storage_mass
physics_denom = floor_total + denom_offset
if "power_density" in raw:
raw["power_density"] = power_density
raw["power_density"] = (k_act * power_mass) / physics_denom if physics_denom else 0.0
if "range_fuel" in raw:
if storage_energy_form in AMBIENT_ENERGY_FORMS or k_med is None:
# Ambient sources aren't a depletable store (see module note
# above). Space/rocket platforms (k_med undeclared for
# "space") are the same conclusion from different physics:
# in vacuum coast there's no resistance to fight, so a
# working engine covers arbitrary distance given enough
# time -- "range" isn't fuel-quantity-limited the way it is
# for a vehicle fighting drag. The real constraint for a
# rocket is its delta-v budget (maneuvering capability),
# which isn't a distance and isn't what this metric asks --
# reporting the domain's ceiling is the honest answer, not
# the old magic-constant guess (e_dens * 2.78) it replaces.
mb = bounds_by_name.get("range_fuel")
raw["range_fuel"] = mb.norm_max if mb else 0.0
elif floor_total > 0:
raw["range_fuel"] = min((e_dens * storage_mass) / (k_med * floor_total), 1e13)
if "cost_efficiency" in raw:
base_cost = ENERGY_FORM_BASE_COST.get(energy_form, 5e-4)
cost_mult = THRUST_PROFILE_COST_MULTIPLIER.get(thrust_profile, 1.0)
cost_per_meter = base_cost * cost_mult
structural_cost = p_rep * STRUCTURAL_COST_PER_KG_BY_MEDIUM.get(medium, STRUCTURAL_COST_PER_KG_BY_MEDIUM["ground"])
actuator_hw_cost = actuator_mass * HARDWARE_COST_PER_KG_BY_ENERGY_FORM.get(actuator_energy_form, 50.0)
storage_hw_cost = storage_mass * HARDWARE_COST_PER_KG_BY_ENERGY_FORM.get(storage_energy_form, 50.0)
upfront_cost = structural_cost + actuator_hw_cost + storage_hw_cost
lifetime_m = LIFETIME_DISTANCE_M_BY_MEDIUM.get(medium, LIFETIME_DISTANCE_M_BY_MEDIUM["ground"])
amortized_per_m = upfront_cost / lifetime_m
fuel_price_per_mj = FUEL_PRICE_PER_MJ.get(storage_energy_form, 0.04)
energy_per_m_mj = ((k_med or SPECIFIC_ENERGY_CONSUMPTION_J_PER_KG_M["ground"]) * floor_total) / 1e6
operating_per_m = energy_per_m_mj * fuel_price_per_mj
cost_per_m = amortized_per_m + operating_per_m
if units_by_name.get("cost_efficiency") == "$/(kg·m)":
raw["cost_efficiency"] = cost_per_meter / max(cargo_capacity_kg, 1.0)
raw["cost_efficiency"] = cost_per_m / max(cargo_capacity_kg, 1.0)
else:
raw["cost_efficiency"] = cost_per_meter
raw["cost_efficiency"] = cost_per_m
if "safety" in raw:
candidates = [
@@ -612,9 +932,6 @@ class Pipeline:
sum(infra_matches) / len(infra_matches) if infra_matches else 0.5
)
if "range_fuel" in raw:
raw["range_fuel"] = min(energy_density * 2.78, 1e13)
if "range_degradation" in raw:
raw["range_degradation"] = 365 * 86400

View File

@@ -36,8 +36,20 @@ class LLMProvider(ABC):
@abstractmethod
def review_plausibility(
self, combination_description: str, scores: dict[str, float]
self,
combination_description: str,
raw_metrics: dict[str, float],
normalized_scores: dict[str, float],
metrics: list[MetricBound],
) -> tuple[str, bool]:
"""Given a combination and its scores, return a (text, is_plausible)
tuple: natural-language assessment and whether the concept is plausible."""
"""Given a combination, its raw physical estimates, and their
normalized scores, return a (text, is_plausible) tuple:
natural-language assessment and whether the concept is plausible.
Both raw_metrics and normalized_scores are given (not just the
normalized score) so the review can reason from the actual physics
rather than only a compressed 0-1 number, which can look
deceptively bad for a metric whose scale was built for a different
kind of vehicle. `metrics` carries each metric's unit for
formatting the raw value meaningfully."""
...

View File

@@ -20,6 +20,35 @@ def format_metrics_for_prompt(metrics: list["MetricBound"]) -> str:
return "\n".join(lines)
def format_scores_for_prompt(
raw_metrics: dict[str, float],
normalized_scores: dict[str, float],
metrics: list["MetricBound"],
) -> str:
"""Render each metric with BOTH its raw physical value and its
normalized score, so the reviewing pass can reason from the actual
physics instead of only ever seeing a compressed 0-1 number.
A real, correct estimate can still look damning once log-normalized
against a scale built for a different kind of vehicle (a cyclist's
real ~5 W/kg reads as "0.159" next to a car's 2000 W/kg ceiling) --
a reviewer that only sees the 0.159 has no way to notice that. See
the labeled-set calibration note on PLAUSIBILITY_REVIEW_PROMPT below.
"""
lines = []
for mb in metrics:
normed = normalized_scores.get(mb.metric_name)
if normed is None:
continue
raw = raw_metrics.get(mb.metric_name)
unit = mb.unit or "dimensionless"
raw_str = f"{raw:g} {unit}" if raw is not None else "unknown"
lines.append(
f"- {mb.metric_name}: raw estimate {raw_str} — normalized score {normed:.3f}"
)
return "\n".join(lines)
PHYSICS_ESTIMATION_PROMPT = """\
You are a physics estimation assistant. Given the following transportation concept, \
estimate the requested metrics using order-of-magnitude physics reasoning.
@@ -44,15 +73,22 @@ match that magnitude, don't guess a generically "reasonable-looking" decimal.
{{"some_metric": <number>, "another_metric": <number>}} — no explanatory text.
"""
# ponytail: pass 4 only sees pass 2's raw numbers, not its reasoning. Sharpened
# prompts on both sides closed most of the gap (a bad safety estimate went from
# 0.95 to 0.80 on the same combo once pass 2 was told to consider combination-
# specific hazards). Upgrade path if this isn't good enough in practice: have
# estimate_physics() also return a short per-metric reason, persist it
# alongside raw_value (new nullable column), and feed it into this prompt so
# pass 4 has something concrete to agree or disagree with. Deferred because it
# needs a schema/interface change across LLMProvider + both providers +
# pipeline + scorer + repository, and more generated tokens per combo.
# ponytail: pass 4 used to see only pass 2's normalized scores, not the raw
# physical numbers or any reasoning behind them. Fixed the raw-value half of
# that gap: format_scores_for_prompt() now shows both, since a correct raw
# estimate can look damning once log-normalized against a scale built for a
# different kind of vehicle (a cyclist's real ~5 W/kg reads as "0.159" next
# to a car's 2000 W/kg ceiling) -- gemma2:27b did exactly this on a real
# bicycle combo, citing "extremely low power density (0.159)" as grounds for
# IMPLAUSIBLE while never reasoning from the actual (correct) 5 W/kg. The
# reasoning-text half of the gap is still open: estimate_physics() doesn't
# return a per-metric rationale, so pass 4 still can't see WHY pass 2 landed
# on a number, only what the number is. Upgrade path if the raw value alone
# isn't enough in practice: have estimate_physics() also return a short
# per-metric reason, persist it alongside raw_value (new nullable column),
# and feed it into this prompt. Deferred because it needs a schema/interface
# change across LLMProvider + both providers + pipeline + scorer +
# repository, and more generated tokens per combo.
#
# If we plan to LLM-review every p2 pass then maybe p2 and p4 should be combined.
#
@@ -84,10 +120,22 @@ is NOT the question.
{description}
## Metric Scores
All scores below are normalized to 0-1, where HIGHER IS ALWAYS BETTER for
every metric listed, regardless of what the metric measures (this already
accounts for things like "lower cost is better" — you don't need to invert
anything). A score of 1.0 means excellent, not "pegged" or "maxed out badly."
Each metric below is given as its raw estimated physical value (in the unit
shown) AND a normalized score from 0-1, where HIGHER IS ALWAYS BETTER for
every metric listed regardless of what it measures (this already accounts
for things like "lower cost is better" — you don't need to invert anything).
A score of 1.0 means excellent, not "pegged" or "maxed out badly."
Reason from the RAW value first — it's the actual physics. The normalized
score is a summary, not a fact on its own: a real, correct estimate can
still normalize to a low-looking number simply because the domain's scale
was built for a different, more demanding kind of vehicle (a cyclist's real
~5 W/kg legitimately normalizes to ~0.16 next to a car engine's 2000 W/kg
ceiling — that low score doesn't mean the estimate is bad or the concept is
weak, it means human power is small next to a car engine, which everyone
already knows). If a normalized score looks alarming, check whether the raw
value is actually reasonable for what this component fundamentally is
before treating the score as evidence of a problem.
{scores}

View File

@@ -11,6 +11,7 @@ from physcom.llm.prompts import (
PHYSICS_ESTIMATION_PROMPT,
PLAUSIBILITY_REVIEW_PROMPT,
format_metrics_for_prompt,
format_scores_for_prompt,
)
from physcom.models.domain import MetricBound
@@ -46,9 +47,13 @@ class GeminiLLMProvider(LLMProvider):
return parse_metric_json(response.text, metrics)
def review_plausibility(
self, combination_description: str, scores: dict[str, float]
self,
combination_description: str,
raw_metrics: dict[str, float],
normalized_scores: dict[str, float],
metrics: list[MetricBound],
) -> tuple[str, bool]:
scores_str = "\n".join(f"- {k}: {v:.3f}" for k, v in scores.items())
scores_str = format_scores_for_prompt(raw_metrics, normalized_scores, metrics)
prompt = PLAUSIBILITY_REVIEW_PROMPT.format(
description=combination_description,
scores=scores_str,

View File

@@ -21,9 +21,13 @@ class MockLLMProvider(LLMProvider):
return result
def review_plausibility(
self, combination_description: str, scores: dict[str, float]
self,
combination_description: str,
raw_metrics: dict[str, float],
normalized_scores: dict[str, float],
metrics: list[MetricBound],
) -> tuple[str, bool]:
avg = sum(scores.values()) / max(len(scores), 1)
avg = sum(normalized_scores.values()) / max(len(normalized_scores), 1)
if avg > 0.5:
return ("This concept appears plausible and worth further investigation.", True)
return ("This concept has significant feasibility challenges.", False)

View File

@@ -12,6 +12,7 @@ from physcom.llm.prompts import (
PHYSICS_ESTIMATION_PROMPT,
PLAUSIBILITY_REVIEW_PROMPT,
format_metrics_for_prompt,
format_scores_for_prompt,
)
from physcom.models.domain import MetricBound
@@ -34,9 +35,13 @@ class OllamaLLMProvider(LLMProvider):
return parse_metric_json(text, metrics)
def review_plausibility(
self, combination_description: str, scores: dict[str, float]
self,
combination_description: str,
raw_metrics: dict[str, float],
normalized_scores: dict[str, float],
metrics: list[MetricBound],
) -> tuple[str, bool]:
scores_str = "\n".join(f"- {k}: {v:.3f}" for k, v in scores.items())
scores_str = format_scores_for_prompt(raw_metrics, normalized_scores, metrics)
prompt = PLAUSIBILITY_REVIEW_PROMPT.format(
description=combination_description,
scores=scores_str,
@@ -54,7 +59,7 @@ class OllamaLLMProvider(LLMProvider):
headers={"Content-Type": "application/json"},
)
try:
with urllib.request.urlopen(req, timeout=120) as resp:
with urllib.request.urlopen(req, timeout=300) as resp:
return json.loads(resp.read())["response"]
except urllib.error.URLError as exc:
raise ConnectionError(

View File

@@ -24,6 +24,7 @@ GROUND_PLATFORMS: list[Entity] = [
Dependency("physical", "mass", "50", "kg", "range_min"),
Dependency("infrastructure", "road_network", "true", None, "requires"),
Dependency("environment", "medium", "ground", None, "requires"),
Dependency("physical", "target_velocity", "25", "m/s", "provides"),
],
),
Entity(
@@ -41,6 +42,7 @@ GROUND_PLATFORMS: list[Entity] = [
Dependency("physical", "mass", "5", "kg", "range_min"),
Dependency("infrastructure", "road_network", "true", None, "requires"),
Dependency("environment", "medium", "ground", None, "requires"),
Dependency("physical", "target_velocity", "6", "m/s", "provides"),
],
),
Entity(
@@ -58,6 +60,7 @@ GROUND_PLATFORMS: list[Entity] = [
Dependency("physical", "mass", "10000", "kg", "range_min"),
Dependency("infrastructure", "rail_network", "true", None, "requires"),
Dependency("environment", "medium", "ground", None, "requires"),
Dependency("physical", "target_velocity", "30", "m/s", "provides"),
],
),
]
@@ -79,6 +82,7 @@ WATER_PLATFORMS: list[Entity] = [
Dependency("physical", "mass", "100000", "kg", "range_max"),
Dependency("physical", "mass", "30", "kg", "range_min"),
Dependency("environment", "medium", "water", None, "requires"),
Dependency("physical", "target_velocity", "8", "m/s", "provides"),
],
),
Entity(
@@ -94,6 +98,7 @@ WATER_PLATFORMS: list[Entity] = [
Dependency("physical", "mass", "10000", "kg", "range_min"),
Dependency("environment", "medium", "water", None, "requires"),
Dependency("physical", "energy_density", "720000", "J/kg", "range_min"),
Dependency("physical", "target_velocity", "8", "m/s", "provides"),
],
),
]
@@ -118,6 +123,7 @@ AIR_PLATFORMS: list[Entity] = [
Dependency("environment", "medium", "air", None, "requires"),
Dependency("physical", "energy_density", "1440000", "J/kg", "range_min"),
Dependency("physical", "min_effective_accel", "2.0", "m/s²", "range_min"),
Dependency("physical", "target_velocity", "60", "m/s", "provides"),
],
),
Entity(
@@ -135,6 +141,7 @@ AIR_PLATFORMS: list[Entity] = [
Dependency("environment", "medium", "air", None, "requires"),
Dependency("physical", "energy_density", "720000", "J/kg", "range_min"),
Dependency("physical", "min_effective_accel", "10", "m/s²", "range_min"),
Dependency("physical", "target_velocity", "30", "m/s", "provides"),
],
),
Entity(
@@ -214,6 +221,7 @@ FICTIONAL_PLATFORMS: list[Entity] = [
Dependency("physical", "mass", "5000", "kg", "range_min"),
Dependency("infrastructure", "hyperloop_tube", "true", None, "requires"),
Dependency("environment", "medium", "ground", None, "requires"),
Dependency("physical", "target_velocity", "270", "m/s", "provides"), # near-sonic, per its own description
],
),
]
@@ -302,7 +310,7 @@ BIOLOGICAL_ACTUATORS: list[Entity] = [
Dependency("energy", "energy_form", "biological", None, "requires"),
Dependency("physical", "mass", "0", "kg", "range_min"),
Dependency("force", "thrust_profile", "low_continuous", None, "provides"),
Dependency("force", "power_density", "1.5", "W/kg", "provides"),
Dependency("force", "power_density", "5.5", "W/kg", "provides"),
],
),
Entity(