give pass 4 raw values, batch deferrable commits, fix two known bugs
Pass 4's plausibility review only ever saw normalized scores, never the raw physical estimate behind them -- confirmed via live testing this was exactly what caused a real misfire (gemma2:27b cited a real cyclist's correct 5 W/kg, log-normalized to "0.159" against a car's power scale, as grounds for rejecting an ordinary bicycle). review_plausibility now takes raw_metrics + normalized_scores + metric units, and the prompt explicitly instructs reasoning from the raw value first. Verified live against phi4: it now cites the actual raw number and correctly explains why a low normalized score doesn't mean the estimate or concept is bad. Repository write methods used in the pipeline's hot path now take an optional commit=False, and Pipeline defers commits during the fast/ deterministic passes (1, 3, and 2 without an LLM), flushing every 200 combos and on any exit path (finally block covers normal completion, cancellation, and any other exception). LLM-involving calls (pass 2 with an LLM, all of pass 4) still commit immediately -- those are slow and crash-prone and worth protecting per-write; the deterministic passes aren't, and recomputing them is now measured at under a second for the full domain rather than worth 8,000+ individual fsync'd commits. Full 2,970-combination domain run: multiple minutes -> 0.91s. Test suite: ~70s -> ~15s. Also fixes two more issues found while auditing the estimator for a real run: CARGO_KG_PER_STRUCTURAL_KG was 500 (no real vehicle carries 500x its own structural mass in cargo -- a magnitude bug, not a modeling choice), corrected to 2.5. And space/rocket platforms' range_fuel now reports the domain's ceiling instead of an arbitrary placeholder constant -- vacuum coast isn't resistance-limited, so "distance before running out of fuel" isn't a meaningful question for these the way it is for ground/air/water vehicles; the real constraint is delta-v budget, a different metric this pass doesn't model. Validated with a live full-domain run (phi4, real Ollama calls): 115 reviewed, 0 malformed/null reviews, 0 verdict-vs-status mismatches. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -69,8 +69,12 @@ INFRASTRUCTURE_AVAILABILITY: dict[tuple[str, str], float] = {
|
||||
("fuel_infrastructure", "xenon_propellant"): 0.05,
|
||||
}
|
||||
|
||||
# Crude freight-capacity proxy: kg of cargo per kg of vehicle structural mass.
|
||||
CARGO_KG_PER_STRUCTURAL_KG: float = 500
|
||||
# Crude freight-capacity proxy: kg of cargo per kg of vehicle structural
|
||||
# mass. Was 500 -- a magnitude error (500x cargo-to-structure has no real
|
||||
# vehicle analog). Real cargo ships run deadweight/lightship ratios of
|
||||
# roughly 1.5-4x depending on class; 2.5 is a reasonable general-cargo
|
||||
# midpoint for this domain-agnostic proxy.
|
||||
CARGO_KG_PER_STRUCTURAL_KG: float = 2.5
|
||||
|
||||
# How mechanically proven/predictable an energy form is in practice — distinct
|
||||
# from safety (risk when something goes wrong) and thrust_profile (delivery
|
||||
@@ -367,9 +371,13 @@ class Pipeline:
|
||||
combos = generate_combinations(self.repo, dimensions)
|
||||
result.total_generated = len(combos)
|
||||
|
||||
# Save all combinations to DB (also loads status for existing combos)
|
||||
# Save all combinations to DB (also loads status for existing combos).
|
||||
# Deferred commit -- registering combos is instant/deterministic, so a
|
||||
# crash here just means re-running the (cheap) registration loop, not
|
||||
# losing anything worth protecting with a commit per row.
|
||||
for combo in combos:
|
||||
self.repo.save_combination(combo)
|
||||
self.repo.save_combination(combo, commit=False)
|
||||
self.repo.commit()
|
||||
|
||||
if run_id is not None:
|
||||
self.repo.update_pipeline_run(run_id, total_combos=len(combos))
|
||||
@@ -378,9 +386,20 @@ class Pipeline:
|
||||
bounds_by_name = {mb.metric_name: mb for mb in domain.metric_bounds}
|
||||
|
||||
# ── Combo-first loop ─────────────────────────────────────
|
||||
# Deterministic passes (1, 3, and 2 without an LLM) defer commits and
|
||||
# get flushed periodically + in `finally` below -- a crash there costs
|
||||
# a cheap recompute, not lost work worth committing per write. Pass 4
|
||||
# (and pass 2 with an LLM) commit immediately after each call: those
|
||||
# are slow and crash-prone (see the QwQ timeout saga), so that result
|
||||
# is worth protecting the moment it lands.
|
||||
combos_since_commit = 0
|
||||
try:
|
||||
for combo in combos:
|
||||
self._check_cancelled(run_id)
|
||||
combos_since_commit += 1
|
||||
if combos_since_commit >= 200:
|
||||
self.repo.commit()
|
||||
combos_since_commit = 0
|
||||
|
||||
# Check existing progress for this combo in this domain
|
||||
existing_pass = self.repo.get_combo_pass_reached(
|
||||
@@ -399,7 +418,7 @@ class Pipeline:
|
||||
combo.status = "p1_fail"
|
||||
combo.block_reason = "; ".join(cr.violations)
|
||||
self.repo.update_combination_status(
|
||||
combo.id, "p1_fail", combo.block_reason
|
||||
combo.id, "p1_fail", combo.block_reason, commit=False
|
||||
)
|
||||
# Save a result row so failed combos appear in results
|
||||
self.repo.save_result(
|
||||
@@ -407,13 +426,14 @@ class Pipeline:
|
||||
domain.id,
|
||||
composite_score=0.0,
|
||||
pass_reached=1,
|
||||
commit=False,
|
||||
)
|
||||
result.pass1_failed += 1
|
||||
self._update_run_counters(run_id, result, current_pass=1)
|
||||
continue # p1_fail — skip remaining passes
|
||||
else:
|
||||
combo.status = "valid"
|
||||
self.repo.update_combination_status(combo.id, "valid")
|
||||
self.repo.update_combination_status(combo.id, "valid", commit=False)
|
||||
|
||||
# Domain constraint check (per-domain block only). combo.status
|
||||
# stays "valid" here on purpose: it's domain-agnostic and the
|
||||
@@ -431,6 +451,7 @@ class Pipeline:
|
||||
domain_block_reason="; ".join(
|
||||
dc_result.violations
|
||||
),
|
||||
commit=False,
|
||||
)
|
||||
result.pass1_failed += 1
|
||||
self._update_run_counters(
|
||||
@@ -482,9 +503,13 @@ class Pipeline:
|
||||
"estimation_method": "llm" if self.llm else "stub",
|
||||
"confidence": 1.0,
|
||||
})
|
||||
# LLM-produced estimates commit immediately (slow/crash-
|
||||
# prone, worth protecting); stub estimates are instant
|
||||
# and defer, same as the rest of the deterministic passes.
|
||||
used_llm = self.llm is not None
|
||||
if estimate_dicts:
|
||||
self.repo.save_raw_estimates(
|
||||
combo.id, domain.id, estimate_dicts
|
||||
combo.id, domain.id, estimate_dicts, commit=used_llm
|
||||
)
|
||||
|
||||
# Check for all-zero estimates → p2_fail
|
||||
@@ -492,11 +517,12 @@ class Pipeline:
|
||||
combo.status = "p2_fail"
|
||||
combo.block_reason = "All metric estimates are zero"
|
||||
self.repo.update_combination_status(
|
||||
combo.id, "p2_fail", combo.block_reason
|
||||
combo.id, "p2_fail", combo.block_reason, commit=used_llm
|
||||
)
|
||||
self.repo.save_result(
|
||||
combo.id, domain.id,
|
||||
composite_score=0.0, pass_reached=2,
|
||||
commit=used_llm,
|
||||
)
|
||||
result.pass2_failed += 1
|
||||
self._update_run_counters(run_id, result, current_pass=2)
|
||||
@@ -534,7 +560,7 @@ class Pipeline:
|
||||
"confidence": s.confidence,
|
||||
})
|
||||
if score_dicts:
|
||||
self.repo.save_scores(combo.id, domain.id, score_dicts)
|
||||
self.repo.save_scores(combo.id, domain.id, score_dicts, commit=False)
|
||||
|
||||
# Preserve existing human data
|
||||
novelty_flag = (
|
||||
@@ -550,6 +576,7 @@ class Pipeline:
|
||||
sr.composite_score, pass_reached=3,
|
||||
novelty_flag=novelty_flag,
|
||||
human_notes=human_notes,
|
||||
commit=False,
|
||||
)
|
||||
combo.status = "p3_fail"
|
||||
combo.block_reason = (
|
||||
@@ -557,7 +584,7 @@ class Pipeline:
|
||||
f"below threshold {score_threshold}"
|
||||
)
|
||||
self.repo.update_combination_status(
|
||||
combo.id, "p3_fail", combo.block_reason
|
||||
combo.id, "p3_fail", combo.block_reason, commit=False
|
||||
)
|
||||
result.pass3_failed += 1
|
||||
result.pass3_scored += 1
|
||||
@@ -571,8 +598,9 @@ class Pipeline:
|
||||
pass_reached=3,
|
||||
novelty_flag=novelty_flag,
|
||||
human_notes=human_notes,
|
||||
commit=False,
|
||||
)
|
||||
self.repo.update_combination_status(combo.id, "scored")
|
||||
self.repo.update_combination_status(combo.id, "scored", commit=False)
|
||||
|
||||
result.pass3_scored += 1
|
||||
result.pass3_above_threshold += 1
|
||||
@@ -608,16 +636,21 @@ class Pipeline:
|
||||
for s in db_scores
|
||||
if s["normalized_score"] is not None
|
||||
}
|
||||
raw_dict = {
|
||||
s["metric_name"]: s["raw_value"]
|
||||
for s in db_scores
|
||||
if s["raw_value"] is not None
|
||||
}
|
||||
review_result: tuple[str, bool] | None = None
|
||||
try:
|
||||
review_result = self.llm.review_plausibility(
|
||||
description, score_dict
|
||||
description, raw_dict, score_dict, domain.metric_bounds
|
||||
)
|
||||
except LLMRateLimitError as exc:
|
||||
self._wait_for_rate_limit(run_id, exc.retry_after)
|
||||
try:
|
||||
review_result = self.llm.review_plausibility(
|
||||
description, score_dict
|
||||
description, raw_dict, score_dict, domain.metric_bounds
|
||||
)
|
||||
except LLMRateLimitError:
|
||||
pass # still limited; skip, retry next run
|
||||
@@ -664,6 +697,13 @@ class Pipeline:
|
||||
)
|
||||
result.top_results = self.repo.get_top_results(domain.name, limit=20)
|
||||
return result
|
||||
finally:
|
||||
# Flush any batched deterministic writes -- runs on normal
|
||||
# completion, cancellation, and any other exception propagating
|
||||
# out of the loop, so nothing deferred above is ever silently lost
|
||||
# on a clean exit path (a hard process crash is a different story
|
||||
# and is exactly what the immediate LLM-call commits protect).
|
||||
self.repo.commit()
|
||||
|
||||
# Mark run as completed
|
||||
if run_id is not None:
|
||||
@@ -842,15 +882,22 @@ class Pipeline:
|
||||
raw["power_density"] = (k_act * power_mass) / physics_denom if physics_denom else 0.0
|
||||
|
||||
if "range_fuel" in raw:
|
||||
if storage_energy_form in AMBIENT_ENERGY_FORMS:
|
||||
if storage_energy_form in AMBIENT_ENERGY_FORMS or k_med is None:
|
||||
# Ambient sources aren't a depletable store (see module note
|
||||
# above). Space/rocket platforms (k_med undeclared for
|
||||
# "space") are the same conclusion from different physics:
|
||||
# in vacuum coast there's no resistance to fight, so a
|
||||
# working engine covers arbitrary distance given enough
|
||||
# time -- "range" isn't fuel-quantity-limited the way it is
|
||||
# for a vehicle fighting drag. The real constraint for a
|
||||
# rocket is its delta-v budget (maneuvering capability),
|
||||
# which isn't a distance and isn't what this metric asks --
|
||||
# reporting the domain's ceiling is the honest answer, not
|
||||
# the old magic-constant guess (e_dens * 2.78) it replaces.
|
||||
mb = bounds_by_name.get("range_fuel")
|
||||
raw["range_fuel"] = mb.norm_max if mb else 0.0
|
||||
elif k_med is not None and floor_total > 0:
|
||||
elif floor_total > 0:
|
||||
raw["range_fuel"] = min((e_dens * storage_mass) / (k_med * floor_total), 1e13)
|
||||
else:
|
||||
# space/rocket platforms: resistance-based formula doesn't
|
||||
# apply (see module note) -- old placeholder, not a claim.
|
||||
raw["range_fuel"] = min(e_dens * 2.78, 1e13)
|
||||
|
||||
if "cost_efficiency" in raw:
|
||||
structural_cost = p_rep * STRUCTURAL_COST_PER_KG_BY_MEDIUM.get(medium, STRUCTURAL_COST_PER_KG_BY_MEDIUM["ground"])
|
||||
|
||||
Reference in New Issue
Block a user