andrew/physicalCombinatorics
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

drop safety/availability from scoring, holistic p4 rating, phase-parallel pipeline

Browse Source 
safety and availability don't reduce to physics formulas the way
power_density/range_fuel/cost_efficiency do -- they're judgment calls
(risk assessment, infrastructure prevalence), and running them through the
same log-normalize() built for physical quantities produced incoherent
results: safety's raw value is already a "0-1" score, and normalizing it
again turned 0.6 into an unexplainable 0.678 that even the LLM reviewing
it could only cite, never justify (see combo 1540). Removed both from
domain_metric_weights (safety from 4 domains, availability from
urban_commuting) and renormalized the remaining weights to sum to 1.0.

Pass 4 now produces one holistic RATING (LOW/MEDIUM/HIGH) alongside the
existing VERDICT, with safety and accessibility folded in as qualitative
considerations feeding that single judgment rather than scored
separately -- not a checklist of independent numbers. New
qualitative_rating column, filterable in the results UI. Also added
domain name/description to the review prompt so the LLM judges a metric
like range against what the domain actually needs (urban_commuting:
1-50km) instead of generic real-world expectations for the platform
category -- confirmed live on a combo where phi4 had called a 396km range
"limited" by comparing to typical aircraft rather than a domain that
needs 1-50km.

Pass 2 is estimator-only now -- self.llm is never consulted there,
reserved entirely for pass 4. Restructured Pipeline.run() from combo-first
to phase-parallel: each pass now runs to completion across every combo
before the next pass starts, rather than walking each combo through all
four passes before the next combo. This surfaced a real bug: domain-
blocked combos (status stays "valid" by design, not "_fail") were
slipping past a naive status-based skip guard and getting silently
re-processed by pass 2. Fixed with a shared dead-combo check that catches
both generic failures and domain blocks correctly.

Also fixes a results-page display bug found while reviewing a live combo:
the per-metric "position" bar showed raw distance from norm_min without
inverting for lower_is_better metrics, so an excellent cost score (near
the good end) rendered as a ~0%, near-empty bar -- looked bad next to its
own 0.99 normalized score.

Validated live against phi4 (real Ollama calls, not mocked): full-domain
phase-parallel run (2,970 combos, estimator-only p2, 1.6s) followed by a
real pass-4 run (111 reviewed, 11m, 0 crashes, 0 null ratings). Two tests
that relied on the old LLM-driven pass 2 to force deterministic outcomes
were updated to test pass 4's verdict-wiring directly instead. All 100
tests pass.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Andrew Simonson
2026-08-15 16:52:26 -05:00
parent 730a23bac3
commit 76f460499a
16 changed files with 569 additions and 403 deletions
Download Patch File Download Diff File Expand all files Collapse all files

65
src/physcom/seed/transport_example.py
View File

@@ -728,11 +728,22 @@ URBAN_COMMUTING = Domain(
name="urban_commuting",
description="Daily travel within a city, 1-50km range",
metric_bounds=[
MetricBound("power_density", weight=0.25, norm_min=1, norm_max=2000, unit="W/kg"),
MetricBound("cost_efficiency", weight=0.25, norm_min=1e-5, norm_max=2e-3, unit="$/m", lower_is_better=True),
MetricBound("safety", weight=0.25, norm_min=0.0, norm_max=1.0, unit="0-1"),
MetricBound("availability", weight=0.15, norm_min=0.0, norm_max=1.0, unit="0-1"),
MetricBound("range_fuel", weight=0.10, norm_min=5000, norm_max=500000, unit="m"),
# safety and availability removed from the scored/weighted metric set:
# both are judgment calls (risk assessment, infrastructure prevalence),
# not physics quantities with a formula, and running them through the
# same log-normalize() built for physical quantities produced
# incoherent results (a safety raw value already declared as "0-1"
# getting re-normalized into a different, unexplainable 0-1 number --
# see combo 1540's review, where phi4 could only cite the post-
# normalization number with no way to justify it). Safety is now a
# qualitative consideration folded into pass 4's holistic RATING
# instead. Availability needs real per-infrastructure-type research
# this project hasn't done -- not scored anywhere for now rather than
# pretend a quick formula or an equally uninformed LLM guess settles it.
# Weights renormalized to sum to 1.0 across the remaining metrics.
MetricBound("power_density", weight=0.4167, norm_min=1, norm_max=2000, unit="W/kg"),
MetricBound("cost_efficiency", weight=0.4167, norm_min=1e-5, norm_max=2e-3, unit="$/m", lower_is_better=True),
MetricBound("range_fuel", weight=0.1666, norm_min=5000, norm_max=500000, unit="m"),
],
constraints=[DomainConstraint("medium", ["ground", "air"])],
)
@@ -741,11 +752,12 @@ INTERPLANETARY = Domain(
name="interplanetary_travel",
description="Travel between planets within a solar system",
metric_bounds=[
MetricBound("power_density", weight=0.30, norm_min=10, norm_max=10000, unit="W/kg"),
MetricBound("range_fuel", weight=0.30, norm_min=1e9, norm_max=1e13, unit="m"),
MetricBound("safety", weight=0.20, norm_min=0.0, norm_max=1.0, unit="0-1"),
MetricBound("cost_efficiency", weight=0.10, norm_min=1.0, norm_max=1e6, unit="$/m", lower_is_better=True),
MetricBound("range_degradation", weight=0.10, norm_min=8640000, norm_max=3.1536e9, unit="s"),
# safety removed -- see URBAN_COMMUTING comment above. Weights
# renormalized across the remaining metrics.
MetricBound("power_density", weight=0.375, norm_min=10, norm_max=10000, unit="W/kg"),
MetricBound("range_fuel", weight=0.375, norm_min=1e9, norm_max=1e13, unit="m"),
MetricBound("cost_efficiency", weight=0.125, norm_min=1.0, norm_max=1e6, unit="$/m", lower_is_better=True),
MetricBound("range_degradation", weight=0.125, norm_min=8640000, norm_max=3.1536e9, unit="s"),
],
constraints=[DomainConstraint("medium", ["space"])],
)
@@ -754,11 +766,12 @@ MARITIME_SHIPPING = Domain(
name="maritime_shipping",
description="Ocean cargo transport between ports, 100-40000km range",
metric_bounds=[
MetricBound("power_density", weight=0.15, norm_min=1, norm_max=1000, unit="W/kg"),
MetricBound("cargo_capacity", weight=0.25, norm_min=1000, norm_max=2e8, unit="kg"),
MetricBound("cost_efficiency", weight=0.25, norm_min=1e-9, norm_max=1e-6, unit="$/(kg\u00b7m)", lower_is_better=True),
MetricBound("safety", weight=0.20, norm_min=0.0, norm_max=1.0, unit="0-1"),
MetricBound("range_fuel", weight=0.15, norm_min=100000, norm_max=40000000, unit="m"),
# safety removed -- see URBAN_COMMUTING comment above. Weights
# renormalized across the remaining metrics.
MetricBound("power_density", weight=0.1875, norm_min=1, norm_max=1000, unit="W/kg"),
MetricBound("cargo_capacity", weight=0.3125, norm_min=1000, norm_max=2e8, unit="kg"),
MetricBound("cost_efficiency", weight=0.3125, norm_min=1e-9, norm_max=1e-6, unit="$/(kg\u00b7m)", lower_is_better=True),
MetricBound("range_fuel", weight=0.1875, norm_min=100000, norm_max=40000000, unit="m"),
],
constraints=[DomainConstraint("medium", ["water"])],
)
@@ -767,11 +780,12 @@ LAST_MILE_DELIVERY = Domain(
name="last_mile_delivery",
description="Short-range package delivery within neighborhoods, 0.5-15km",
metric_bounds=[
MetricBound("power_density", weight=0.25, norm_min=1, norm_max=500, unit="W/kg"),
MetricBound("cost_efficiency", weight=0.30, norm_min=1e-5, norm_max=5e-3, unit="$/m", lower_is_better=True),
MetricBound("cargo_capacity_kg", weight=0.20, norm_min=1, norm_max=500, unit="kg"),
MetricBound("safety", weight=0.15, norm_min=0.0, norm_max=1.0, unit="0-1"),
MetricBound("environmental_impact", weight=0.10, norm_min=0, norm_max=5e-4, unit="kg/m", lower_is_better=True),
# safety removed -- see URBAN_COMMUTING comment above. Weights
# renormalized across the remaining metrics.
MetricBound("power_density", weight=0.2941, norm_min=1, norm_max=500, unit="W/kg"),
MetricBound("cost_efficiency", weight=0.3529, norm_min=1e-5, norm_max=5e-3, unit="$/m", lower_is_better=True),
MetricBound("cargo_capacity_kg", weight=0.2353, norm_min=1, norm_max=500, unit="kg"),
MetricBound("environmental_impact", weight=0.1177, norm_min=0, norm_max=5e-4, unit="kg/m", lower_is_better=True),
],
constraints=[DomainConstraint("medium", ["ground", "air"])],
)
@@ -839,12 +853,11 @@ def load_transport_seed(repo) -> dict:
counts["domains"] += 1
except sqlite3.IntegrityError:
pass
# Backfill metric units and lower_is_better on existing DBs.
for mb in domain.metric_bounds:
repo.ensure_metric(mb.metric_name, unit=mb.unit)
repo.backfill_metric_unit(domain.name, mb.metric_name, mb.unit)
if mb.lower_is_better:
repo.backfill_lower_is_better(domain.name, mb.metric_name)
# Sync domain_metric_weights to exactly match this domain's current
# metric_bounds on existing DBs -- upserts weight/norm_min/norm_max/
# unit for current metrics and removes any that were dropped (e.g.
# safety/availability no longer scored).
repo.sync_domain_metric_weights(domain)
# Backfill domain constraints
repo.replace_domain_constraints(domain)