drop safety/availability from scoring, holistic p4 rating, phase-parallel pipeline
safety and availability don't reduce to physics formulas the way power_density/range_fuel/cost_efficiency do -- they're judgment calls (risk assessment, infrastructure prevalence), and running them through the same log-normalize() built for physical quantities produced incoherent results: safety's raw value is already a "0-1" score, and normalizing it again turned 0.6 into an unexplainable 0.678 that even the LLM reviewing it could only cite, never justify (see combo 1540). Removed both from domain_metric_weights (safety from 4 domains, availability from urban_commuting) and renormalized the remaining weights to sum to 1.0. Pass 4 now produces one holistic RATING (LOW/MEDIUM/HIGH) alongside the existing VERDICT, with safety and accessibility folded in as qualitative considerations feeding that single judgment rather than scored separately -- not a checklist of independent numbers. New qualitative_rating column, filterable in the results UI. Also added domain name/description to the review prompt so the LLM judges a metric like range against what the domain actually needs (urban_commuting: 1-50km) instead of generic real-world expectations for the platform category -- confirmed live on a combo where phi4 had called a 396km range "limited" by comparing to typical aircraft rather than a domain that needs 1-50km. Pass 2 is estimator-only now -- self.llm is never consulted there, reserved entirely for pass 4. Restructured Pipeline.run() from combo-first to phase-parallel: each pass now runs to completion across every combo before the next pass starts, rather than walking each combo through all four passes before the next combo. This surfaced a real bug: domain- blocked combos (status stays "valid" by design, not "_fail") were slipping past a naive status-based skip guard and getting silently re-processed by pass 2. Fixed with a shared dead-combo check that catches both generic failures and domain blocks correctly. Also fixes a results-page display bug found while reviewing a live combo: the per-metric "position" bar showed raw distance from norm_min without inverting for lower_is_better metrics, so an excellent cost score (near the good end) rendered as a ~0%, near-empty bar -- looked bad next to its own 0.99 normalized score. Validated live against phi4 (real Ollama calls, not mocked): full-domain phase-parallel run (2,970 combos, estimator-only p2, 1.6s) followed by a real pass-4 run (111 reviewed, 11m, 0 crashes, 0 null ratings). Two tests that relied on the old LLM-driven pass 2 to force deterministic outcomes were updated to test pass 4's verdict-wiring directly instead. All 100 tests pass. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
@@ -25,9 +25,11 @@ def results_domain(domain_name: str):
|
||||
return redirect(url_for("results.results_index"))
|
||||
|
||||
status_filter = request.args.get("status")
|
||||
results = repo.get_all_results(domain_name, status=status_filter)
|
||||
rating_filter = request.args.get("rating")
|
||||
results = repo.get_all_results(domain_name, status=status_filter, rating=rating_filter)
|
||||
# Domain-scoped status counts (only combos that have results in this domain)
|
||||
statuses = repo.count_combinations_by_status(domain_name=domain_name)
|
||||
ratings = repo.count_results_by_rating(domain_name)
|
||||
|
||||
return render_template(
|
||||
"results/list.html",
|
||||
@@ -35,7 +37,9 @@ def results_domain(domain_name: str):
|
||||
domain=domain,
|
||||
results=results,
|
||||
status_filter=status_filter,
|
||||
rating_filter=rating_filter,
|
||||
statuses=statuses,
|
||||
ratings=ratings,
|
||||
total_results=sum(statuses.values()),
|
||||
)
|
||||
|
||||
@@ -101,6 +105,7 @@ def submit_review(domain_name: str, combo_id: int):
|
||||
novelty_flag=novelty_flag,
|
||||
llm_review=existing.get("llm_review") if existing else None,
|
||||
human_notes=human_notes,
|
||||
qualitative_rating=existing.get("qualitative_rating") if existing else None,
|
||||
)
|
||||
repo.update_combination_status(combo_id, "reviewed")
|
||||
|
||||
|
||||
@@ -214,6 +214,9 @@ table.compact th, table.compact td { padding: 0.25rem 0.4rem; font-size: 0.83rem
|
||||
.badge-llm_reviewed { background: rgba(107,163,160,0.12); color: var(--accent-teal); border-color: rgba(107,163,160,0.25); }
|
||||
.badge-reviewed { background: rgba(155,142,196,0.12); color: var(--accent-violet); border-color: rgba(155,142,196,0.25); }
|
||||
.badge-pending { background: rgba(184,147,92,0.12); color: var(--accent-amber); border-color: rgba(184,147,92,0.25); }
|
||||
.badge-rating-low { background: rgba(184,92,92,0.12); color: var(--accent-red); border-color: rgba(184,92,92,0.25); }
|
||||
.badge-rating-medium { background: rgba(184,147,92,0.12); color: var(--accent-amber); border-color: rgba(184,147,92,0.25); }
|
||||
.badge-rating-high { background: rgba(122,171,138,0.12); color: var(--accent-green); border-color: rgba(122,171,138,0.25); }
|
||||
|
||||
/* ── Buttons ─────────────────────────────────────────────── */
|
||||
.btn {
|
||||
|
||||
@@ -50,12 +50,13 @@
|
||||
<div class="step-body">
|
||||
<h3>Physics Estimation</h3>
|
||||
<p>
|
||||
Surviving combinations get raw metric estimates — speed, cost,
|
||||
safety, range — via heuristic stubs or an LLM provider that
|
||||
reasons about the physical properties of each pairing.
|
||||
Surviving combinations get raw metric estimates — power
|
||||
density, cost, range — from a deterministic physics engine
|
||||
that sizes each combination from its own declared attributes, not
|
||||
a guess.
|
||||
</p>
|
||||
<div class="step-example">
|
||||
Bicycle + Human Pedalling → speed: 20 km/h, cost: $0.01/km
|
||||
Bicycle + Human Muscle → power density: 4.4 W/kg, range: 500km
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -72,8 +73,8 @@
|
||||
Combinations are ranked within their domain.
|
||||
</p>
|
||||
<div class="step-example">
|
||||
Domain <code>urban_commuting</code> weights: speed 25%, cost 25%,
|
||||
safety 25%, availability 15%, range 10%
|
||||
Domain <code>urban_commuting</code> weights: power density 42%,
|
||||
cost 42%, range 17%
|
||||
</div>
|
||||
</div>
|
||||
</div>
|
||||
@@ -85,9 +86,11 @@
|
||||
<div class="step-body">
|
||||
<h3>LLM Review</h3>
|
||||
<p>
|
||||
Top-scoring combinations are sent to a language model for plausibility
|
||||
and novelty assessment — catching physically valid but practically
|
||||
absurd pairings.
|
||||
Top-scoring combinations are sent to a language model for a
|
||||
plausibility verdict plus a holistic LOW/MEDIUM/HIGH rating —
|
||||
weighing safety and accessibility as qualitative judgment calls
|
||||
alongside the physics scores, catching physically valid but
|
||||
practically absurd pairings.
|
||||
</p>
|
||||
<div class="step-example">
|
||||
"Train + Solar Sail: structurally valid constraints, but solar radiation
|
||||
@@ -163,14 +166,15 @@
|
||||
<div class="card concept-card">
|
||||
<h3>Metrics</h3>
|
||||
<p>
|
||||
Quantitative axes like speed, cost, safety, and range. Each metric
|
||||
has a domain-specific weight and normalization range. Some are
|
||||
inverted — lower cost is better.
|
||||
Quantitative physics axes like power density, cost, and range. Each
|
||||
metric has a domain-specific weight and normalization range. Some
|
||||
are inverted — lower cost is better. Safety and accessibility
|
||||
are judgment calls, not physics quantities — they're weighed
|
||||
qualitatively in the LLM review pass instead of scored here.
|
||||
</p>
|
||||
<div class="concept-examples">
|
||||
<span class="badge">speed</span>
|
||||
<span class="badge">power_density</span>
|
||||
<span class="badge">cost_efficiency</span>
|
||||
<span class="badge">safety</span>
|
||||
<span class="badge">range_fuel</span>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
@@ -27,6 +27,9 @@
|
||||
{% if result %}
|
||||
<dt>Composite Score</dt><dd class="score-cell">{{ "%.4f"|format(result.composite_score) }}</dd>
|
||||
<dt>Pass Reached</dt><dd>{{ result.pass_reached }}</dd>
|
||||
{% if result.qualitative_rating %}
|
||||
<dt>Rating</dt><dd><span class="badge badge-rating-{{ result.qualitative_rating|lower }}">{{ result.qualitative_rating }}</span></dd>
|
||||
{% endif %}
|
||||
{% if result.novelty_flag %}
|
||||
<dt>Novelty</dt><dd>{{ result.novelty_flag }}</dd>
|
||||
{% endif %}
|
||||
@@ -102,11 +105,16 @@
|
||||
{%- elif s.raw_value >= mb.norm_max -%}
|
||||
<span class="badge badge-{{ 'p1_fail' if mb.lower_is_better else 'valid' }}">at/above max{{ ' (worst)' if mb.lower_is_better else '' }}</span>
|
||||
{%- else -%}
|
||||
{% set pct = ((s.raw_value - mb.norm_min) / (mb.norm_max - mb.norm_min) * 100) | int %}
|
||||
{% set raw_pct = (s.raw_value - mb.norm_min) / (mb.norm_max - mb.norm_min) * 100 %}
|
||||
{# For lower_is_better metrics, raw_pct alone measures distance from norm_min,
|
||||
not quality -- a value near norm_min (excellent, cost near its floor) would
|
||||
otherwise render as a near-empty bar. Invert so the bar and percentage always
|
||||
mean "how good", matching the normalized score's own higher-is-better convention. #}
|
||||
{% set pct = ((100 - raw_pct) if mb.lower_is_better else raw_pct) | int %}
|
||||
<div class="metric-bar-container">
|
||||
<div class="metric-bar" style="width: {{ pct }}%"></div>
|
||||
</div>
|
||||
<span class="metric-bar-label">~{{ pct }}%{{ ' ↓' if mb.lower_is_better else '' }}</span>
|
||||
<span class="metric-bar-label">~{{ pct }}%{{ ' (lower is better)' if mb.lower_is_better else '' }}</span>
|
||||
{%- endif -%}
|
||||
{%- else -%}
|
||||
—
|
||||
|
||||
@@ -26,11 +26,11 @@
|
||||
|
||||
{% if statuses %}
|
||||
<div class="filter-row">
|
||||
<span>Filter:</span>
|
||||
<a href="{{ url_for('results.results_domain', domain_name=domain.name) }}"
|
||||
<span>Status:</span>
|
||||
<a href="{{ url_for('results.results_domain', domain_name=domain.name, rating=rating_filter) }}"
|
||||
class="btn btn-sm {{ '' if status_filter else 'btn-primary' }}">All ({{ total_results }})</a>
|
||||
{% for s, cnt in statuses.items() %}
|
||||
<a href="{{ url_for('results.results_domain', domain_name=domain.name, status=s) }}"
|
||||
<a href="{{ url_for('results.results_domain', domain_name=domain.name, status=s, rating=rating_filter) }}"
|
||||
class="btn btn-sm {{ 'btn-primary' if status_filter == s else '' }}">
|
||||
{{ s }} ({{ cnt }})
|
||||
</a>
|
||||
@@ -38,9 +38,25 @@
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{% if ratings %}
|
||||
<div class="filter-row">
|
||||
<span>Rating:</span>
|
||||
<a href="{{ url_for('results.results_domain', domain_name=domain.name, status=status_filter) }}"
|
||||
class="btn btn-sm {{ '' if not rating_filter else 'btn-primary' }}">All</a>
|
||||
{% for rt in ['HIGH', 'MEDIUM', 'LOW'] %}
|
||||
{% if rt in ratings %}
|
||||
<a href="{{ url_for('results.results_domain', domain_name=domain.name, status=status_filter, rating=rt) }}"
|
||||
class="btn btn-sm {{ 'btn-primary' if rating_filter == rt else '' }}">
|
||||
{{ rt }} ({{ ratings[rt] }})
|
||||
</a>
|
||||
{% endif %}
|
||||
{% endfor %}
|
||||
</div>
|
||||
{% endif %}
|
||||
|
||||
{% if not results %}
|
||||
{% if status_filter %}
|
||||
<p class="empty">No results with status "{{ status_filter }}" in this domain.</p>
|
||||
{% if status_filter or rating_filter %}
|
||||
<p class="empty">No results matching that filter in this domain.</p>
|
||||
{% else %}
|
||||
<p class="empty">No results for this domain yet. <a href="{{ url_for('pipeline.pipeline_form') }}">Run the pipeline</a> first.</p>
|
||||
{% endif %}
|
||||
@@ -52,6 +68,7 @@
|
||||
<th>Score</th>
|
||||
<th>Entities</th>
|
||||
<th>Status</th>
|
||||
<th>Rating</th>
|
||||
<th>Details</th>
|
||||
<th></th>
|
||||
</tr>
|
||||
@@ -69,6 +86,13 @@
|
||||
<span class="badge badge-{{ r.combination.status }}">{{ r.combination.status }}</span>
|
||||
{%- endif -%}
|
||||
</td>
|
||||
<td>
|
||||
{%- if r.qualitative_rating -%}
|
||||
<span class="badge badge-rating-{{ r.qualitative_rating|lower }}">{{ r.qualitative_rating }}</span>
|
||||
{%- else -%}
|
||||
—
|
||||
{%- endif -%}
|
||||
</td>
|
||||
<td class="block-reason-cell">
|
||||
{%- if r.domain_block_reason -%}
|
||||
{{ r.domain_block_reason }}
|
||||
|
||||
Reference in New Issue
Block a user