Add async pipeline with progress monitoring, resumability, and result transparency

Pipeline engine rewritten with combo-first loop: each combination is processed
through all requested passes before moving to the next, with incremental DB
saves after every step (crash-safe). Blocked combos now get result rows so they
appear in the results page with constraint violation reasons.

New pipeline_runs table tracks run lifecycle (pending/running/completed/failed/
cancelled). Web route launches pipeline in a background thread with its own DB
connection. HTMX polling partial shows live progress with per-pass breakdown.

Also: status guard prevents reviewed->scored downgrade, save_combination loads
existing status on dedup for correct resume, per-metric scores show domain
bounds + units + position bars, ensure_metric backfills units on existing rows.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
Simonson, Andrew
2026-02-18 15:30:52 -06:00
parent 8118a62242
commit d2028a642b
17 changed files with 1263 additions and 217 deletions

View File

@@ -21,7 +21,7 @@
<div class="filter-row">
<span>Filter:</span>
<a href="{{ url_for('results.results_domain', domain_name=domain.name) }}"
class="btn btn-sm {{ '' if status_filter else 'btn-primary' }}">All</a>
class="btn btn-sm {{ '' if status_filter else 'btn-primary' }}">All ({{ total_results }})</a>
{% for s, cnt in statuses.items() %}
<a href="{{ url_for('results.results_domain', domain_name=domain.name, status=s) }}"
class="btn btn-sm {{ 'btn-primary' if status_filter == s else '' }}">
@@ -32,7 +32,11 @@
{% endif %}
{% if not results %}
<p class="empty">No results yet. <a href="{{ url_for('pipeline.pipeline_form') }}">Run the pipeline</a> first.</p>
{% if status_filter %}
<p class="empty">No results with status "{{ status_filter }}" in this domain.</p>
{% else %}
<p class="empty">No results for this domain yet. <a href="{{ url_for('pipeline.pipeline_form') }}">Run the pipeline</a> first.</p>
{% endif %}
{% else %}
<table>
<thead>
@@ -41,7 +45,7 @@
<th>Score</th>
<th>Entities</th>
<th>Status</th>
<th>Novelty</th>
<th>Details</th>
<th></th>
</tr>
</thead>
@@ -49,10 +53,18 @@
{% for r in results %}
<tr>
<td>{{ loop.index }}</td>
<td class="score-cell">{{ "%.4f"|format(r.composite_score) }}</td>
<td class="score-cell">{{ "%.4f"|format(r.composite_score) if r.composite_score else '—' }}</td>
<td>{{ r.combination.entities|map(attribute='name')|join(' + ') }}</td>
<td><span class="badge badge-{{ r.combination.status }}">{{ r.combination.status }}</span></td>
<td>{{ r.novelty_flag or '—' }}</td>
<td class="block-reason-cell">
{%- if r.combination.status == 'blocked' and r.combination.block_reason -%}
{{ r.combination.block_reason }}
{%- elif r.novelty_flag -%}
{{ r.novelty_flag }}
{%- else -%}
{%- endif -%}
</td>
<td>
<a href="{{ url_for('results.result_detail', domain_name=domain.name, combo_id=r.combination.id) }}"
class="btn btn-sm">View</a>