200 lines
13 KiB
Markdown
200 lines
13 KiB
Markdown
# Implementation Plan: Article Consolidation and Hygiene Runtime
|
|
|
|
**Branch**: `006-article-consolidation-runtime` | **Date**: 2026-08-23 | **Spec**: [`specs/006-article-consolidation-runtime/spec.md`](file:///c:/Users/aferr/Projects/AFTech/DunaMedia/TextNLPClassifierApp/specs/006-article-consolidation-runtime/spec.md)
|
|
|
|
**Input**: Feature specification from `specs/006-article-consolidation-runtime/spec.md`
|
|
|
|
---
|
|
|
|
## Summary
|
|
|
|
The Article Consolidation and Hygiene Runtime is an ephemeral, deterministic Python CLI tool that processes exactly one article unit per run alongside a canonical Entity Context Profile (ECP) snapshot. It coordinates extraction payloads from three extractors (`trafilatura`, `newspaper4k`, `readability`), performs 100% LLM extractive hygiene via certified low-cost models, validates grounded candidate selections and micro-repairs without regular expressions or manual keyword dictionaries, applies an ECP relevance gate via `src.classifier.InherenceClassifier`, adds entity-relative sentiment and native-language tags, renders canonical Markdown with YAML front matter, persists machine-readable manifests and state atomically in SQLite (WAL), and transmits sanitized observability telemetry directly to Langfuse.
|
|
|
|
---
|
|
|
|
## Technical Context
|
|
|
|
**Language/Version**: Python >= 3.10 (strictly matching `requires-python` in `pyproject.toml`)
|
|
**Primary Dependencies**:
|
|
- Standard library (`json`, `urllib.parse`, `unicodedata`, `difflib`, `sqlite3`, `pathlib`, `hashlib`, `typing`, `signal`)
|
|
- `beautifulsoup4` (DOM parsing)
|
|
- `marko` (CommonMark AST parsing)
|
|
- `jsonschema` + `referencing` (JSON Schema Draft 2020-12 validation with immutable local schema registry)
|
|
- `python-dateutil` (ISO 8601 date parsing)
|
|
- `pyyaml` (Safe YAML front matter serialization)
|
|
- `httpx` (HTTP client for Model Gateway)
|
|
- `langfuse` (Observability SDK >= 4.7)
|
|
- Existing monorepo modules (`src.language`, `src.classifier.InherenceClassifier`)
|
|
|
|
**Storage**: SQLite 3 (WAL mode, configurable busy timeout, short transactions, native backup API) + Local filesystem (atomic temporary files and renames)
|
|
**Testing**: `pytest` (unit, contract, mock integration, security, load, fault injection, operations resilience), static zero-regex multi-parser analyzer (Python AST + JSON pattern check + Promptfoo YAML check), `promptfoo` (offline prompt evaluations in dev/CI)
|
|
**Target Platform**: Ambiente suportado pelo repositório e pelo deployment pipeline
|
|
**Project Type**: Python CLI Tool (Ephemeral process, no API, no internal worker pool)
|
|
**Performance Goals**: Sustained throughput >= 100 articles/hour under external orchestrator concurrency; p50, p95, and p99 latency and cost empirically measured and approved in staging before go-live
|
|
**Constraints**: Zero regular expressions (`re`) in text processing, schemas, and assertions; zero manual keyword dictionaries; zero expensive/powerful models in runtime roles (or internal ECP classifier); 11 critical release invariants (all 0); secret exposure = 0; strict 10-step hygiene harness; input size limit enforcement (`INVALID_ARTICLE_SCHEMA` on overflow); no generic unused abstractions
|
|
**Scale/Scope**: 1 article per CLI invocation, 9 versioned contracts, 10 fault injection scenarios, 8 security scenarios (SEC-001 to SEC-008), operational resilience testing (FR-081), complete golden set with holdout, contract tests over all 20 real reference units
|
|
|
|
---
|
|
|
|
## Constitution Check
|
|
|
|
*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
|
|
|
|
| Principle / Rule | Compliance Status | Description |
|
|
|:---|:---|:---|
|
|
| **I. Simplicity Mandate (FR-001)** | **PASS** | Minimum sufficient code, no generic unused abstractions, small responsibilities share modules, zero redundant local metrics system. |
|
|
| **II. Strict Dependency Policy (FR-002)** | **PASS** | Standard library prioritized; 7-factor qualitative evaluation completed; no agent frameworks, no trivial libraries, no duplicate ECP schema, no custom regex parsers. |
|
|
| **III. CLI Ephemeral Interface (FR-003)** | **PASS** | Python CLI processing 1 article per invocation, no internal batch loops, no internal worker pools, external concurrency. |
|
|
| **IV. Regex Prohibition (FR-015, FR-070)** | **PASS** | Multi-parser static check in CI enforces zero `re` calls in Python AST, zero `pattern` keys in JSON schemas, and zero regex assertions in Promptfoo YAML. |
|
|
| **V. Agnostic Gateway & Cheap Models (FR-041, FR-042)** | **PASS** | Logical roles `runtime_primary` and `runtime_fallback` limited to certified cheap models; zero powerful models in runtime or internal classifier. |
|
|
| **VI. Zero Hallucination Grounding (FR-023, FR-026)** | **PASS** | LLM returns only candidate IDs and bounded diffs; harness strictly enforces candidate grounding and reverses ungrounded micro-repairs. |
|
|
| **VII. Atomic Persistence & Idempotency (FR-009, FR-050)** | **PASS** | SQLite WAL mode + file atomic renames; deterministic fingerprinting; hash-based crash reconciliation. |
|
|
|
|
---
|
|
|
|
## Project Structure
|
|
|
|
### Documentation & 9 Versioned Contracts (this feature)
|
|
|
|
```text
|
|
specs/006-article-consolidation-runtime/
|
|
├── spec.md # Feature specification (v1.0.0)
|
|
├── plan.md # Implementation plan (this file)
|
|
├── research.md # Technical research & decisions (Phase 0)
|
|
├── data-model.md # Entity definitions, SQLite tables & formats (Phase 1)
|
|
├── quickstart.md # Runnable verification guide (Phase 1)
|
|
├── contracts/ # All 9 independent versioned contracts (Phase 1)
|
|
│ ├── article-input.schema.json # Contract 1: Article Input Unit (v1.0.0)
|
|
│ ├── ecp-snapshot.schema.json # Contract 2: ECP Snapshot Canonical Reference (v1.0.0)
|
|
│ ├── runtime-config.schema.json # Contract 3: Runtime Configuration (v1.0.0)
|
|
│ ├── candidates-payload.schema.json # Contract 4: Candidate Payload to LLM (v1.0.0)
|
|
│ ├── hygiene-response.schema.json # Contract 5: Hygiene LLM Response (v1.0.0)
|
|
│ ├── repair-operations.schema.json # Contract 6: Repair Operations Schema (v1.0.0)
|
|
│ ├── enrichment-response.schema.json # Contract 7: Enrichment LLM Response (v1.0.0)
|
|
│ ├── manifest-output.schema.json # Contract 8: Output Manifest (v1.0.0)
|
|
│ ├── prompts-contract.md # Contract 9: Versioned Prompts Contract (v1.0.0)
|
|
│ └── cli-interface.md # Interface: CLI Command Interface (v1.0.0)
|
|
└── checklists/
|
|
└── requirements.md # Quality checklist
|
|
```
|
|
|
|
### Source Code & Fixtures (repository layout)
|
|
|
|
```text
|
|
src/
|
|
├── __init__.py
|
|
├── cli/
|
|
│ ├── __init__.py
|
|
│ ├── consolidate.py # Main single-article CLI entry point (--input-article, --ecp-snapshot, --config) with SIGTERM/SIGINT graceful shutdown
|
|
│ ├── preflight.py # Preflight validation CLI (--config verified against src/core/release-metadata.json)
|
|
│ ├── smoke.py # Smoke test CLI (--config, --fixture)
|
|
│ ├── reconcile.py # State & artifact reconciliation CLI (--config)
|
|
│ └── telemetry_flush.py # Deferred telemetry flush CLI (--config)
|
|
├── core/
|
|
│ ├── __init__.py
|
|
│ ├── config.py # Runtime configuration loading & SHA-256 release metadata verification (exact byte hash)
|
|
│ ├── limits.py # Input size limit validation (FR-056)
|
|
│ ├── fingerprint.py # Deterministic fingerprint calculator
|
|
│ ├── state_machine.py # Python explicit state machine & SQLite logger
|
|
│ └── release-metadata.json # Packaged release metadata (SHA-256 hashes of config, prompts, schemas, ECP config)
|
|
├── candidate/
|
|
│ ├── __init__.py
|
|
│ ├── parser.py # DOM, CommonMark AST, JSON-LD structural parsing
|
|
│ ├── equivalence.py # Backbone ordering & equivalence mapping (no deletion)
|
|
│ └── models.py # CandidateObject definitions
|
|
├── hygiene/
|
|
│ ├── __init__.py
|
|
│ ├── harness.py # 10-step validation harness
|
|
│ ├── repairs.py # Controlled micro-repair validator (difflib/unicodedata)
|
|
│ └── assembler.py # Grounded intermediate Markdown assembler
|
|
├── ecp/
|
|
│ ├── __init__.py
|
|
│ └── adapter.py # ECP adapter invoking src.classifier.InherenceClassifier & referencing.Registry local schema
|
|
├── enrichment/
|
|
│ ├── __init__.py
|
|
│ └── harness.py # Entity sentiment & native tags validator
|
|
├── gateway/
|
|
│ ├── __init__.py
|
|
│ ├── client.py # Agnostic Model Gateway (primary/fallback)
|
|
│ └── adapters.py # Provider-specific minimal HTTP adapters (Groq, DeepSeek)
|
|
├── storage/
|
|
│ ├── __init__.py
|
|
│ ├── sqlite_store.py # SQLite WAL state store, transition logs, and native backup/restore API
|
|
│ ├── file_store.py # Atomic filesystem writer (temp + rename)
|
|
│ └── markdown_renderer.py # Canonical YAML front matter & body renderer
|
|
└── observability/
|
|
├── __init__.py
|
|
├── langfuse_tracer.py # Spans, generations, metrics, scores & secret redaction
|
|
└── structured_logger.py # Sanitized JSON logger
|
|
|
|
prompts/
|
|
├── article_content_hygiene.v1.txt
|
|
└── article_sentiment_tags.v1.txt
|
|
|
|
evals/
|
|
├── promptfoo.config.yaml # Promptfoo test suite configuration
|
|
├── golden_set/ # Full golden dataset with reference truths
|
|
├── holdout/ # Holdout dataset (never used in few-shot)
|
|
└── reference_20/ # 20 reference regression cases
|
|
|
|
examples/
|
|
├── sample_article_valid.json # Executable fixture: valid inherent article unit
|
|
├── sample_article_tangential.json # Executable fixture: non-inherent article unit
|
|
└── sample_ecp_snapshot.json # Executable fixture: canonical ECP snapshot
|
|
|
|
runtime_config.local.json # Executable local validation configuration fixture
|
|
|
|
tests/
|
|
├── contract/ # Contract tests for all 9 versioned contracts (evaluated over 20 real reference units)
|
|
│ ├── test_article_input_contract.py
|
|
│ ├── test_ecp_snapshot_contract.py
|
|
│ ├── test_runtime_config_contract.py
|
|
│ ├── test_candidates_payload_contract.py
|
|
│ ├── test_hygiene_response_contract.py
|
|
│ ├── test_repair_operations_contract.py
|
|
│ ├── test_enrichment_response_contract.py
|
|
│ ├── test_manifest_output_contract.py
|
|
│ └── test_prompts_contract.py
|
|
├── unit/
|
|
│ ├── test_fingerprint.py
|
|
│ ├── test_input_limits.py
|
|
│ ├── test_candidate_parser.py
|
|
│ ├── test_equivalence_mapping.py
|
|
│ ├── test_hygiene_harness.py
|
|
│ ├── test_repairs_validator.py
|
|
│ ├── test_ecp_adapter.py
|
|
│ ├── test_enrichment_harness.py
|
|
│ ├── test_model_gateway.py
|
|
│ ├── test_sqlite_store.py
|
|
│ └── test_file_store.py
|
|
├── integration/
|
|
│ ├── test_e2e_pipeline_mock.py
|
|
│ ├── test_idempotency_concurrency.py
|
|
│ └── test_operations_resilience.py # Backup/restore, graceful shutdown, signals, pending telemetry, rollback, credential rotation, certified model rotation, and uncertified rotation rejection (FR-081)
|
|
├── security/
|
|
│ └── test_security_scenarios.py # Parametrized tests for SEC-001 to SEC-008
|
|
├── load/
|
|
│ └── test_load_100_art_per_hour.py # 100 articles/hour staging load benchmark
|
|
├── fault_injection/
|
|
│ └── test_fault_injection_scenarios.py # 10 normative fault scenarios (FLT-001 to FLT-010)
|
|
└── scripts/
|
|
└── check_zero_regex.py # Multi-parser static check (Python AST + JSON pattern + Promptfoo YAML)
|
|
```
|
|
|
|
---
|
|
|
|
## Complexity Tracking
|
|
|
|
> **No architectural violations detected. All modules adhere strictly to simplicity and dependency rules.**
|
|
|
|
| Component | Design Choice | Simplicity Justification |
|
|
|:---|:---|:---|
|
|
| Orchestration | Explicit Python state machine | Eliminates LangGraph / LangChain overhead while providing full deterministic lifecycle tracking. |
|
|
| Model Gateway | Direct 2-adapter primary/fallback | Eliminates smart router complexity while guaranteeing cheap model enforcement and fallback. |
|
|
| Candidate Model | Unified `CandidateObject` dictionary/dataclass | Avoids class hierarchies for candidate types while capturing all required structural flags. |
|
|
| Persistence | SQLite (WAL) + Local Filesystem | Eliminates distributed database/queue dependencies; provides ACID state within local process boundaries. |
|
|
| ECP Integration | Direct invocation of `src.classifier.InherenceClassifier` | Leverages existing monorepo module without creating redundant remote microservices. |
|
|
| Observability | Native Langfuse dashboards + SQLite outage queue | Eliminates redundant local metric systems while fulfilling all reporting requirements. |
|
|
| Operations | Native SQLite backup API + Signal handling in CLI | Fulfills FR-081 requirements using Python standard library without external daemons. |
|
|
| Regex Prohibition | `unicodedata`, `difflib`, NLP tokenizers | Multi-parser validation ensures zero regex across code, schemas, and evals without runtime overhead. |
|