andreferraro/TextNLPClassifierApp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(runtime): implement single-article consolidation runtime and modularize codebase

Browse Source
This commit is contained in:
andreferraro
2026-08-24 00:14:07 -03:00
parent e1e0be1353
commit 23de7d8fe7
176 changed files with 266754 additions and 10179 deletions
Download Patch File Download Diff File Expand all files Collapse all files
specs/006-article-consolidation-runtime/data-model.md
+263
View File
@@ -0,0 +1,263 @@
# Data Model: Article Consolidation and Hygiene Runtime
**Feature Branch**: `006-article-consolidation-runtime`
**Date**: 2026-08-23
**Status**: Complete
---
## 1. Domain Entities & Relationships
```mermaid
erDiagram
ArticleInputUnit ||--o{ CandidateObject : extracts
ArticleInputUnit ||--|| ECPSnapshot : references
ArticleInputUnit ||--|| StateMachineRecord : tracks
CandidateObject ||--o{ TextRepairOperation : receives
StateMachineRecord ||--o| OutputManifest : persists
StateMachineRecord ||--o| PublishedMarkdown : renders
StateMachineRecord ||--o{ StateTransitionLog : logs
StateMachineRecord ||--o{ TelemetryEvent : queues
```
---
## 2. Entity Definitions
### 2.1 Article Input Unit (`article_input`)
Represents the incoming single article JSON payload. Rejects explicit batch wrappers via `"articles": false`. Unknown fields in the input are preserved in the original object without alteration.
| Field | Type | Description | Required |
|:---|:---|:---|:---|
| `selected_extractor` | enum | `trafilatura` \| `newspaper4k` \| `readability` | Yes |
| `crawled_url` | string (URL) \| null | Crawled URL | No |
| `error_message` | string \| null | Upstream error message | No |
| `extraction_status` | string \| null | Upstream status | No |
| `http_status` | integer \| null | HTTP response status code | No |
| `input_meta` | object \| null | Metadata map (`url`, `titulo`, `subtitulo`, `quando_publicado`) | No |
| `page_title` | string \| null | Raw HTML page title | No |
| `trafilatura` | object \| null | Trafilatura extraction output (accepts `raw_json` as object, string, or null) | No |
| `newspaper4k` | object \| null | Newspaper4k extraction output | No |
| `readability` | object \| null | Readability extraction output | No |
| `articles` | false | Explicitly forbidden (batch wrapper rejection) | No |
*Note on Validation*: The runtime validates that the collective extraction sources provide at least one resolvable source URL, at least one non-empty candidate title, processable text, and usable content in `selected_extractor`. Total byte size is checked against `limits.max_input_bytes` before invoking remote providers (failing with `INVALID_ARTICLE_SCHEMA` if exceeded).
---
### 2.2 Entity Context Profile Snapshot (`ecp_snapshot`)
Represents the complete, integral ECP Snapshot received by the runtime. The runtime validates the payload locally against the monorepo's canonical ECP schema (`src/adapters/ecp/schemas/ecp-profile.schema.json`) registered in `referencing.Registry` matching `$ref: "https://schemas.aftech.internal/ecp/v1/ecp-profile.schema.json"` without HTTP lookups. It extracts identity and version metadata (`qid`, `canonical_name`, `version`) for manifest and trace recording.
| Field | Type | Description | Required |
|:---|:---|:---|:---|
| *(opaque payload)* | object | Complete canonical ECP snapshot validated via `referencing.Registry` | Yes |
| `qid` | string | Extracted canonical Wikidata / Entity QID (e.g. `Q148`) | Extracted |
| `canonical_name` | string | Extracted entity canonical name | Extracted |
| `version` | string | Extracted semantic version of the referenced ECP profile | Extracted |
---
### 2.3 Candidate Object (`candidate_object`)
Single structural element extracted from the article payloads. Preserves all candidates without destructive deduplication.
| Field | Type | Description | Required |
|:---|:---|:---|:---|
| `candidate_id` | string | Opaque unique ID (e.g. `cand_blk_001`, `cand_title_001`) | Yes |
| `type` | enum | `title` \| `subtitle` \| `author` \| `date` \| `paragraph` \| `heading` \| `list_item` \| `quote` \| `link` \| `image` | Yes |
| `extractor_source` | enum | `trafilatura` \| `newspaper4k` \| `readability` \| `input_meta` \| `page_title` | Yes |
| `source_field` | string | Origin field (e.g. `text`, `article_html`, `title`) | Yes |
| `original_text_or_url` | string | Exact original text or URL content | Yes |
| `structural_representation` | string | Markdown/HTML/AST structural snippet | Yes |
| `order_index` | integer | Position index in extractor backbone | Yes |
| `parent_candidate_id` | string \| null | ID of parent element (for nested list items, blockquotes, etc.) | No |
| `equivalences` | array[string] | List of candidate IDs representing equivalent content from other extractors | Yes |
| `content_hash` | string (64-char hex) | Deterministic content hash | Yes |
| `structural_flags` | object | Purely structural flags (e.g. `{"heading_level": 2}`) | Yes |
*Note on Projection*: The LLM prompt receives `CandidatesPayload`, which is a clean, normalized projection of these internal `CandidateObject` instances.
---
### 2.4 Text Repair Operation (`text_repair`)
Micro-repair proposed by the LLM and validated by the harness.
| Field | Type | Description | Required |
|:---|:---|:---|:---|
| `target_candidate_id` | string | ID of the target block or metadata candidate | Yes |
| `original_fragment` | string | Exact substring in candidate to replace | Yes |
| `replacement_fragment` | string | Validated replacement text | Yes |
| `category` | enum | `encoding` \| `unicode` \| `spacing` \| `punctuation_corruption` \| `obvious_typo` | Yes |
| `rationale` | string | Short explanation | Yes |
| `is_accepted` | boolean | Validation outcome from harness | Yes |
| `rejection_reason` | string \| null | Code if rejected (`SENSITIVE_ENTITY`, `AMBIGUOUS_TARGET`, `OUT_OF_CATEGORY`, etc.) | No |
---
### 2.5 State Machine Record (`state_record`)
SQLite table `article_states` storing runtime execution status.
| Column | SQLite Type | Description |
|:---|:---|:---|
| `fingerprint` | TEXT (PK, 64-char hex) | Deterministic content hash of the execution |
| `source_url` | TEXT | Resolved source URL |
| `selected_extractor` | TEXT | Extractor used as backbone |
| `current_state` | TEXT | `received` \| `validated` \| `content_cleaned` \| `ecp_approved` \| `ecp_rejected` \| `enriched` \| `completed_text` \| `failed` |
| `final_status` | TEXT | `completed_text` \| `rejected_ecp` \| `failed_validation` \| `failed_processing` \| NULL |
| `generate_markdown` | INTEGER | 1 if Markdown generated, 0 otherwise |
| `markdown_path` | TEXT | Path to generated `.md` file (or NULL) |
| `manifest_path` | TEXT | Path to generated `.result.json` file |
| `markdown_hash` | TEXT (64-char hex) | Content hash of generated `.md` file (or NULL) |
| `manifest_hash` | TEXT (64-char hex) | Content hash of generated `.result.json` file |
| `ecp_category` | TEXT | `DIRECT_INHERENT` \| `CONTEXTUAL_INHERENT` \| `TANGENTIAL` \| `NOT_RELATED` \| NULL |
| `ecp_confidence` | REAL | Confidence score (0.0 to 1.0) |
| `functional_versions_json` | TEXT (JSON) | Consolidated versions of contracts, ECP reference, config, prompts, and models |
| `trace_id` | TEXT | Langfuse trace identifier |
| `terminal_error_code` | TEXT | Normative error code if failed |
| `error_metadata_json` | TEXT (JSON) | Sanitized minimal error metadata (stack trace in technical log only) |
| `created_at` | TEXT (ISO 8601) | Timestamp of ingestion |
| `updated_at` | TEXT (ISO 8601) | Timestamp of last transition |
---
### 2.6 State Transition Log (`state_transitions`)
SQLite table `state_transitions` tracking execution lifecycle.
| Column | SQLite Type | Description |
|:---|:---|:---|
| `id` | INTEGER (PK AUTO) | Unique transition ID |
| `fingerprint` | TEXT (FK) | Reference to `article_states.fingerprint` |
| `from_state` | TEXT | Starting state |
| `to_state` | TEXT | Destination state |
| `start_time` | TEXT (ISO 8601) | Transition start timestamp |
| `end_time` | TEXT (ISO 8601) | Transition end timestamp |
| `duration_ms` | REAL | Elapsed milliseconds |
| `result` | TEXT | `success` \| `failure` \| `skipped` |
| `metadata_json` | TEXT (JSON) | Transition context metadata |
---
### 2.7 Telemetry Event (`pending_telemetry`)
SQLite table `pending_telemetry` for resilient deferred delivery to Langfuse when the network/service is unreachable.
| Column | SQLite Type | Description |
|:---|:---|:---|
| `event_id` | TEXT (PK) | UUID / Unique event ID |
| `fingerprint` | TEXT | Associated article fingerprint |
| `trace_id` | TEXT | Associated trace ID |
| `event_type` | TEXT | `span` \| `generation` \| `score` |
| `payload_json` | TEXT (JSON) | Sanitized telemetry event payload |
| `created_at` | TEXT (ISO 8601) | Creation timestamp |
| `retry_count` | INTEGER | Number of transmission attempts |
| `last_error` | TEXT | Last error message |
---
### 2.8 Release Metadata Contract (`src/core/release-metadata.json`)
Immutable packaged artifact recording certified configurations for preflight verification. `runtime_config_sha256` is strictly calculated as the exact file byte SHA-256 hash (`hashlib.sha256(Path(config_path).read_bytes()).hexdigest()`).
```json
{
"release_version": "1.0.0",
"runtime_config_sha256": "06a2769f7aa3a15a1e61880f171ecebc0d29094ab2499616243e59c0aecf340f",
"prompts_hashes": {
"article_content_hygiene": "f8a9c2b1d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0",
"article_sentiment_tags": "d4e1b7a2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0"
},
"schemas_versions": {
"article_input": "1.0.0",
"ecp_snapshot": "1.0.0",
"runtime_config": "1.0.0",
"candidates_payload": "1.0.0",
"hygiene_response": "1.0.0",
"repair_operations": "1.0.0",
"enrichment_response": "1.0.0",
"manifest_output": "1.0.0"
},
"certified_models": {
"runtime_primary": { "provider": "groq", "model": "openai/gpt-oss-20b", "role_config_version": "1.0.0" },
"runtime_fallback": { "provider": "deepseek", "model": "deepseek-v4-flash", "role_config_version": "1.0.0" }
},
"ecp_classifier_config_hash": "a1b2c3d4e5f67890a1b2c3d4e5f67890a1b2c3d4e5f67890a1b2c3d4e5f67890"
}
```
---
### 2.9 Output Manifest (`<fingerprint>.result.json`)
Structure of the machine-readable output manifest complying with `manifest-output.schema.json`.
```json
{
"schema_version": "1.0.0",
"fingerprint": "a1b2c3d4e5f67890a1b2c3d4e5f67890a1b2c3d4e5f67890a1b2c3d4e5f67890",
"source_url": "https://example.com/noticia-123",
"selected_extractor": "trafilatura",
"final_status": "completed_text",
"generate_markdown": true,
"markdown_path": "out/a1b2c3d4e5f67890a1b2c3d4e5f67890a1b2c3d4e5f67890a1b2c3d4e5f67890.md",
"markdown_hash": "06a2769f7aa3a15a1e61880f171ecebc0d29094ab2499616243e59c0aecf340f",
"ecp_classification": {
"category": "DIRECT_INHERENT",
"confidence": 0.95,
"rationale": "Article directly analyzes the economic policy of the entity.",
"evidences": ["trecho textual fundamentado 1", "trecho textual fundamentado 2"]
},
"enrichment": {
"sentiment": "positive",
"tags": ["Economia", "Política Monetária", "Inflação"]
},
"provider_versions": {
"hygiene": { "provider": "groq", "model": "openai/gpt-oss-20b", "role_config_version": "1.0.0" },
"enrichment": { "provider": "groq", "model": "openai/gpt-oss-20b", "role_config_version": "1.0.0" }
},
"model_versions": {
"runtime_primary": { "provider": "groq", "model": "openai/gpt-oss-20b", "role_config_version": "1.0.0" },
"runtime_fallback": { "provider": "deepseek", "model": "deepseek-v4-flash", "role_config_version": "1.0.0" }
},
"prompt_versions": {
"article_content_hygiene": { "version": "1.0.0", "hash": "f8a9c2b1d3e4f5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f9a0" },
"article_sentiment_tags": { "version": "1.0.0", "hash": "d4e1b7a2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0" }
},
"config_version": "1.0.0",
"trace_id": "trace_run_20260823_001",
"error_codes": []
}
```
---
### 2.10 Published Markdown (`<fingerprint>.md`)
Format of the rendered Markdown document with YAML front matter.
```markdown
---
title: "Título Principal do Artigo Publicado"
subtitle: "Subtítulo editorial detalhado"
author: "Nome do Autor"
published_at: "2026-08-23T14:00:00Z"
source_url: "https://example.com/noticia-123"
sentiment: positive
tags:
- Economia
- Política Monetária
- Inflação
ecp_qid: "Q148"
ecp_canonical_name: "Entidade Alvo"
ecp_category: DIRECT_INHERENT
ecp_confidence: 0.95
---
# Título Principal do Artigo Publicado
*Subtítulo editorial detalhado*
Primeiro parágrafo do artigo com [link grounded](https://example.com/referencia) e texto limpo.
## Intertítulo Editorial
Segundo parágrafo contendo citação textual sem alterações indevidas.
![Legenda da imagem grounded](https://example.com/imagem.jpg)
Parágrafo de encerramento sem notas de rodapé publicitárias ou chamadas de redes sociais.
```