feat(extractor): implement multi-engine article content extractor

- Added scripts/extract_article_contents.py for batch scraping with stealth Foxcape and triple extraction (Trafilatura, Newspaper4k, Readability) - Created unit, integration, and E2E test suite in tests/test_extract_article_contents.py (90/90 passing) - Updated specs/003-article-content-extractor and README.md with usage documentation and CLI contracts - Passed ruff linting/formatting and mypy type checking cleanly
2026-08-20 19:22:20 -03:00
parent 6e3d57619b
commit 6a45368cb0
85 changed files with 18345 additions and 3897 deletions
@@ -0,0 +1,88 @@
+# JSON Schema Contract: Article Content Multi-Engine Extractor
+
+## 1. Schema de Entrada (Input JSON)
+
+O arquivo de entrada deve conter a seguinte estrutura JSON:
+
+```json
+{
+  "query": "string (opcional)",
+  "language": "string (opcional, ex: 'pt', 'es', 'en')",
+  "locale": "string (opcional, ex: 'BR', 'AR', 'US')",
+  "total_itens": "integer (opcional)",
+  "items": [
+    {
+      "titulo": "string (obrigatório)",
+      "url": "string (obrigatório, URL HTTP/HTTPS)",
+      "subtitulo": "string (opcional)",
+      "quando_publicado": "string (opcional)",
+      "pagina": "integer (opcional, default: 1)"
+    }
+  ]
+}
+```
+
+---
+
+## 2. Schema de Saída (Output JSON)
+
+O arquivo gerado conterá a estrutura consolidada:
+
+```json
+{
+  "source_file": "out/river_plate.json",
+  "processed_at": "2026-08-20T15:30:00.000000+00:00",
+  "total_articles": 1,
+  "successful_articles": 1,
+  "failed_articles": 0,
+  "articles": [
+    {
+      "input_meta": {
+        "titulo": "Los puntajes de River vs. Independiente Santa Fe",
+        "url": "https://www.tycsports.com/river-plate/los-puntajes-id755914.html",
+        "subtitulo": "River empató sem gols...",
+        "quando_publicado": "Thu, 20 Aug 2026 03:27:26 GMT",
+        "pagina": 1
+      },
+      "extraction_status": "success",
+      "error_message": null,
+      "crawled_url": "https://www.tycsports.com/river-plate/los-puntajes-id755914.html",
+      "page_title": "Los puntajes de River vs. Independiente Santa Fe - TyC Sports",
+      "http_status": 200,
+      "trafilatura": {
+        "title": "Los puntajes de River vs. Independiente Santa Fe",
+        "author": "Ernesto Provitilo",
+        "date": "2026-08-20",
+        "description": "El análisis uno por uno...",
+        "categories": ["River Plate", "Copa Sudamericana"],
+        "tags": ["River", "Santa Fe"],
+        "canonical_url": "https://www.tycsports.com/river-plate/los-puntajes-id755914.html",
+        "text": "Franco Armani (6): Seguro en las pocas llegadas del rival...",
+        "raw_json": { ... },
+        "error": null
+      },
+      "newspaper4k": {
+        "title": "Los puntajes de River vs. Independiente Santa Fe",
+        "authors": ["Ernesto Provitilo"],
+        "publish_date": "2026-08-20T03:27:26",
+        "text": "Franco Armani (6): Seguro en las pocas llegadas del rival...",
+        "summary": "Resumo gerado por NLP com as sentenças principais...",
+        "keywords": ["river", "santa fe", "puntajes", "armani"],
+        "top_image": "https://media.tycsports.com/adjuntos/800/2026/08/20/armani.jpg",
+        "images": [
+          "https://media.tycsports.com/adjuntos/800/2026/08/20/armani.jpg"
+        ],
+        "meta_data": { ... },
+        "error": null
+      },
+      "readability": {
+        "title": "Los puntajes de River vs. Independiente Santa Fe",
+        "short_title": "Los puntajes de River",
+        "cleaned_html": "<div><p>Franco Armani (6): Seguro en las pocas llegadas...</p></div>",
+        "cleaned_text": "Franco Armani (6): Seguro en las pocas llegadas...",
+        "error": null
+      }
+    }
+  ]
+}
+```