andreferraro
6a45368cb0
- Added scripts/extract_article_contents.py for batch scraping with stealth Foxcape and triple extraction (Trafilatura, Newspaper4k, Readability) - Created unit, integration, and E2E test suite in tests/test_extract_article_contents.py (90/90 passing) - Updated specs/003-article-content-extractor and README.md with usage documentation and CLI contracts - Passed ruff linting/formatting and mypy type checking cleanly
3.0 KiB
3.0 KiB
JSON Schema Contract: Article Content Multi-Engine Extractor
1. Schema de Entrada (Input JSON)
O arquivo de entrada deve conter a seguinte estrutura JSON:
{
"query": "string (opcional)",
"language": "string (opcional, ex: 'pt', 'es', 'en')",
"locale": "string (opcional, ex: 'BR', 'AR', 'US')",
"total_itens": "integer (opcional)",
"items": [
{
"titulo": "string (obrigatório)",
"url": "string (obrigatório, URL HTTP/HTTPS)",
"subtitulo": "string (opcional)",
"quando_publicado": "string (opcional)",
"pagina": "integer (opcional, default: 1)"
}
]
}
2. Schema de Saída (Output JSON)
O arquivo gerado conterá a estrutura consolidada:
{
"source_file": "out/river_plate.json",
"processed_at": "2026-08-20T15:30:00.000000+00:00",
"total_articles": 1,
"successful_articles": 1,
"failed_articles": 0,
"articles": [
{
"input_meta": {
"titulo": "Los puntajes de River vs. Independiente Santa Fe",
"url": "https://www.tycsports.com/river-plate/los-puntajes-id755914.html",
"subtitulo": "River empató sem gols...",
"quando_publicado": "Thu, 20 Aug 2026 03:27:26 GMT",
"pagina": 1
},
"extraction_status": "success",
"error_message": null,
"crawled_url": "https://www.tycsports.com/river-plate/los-puntajes-id755914.html",
"page_title": "Los puntajes de River vs. Independiente Santa Fe - TyC Sports",
"http_status": 200,
"trafilatura": {
"title": "Los puntajes de River vs. Independiente Santa Fe",
"author": "Ernesto Provitilo",
"date": "2026-08-20",
"description": "El análisis uno por uno...",
"categories": ["River Plate", "Copa Sudamericana"],
"tags": ["River", "Santa Fe"],
"canonical_url": "https://www.tycsports.com/river-plate/los-puntajes-id755914.html",
"text": "Franco Armani (6): Seguro en las pocas llegadas del rival...",
"raw_json": { ... },
"error": null
},
"newspaper4k": {
"title": "Los puntajes de River vs. Independiente Santa Fe",
"authors": ["Ernesto Provitilo"],
"publish_date": "2026-08-20T03:27:26",
"text": "Franco Armani (6): Seguro en las pocas llegadas del rival...",
"summary": "Resumo gerado por NLP com as sentenças principais...",
"keywords": ["river", "santa fe", "puntajes", "armani"],
"top_image": "https://media.tycsports.com/adjuntos/800/2026/08/20/armani.jpg",
"images": [
"https://media.tycsports.com/adjuntos/800/2026/08/20/armani.jpg"
],
"meta_data": { ... },
"error": null
},
"readability": {
"title": "Los puntajes de River vs. Independiente Santa Fe",
"short_title": "Los puntajes de River",
"cleaned_html": "<div><p>Franco Armani (6): Seguro en las pocas llegadas...</p></div>",
"cleaned_text": "Franco Armani (6): Seguro en las pocas llegadas...",
"error": null
}
}
]
}