andreferraro
6a45368cb0
- Added scripts/extract_article_contents.py for batch scraping with stealth Foxcape and triple extraction (Trafilatura, Newspaper4k, Readability) - Created unit, integration, and E2E test suite in tests/test_extract_article_contents.py (90/90 passing) - Updated specs/003-article-content-extractor and README.md with usage documentation and CLI contracts - Passed ruff linting/formatting and mypy type checking cleanly
89 lines
3.0 KiB
Markdown
89 lines
3.0 KiB
Markdown
# JSON Schema Contract: Article Content Multi-Engine Extractor
|
|
|
|
## 1. Schema de Entrada (Input JSON)
|
|
|
|
O arquivo de entrada deve conter a seguinte estrutura JSON:
|
|
|
|
```json
|
|
{
|
|
"query": "string (opcional)",
|
|
"language": "string (opcional, ex: 'pt', 'es', 'en')",
|
|
"locale": "string (opcional, ex: 'BR', 'AR', 'US')",
|
|
"total_itens": "integer (opcional)",
|
|
"items": [
|
|
{
|
|
"titulo": "string (obrigatório)",
|
|
"url": "string (obrigatório, URL HTTP/HTTPS)",
|
|
"subtitulo": "string (opcional)",
|
|
"quando_publicado": "string (opcional)",
|
|
"pagina": "integer (opcional, default: 1)"
|
|
}
|
|
]
|
|
}
|
|
```
|
|
|
|
---
|
|
|
|
## 2. Schema de Saída (Output JSON)
|
|
|
|
O arquivo gerado conterá a estrutura consolidada:
|
|
|
|
```json
|
|
{
|
|
"source_file": "out/river_plate.json",
|
|
"processed_at": "2026-08-20T15:30:00.000000+00:00",
|
|
"total_articles": 1,
|
|
"successful_articles": 1,
|
|
"failed_articles": 0,
|
|
"articles": [
|
|
{
|
|
"input_meta": {
|
|
"titulo": "Los puntajes de River vs. Independiente Santa Fe",
|
|
"url": "https://www.tycsports.com/river-plate/los-puntajes-id755914.html",
|
|
"subtitulo": "River empató sem gols...",
|
|
"quando_publicado": "Thu, 20 Aug 2026 03:27:26 GMT",
|
|
"pagina": 1
|
|
},
|
|
"extraction_status": "success",
|
|
"error_message": null,
|
|
"crawled_url": "https://www.tycsports.com/river-plate/los-puntajes-id755914.html",
|
|
"page_title": "Los puntajes de River vs. Independiente Santa Fe - TyC Sports",
|
|
"http_status": 200,
|
|
"trafilatura": {
|
|
"title": "Los puntajes de River vs. Independiente Santa Fe",
|
|
"author": "Ernesto Provitilo",
|
|
"date": "2026-08-20",
|
|
"description": "El análisis uno por uno...",
|
|
"categories": ["River Plate", "Copa Sudamericana"],
|
|
"tags": ["River", "Santa Fe"],
|
|
"canonical_url": "https://www.tycsports.com/river-plate/los-puntajes-id755914.html",
|
|
"text": "Franco Armani (6): Seguro en las pocas llegadas del rival...",
|
|
"raw_json": { ... },
|
|
"error": null
|
|
},
|
|
"newspaper4k": {
|
|
"title": "Los puntajes de River vs. Independiente Santa Fe",
|
|
"authors": ["Ernesto Provitilo"],
|
|
"publish_date": "2026-08-20T03:27:26",
|
|
"text": "Franco Armani (6): Seguro en las pocas llegadas del rival...",
|
|
"summary": "Resumo gerado por NLP com as sentenças principais...",
|
|
"keywords": ["river", "santa fe", "puntajes", "armani"],
|
|
"top_image": "https://media.tycsports.com/adjuntos/800/2026/08/20/armani.jpg",
|
|
"images": [
|
|
"https://media.tycsports.com/adjuntos/800/2026/08/20/armani.jpg"
|
|
],
|
|
"meta_data": { ... },
|
|
"error": null
|
|
},
|
|
"readability": {
|
|
"title": "Los puntajes de River vs. Independiente Santa Fe",
|
|
"short_title": "Los puntajes de River",
|
|
"cleaned_html": "<div><p>Franco Armani (6): Seguro en las pocas llegadas...</p></div>",
|
|
"cleaned_text": "Franco Armani (6): Seguro en las pocas llegadas...",
|
|
"error": null
|
|
}
|
|
}
|
|
]
|
|
}
|
|
```
|