Files

T

andreferraro 6e3d57619b feat(extractor): add Google News headlines extractor with Foxcape headless and URL resolution

- Add standalone CLI script scripts/extract_google_news.py for Google News RSS scraping
- Integrate foxcape in headless mode as primary stealth anti-bot engine
- Implement parallel article URL resolution using googlenewsdecoder and ThreadPoolExecutor
- Support language and regional locale mapping (-l, --lang, --locale)
- Implement real-time progress logging in stderr and --silent flag
- Add unit, integration, and live E2E tests in tests/test_extract_google_news.py
- Add full SpecKit documentation (specs/002-google-news-extractor/)
- Create comprehensive README.md covering both NLP Classifier and Google News Extractor

2026-08-20 11:50:16 -03:00

22 KiB

Raw Blame History

Graph Report - TextNLPClassifierApp (2026-08-20)

Corpus Check

147 files · ~65,826 words
Verdict: corpus is large enough that graph structure adds value.

Summary

775 nodes · 931 edges · 99 communities (61 shown, 38 thin omitted)
Extraction: 96% EXTRACTED · 4% INFERRED · 0% AMBIGUOUS · INFERRED: 36 edges (avg confidence: 0.94)
Token cost: 0 input · 0 output

Graph Freshness

Built from commit: 67cc40f9
Run git rev-parse HEAD and compare to check if the graph is stale.
Run graphify update . after code changes (no API cost).

Task Planning
Convergence Workflow
SpecKit Utilities
Graphify Commands
speckit-analyze/SKILL.md
POC Readiness & Requirements Quality Checklist: Multilingual NLP Entity Inherence Classifier
Feature Specification Template
Graphify Rules
Implementation Planning
Feature Specification
Task Generation
Project Constitution
Constitution Template
Graphify Exports
Ponytail Configuration
Implementation Planning Template
Ponytail Help
Checklist Generation
Clarification Workflow
Implementation Workflow
Graph Query
Constitution Workflow
Feature Branch Creation
Ponytail Audit
Ponytail Metrics
Ponytail Review
Task Issue Conversion
Checklist Template
Graphify Watch Mode
Graphify Hooks
Graphify Updates
Ponytail Debt
Repository Merge
Media Transcription
Extraction Specification
Graphify Workflows
main
1. Technical Decisions & Tradeoffs
1. Input Schemas
1. Basic CLI Usage Examples
1. Standard Streams & Exit Codes
ECPSnapshot
Tasks: Multilingual NLP Entity Inherence Classifier (POC)
detect_language
test_models.py
content_northvolt_de.md
content_presal_pt.md
content_tangential_es.md
adapters/__init__.py
src/__init__.py
de/contextual.md
de/direct.md
de/not_related.md
de/tangential.md
en/contextual.md
en/direct.md
en/not_related.md
en/tangential.md
es/contextual.md
es/direct.md
es/not_related.md
es/tangential.md
fr/contextual.md
fr/direct.md
fr/not_related.md
fr/tangential.md
it/contextual.md
it/direct.md
it/not_related.md
it/tangential.md
pt/contextual.md
pt/direct.md
pt/not_related.md
pt/tangential.md
tests/__init__.py
text-nlp-classifier
get_hl_gl_ceid
Extrator de Notícias do Google News — Guia Completo de Funcionamento
extract_google_news.py
ExtractionResult
test_extract_google_news.py
Implementation Tasks: Google News Headlines Extractor
Feature Specification: Google News Headlines Extractor
1. Cenários Práticos de Uso
Implementation Plan: Google News Headlines Extractor
scripts/__init__.py
SearchQuery
1. Technical Decisions & Tradeoffs
General Readiness Checklist: Google News Headlines Extractor
1. Entidades de Domínio & DTOs
Specification Quality Checklist: Google News Headlines Extractor
CLI Contract: Google News Headlines Extractor
build_parser
sample_rss_xml

God Nodes (most connected - your core abstractions)

ECPSnapshot - 31 edges
InherenceClassifier - 25 edges
DecisionCategory - 18 edges
ClassificationResult - 17 edges
main() - 14 edges
LocalEmbeddingsAdapter - 14 edges
LLMFallbackAdapter - 14 edges
detect_language() - 14 edges
Tasks: [FEATURE NAME] - 13 edges
SearchQuery - 12 edges

Surprising Connections (you probably didn't know these)

main() --uses--> ECPSnapshot [INFERRED] classify.py → src/models.py
test_extract_google_news_orchestration_mocked() --uses--> ExtractionResult [INFERRED] tests/test_extract_google_news.py → scripts/extract_google_news.py
test_classification_result_serialization() --uses--> DecisionCategory [INFERRED] tests/test_models.py → src/models.py
test_ecp_snapshot_defaults() --uses--> ECPSnapshot [INFERRED] tests/test_models.py → src/models.py
test_ecp_snapshot_missing_required() --uses--> ECPSnapshot [INFERRED] tests/test_models.py → src/models.py

Import Cycles

None detected.

Communities (99 total, 38 thin omitted)

Community 0 - "Task Planning"

Cohesion: 0.07 Nodes (26): Dependencies & Execution Order, Format: [ID] [P?] [Story] Description, Implementation for User Story 1, Implementation for User Story 2, Implementation for User Story 3, Implementation Strategy, Incremental Delivery, MVP First (User Story 1 Only) (+18 more)

Community 1 - "Convergence Workflow"

Cohesion: 0.12 Nodes (15): 1. Initialize Convergence Context, 2. Load Artifacts (Progressive Disclosure), 3. Build the Intent Inventory, 4. Assess the Codebase and Classify Findings, 5. Assign Severity, 6. Present the In-Session Findings Summary, 7. Append Convergence Tasks (or report converged), 8. Provide Next Actions (Handoff) (+7 more)

Community 2 - "SpecKit Utilities"

Cohesion: 0.23 Nodes (13): Find-SpecifyRoot(), Format-SpecKitCommand(), Get-CurrentBranch(), Get-FeaturePathsEnv(), Get-InvokeSeparator(), Get-NormalizedPriority(), Get-Python3Command(), Get-RepoRoot() (+5 more)

Community 3 - "Graphify Commands"

Cohesion: 0.08 Nodes (24): For /graphify add and --watch, For /graphify query, For the commit hook and native CLAUDE.md integration, For --update and --cluster-only, /graphify, Honesty Rules, Interpreter guard for subcommands, Part A - Structural extraction for code files (+16 more)

Community 4 - "speckit-analyze/SKILL.md"

Cohesion: 0.08 Nodes (25): 1. Initialize Analysis Context, 2. Load Artifacts (Progressive Disclosure), 3. Build Semantic Models, 4. Detection Passes (Token-Efficient Analysis), 5. Severity Assignment, 6. Produce Compact Analysis Report, 7. Provide Next Actions, 8. Offer Remediation (+17 more)

Community 5 - "POC Readiness & Requirements Quality Checklist: Multilingual NLP Entity Inherence Classifier"

Cohesion: 0.05 Nodes (34): 1. Requirement Completeness & Scope Boundaries, 2. Requirement Clarity & Decision Semantics, 3. Requirement Consistency & Alignment, 4. Acceptance Criteria & Measurability, 5. Scenario & Edge Case Coverage, Notes, POC Readiness & Requirements Quality Checklist: Multilingual NLP Entity Inherence Classifier, Content Quality (+26 more)

Community 6 - "Feature Specification Template"

Cohesion: 0.15 Nodes (12): Assumptions, Edge Cases, Feature Specification: [FEATURE NAME], Functional Requirements, Key Entities (include if feature involves data), Measurable Outcomes, Requirements (mandatory), Success Criteria (mandatory) (+4 more)

Community 8 - "Implementation Planning"

Cohesion: 0.18 Nodes (10): Completion Report, Done When, Key rules, Mandatory Post-Execution Hooks, Outline, Phase 0: Outline & Research, Phase 1: Design & Contracts, Phases (+2 more)

Community 9 - "Feature Specification"

Cohesion: 0.18 Nodes (10): Completion Report, Done When, For AI Generation, Mandatory Post-Execution Hooks, Outline, Pre-Execution Checks, Quick Guidelines, Section Requirements (+2 more)

Community 10 - "Task Generation"

Cohesion: 0.18 Nodes (10): Checklist Format (REQUIRED), Completion Report, Done When, Mandatory Post-Execution Hooks, Outline, Phase Structure, Pre-Execution Checks, Task Generation Rules (+2 more)

Community 11 - "Project Constitution"

Cohesion: 0.18 Nodes (10): Core Principles, Governance, [PRINCIPLE_1_NAME], [PRINCIPLE_2_NAME], [PRINCIPLE_3_NAME], [PRINCIPLE_4_NAME], [PRINCIPLE_5_NAME], [PROJECT_NAME] Constitution (+2 more)

Community 12 - "Constitution Template"

Cohesion: 0.18 Nodes (10): Core Principles, Governance, [PRINCIPLE_1_NAME], [PRINCIPLE_2_NAME], [PRINCIPLE_3_NAME], [PRINCIPLE_4_NAME], [PRINCIPLE_5_NAME], [PROJECT_NAME] Constitution (+2 more)

Community 13 - "Graphify Exports"

Cohesion: 0.22 Nodes (8): graphify reference: extra exports and benchmark, Step 6b - Wiki (only if --wiki flag), Step 7 - Neo4j export (only if --neo4j or --neo4j-push flag), Step 7a - FalkorDB export (only if --falkordb or --falkordb-push flag), Step 7b - SVG export (only if --svg flag), Step 7c - GraphML export (only if --graphml flag), Step 7d - MCP server (only if --mcp flag), Step 8 - Token reduction benchmark (only if total_words > 5000)

Community 14 - "Ponytail Configuration"

Cohesion: 0.22 Nodes (8): Boundaries, Intensity, Output, Persistence, Ponytail, Rules, The ladder, When NOT to be lazy

Community 15 - "Implementation Planning Template"

Cohesion: 0.22 Nodes (8): Complexity Tracking, Constitution Check, Documentation (this feature), Implementation Plan: [FEATURE], Project Structure, Source Code (repository root), Summary, Technical Context

Community 16 - "Ponytail Help"

Cohesion: 0.25 Nodes (7): Configure Default Mode, Deactivate, Levels, More, Ponytail Help, Skills, Update

Community 17 - "Checklist Generation"

Cohesion: 0.25 Nodes (7): Anti-Examples: What NOT To Do, Checklist Purpose: "Unit Tests for English", Example Checklist Types & Sample Items, Execution Steps, Post-Execution Checks, Pre-Execution Checks, User Input

Community 18 - "Clarification Workflow"

Cohesion: 0.29 Nodes (6): Completion Report, Done When, Mandatory Post-Execution Hooks, Outline, Pre-Execution Checks, User Input

Community 19 - "Implementation Workflow"

Cohesion: 0.29 Nodes (6): Completion Report, Done When, Mandatory Post-Execution Hooks, Outline, Pre-Execution Checks, User Input

Community 20 - "Graph Query"

Cohesion: 0.33 Nodes (5): For /graphify explain, For /graphify path, graphify reference: query, path, explain, Step 0 — Constrained query expansion (REQUIRED before traversal), Step 1 — Traversal

Community 21 - "Constitution Workflow"

Cohesion: 0.33 Nodes (5): Outline, Post-Execution Checks, Pre-Execution Checks, Scope Guard, User Input

Community 23 - "Ponytail Audit"

Cohesion: 0.40 Nodes (4): Boundaries, Hunt, Output, Tags

Community 24 - "Ponytail Metrics"

Cohesion: 0.40 Nodes (4): Boundaries, Honesty boundary, Ponytail Gain, Scoreboard

Community 25 - "Ponytail Review"

Cohesion: 0.40 Nodes (4): Boundaries, Examples, Format, Scoring

Community 26 - "Task Issue Conversion"

Cohesion: 0.40 Nodes (4): Outline, Post-Execution Checks, Pre-Execution Checks, User Input

Community 27 - "Checklist Template"

Cohesion: 0.40 Nodes (4): [Category 1], [Category 2], [CHECKLIST TYPE] Checklist: [FEATURE NAME], Notes

Community 28 - "Graphify Watch Mode"

Cohesion: 0.50 Nodes (3): For /graphify add, For --watch, graphify reference: add a URL and watch a folder

Community 29 - "Graphify Hooks"

Cohesion: 0.50 Nodes (3): For git commit hook, For native CLAUDE.md integration, graphify reference: commit hook and native CLAUDE.md integration

Community 30 - "Graphify Updates"

Cohesion: 0.50 Nodes (3): For --cluster-only, For --update (incremental re-extraction), graphify reference: incremental update and cluster-only

Community 31 - "Ponytail Debt"

Cohesion: 0.50 Nodes (3): Boundaries, Output, Scan

Community 40 - "main"

Cohesion: 0.19 Nodes (14): CaptureFixture, Path, main(), Ponto de entrada do script CLI., Valida execução padrão do CLI com saída JSON no stdout., Valida gravação em arquivo com criação automática de diretórios pais., Valida tratamento de erro e código de saída 1 para parâmetro vazio., Valida tratamento de erro e código de saída 2 para falhas de rede. (+6 more)

Community 41 - "1. Technical Decisions & Tradeoffs"

Cohesion: 0.22 Nodes (8): 1. Technical Decisions & Tradeoffs, 2. Standardized Error Handling Strategy, Decision 1: Execution Engine & CLI Architecture, Decision 2: Multilingual Language Detection & Normalization (Tier 1 Core), Decision 3: Materialized ECP Snapshot Contract & Matching Logic, Decision 4: Tier 2 (Embeddings) & Tier 3 (LLM) Optional Adapters, Decision 5: Controlled 24-Case POC Benchmark Suite, Technical Research & Architecture Decisions (POC)

Community 42 - "1. Input Schemas"

Cohesion: 0.25 Nodes (7): 1.1 ECP Snapshot Schema (snapshot.json), 1.2 Content Item Schema (content.md), 1. Input Schemas, 2.1 Classification Success Result Schema (result.json), 2.2 Error Result Schema, 2. Output Schemas, Data Models & Schemas (POC)

Community 43 - "2. Basic CLI Usage Examples"

Cohesion: 0.25 Nodes (7): 1. Prerequisites & Installation, 2.1 Direct Inherence (Portuguese), 2.2 Contextual Inherence via Graph Snapshot (German), 2.3 Tangential Mention (Spanish), 2. Basic CLI Usage Examples, 3. Running the Controlled 24-Case Benchmark, Quickstart & Validation Guide (POC)

Community 44 - "2. Standard Streams & Exit Codes"

Cohesion: 0.29 Nodes (6): 1.1 Arguments & Options, 1. Command Line Interface, 2.1 Exit Codes, 2.2 Standard Output (stdout) / Standard Error (stderr), 2. Standard Streams & Exit Codes, CLI Contract & Interface Specification (POC)

Community 45 - "ECPSnapshot"

Cohesion: 0.06 Nodes (55): ABC, Enum, parametrize, BaseNLPAdapter, Base abstract adapter interface for optional Tier 2 / Tier 3 NLP enhancers., Abstract interface for pluggable NLP classification adapters., Return True if the underlying provider or model is installed and configured., Compute semantic similarity score between text and a set of candidate terms. (+47 more)

Community 46 - "Tasks: Multilingual NLP Entity Inherence Classifier (POC)"

Cohesion: 0.15 Nodes (13): Dependencies & Execution Order, Implementation for User Story 1, Implementation for User Story 2, Implementation for User Story 3, Implementation Strategy, Phase 1: Setup (Shared Infrastructure), Phase 2: Foundational (Blocking Prerequisites), Phase 3: User Story 1 - Core Tier 1 Deterministic Classification & CLI (Priority: P1) [MVP] (+5 more)

Community 47 - "detect_language"

Cohesion: 0.14 Nodes (21): count_phrase_occurrences(), match_phrase_in_text(), Check if a normalized phrase appears in normalized text with word boundary…, Count occurrences of a phrase in text., Classify inherence of content against an ECP snapshot., detect_language(), extract_words(), normalize_text() (+13 more)

Community 48 - "test_models.py"

Cohesion: 0.09 Nodes (29): emit_error(), main(), parse_args(), Namespace, ClassificationError, ErrorCode, Any, extract_evidence_snippets() (+21 more)

Community 80 - "get_hl_gl_ceid"

Cohesion: 0.25 Nodes (8): get_hl_gl_ceid(), Mapeia idioma e locale para os parâmetros hl, gl e ceid do Google News., Valida o mapeamento padrão de idiomas para pares (hl, gl, ceid)., Valida a sobrescrita geográfica quando o argumento locale é especificado., Valida fallback dinâmico para idiomas regionais não listados explicitamente., test_get_hl_gl_ceid_default_mappings(), test_get_hl_gl_ceid_dynamic_fallback(), test_get_hl_gl_ceid_with_custom_locale()

Community 81 - "Extrator de Notícias do Google News — Guia Completo de Funcionamento"

Cohesion: 0.08 Nodes (24): 1. Visão geral (arquitetura), 2.1 DTO de entrada (googlenews_etl/application/dtos/extract_news_dto.py), 2.2 Value Object de validação (googlenews_etl/domain/entities/search_query.py), 2. Entrada, 3.1 O caso de uso (googlenews_etl/application/use_cases/extract_news_use_case.py), 3.2 A porta (googlenews_etl/domain/ports/news_extractor_port.py), 3.3.1 Inicialização: sessão HTTP com impersonação de browser, 3.3.2 Mapeamento idioma → parâmetros hl/gl (_get_hl_gl) (+16 more)

Community 82 - "extract_google_news.py"

Cohesion: 0.20 Nodes (14): extract_google_news(), _fetch_rss_content(), NewsArticle, _normalize_text_for_comparison(), parse_google_news_rss(), Remove pontuação e espaços extras para comparação de redundância., Parseia o XML do RSS do Google News e extrai os itens estruturados., Resolve em paralelo as URLs intermediárias do Google News para os links finais… (+6 more)

Community 83 - "ExtractionResult"

Cohesion: 0.29 Nodes (5): ExtractionResult, Any, Resultado consolidado da extração., Valida E2E o fluxo completo de busca, parsing e resolução de URLs reais ao vivo., test_e2e_extract_google_news_live_pipeline()

Community 84 - "test_extract_google_news.py"

Cohesion: 0.21 Nodes (11): Resolve a URL intermediária do Google News para a URL real do veículo., resolve_article_url(), Testes unitários e de integração para o Extrator de Manchetes do Google News.…, Valida o parsing do feed RSS, higienização de tags HTML e deduplicação., Valida fallback gracioso de URL quando não é link do Google News ou em erro., Valida resolução bem-sucedida de URL do Google News para o portal destino., Valida E2E que o decodificador resolve uma URL real do Google News para o…, test_e2e_resolve_real_google_news_url() (+3 more)

Community 85 - "Implementation Tasks: Google News Headlines Extractor"

Cohesion: 0.14 Nodes (14): Implementation for User Story 1, Implementation for User Story 2, Implementation for User Story 3, Implementation Tasks: Google News Headlines Extractor, Phase 1: Setup (Shared Infrastructure), Phase 2: Foundational (Blocking Prerequisites), Phase 3: User Story 1 - Extração Básica de Notícias por Assunto e Idioma (Priority: P1) 🌟 MVP, Phase 4: User Story 2 - Filtragem Regional e Edição Geográfica (Priority: P2) (+6 more)

Community 86 - "Feature Specification: Google News Headlines Extractor"

Cohesion: 0.18 Nodes (11): Clarifications, Edge Cases, Feature Specification: Google News Headlines Extractor, Functional Requirements, Requirements (mandatory), Session 2026-08-20, Success Criteria (mandatory), User Scenarios & Testing (mandatory) (+3 more)

Community 87 - "2. Cenários Práticos de Uso"

Cohesion: 0.20 Nodes (9): 1. Pré-requisitos e Instalação, 2. Cenários Práticos de Uso, 3. Validação dos Testes Automatizados e Linter, Cenário 1: River Plate — Argentina (Espanhol / 2 Páginas / Salvar em Arquivo), Cenário 2: Cruzeiro — Brasil (Português / Formatado no Terminal), Cenário 3: Fórmula 1 — Reino Unido (Inglês), Cenário 4: Integração em Pipeline com jq (Modo Silencioso), Cenário 5: Extração Rápida com Links Brutos (Sem Resolução de URLs) (+1 more)

Community 88 - "Implementation Plan: Google News Headlines Extractor"

Cohesion: 0.29 Nodes (7): Architecture & Pipeline, Documentation (this feature), Implementation Plan: Google News Headlines Extractor, Project Structure, Source Code, Summary, Technical Context

Community 90 - "SearchQuery"

Cohesion: 0.20 Nodes (6): Value Object com parâmetros de busca validados., SearchQuery, Valida a consolidação do ExtractionResult a partir da busca mockada com URLs…, Valida as regras de negócio e limites de SearchQuery., test_extract_google_news_orchestration_mocked(), test_search_query_validation()

Community 91 - "1. Technical Decisions & Tradeoffs"

Cohesion: 0.25 Nodes (7): 1. Technical Decisions & Tradeoffs, Decision 1: Motor de Requisição e Scraping com foxcape em Modo Headless, Decision 2: Endpoint RSS do Google News vs. Scraping de DOM, Decision 3: Mapeamento de Idioma e Locale (hl, gl, ceid), Decision 4: Resolução de URLs do Google News via googlenewsdecoder, Decision 5: Logging em Tempo Real no stderr e Segregação de Streams, Research: Google News Headlines Extractor

Community 92 - "General Readiness Checklist: Google News Headlines Extractor"

Cohesion: 0.29 Nodes (7): CLI Interface & Parameter Contracts, Data Sanitization & Article Extraction, Error Handling & Edge Cases, General Readiness Checklist: Google News Headlines Extractor, Non-Functional & Operational Readiness, Notes, Scraping Engine & Feed Mapping

Community 93 - "1. Entidades de Domínio & DTOs"

Cohesion: 0.29 Nodes (6): 1.1 SearchQuery (Parâmetros da Busca), 1.2 NewsArticle (Item de Notícia), 1.3 ExtractionResult (Saída Estruturada Consolidada), 1. Entidades de Domínio & DTOs, 2. Esquema JSON de Saída, Data Model: Google News Headlines Extractor

Community 94 - "Specification Quality Checklist: Google News Headlines Extractor"

Cohesion: 0.33 Nodes (5): Content Quality, Feature Readiness, Notes, Requirement Completeness, Specification Quality Checklist: Google News Headlines Extractor

Community 95 - "CLI Contract: Google News Headlines Extractor"

Cohesion: 0.33 Nodes (6): 1. Comando e Argumentos, 2. Códigos de Saída (Exit Codes), 3. Protocolo de Streams (Stdout / Stderr), Argumentos de Linha de Comando, CLI Contract: Google News Headlines Extractor, Sintaxe

Community 98 - "build_parser"

Cohesion: 0.67 Nodes (3): ArgumentParser, build_parser(), Cria e configura o parser de argumentos CLI.

Community 99 - "sample_rss_xml"

Cohesion: 0.67 Nodes (3): fixture, Fixture que fornece o conteúdo do XML de exemplo para testes offline., sample_rss_xml()

Knowledge Gaps

361 isolated node(s): text-nlp-classifier, MatchedGraphEntity, graphify, Usage, What graphify is for (+356 more) These have ≤1 connection - possible missing edges or undocumented components.
38 thin communities (<3 nodes) omitted from report — run graphify query to explore isolated nodes.

22 KiB Raw Blame History