andreferraro/TextNLPClassifierApp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: add deterministic content extractor selector engine with F1 consensus

Browse Source
This commit is contained in:
andreferraro
2026-08-20 22:09:43 -03:00
parent 6a45368cb0
commit ff7a50e0eb
46 changed files with 18503 additions and 2813 deletions
Download Patch File Download Diff File Expand all files Collapse all files
graphify-out/GRAPH_REPORT.md
+122 -47
View File
@@ -1,16 +1,16 @@
# Graph Report - TextNLPClassifierApp (2026-08-20)
## Corpus Check
- 161 files · ~78,549 words
- 174 files · ~91,961 words
- Verdict: corpus is large enough that graph structure adds value.
## Summary
- 1000 nodes · 1220 edges · 115 communities (77 shown, 38 thin omitted)
- Extraction: 97% EXTRACTED · 3% INFERRED · 0% AMBIGUOUS · INFERRED: 39 edges (avg confidence: 0.95)
- 1235 nodes · 1548 edges · 131 communities (93 shown, 38 thin omitted)
- Extraction: 97% EXTRACTED · 3% INFERRED · 0% AMBIGUOUS · INFERRED: 51 edges (avg confidence: 0.95)
- Token cost: 0 input · 0 output
## Graph Freshness
- Built from commit: `6e3d5761`
- Built from commit: `6a45368c`
- Run `git rev-parse HEAD` and compare to check if the graph is stale.
- Run `graphify update .` after code changes (no API cost).
@@ -56,10 +56,10 @@
- 1. Input Schemas
- 2. Basic CLI Usage Examples
- 2. Standard Streams & Exit Codes
- classifier.py
- ClassificationResult
- InherenceClassifier
- detect_language
- ClassificationError
- classifier.py
- main
- content_northvolt_de.md
- content_presal_pt.md
- content_tangential_es.md
@@ -110,12 +110,12 @@
- 🧠 TextNLPClassifierApp
- Extraction Pipeline Checklist: Article Content Multi-Engine Extractor
- sample_rss_xml
- main
- models.py
- ECPSnapshot
- Feature Specification: Multilingual NLP Entity Inherence Classifier (POC)
- 4. Requisitos Funcionais (FR)
- Tasks: Article Content Multi-Engine Extractor
- models.py
- LocalEmbeddingsAdapter
- Implementation Plan: Article Content Multi-Engine Extractor
- 2. Cenários de Validação
- 1. Technical Decisions & Tradeoffs
@@ -124,35 +124,50 @@
- Specification Quality Checklist: Multilingual NLP Entity Inherence Classifier
- CLI Contract: Article Content Multi-Engine Extractor
- JSON Schema Contract: Article Content Multi-Engine Extractor
- PRD — Seleção determinística da biblioteca de extração de conteúdo
- select_article_extractor.py
- 1. Text Normalization Pipeline
- Tasks: Deterministic Article Content Selection
- select_article_extractor
- process_batch
- test_select_article_extractor.py
- Feature Specification: Deterministic Content Selection
- 2. Entity Descriptions & Fields
- Implementation Plan: Deterministic Article Content Selection
- Deterministic Content Selection Checklist: End-to-End Requirements Quality
- Quickstart: Deterministic Article Content Selection
- Specification Quality Checklist: Deterministic Content Selection
- CLI Interface Contract: Deterministic Article Content Selection
- JSON Schema Contract: Deterministic Article Content Selection
## God Nodes (most connected - your core abstractions)
1. `ECPSnapshot` - 31 edges
2. `InherenceClassifier` - 25 edges
3. `DecisionCategory` - 17 edges
4. `ClassificationResult` - 17 edges
5. `process_batch()` - 15 edges
6. `LocalEmbeddingsAdapter` - 14 edges
7. `LLMFallbackAdapter` - 14 edges
8. `detect_language()` - 14 edges
9. `main()` - 13 edges
10. `ArticleCrawler` - 13 edges
3. `select_article_extractor()` - 23 edges
4. `ExtractorName` - 21 edges
5. `DecisionCategory` - 17 edges
6. `ClassificationResult` - 17 edges
7. `process_batch()` - 15 edges
8. `process_batch()` - 14 edges
9. `LocalEmbeddingsAdapter` - 14 edges
10. `LLMFallbackAdapter` - 14 edges
## Surprising Connections (you probably didn't know these)
- `main()` --uses--> `ECPSnapshot` [INFERRED]
classify.py → src/models.py
- `main()` --uses--> `ErrorCode` [INFERRED]
classify.py → src/models.py
- `test_extract_google_news_orchestration_mocked()` --uses--> `ExtractionResult` [INFERRED]
tests/test_extract_google_news.py → scripts/extract_google_news.py
- `test_llm_adapter_interface()` --calls--> `LLMFallbackAdapter` [EXTRACTED]
tests/test_adapters.py → src/adapters/llm.py
- `classifier()` --uses--> `InherenceClassifier` [INFERRED]
tests/test_benchmark_24.py → src/classifier.py
- `test_classification_result_serialization()` --uses--> `DecisionCategory` [INFERRED]
tests/test_models.py → src/models.py
- `test_classification_error_serialization()` --uses--> `ErrorCode` [INFERRED]
tests/test_models.py → src/models.py
## Import Cycles
- None detected.
## Communities (115 total, 38 thin omitted)
## Communities (131 total, 38 thin omitted)
### Community 0 - "Task Planning"
Cohesion: 0.07
@@ -294,21 +309,21 @@ Nodes (7): 1. Prerequisites & Installation, 2.1 Direct Inherence (Portuguese), 2
Cohesion: 0.29
Nodes (6): 1.1 Arguments & Options, 1. Command Line Interface, 2.1 Exit Codes, 2.2 Standard Output (`stdout`) / Standard Error (`stderr`), 2. Standard Streams & Exit Codes, CLI Contract & Interface Specification (POC)
### Community 45 - "classifier.py"
### Community 45 - "ClassificationResult"
Cohesion: 0.14
Nodes (9): LocalEmbeddingsAdapter, Optional adapter for local multilingual semantic vector embeddings., LLMFallbackAdapter, Optional adapter for LLM fallback boundary disambiguation., Core deterministic classification engine (Tier 1 core)., Unit tests for optional adapter interfaces (Tier 2 / Tier 3)., test_classifier_with_adapter_flags(), test_embeddings_adapter_interface() (+1 more)
Nodes (12): ABC, BaseNLPAdapter, Base abstract adapter interface for optional Tier 2 / Tier 3 NLP enhancers., Abstract interface for pluggable NLP classification adapters., Return True if the underlying provider or model is installed and configured., Compute semantic similarity score between text and a set of candidate terms., Optionally refine an ambiguous classification result., LLMFallbackAdapter (+4 more)
### Community 46 - "InherenceClassifier"
Cohesion: 0.12
Nodes (27): InherenceClassifier, Tier 1 Deterministic NLP Entity Inherence Classifier., DecisionCategory, RelatedEntity, Adversarial and robustness test suite for Multilingual NLP Entity Inherence…, Run CLI via subprocess without --output and verify stdout is pure parseable…, Run CLI via subprocess with empty content and verify error code and exit code., Content about city/state governance of São Paulo against ECP for São Paulo FC. (+19 more)
### Community 47 - "detect_language"
Cohesion: 0.09
Nodes (30): count_phrase_occurrences(), match_phrase_in_text(), Check if a normalized phrase appears in normalized text with word boundary…, Count occurrences of a phrase in text., Classify inherence of content against an ECP snapshot., detect_language(), extract_words(), normalize_text() (+22 more)
### Community 47 - "classifier.py"
Cohesion: 0.10
Nodes (31): count_phrase_occurrences(), match_phrase_in_text(), Core deterministic classification engine (Tier 1 core)., Check if a normalized phrase appears in normalized text with word boundary…, Count occurrences of a phrase in text., Classify inherence of content against an ECP snapshot., detect_language(), extract_words() (+23 more)
### Community 48 - "ClassificationError"
Cohesion: 0.29
Nodes (3): ClassificationError, Any, test_classification_error_serialization()
### Community 48 - "main"
Cohesion: 0.31
Nodes (9): main(), parse_args(), Namespace, CLI execution tests covering flags, arguments, stdout, and error handling., test_cli_empty_content_file(), test_cli_missing_ecp_file(), test_cli_missing_required_ecp_field(), test_cli_output_file() (+1 more)
### Community 80 - "test_extract_article_contents.py"
Cohesion: 0.06
@@ -372,7 +387,7 @@ Nodes (6): 1. Comando e Argumentos, 2. Códigos de Saída (Exit Codes), 3. Proto
### Community 97 - "🧠 TextNLPClassifierApp"
Cohesion: 0.06
Nodes (33): 1. 🧠 Classificador de Conteúdo e Inerência (NLP / LLM / ECP), 1. Clonar o Repositório e Criar Ambiente Virtual, 1. Extração Completa Automática, 1. River Plate (Argentina / Espanhol / 2 Páginas / Salvar em Arquivo), 2. Amostragem Rápida (Limit 2 Notícias), 2. Cruzeiro (Brasil / Português / Formatado no Terminal), 2. 📰 Extrator de Manchetes do Google News, 2. Instalar Dependências (+25 more)
Nodes (35): 1. 🧠 Classificador de Conteúdo e Inerência (NLP / LLM / ECP), 1. Clonar o Repositório e Criar Ambiente Virtual, 1. Execução Padrão Automática, 2. Execução com Modo Verboso, 2. 📰 Extrator de Manchetes do Google News, 2. Instalar Dependências, 3. Baixar Binários do Navegador Stealth (Camoufox), 3. 📄 Extrator e Parser Multimotor de Artigos (+27 more)
### Community 98 - "Extraction Pipeline Checklist: Article Content Multi-Engine Extractor"
Cohesion: 0.05
@@ -382,13 +397,13 @@ Nodes (34): 1. Requirement Completeness, 2. Requirement Clarity & Non-Ambiguity,
Cohesion: 0.67
Nodes (3): fixture, Fixture que fornece o conteúdo do XML de exemplo para testes offline., sample_rss_xml()
### Community 100 - "main"
Cohesion: 0.23
Nodes (13): emit_error(), main(), parse_args(), Namespace, Enum, ErrorCode, str, CLI execution tests covering flags, arguments, stdout, and error handling. (+5 more)
### Community 100 - "models.py"
Cohesion: 0.24
Nodes (8): emit_error(), ClassificationError, ErrorCode, MatchedGraphEntity, Enum, str, Data models and validation schemas for Multilingual NLP Entity Inherence…, test_classification_error_serialization()
### Community 101 - "ECPSnapshot"
Cohesion: 0.22
Nodes (11): parametrize, ECPSnapshot, classifier(), fixture, Controlled 24-case benchmark suite for Multilingual NLP Entity Inherence…, test_benchmark_case(), Unit tests for ECP models, schema validation, and structured error handling., test_classification_result_serialization() (+3 more)
Cohesion: 0.19
Nodes (11): parametrize, ECPSnapshot, Any, classifier(), fixture, Controlled 24-case benchmark suite for Multilingual NLP Entity Inherence…, test_benchmark_case(), Unit tests for ECP models, schema validation, and structured error handling. (+3 more)
### Community 102 - "Feature Specification: Multilingual NLP Entity Inherence Classifier (POC)"
Cohesion: 0.14
@@ -402,9 +417,9 @@ Nodes (20): 1.1 Objetivo do Produto, 1. Visão Geral e Contexto, 2. Personas e C
Cohesion: 0.11
Nodes (18): Dependencies & Execution Order, Entrega Incremental, Implementation Strategy, Implementação da User Story 1, Implementação da User Story 2, Implementação da User Story 3, MVP First (User Story 1 Only), Oportunidades de Execução Paralela (+10 more)
### Community 105 - "models.py"
Cohesion: 0.16
Nodes (12): ABC, BaseNLPAdapter, Base abstract adapter interface for optional Tier 2 / Tier 3 NLP enhancers., Abstract interface for pluggable NLP classification adapters., Return True if the underlying provider or model is installed and configured., Compute semantic similarity score between text and a set of candidate terms., Optionally refine an ambiguous classification result., Optional local vector embeddings adapter (Tier 2). Disabled by default.… (+4 more)
### Community 105 - "LocalEmbeddingsAdapter"
Cohesion: 0.18
Nodes (7): LocalEmbeddingsAdapter, Optional local vector embeddings adapter (Tier 2). Disabled by default.…, Optional adapter for local multilingual semantic vector embeddings., Unit tests for optional adapter interfaces (Tier 2 / Tier 3)., test_classifier_with_adapter_flags(), test_embeddings_adapter_interface(), test_llm_adapter_interface()
### Community 106 - "Implementation Plan: Article Content Multi-Engine Extractor"
Cohesion: 0.17
@@ -438,25 +453,85 @@ Nodes (5): 1. Comando de Execução, 2. Argumentos e Flags, 3. Códigos de Saíd
Cohesion: 0.50
Nodes (3): 1. Schema de Entrada (Input JSON), 2. Schema de Saída (Output JSON), JSON Schema Contract: Article Content Multi-Engine Extractor
### Community 115 - "PRD — Seleção determinística da biblioteca de extração de conteúdo"
Cohesion: 0.07
Nodes (29): 10. Requisitos não funcionais, 11. Critérios de aceite, 12. Casos obrigatórios de teste, 13. Definition of Done, 1. Contexto, 2. Objetivo, 3.1 Incluído, 3.2 Fora do escopo (+21 more)
### Community 116 - "select_article_extractor.py"
Cohesion: 0.16
Nodes (17): BatchProcessingResult, break_priority_tie(), calculate_consensus_metrics(), ExtractorCandidate, form_active_set(), main(), parse_args(), Namespace (+9 more)
### Community 117 - "1. Text Normalization Pipeline"
Cohesion: 0.11
Nodes (18): 1. Text Normalization Pipeline, 2. 5-Token Shingles & Consensus Metrics, 3. Regras de Decisão, Empate Técnico e Desempate Hierárquico, 4. Estratégia de I/O Não Destrutiva e Escrita Atômica, Alternatives Considered, Context, Context, Context (+10 more)
### Community 118 - "Tasks: Deterministic Article Content Selection"
Cohesion: 0.11
Nodes (18): Dependencies & Execution Order, Implementation for User Story 1, Implementation for User Story 2, Implementation for User Story 3, Implementation Strategy, Incremental Delivery (Phases 4, 5 & 6), MVP First (Phases 1, 2 & 3), Parallel Opportunities (+10 more)
### Community 119 - "select_article_extractor"
Cohesion: 0.08
Nodes (37): ArticleSelectionResult, CandidateStatus, extract_candidate_data(), ExtractorName, Any, Enum, str, Extrai campo de texto, erro e calcula tokens/shingles para um motor. (+29 more)
### Community 120 - "process_batch"
Cohesion: 0.10
Nodes (26): atomic_save_json(), process_batch(), Path, Salva dados em JSON de forma atômica utilizando arquivo temporário e rename., Lê o JSON de entrada, valida a estrutura, processa todos os artigos e grava o…, Path, CT-012: A entrada já contém selected_extractor -> Recalcular e substituir…, CT-013: articles está vazio -> Gerar saída válida com articles vazio. (+18 more)
### Community 122 - "test_select_article_extractor.py"
Cohesion: 0.21
Nodes (14): generate_shingles(), normalize_text(), Executa a normalização determinística para comparação: 1. Decodificar entidades…, Gera conjunto de shingles ordenados de tamanho window_size (padrão 5). - Se…, Suíte de Testes Automatizados para o Seletor Determinístico de Extrator. Cobre…, Garante que marcação de imagem Markdown ![alt](url) seja descartada e link…, test_generate_shingles_empty(), test_generate_shingles_short_text() (+6 more)
### Community 123 - "Feature Specification: Deterministic Content Selection"
Cohesion: 0.17
Nodes (12): Assumptions, Edge Cases, Feature Specification: Deterministic Content Selection, Functional Requirements, Key Entities *(include if feature involves data)*, Measurable Outcomes, Requirements *(mandatory)*, Success Criteria *(mandatory)* (+4 more)
### Community 124 - "2. Entity Descriptions & Fields"
Cohesion: 0.18
Nodes (11): 1. Domain Entities & Value Types, 2. Entity Descriptions & Fields, 3. JSON Schema Mapping, `ArticleSelectionResult` (Dataclass), `BatchProcessingResult` (Dataclass), `CandidateStatus` (Enum), Data Model: Deterministic Content Selection, Entrada (+3 more)
### Community 125 - "Implementation Plan: Deterministic Article Content Selection"
Cohesion: 0.18
Nodes (11): Constitution Check, Documentation (this feature), Implementation Phases, Implementation Plan: Deterministic Article Content Selection, Phase 0: Outline & Research *(Completed)*, Phase 1: Design & Contracts *(Completed)*, Phase 2: Tasks & Execution *(Next Step via `/speckit-tasks`)*, Project Structure (+3 more)
### Community 126 - "Deterministic Content Selection Checklist: End-to-End Requirements Quality"
Cohesion: 0.25
Nodes (7): Candidate State Transitions & Resilience, Decision & Tie-Breaking Hierarchy, Deterministic Content Selection Checklist: End-to-End Requirements Quality, JSON Schema Integrity & Atomic I/O, Notes, Shingles & Consensus Metric Formulation, Text Normalization & Tokenization Quality
### Community 127 - "Quickstart: Deterministic Article Content Selection"
Cohesion: 0.29
Nodes (7): 1. Pré-requisitos, 2. Execução Rápida via CLI, 3. Execução dos Testes Automatizados, 4. Validação Programática / Uso como Módulo Python, Cenário 1: Selecionar o melhor extrator para uma extração existente, Cenário 2: Especificar caminho de saída customizado e modo verboso, Quickstart: Deterministic Article Content Selection
### Community 128 - "Specification Quality Checklist: Deterministic Content Selection"
Cohesion: 0.33
Nodes (5): Content Quality, Feature Readiness, Notes, Requirement Completeness, Specification Quality Checklist: Deterministic Content Selection
### Community 129 - "CLI Interface Contract: Deterministic Article Content Selection"
Cohesion: 0.33
Nodes (5): 1. Command Syntax, 2. Arguments and Flags, 3. Standard Streams (I/O), 4. Exit Codes, CLI Interface Contract: Deterministic Article Content Selection
### Community 131 - "JSON Schema Contract: Deterministic Article Content Selection"
Cohesion: 0.50
Nodes (3): 1. Input JSON Schema, 2. Output JSON Schema, JSON Schema Contract: Deterministic Article Content Selection
## Knowledge Gaps
- **465 isolated node(s):** `text-nlp-classifier`, `MatchedGraphEntity`, `graphify`, `Usage`, `What graphify is for` (+460 more)
- **562 isolated node(s):** `text-nlp-classifier`, `MatchedGraphEntity`, `graphify`, `Usage`, `What graphify is for` (+557 more)
These have ≤1 connection - possible missing edges or undocumented components.
- **38 thin communities (<3 nodes) omitted from report** — run `graphify query` to explore isolated nodes.
## Suggested Questions
_Questions this graph is uniquely positioned to answer:_
- **Why does `ECPSnapshot` connect `ECPSnapshot` to `main`, `models.py`, `classifier.py`, `InherenceClassifier`, `detect_language`?**
_High betweenness centrality (0.006) - this node is a cross-community bridge._
- **Why does `InherenceClassifier` connect `InherenceClassifier` to `main`, `ECPSnapshot`, `models.py`, `classifier.py`, `detect_language`?**
- **Why does `ECPSnapshot` connect `ECPSnapshot` to `models.py`, `LocalEmbeddingsAdapter`, `ClassificationResult`, `InherenceClassifier`, `classifier.py`, `main`?**
_High betweenness centrality (0.003) - this node is a cross-community bridge._
- **Why does `detect_language()` connect `detect_language` to `classifier.py`?**
- **Why does `LLMFallbackAdapter` connect `ClassificationResult` to `LocalEmbeddingsAdapter`, `ECPSnapshot`, `InherenceClassifier`, `classifier.py`?**
_High betweenness centrality (0.003) - this node is a cross-community bridge._
- **Why does `Research & Architectural Decisions: Deterministic Content Selection` connect `1. Text Normalization Pipeline` to `004-deterministic-content-selection/spec.md`?**
_High betweenness centrality (0.003) - this node is a cross-community bridge._
- **Are the 10 inferred relationships involving `ECPSnapshot` (e.g. with `main()` and `BaseNLPAdapter`) actually correct?**
_`ECPSnapshot` has 10 INFERRED edges - model-reasoned connections that need verification._
- **Are the 6 inferred relationships involving `InherenceClassifier` (e.g. with `LocalEmbeddingsAdapter` and `LLMFallbackAdapter`) actually correct?**
_`InherenceClassifier` has 6 INFERRED edges - model-reasoned connections that need verification._
- **Are the 12 inferred relationships involving `ExtractorName` (e.g. with `test_article_1_regression_technical_tie_markdown_images()` and `test_ct_001_three_candidates_clear_winner()`) actually correct?**
_`ExtractorName` has 12 INFERRED edges - model-reasoned connections that need verification._
- **Are the 10 inferred relationships involving `DecisionCategory` (e.g. with `InherenceClassifier` and `test_adversarial_apple_fruit_recipe()`) actually correct?**
_`DecisionCategory` has 10 INFERRED edges - model-reasoned connections that need verification._
- **Are the 4 inferred relationships involving `ClassificationResult` (e.g. with `BaseNLPAdapter` and `LocalEmbeddingsAdapter`) actually correct?**
_`ClassificationResult` has 4 INFERRED edges - model-reasoned connections that need verification._
_`DecisionCategory` has 10 INFERRED edges - model-reasoned connections that need verification._