andreferraro/TextNLPClassifierApp
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(extractor): add Google News headlines extractor with Foxcape headless and URL resolution

Browse Source 
- Add standalone CLI script scripts/extract_google_news.py for Google News RSS scraping
- Integrate foxcape in headless mode as primary stealth anti-bot engine
- Implement parallel article URL resolution using googlenewsdecoder and ThreadPoolExecutor
- Support language and regional locale mapping (-l, --lang, --locale)
- Implement real-time progress logging in stderr and --silent flag
- Add unit, integration, and live E2E tests in tests/test_extract_google_news.py
- Add full SpecKit documentation (specs/002-google-news-extractor/)
- Create comprehensive README.md covering both NLP Classifier and Google News Extractor
This commit is contained in:
andreferraro
2026-08-20 11:50:16 -03:00
parent 67cc40f91a
commit 6e3d57619b
59 changed files with 16118 additions and 2160 deletions
Download Patch File Download Diff File Expand all files Collapse all files
requirements.txt
+4
View File
@@ -1,4 +1,8 @@
pytest>=7.0.0
foxcape>=0.1.1
beautifulsoup4>=4.12.0
googlenewsdecoder>=0.1.7
selectolax>=0.3.27
# Optional Tier 2 / Tier 3 dependencies (not required for POC core execution)
# sentence-transformers>=2.2.0
# httpx>=0.24.0