Python news scraper & article extractor — extract title, text, authors, date, image, and publisher from any news URL. No API key. Confidence scores included.
Fetch news. Know why it worked.
pip install news-fetchfrom newsfetch import fetch
article = fetch("https://www.thehindu.com/...")
print(article.title)
print(article.text)
print(article.authors)
print(article.published_at)
print(article.image)
print(article.confidence.overall) # 0.0–1.0
print(article.content_source) # e.g. json-ld.articleBodynews-fetch https://example.com/article
news-fetch https://example.com/article --json
news-fetch batch urls.txt -o articles.jsonlA lightweight alternative to newspaper3k / newspaper4k / trafilatura wrappers — with its own extraction engine, confidence scores, and bulk + proxy support.
| Feature | news-fetch |
|---|---|
| News article extraction (title, body, authors, date, image) | ✅ |
| Confidence scores + extraction provenance | ✅ |
Bulk scraping (fetch_many / fetch_iter / CLI JSONL) |
✅ |
| Proxy + proxy rotation for thousands of URLs | ✅ |
| RSS / sitemap article discovery | ✅ |
Async (pip install news-fetch[async]) |
✅ |
Optional browser render (pip install news-fetch[browser]) |
✅ |
| Disk cache + robots.txt respect | ✅ |
| No API key / no account | ✅ |
Small deps (lxml, requests, python-dateutil, cssselect) |
✅ |
pip install news-fetch
pip install news-fetch[async] # httpx async fetch
pip install news-fetch[browser] # Playwright fallback (then: playwright install chromium)Requirements: Python 3.10+
from newsfetch import fetch
article = fetch(url)
print(article.title, article.text, article.confidence.overall)from newsfetch import extract
article = extract(html_bytes, url="https://example.com/story")from newsfetch import fetch_many, fetch_iter
results = fetch_many(
urls,
max_workers=20,
proxies=["http://user:pass@p1:8080", "http://user:pass@p2:8080"],
request_delay=0.05,
)
for url, article in fetch_iter(urls, max_workers=16):
if article:
print(article.title)from newsfetch import fetch, LowConfidenceExtractionError
try:
article = fetch(url, strict=True)
except LowConfidenceExtractionError as e:
print(e.failed_fields, e.confidence.overall)from newsfetch import discover
for item in discover("https://www.bbc.com", limit=10):
print(item["url"], item.get("title"))from newsfetch import fetch_async, fetch_many_async
article = await fetch_async(url)
articles = await fetch_many_async(urls, max_concurrency=50, proxies=PROXIES)news-fetch https://example.com/article
news-fetch get URL --json
news-fetch batch urls.txt -o out.jsonl --workers 20
news-fetch discover https://www.theguardian.com --limit 10from newsfetch import fetch, Config, NewsFetcher
fetch(url, cache=True, respect_robots=True)
fetch(url, render=True) # needs news-fetch[browser]
fetch(url, browser_fallback=True) # retry with Playwright if confidence is lowfrom newsfetch import NewsFetcher, CallableStrategy
from newsfetch.strategies.base import Candidate
def my_strategy(doc):
return {"title": [Candidate("Custom", "plugin.custom", 0.99)]}
fetcher = NewsFetcher()
fetcher.register_strategy(CallableStrategy("custom", my_strategy))url · canonical_url · title · description · text · authors · published_at · modified_at · publisher · language · image · keywords · section · summary · word_count · reading_time_minutes · page_type · is_article · confidence · extraction · sources
article.to_dict()
article.to_json()- PyPI: https://pypi.org/project/news-fetch/
- Docs: https://santhoshse7en.github.io/newsfetch_doc/
- GitHub: https://github.com/santhoshse7en/news-fetch
- Issues: https://github.com/santhoshse7en/news-fetch/issues
- Changelog: CHANGELOG.md
MIT License · Built for developers who need reliable Python news scraping without an API.