Commoncrawl Collection

◆

◆

◆

◆

🔮Psychic

★★

“news-please - an integrated web crawler and information extractor for news that just works”

◆

◆

◆

◆

🔮Psychic

★

“A very simple news crawler with a funny name”

◆

◆

◆

◆

🔮Psychic

★

“Process Common Crawl data with Python and Spark”

◆

◆

◆

◆

⚡Thunder

★

“News crawling with StormCrawler - stores content as WARC”

◆

◆

◆

◆

🔮Psychic

★

“Statistics of Common Crawl monthly archives mined from URL index files”

◆

◆

◆

◆

🔮Psychic

★

“A toolkit for CDX indices such as Common Crawl and the Internet Archive's Wayback Machine”

◆

◆

◆

◆

🔮Psychic

★

“Price Crawler - Tracking Price Inflation”

◆

◆

◆

◆

🌊Water

★

“Extract web archive data using Wayback Machine and Common Crawl”

◆

◆

◆

◆

⚙️Steel

★

“:spider: The pipeline for the OSCAR corpus”

◆

◆

◆

◆

⚡Electric

★

“Paskto - Passive Web Scanner”

◆

◆

◆

◆

🔮Psychic

★

“Inspired by google c4, here is a series of colossal clean data cleaning scripts focused on CommonCrawl data processing. Including Chinese data processing and cleaning methods in MassiveText.”

◆

◆

◆

◆

⚡Thunder

★

“Index Common Crawl archives in tabular format”

◆

◆

◆

◆

⚡Thunder

★

“Tools to construct and process Common Crawl webgraphs”

◆

◆

◆

◆

⚙️Steel

★

“A polite and user-friendly downloader for Common Crawl data”

◆

◆

◆

◆

⚡Thunder

★

“A small tool which uses the CommonCrawl URL Index to download documents with certain file types or mime-types. This is used for mass-testing of frameworks like Apache POI and Apache Tika”

◆

◆

◆

◆

🔮Psychic

★

“[码云](https://gitee.com/generals-space/site-mirror-py) 通用爬虫, 仿站工具, 整站下载”

◆

◆

◆

◆

📦Normal

★

“Various Jupyter notebooks about Common Crawl data”

◆

◆

◆

◆

📦Normal

★

“Word analysis, by domain, on the Common Crawl data set for the purpose of finding industry trends”