Crawling Collection

◆

◆

◆

◆

🔮Psychic

★★★★

“🕷️ An adaptive Web Scraping framework that handles everything from a single request to a full-scale crawl!”

◆

◆

◆

◆

🔮Psychic

★★★★

“Scrapy, a fast high-level web crawling & scraping framework for Python.”

◆

◆

◆

◆

🌊Water

★★★★

“Elegant Scraper and Crawler Framework for Golang”

◆

◆

◆

◆

💎Aqua

★★★★

“Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.”

◆

◆

◆

◆

💎Aqua

★★★

“🔥 The open-source no-code platform for web scraping, crawling, search and AI data extraction • Turn websites into structured APIs in minutes 🔥 ”

◆

◆

◆

◆

🔮Psychic

★★★

“newspaper3k is a news, full-text, and article metadata extraction in Python 3. Advanced docs:”

◆

◆

◆

◆

🔮Psychic

★★★

“Crawlee—A web scraping and browser automation library for Python to build reliable crawlers. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Parsel, BeautifulSoup, Playwright, and raw HTTP. Both headful and headless mode. With proxy rotation.”

◆

◆

◆

◆

📦Normal

★★★

“List of libraries, tools and APIs for web scraping and data processing.”

◆

◆

◆

◆

🌊Water

★★★

“A Chrome DevTools Protocol driver for web automation and scraping.”

◆

◆

◆

◆

⚡Electric

★★★

“Distributed crawler powered by Headless Chrome”

◆

◆

◆

◆

🌊Water

★★★

“Simple, fast web crawler designed for easy, quick discovery of endpoints and assets within a web application”

◆

◆

◆

◆

💎Aqua

★★

“Exa MCP for web search and web crawling!”

◆

◆

◆

◆

🔮Psychic

★★

“A list of AI agents and robots to block.”

◆

◆

◆

◆

🔮Psychic

★★

“Headless Chrome .NET API”

◆

◆

◆

◆

🌊Water

★★

“Take a list of domains, crawl urls and scan for endpoints, secrets, api keys, file extensions, tokens and more”

◆

◆

◆

◆

⚡Thunder

★★

“Apache Nutch is an extensible and scalable web crawler”

◆

◆

◆

◆

📦Normal

★★

“A curated list of awesome puppeteer resources.”

◆

◆

◆

◆

🔮Psychic

★★

“Web Scraping Framework”