Most popular crawler repositories and open source projects

Web_Vulnerability_Scanner-AI SagarBiswas-MultiHAT Python

PoliteWebScanner is a learning-grade web vulnerability scanner plus AI-assisted report viewer. It crawls safely (queue-based, robots-aware, rate-limit...

32 0 0

scrapedo-scrapers scrape-do Python

Web scraping examples with Scrape.do 😎

32 1 3

redbooks xiaofuqing13 Python

小红书爬虫工具 - 支持关键词搜索、主页推荐、博主主页爬取，图片视频批量下载，评论采集，Excel导出，SQLite存储，GUI图形界面 | Python + DrissionPage

32 10 2

WebCrawler debugtalk Python

A web crawler based on requests-html, mainly targets for url validation test.

31 12 3

pysaint gomjellie Python

[deprecated] 유세인트 파이썬 클라이언트

31 3 4

frisbee 9b Python

Collect email addresses by crawling search engine results.

31 6 3

scrapy_pro yangge11 Python

关于5000+站点的scrapy爬虫开发，涉及一些技术架构搭建以及各种反爬方案，详见readme文件

31 12 0

CrowLeer erap320 C

Powerful C++ web crawler based on libcurl

31 4 2

bet365API BET365-API

The latest way to get bet365 data odds, with a delay of 0.2 seconds bet365api

31 8 31

little-python JeffyLu Python

little python projects, 一些小的python项目.

31 14 3

octopus dept JavaScript

Recursive and multi-threaded broken link checker

31 12 4

Douban-Crawler ysh329 Python

抓取豆瓣小组相关信息（小组、用户、帖子）。

31 7 2

local-api-client-csharp kameleo-io C#

This .NET Standard package provides convenient access to the Local API REST interface of the Kameleo Client.

31 2 31

site-mirror-go generals-space Go

来自[码云](https://gitee.com/generals-space/site-mirror-go) 通用爬虫, 仿站工具, 整站下载

31 5 1

FTPSearcher Sunlight-Rim Python

Asynchronous file scanner and downloader for FTP servers.

31 6 2

Broken-Links-Crawler-Action ScholliYT Python

GitHub Action to check a website for broken links

31 4 1

ghrr schosterbarak Python

A utility to collect data from github stargazers, subscribers and contributors of a selected project

31 2 1

reddit_scraper_and_sentiment_analyzer pratikpv Python

Download reddit posts based on keywords and perform sentiment analysis on the posts.

31 8 2

web-scraping-Riyasewana.lk gimnathperera Python

Web scraping script written in python using scrapy library in order to scrape product data from popular Sri Lankan vehicle selling web sites.

31 2 2

Comic-Toolbox freedy82 Python

Comic toolbox, GUI interface, download, reader, translation, conversion, image processing, cropping, compression, convenient for e-book reader users...

31 5 1

GithubCollectAgent whz39coding Python

Github自动收集比较热门的项目,并自动阅读项目,生成相关报告发送到自己的飞书中

31 7 0

WebSource-Harvester SagarBiswas-MultiHAT Python

WebSource Harvester is an educational web-source harvester that crawls a site (BFS, depth-controlled), downloads browser-visible assets (HTML, CSS, JS...

31 1 1

weread-mp-fetcher Pengyf04 JavaScript

通过微信读书抓取公众号最新文章。零依赖，复用你已登录的 Chrome，输出 mp.weixin 原文直链。

31 4 0

Mechanize.NET WilliamABradley C#

Stateful programmatic web browsing, based on Python-Mechanize, which is based on Andy Lester’s Perl module WWW::Mechanize.

30 8 1

Search_Ads_Web_Service youhusky Java

Online search advertisement platform & Realtime Campaign Monitoring [Maybe Deprecated]

30 1 1

TTBot2.0 01ly Python

app版本今日头条用户登录/个人主页/关注列表/粉丝列表/评论点赞收藏关键词搜索头条视频文章信息/评论数据采集

30 16 30

vietnam-ecommerce-crawler nhat2008 Python

Crawling the data from lazada, websosanh, compare.vn, cdiscount and cungmua with flexible configs

30 10 4

google-image-downloader koallen Python

A script to download images from images.google.com

30 14 1

DouyuBarrage Crawler995 JavaScript

(2020年最新)斗鱼弹幕抓取及实时弹幕数据可视化，分为crawler(弹幕抓取)，server(弹幕统计数据服务器)，web(统计数据可视化前端)三部分

30 10 1

BitInsight simionrobert JavaScript

:earth_africa: Bittorrent Network Overview through Infohash Indexing, Metadata and IP visualisations of the DHT network

30 7 3

USPTO-PatFT-Web-Crawler mattwang44 Python

Crawler for fetching information of US Patents and PDF bulk download

30 16 0

publiccode-crawler italia Go

publiccode.yml crawler for the Open Source software catalog of Developers Italia

30 54 8

weibo_keyword_crawl huifeng-kooboo Python

爬取微博关键词相关信息,有远程开发需求可联系，有需要合作加微信: ytouching

30 5 1

gurl luisra51 Go

An intelligent web crawler built in Go that extracts email addresses from websites with precision and speed.

30 4 1

crawl-analysis-data-facebook ntthanh2603

📊 Project: Analysis & Data Crawling for Two Football Pages – Manchester United & Liverpool FC ⚽🔍

30 2 1

supa-crawl-chat bigsk1 Python

Integrates Supabase with Crawl4AI and AI Chat to create a powerful web crawling and semantic search solution. Streamlit supabase data visualization. R...

30 7 4

job-scraper DEENUU1 Python

🔎 Job offers scraper: bulldogjob.pl, indeed.com, it.pracuj.pl, jooble.org, justjoin.it, nofluffjob.com, olx.pl, pracuj.pl, theprotocol.it, useme.com

30 4 1

AI-Gov-Content-Curator hoangsonww TypeScript

💡An end-to-end solution for aggregating, summarizing, and displaying news articles using an AI-powered backend, an automated CRON crawler & newslette...

30 18 9