Topic

crawler

Repositories (1460)

ebedke
ebedke ijanos Python

crawl pages to check what is for lunch today

33
toutiaocrawler
toutiaocrawler a252937166 Java

头条号爬虫案例

33
proxi
proxi nicksherron Go

Proxy pool. Finds and checks proxies with rest api for querying results. Can find over 25k proxies in under 5 minutes.

33
iranian-calendar-events
iranian-calendar-events mamal72 JavaScript

Fetch Iranian calendar events (Jalali, Hijri and Gregorian) from time.ir website

33
goGamer
goGamer davidleitw Go

巴哈姆特自訂API

33
ioweb
ioweb lorien Python

Web Scraping Framework

33
serritor
serritor peterbencze Java

Serritor is an open source web crawler framework built upon Selenium and written in Java. It can be used to crawl dynamic web pages that require JavaS...

33
advanced-php-crawler
advanced-php-crawler juzeon PHP

新浪博客文章/wenku8轻小说文库爬虫，可抓取图片保存，一键制作电子书。kindle读书党的神器！

33
PySitemap
PySitemap Cartman720 Python

🕸️ Spider Sitemap - Simple Python 3 crawler that automatically navigates your website, discovers all pages, and generates a complete XML sitemap. Easy...

33
Crawling-Emails
Crawling-Emails pH-7 Shell

Very simple bash script to crawl email addresses from a specific website.

33
emarketcrawlR
emarketcrawlR wagnertimo R

This R package provides a crawler to scrape the European Energy Market EPEX SPOT at https://www.epexspot.com and the European Energy Exchange at https...

33
xiaohongshu-spider-visualizer
xiaohongshu-spider-visualizer KaitoHH Python

A distributed web crawler for xiaohongshu.com and visualization for the crawled content.

33
dorker
dorker 0xdln1 Python

Better Google Dorking with Dorker.

33
scrapy-tor-proxy-rotation
scrapy-tor-proxy-rotation elvesmrodrigues Python

An IP rotator via Tor for Scrapy.

33
BiliBiliCommentsAnalysis
BiliBiliCommentsAnalysis Timecollector Python

对b站弹幕、评论进行爬虫，然后使用Word2Vec模型将其转化为词向量进行分析

33
flixhq-core
flixhq-core shin202 TypeScript

Nodejs library that provides an Api for obtaining the movies information from FlixHQ website.

33
Douban-MovieReview-Crawler
Douban-MovieReview-Crawler king-wang123 Python

豆瓣影评爬虫助手 这个项目可以让你对感兴趣的电影进行影评数据抓取、分析。不仅可以看到影评的星级分布，还能查看根据点赞数加权后的平均星级，同时生成直观的...

33
chad
chad ivan-sincek Python

Search Google Dorks like Chad. / Broken link hijacking tool.

33
wfdownloader-script
wfdownloader-script notarom

A collection of WFDownloader scripts

33
instagram-scraper-tool
instagram-scraper-tool Z786ZA

instagram scraper tool automated insights

33
telegram_bbbot
telegram_bbbot maddevsio Go

Telegram Bug Bounty Bot

32
see
see tmaciejewski Erlang

Search Engine in Erlang

32
SINA_Spider
SINA_Spider yinhao0214 Python

新浪微博爬虫：登录、关键词微博查询、微博监控

32
php-google
php-google howie6879 PHP

Google search results crawler, get google search results that you need - php

32
gateway_to_DeepReinforcementLearning_DeepNN
gateway_to_DeepReinforcementLearning_DeepNN DeepHiveMind Jupyter Notebook

:trophy: Welcome to the wonderland of "AI" = f(DL, RL, DRL, ML, NLP, KG, MLOPS)

32
kontests
kontests AliOsm Ruby

Competitive programming contests schedule

32
crowlet
crowlet Pixep Go

Tiny sitemap crawler for cache warming, and website status monitoring

32
squirm
squirm squirm-framework Crystal

This was the night of the crawling terror!

32
colymer-acquirers
colymer-acquirers touuki Python

各种爬虫（目前支持Instagram、Weibo、Twitter）Miscellaneous crawlers (currently including instagram, twitter, weibo etc.).

32
Google-Reverse-Image-Search
Google-Reverse-Image-Search ramonclaudio Python

A lightweight python wrapper designed for leveraging Google's search by image capabilities to perform reverse image searches programatically.

32
copymanga-nasdownloader
copymanga-nasdownloader misaka10843 Python

copymanga-downloader的mini ver，专为nas设计，不止于copymanga，支持多种平台！支持Web管理以及Docker部署！(当前支持copymanga、泰拉记事社、antbyw、gangano...

32
WebCrawler
WebCrawler debugtalk Python

A web crawler based on requests-html, mainly targets for url validation test.

31
pysaint
pysaint gomjellie Python

[deprecated] 유세인트 파이썬 클라이언트

31
frisbee
frisbee 9b Python

Collect email addresses by crawling search engine results.

31
scrapy_pro
scrapy_pro yangge11 Python

关于5000+站点的scrapy爬虫开发，涉及一些技术架构搭建以及各种反爬方案，详见readme文件

31
CrowLeer
CrowLeer erap320 C

Powerful C++ web crawler based on libcurl

31
bet365API
bet365API BET365-API

The latest way to get bet365 data odds, with a delay of 0.2 seconds bet365api

31
USPTO-PatFT-Web-Crawler
USPTO-PatFT-Web-Crawler mattwang44 Python

Crawler for fetching information of US Patents and PDF bulk download

31
little-python
little-python JeffyLu Python

little python projects, 一些小的python项目.

31
octopus
octopus dept JavaScript

Recursive and multi-threaded broken link checker

31
Douban-Crawler
Douban-Crawler ysh329 Python

抓取豆瓣小组相关信息（小组、用户、帖子）。

31
froxy
froxy matheusfelipeog Python

Hide your IP with free proxies using Froxy 🔄

31
local-api-client-csharp
local-api-client-csharp kameleo-io C#

This .NET Standard package provides convenient access to the Local API REST interface of the Kameleo Client.

31
site-mirror-go
site-mirror-go generals-space Go

来自[码云](https://gitee.com/generals-space/site-mirror-go) 通用爬虫, 仿站工具, 整站下载

31
FTPSearcher
FTPSearcher Sunlight-Rim Python

Asynchronous file scanner and downloader for FTP servers.

31
Broken-Links-Crawler-Action
Broken-Links-Crawler-Action ScholliYT Python

GitHub Action to check a website for broken links

31
ghrr
ghrr schosterbarak Python

A utility to collect data from github stargazers, subscribers and contributors of a selected project

31
reddit_scraper_and_sentiment_analyzer
reddit_scraper_and_sentiment_analyzer pratikpv Python

Download reddit posts based on keywords and perform sentiment analysis on the posts.

31
web-scraping-Riyasewana.lk
web-scraping-Riyasewana.lk gimnathperera Python

Web scraping script written in python using scrapy library in order to scrape product data from popular Sri Lankan vehicle selling web sites.

31
SouWen
SouWen BlueSkyXN Python

Unified search, crawling, and archiving toolbox for AI agents and automation scripts.

31