crawler
Repositories (1232)
:page_with_curl: Scrape football data from Bet365
Google search results crawler, get google search results that you need - php
scrapy 扩展库：其主要功能使 scrapy 开发不用在意 item，pipeline，middleware 等通用场景下模块的编写，结合模板功能和常见开发的一些方法集成，可以让爬虫开...
Fetches PubMed article IDs (PMIDs) from email inbox, then crawls PubMed, Google Scholar and Sci-Hub for respective PDF files.
📱 A tool to download android apps from Google Play Store and Xiaomi App Store (the famous Chinese Store).
Generic altcoin DNS seeder. Compatible with virtually any cryptocurrency cloned from bitcoin. Built-in lightweight DNS server ~ Cloudflare DNS support...
a crawler to fetch last news from Iranian(Persian) news agencies.
A simple python3 script used to download a users's friend list from facebook.
BotCity Framework Web - Python
A simple library to scrape Pinterest images written in Python
深圳大学抢课系统
A web spider framework
Serverless Architecture Crawler demo
模拟登录各类网站，操作 API 完成各种不可描述的事情
Pimcore Website Indexer (powered by Zend Search Lucene)
OD-Database Go crawler
A tool for crawling the description and accepted submitted code of problems on the LeetCode and LeetCode-Cn website.
拟物校园，一个开源的高校教务移动化解决方案。
A personal tool using Python's Scrapy framework to scrape Best Buy's product pages for RTX 3080 TIs and notify if available/not sold out.
An example of Tor IP rotation in Python
Terminal version of Cambridge Dictionary by default. Also supports Merrian-Webster Dictionary.
Crawler for fetching information of US Patents and PDF bulk download
联系微信（1764328791）、抖音SDK、抖音数据、抖音直播数据、抖音直播Api、抖音视频Api、抖音爬虫、抖音去水印、抖音视频下载、抖音视频解析、抖音直播监控、抖...
Simple sitemap generator with Python 3
Asynchronous file scanner and downloader for FTP servers. Also takes IP ranges.
Hide your IP with free proxies using Froxy 🔄
Nogizaka46 Blog Crawler - 乃木坂46卒業成員部落格備份程式
🔗 Get all of the URL's from a website.
a crawler for wallstreetcn,finance.sina by Scrapy-新浪财经，同花顺财经，华尔街见闻的爬虫
Nomen est omen. It exports tucan grades/vv etc.
轻量级知乎爬虫，支持问题、收藏夹和本月最热
风铃虫是一款轻量级的爬虫工具，似风铃一样灵敏，如蜘蛛一般敏捷，能感知任何细小的风吹草动，轻松抓取互联网上的内容。它是一款对目标服务器相对友好的蜘蛛程序...
代理爬虫服务，爬取代理IP并保存到 Redis 中, topshelf+Quartz.Net+redis
A Symfony bundle for the Crawler-Detect library (detects bots/crawlers/spiders via the user agent)
CyberCrowl is a python Web path scanner tool
DevOps pipeline for Real Time Social/Web Mining
Based on Swoole,a PHP DHT crawler, which have insane productivity（依托于swoole的PHP版本的DHT爬虫，有着奇高的效率）
医疗知识图谱构建实战，通过爬虫获取百度百科数据，使用Mongodb存储结构化三元组，并使用neo4j进行知识图谱的构建及可视化; Medical Knowledge Graph; Crawler;...
A collection of webnovel sources offering varying amounts of scraping capability.
A Node.js template to be implemented to archive post from any social media.
💐Marmot A Golang HTTP Download
高可配的技术周报邮件推送工具
A scraper that gathers data from real estate ads
Crawler used to crawl papers
Kotlin library, Validator box that can inspect any type of form, provides multiple validation functions with an inclusion of clearing views
Downloads HD images from pinterest by keywords
Better Google Dorking with Dorker.
Competitive programming contests schedule
宁稳网(旧富投网)、集思录可转债数据&策略分析