crawler
Repositories (1232)
Online search advertisement platform & Realtime Campaign Monitoring [Maybe Deprecated]
[deprecated] 유세인트 파이썬 클라이언트
A web spider for shodan.io without using the Developer API.
🤖 Prerendering for JavaScript powered websites. Great solution for PWAs (Progressive Web Apps), SPAs (Single Page Applications), and other websites b...
Search Engine in Erlang
Extract instagram users informations from hashtags. This scraper can extract emails addresses from Bio section and business email.
The latest way to get bet365 data odds, with a delay of 0.2 seconds bet365api
This was the night of the crawling terror!
Recursive and multi-threaded broken link checker
🖼️ Get all images from pixiv/twitter/deviantart
Scrape bot, to get you an offline copy of tutorials
This .NET Standard package provides convenient access to the Local API REST interface of the Kameleo Client.
Stateful programmatic web browsing, based on Python-Mechanize, which is based on Andy Lester’s Perl module WWW::Mechanize.
A Web Crawler that scrapes using YAML and python code.
app版本今日头条 用户登录/个人主页/关注列表/粉丝列表/评论点赞收藏 关键词搜索 头条视频文章信息/评论数据采集
A tool to generate a static website by crawling the original site.
新浪微博爬虫：登录、关键词微博查询、微博监控
练习NLP，分析淘宝评论的项目
巴哈姆特自訂API
Node.js/Express app to retrive instagram video/image download urls
Automatically get the csgo skins sale data on igxe.cn and buff and c5game.com.You can choose the specific skins to get data.
Fetch Iranian calendar events (Jalali, Hijri and Gregorian) from time.ir website
Download de livros para PDF/EPUB - Integrada.minhabiblioteca / vitalsource
Scraping scripts for various websites.
(2020年最新)斗鱼弹幕抓取及实时弹幕数据可视化，分为crawler(弹幕抓取)，server(弹幕统计数据服务器)，web(统计数据可视化前端)三部分
A client for fetching stock data from the Tehran Stock Exchange (TSETMC). Works in Browser, Node and as CLI.
Powerful C++ web crawler based on libcurl
A web scraper that allows you to fetch Movies and TV series information from the FMovies website. (Download + Stream Links included!)
A stock market text sentiment analysis website. A股舆情分析, web-crawler, bayesian algorithm, SQL, django, data-visualization.
Simple content crawler for joyreactor.cc
爬取tumblr关注博主图片
Telegram Bug Bounty Bot
Library to crawl and extract internal links from domain
Crawling the data from lazada, websosanh, compare.vn, cdiscount and cungmua with flexible configs
A Moodle Crawler that downloads course content from Moodle (eg. lecture pdfs)
A script to download images from images.google.com
🤖🍴 A Python script that scrapes UFMG's restaurants menus and publishes them @bot_do_bandejao Twitter profile
🔥 Composer package to warm up caches of URLs located in XML sitemaps
The project is a spider that uses scrapy and beautifulsoup4 for crawl picture.
Collect email addresses by crawling search engine results.
Web/FileSystem Crawler Library
:earth_africa: Bittorrent Network Overview through Infohash Indexing, Metadata and IP visualisations of the DHT network
a py module to download apk from apkpure.com
A simple web crawler, using Abot, that indexes page contents into Azure Search.
新浪博客文章/wenku8轻小说文库爬虫，可抓取图片保存，一键制作电子书。kindle读书党的神器！
Tiny script to crawl information of a specific application in the Google play/store base on PHP.
Google search results crawler, get google search results that you need - php
Fetches PubMed article IDs (PMIDs) from email inbox, then crawls PubMed, Google Scholar and Sci-Hub for respective PDF files.
A web spider framework
A simple python3 script used to download a users's friend list from facebook.