Most popular crawler repositories and open source projects

convertible-bond-crawler jackluson HTML

宁稳网(旧富投网)、集思录可转债数据&策略分析

24 11 24

kontests AliOsm Ruby

Competitive programming contests schedule

24 8 24

Pinterest-Crawler SajjadAemmi Python

Downloads HD images from pinterest by keywords

24 3 24

Amipy 01ly Python

A micro asynchronous Python website crawler framework .（Python微型异步爬虫框架）

23 11 23

crawler a11ywatch Rust

gRPC web crawler turbo charged for performance

23 2 23

crawlerr Bartozzz JavaScript

A simple and fully customizable web crawler/spider for Node.js with server-side DOM. Comes with elegant and hell-simple APIs.

23 7 23

Mimo-Crawler NikosRig JavaScript

A web crawler that uses Firefox and js injection to interact with webpages and crawl their content, written in nodejs.

23 2 23

onionstack ntddk Python

A Pictorial Book of Tor Hidden Services.

23 3 23

WebCrawler QinghuaBao Go

one web crawler frame based on golang

23 15 23

ghrr schosterbarak Python

A utility to collect data from github stargazers, subscribers and contributors of a selected project

23 1 23

little-python JeffyLu Python

little python projects, 一些小的python项目.

23 12 23

proxycrawl-node crawlbase JavaScript

ProxyCrawl Node library for scraping and crawling

23 5 23

findmeaflat adriankumpf JavaScript

Get notified of new listings on popular German real estate portals.

23 5 23

DarkSpider PROxZIMA Python

Anatomy and Visualization of the Network structure of the Dark web using multi-threaded crawler

23 3 23

googleplay_api alessandrodd Python

Google Play Unofficial Python 3 API Library

23 8 23

python-hacking-tools cristianzsh Python

Python tools for ethical hacking

23 11 23

Helios stefan2200 Python

A Python based Web Application security scanner

23 11 23

Python fmw666 Python

🍋 Python基础、Pygame游戏编程、Python算法与面试题、四种常用的Python Web框架、爬虫、数据可视化、机器学习。一共七个Python大方向！

23 4 23

github-action-rss-crawler minhhungit HTML

Auto crawl RSS feeds using Github Action

23 9 23

app-crawler maguowei Python

crawling App by uiautomator2 & mitmproxy

23 9 23

doc_crawler.py Siltaar

Explore a website recursively and download all the wanted documents (PDF, ODT…)

22 7 22

okcoin-socket-crawler Asoul Python

A okcoin crawler based on websocket, save data to mysql

22 9 22

Gumo nvk681 JavaScript

A crawler that extracts data from a dynamic webpage. Written in node js.

22 0 22

spider-video tibaiwan JavaScript

Node 批量爬取头条视频

22 7 22

RedBetter-WM2 Mechazawa Python

Better.php crawler for Redacted that uses WhatManager

22 0 22

exoskeleton RuedigerVoigt Python

A Python framework to build polite, but tenacious crawlers / scrapers with a MariaDB backend

22 1 22

linkedin-public-dir-companies robertoarruda Python

Crawler and scraper of the public directory of companies on LinkedIn.

22 5 22

minicrawler testomato C

Multiplexing web client supporting HTTP/2 and WHATWG URL compliant parser written in C

22 5 22

tistore Kagami JavaScript

:camera: Tistory photo grabber

22 10 22

armiarma migalabs Go

Armiarma is a Libp2p open-network crawler with a current focus on Ethereum's CL network

22 9 22

udemy-crawler petehouston JavaScript

Crawling Udemy course info and save into JSON format.

22 7 22

Taiwan-Stocks smalldan1022 Python

台灣上市櫃公司爬蟲，分析盤後股票趨勢以及繪製K線圖、均線圖、三大法人成交量

22 7 22

crawl-original-google-images thaoshibe Python

python scripts for crawling original image from Google Images

22 4 22

Mini-Projects nazaninsbr Python

A collection of short projects, you could try and implement these as short projects or use them as part of a larger project.

22 17 22

gateway_to_DeepReinforcementLearning_DeepNN DeepHiveMind Jupyter Notebook

:trophy: Welcome to the wonderland of "AI" = f(DL, RL, DRL, ML, NLP, KG, MLOPS)

22 5 22

libp2p-dht-scrape-aas alanshaw Go

🧹 A libp2p DHT scraper as a service allowing anyone to collect, consume and use to generate useful reports & visualisations.

22 8 22

udemyscraper sortedcord Python

A Udemy Course Scraper built with bs4 and selenium, that fetches udemy course information. Get udemy course information and convert it to json, csv or...

22 11 22

fifa-stats-crawler sauravhiremath Jupyter Notebook

A web-crawler to scrape FIFA 20 and 21 players' latest information from Sofifa Website

22 11 22

site-mirror-go generals-space Go

来自[码云](https://gitee.com/generals-space/site-mirror-go) 通用爬虫, 仿站工具, 整站下载

22 2 22

JobApplicationBot drkostas HTML

A bot that automatically sends emails to new ads posted in any desired xe.gr search url.

22 2 22

httpsuite whoamisec75 Python

A toolkit for web reconnaissance, it's fast and easy to use.

22 8 22

master-to-pythonista phuocding Python

A list of awesome beginners-friendly projects.

22 20 22

indieweb-search capjamesg Python

Source code for the IndieWeb search engine.

22 2 22

akka-react-cloudant IBM CSS

A Soccer Dashboard created by scraping EPL website using Akka backend and ReactJS frontend and IBM Cloudant for object storage. IBM Cloud Foundry is u...

21 18 21