Topic

crawler

Repositories (1232)

es6-crawler-detect
es6-crawler-detect JefferyHus JavaScript

:spider: This is an ES6 adaptation of the original PHP library CrawlerDetect, this library will help you detect bots/crawlers/spiders vie the useragen...

86
LFITester
LFITester kostas-pa Python

LFITester is a Python3 program that automates the detection and exploitation of Local File Inclusion (LFI) vulnerabilities on a server.

86
scrapy_helper
scrapy_helper facert CSS

Dynamic configurable crawl (动态可配置化爬虫)

85
is-google
is-google roccomuso JavaScript

Verify that a request is from Google crawlers using Google's DNS verification steps

85
Amazon-Price-Alert
Amazon-Price-Alert GaryniL Python

Price tracker of Amazon

84
SeleniumDemo
SeleniumDemo tobecrazy HTML

Selenium automation test framework

84
pagser
pagser foolin Go

Pagser is a simple, extensible, configurable parse and deserialize html page to struct based on goquery and struct tags for golang crawler

84
extension
extension get-set-fetch TypeScript

web scraping extension

84
Hands-on-WebScraping
Hands-on-WebScraping amitupreti Python

This repo is a part of blog series on several web scraping projects where we will explore scraping techniques to crawl data from simple websites to we...

82
instastories-backup
instastories-backup ondrejsojka Python

Backup your friends' Instagram Stories forever and get to keep them even after 24 hours.

82
weibo-scraper
weibo-scraper Xarrow Python

Simple Weibo Scraper

82
Proxy-List-Scrapper
Proxy-List-Scrapper narkhedesam Python

Proxy List Scrapper

82
XVideos-PornHub-RedTube-API
XVideos-PornHub-RedTube-API Joel2B PHP

This script scrapes the HTML from different web pages to get the information from the video (XVideos, PornHub, RedTube) and you can use it in your own...

81
random_user_agent
random_user_agent Luqman-Ud-Din Python

A package to get list of user agents based on filters such as operating system, software name etc..

80
Novel-crawler
Novel-crawler ling7334 Python

这是一个用Python写的小说爬虫软件

79
deepweb-scappering
deepweb-scappering kurogai Python

Discover hidden deepweb pages

79
puppeteer-walker
puppeteer-walker lrlna JavaScript

a puppeteer walker 🕷 🕸

79
ceiba-dl
ceiba-dl lantw44 Python

NTU CEIBA 資料下載工具

79
tumblr_crawler
tumblr_crawler abbey2023 Python

tumblr解析网站

78
arachnid
arachnid watzon Crystal

Powerful web scraping framework for Crystal

78
feedsearch-crawler
feedsearch-crawler DBeath Python

Crawl sites for RSS, Atom, and JSON feeds.

77
crawler_examples
crawler_examples liuslnlp Python

Some classic web crawler projects.一些经典的爬虫

77
scrapy-examples
scrapy-examples feiskyer Python

Some scrapy and web.py exmaples

77
xSMTP
xSMTP aziz0x48 Python

xSMTP 🦟 Lightning fast, multithreaded smtp scanner targeting open-relay and unsecured servers in multiple network ranges.

76
BUbiNG
BUbiNG LAW-Unimi Java

The LAW next generation crawler.

76
WebSecurityArticles
WebSecurityArticles zongdeiqianxing Python

爬取及整理Freebuf\安全客\先知\知道创宇等站点的”web安全“类优质文章

76
ctrip_spider
ctrip_spider evanleungc Python

Scrape Learning (ctrip)

76
tumblr-crawler-cli
tumblr-crawler-cli tzw0745 Python

Tumblr Download Tool with High Speed and Customization. 高性能&高定制化的Tumblr下载工具。

76
fetchman
fetchman DarkSand Python

fetchman is a simple crawler system/简单好用的爬虫框架

76
tg_crawler
tg_crawler vhdmsm Python

Just a crawler based on tg-cli for Telegram. Deprecated by now, please use telegram-export.

75
light-crawler
light-crawler zhang2333 JavaScript

a simplified directed customizable website crawler

75
venom
venom PreferredAI Java

Your preferred open source focused crawler for the deep web.

74
python-tools
python-tools lucasayres Python

A collection of Python tools, scripts and utilities to make your life easier.

74
simpyder
simpyder Jannchie Python

超高速异步协程Python爬虫

74
fund-crawler
fund-crawler nullpointer JavaScript

基于NodeJS的基金数据爬虫,爬取的数据存于github的@nullpointer/fund-data。

73
lrabbit_scrapy
lrabbit_scrapy litter-rabbit Python

a quick start python mutil thread crawl

72
spider
spider jhao104 Python

python crawler spider

72
python-testing-crawler
python-testing-crawler python-testing-crawler Python

A crawler for automated functional testing of a web application

72
achoz
achoz kcubeterm Python

Search through all your personal data efficiently like web search.

72
crawlzone
crawlzone crawlzone PHP

Crawlzone is a fast asynchronous internet crawling framework for PHP.

72
librengine
librengine liameno C++

Privacy Web Search Engine (not meta, own crawler)

72
Instagram-downloader
Instagram-downloader fernandod1 Python

Instagram user's photos and videos downloader. Download all media files from any username. Working 2022!

71
COI
COI AlvinAi96 Jupyter Notebook

练手项目:Comment of Interest 电商文本评论数据挖掘 (爬虫 + 观点抽取 + 句子级和观点级情感分析)

71
car-prices
car-prices go-crawler Go

Golang爬虫 爬取汽车之家 二手车产品库

70
IpProxyPool
IpProxyPool wuchunfu Go

Golang 实现的 IP 代理池, 涉及到的技术点: go gorm proxy proxypool ip crawler 爬虫 mysql viper cobra

70
BOJ-AutoCommit
BOJ-AutoCommit ISKU Python

When you solve the problem of Baekjoon Online Judge, it automatically commits and pushes to the remote repository.

70
robotstxt
robotstxt ropensci R

robots.txt file parsing and checking for R

69
darc
darc JarryShaw Python

Darkweb Crawler Project

69
python-crawler
python-crawler ityouknow Python

Python Crawler

69
tiktok-scraper-php
tiktok-scraper-php snuzi PHP

Tiktok (Musically) PHP scraper

69