Topic

crawler

Repositories (1501)

gurl
gurl luisra51 Go

An intelligent web crawler built in Go that extracts email addresses from websites with precision and speed.

30
USPTO-PatFT-Web-Crawler
USPTO-PatFT-Web-Crawler mattwang44 Python

Crawler for fetching information of US Patents and PDF bulk download

30
DouyuBarrage
DouyuBarrage Crawler995 JavaScript

(2020年最新)斗鱼弹幕抓取及实时弹幕数据可视化,分为crawler(弹幕抓取),server(弹幕统计数据服务器),web(统计数据可视化前端)三部分

30
google-image-downloader
google-image-downloader koallen Python

A script to download images from images.google.com

30
TTBot2.0
TTBot2.0 01ly Python

app版本今日头条 用户登录/个人主页/关注列表/粉丝列表/评论点赞收藏 关键词搜索 头条视频文章信息/评论数据采集

30
Search_Ads_Web_Service
Search_Ads_Web_Service youhusky Java

Online search advertisement platform & Realtime Campaign Monitoring [Maybe Deprecated]

30
Mechanize.NET
Mechanize.NET WilliamABradley C#

Stateful programmatic web browsing, based on Python-Mechanize, which is based on Andy Lester’s Perl module WWW::Mechanize.

30
BitInsight
BitInsight simionrobert JavaScript

:earth_africa: Bittorrent Network Overview through Infohash Indexing, Metadata and IP visualisations of the DHT network

30
Abans-lk-Webscraping
Abans-lk-Webscraping gimnathperera Python

🌐 Web scraping script written in python using scrapy library in order to scrape product data from popular Sri Lankan web sites

29
zhihu_crawler
zhihu_crawler niuniuJQKKK Python

本程序支持关键词搜索、热榜、用户信息、回答、专栏文章、评论等信息的抓取

29
reactor-crw
reactor-crw reactor-joy Go

Simple content crawler for joyreactor.cc

29
screamingFrogR
screamingFrogR Leszek-Sieminski R

R integration with Screaming Frog CLI

29
Academic-Paper-Title-Recommendation
Academic-Paper-Title-Recommendation safakkbilici Python

Supervised text summarization (title generation/recommendation) based on academic paper abstracts, with Seq2Seq LSTM and T5.

29
fmovies-crawler
fmovies-crawler Scrip7 TypeScript

A web scraper that allows you to fetch Movies and TV series information from the FMovies website. (Download + Stream Links included!)

29
PTTmineR
PTTmineR shihjyun R

Parallel Searching and Crawling Data from PTT 🚀

29
utsusemi
utsusemi k1LoW JavaScript

A tool to generate a static website by crawling the original site.

29
fifa-stats-crawler
fifa-stats-crawler sauravhiremath Jupyter Notebook

A web-crawler to scrape FIFA 20 and 21 players' latest information from Sofifa Website

29
webevaluator
webevaluator Aman-Codes JavaScript

A web crawling tool which tests websites for SSL, Cookies and ADA compliance and also suggests ways to fix them.

28
crawlit
crawlit drogbadvc Python

This project is a web crawler based on Scrapy, visualization 2D, PageRank

28
CrawlnChat
CrawlnChat jroakes Python

A modular web crawling and chat system that allows for ingesting website content through XML sitemaps, converting to vector embeddings, and providing...

28
CocoCrawler
CocoCrawler Marcel0024 C#

An declarative and easy to use web crawler and scraper in C#

28
spider_man
spider_man feng19 Elixir

SpiderMan,a base-on Broadway fast high-level web crawling & scraping framework for Elixir.

28
yt-comments-crawler
yt-comments-crawler rdavydov JavaScript

Browser extension that extracts all comments from the YouTube video page, sorts them by the amount of likes and saves them to a csv file.

28
spider
spider cyclone-github Go

Spider - web crawler and local wordlist processor to generate frequency sorted wordlist / ngrams

28
konect-extr
konect-extr kunegis MATLAB

Network dataset extraction library – part of the KONECT project by Jérôme Kunegis, University of Namur

28
spider
spider GeoffZhu JavaScript

A web spider framework

28
ds-video-helper
ds-video-helper kyxw007 JavaScript

群晖Video Station助手,自动获取豆瓣电影信息,并填写Video Station视频信息

28
ytpriv
ytpriv riptl Go

YT metadata exporter

28
scrapit
scrapit Priyankadev Python

Scraping scripts for various websites.

28
Moodle-Downloader
Moodle-Downloader C0D3D3V Python

A Moodle Crawler that downloads course content from Moodle (eg. lecture pdfs)

28
crawler
crawler suyuanhxx Go

爬取tumblr关注博主图片

28
wind-bell
wind-bell yishuifengxiao Java

风铃虫是一款轻量级的爬虫工具,似风铃一样灵敏,如蜘蛛一般敏捷,能感知任何细小的风吹草动,轻松抓取互联网上的内容。它是一款对目标服务器相对友好的蜘蛛程序...

28
bot_do_bandejao
bot_do_bandejao vitor-mafra Python

🤖🍴 A Python script that scrapes UFMG's restaurants menus and publishes them @bot_do_bandejao Twitter profile

28
scrapy-picture-spider
scrapy-picture-spider SylvanasSun Python

The project is a spider that uses scrapy and beautifulsoup4 for crawl picture.

28
social-media-archiver
social-media-archiver Combo819 TypeScript

A Node.js template to be implemented to archive post from any social media.

27
udemyscraper
udemyscraper sortedcord Python

A Udemy Course Scraper built with bs4 and selenium, that fetches udemy course information. Get udemy course information and convert it to json, csv or...

27
soccer-scrape
soccer-scrape o8e JavaScript

:page_with_curl: Scrape football data from Bet365

27
job-funnel-ts
job-funnel-ts alehkot TypeScript

Automated tool for scraping job postings into a .xlsx files inspired by Job Funnel.

27
python-crawl
python-crawl jcesarstef Python

Library to crawl and extract internal links from domain

27
cambridge
cambridge mhwgoo Python

Terminal version of Cambridge Dictionary by default. Also supports Merrian-Webster Dictionary.

26
collector-filesystem
collector-filesystem Norconex Java

Norconex Filesystem Collector is a flexible crawler for collecting, parsing, and manipulating data ranging from local hard drives to network locations...

26
douyin-sdk
douyin-sdk Video-Hub Python

联系微信(1764328791)、抖音SDK、抖音数据、抖音直播数据、抖音直播Api、抖音视频Api、抖音爬虫、抖音去水印、抖音视频下载、抖音视频解析、抖音直播监控、抖...

26
nivinEdu
nivinEdu nivin-studio

拟物校园,一个开源的高校教务移动化解决方案。

26
od-database-crawler
od-database-crawler terorie Go

OD-Database Go crawler

26
pimcore-lucene-search
pimcore-lucene-search dachcom-digital PHP

Pimcore Website Indexer (powered by Zend Search Lucene)

26
PY-Login
PY-Login PY-Trade Python

模拟登录各类网站,操作 API 完成各种不可描述的事情

26
CrawlerDetectBundle
CrawlerDetectBundle nicolasmure PHP

A Symfony bundle for the Crawler-Detect library (detects bots/crawlers/spiders via the user agent)

26
serverless-crawler-demo
serverless-crawler-demo novemberde JavaScript

Serverless Architecture Crawler demo

26
marmot
marmot hunterhug Go

💐Marmot A Golang HTTP Download

25
master-to-pythonista
master-to-pythonista phuocding Python

A list of awesome beginners-friendly projects.

25