crawley
An older project of mine, written in bash. A collection of shell scripts for crawling webpages, counting the number of occurences of keywords, tracing them over time and analysing correlations.
How to download and setup crawley
Open terminal and run command
git clone https://github.com/Ma-Fi-94/crawley.git
git clone is used to create a copy or clone of crawley repositories.
You pass git clone a repository URL. it supports a few different network protocols and corresponding URL formats.
Also you may download zip file with crawley https://github.com/Ma-Fi-94/crawley/archive/master.zip
Or simply clone crawley with SSH
[email protected]:Ma-Fi-94/crawley.git
If you have some problems with crawley
You may open issue on crawley support forum (system) here: https://github.com/Ma-Fi-94/crawley/issuesSimilar to crawley repositories
Here you may see crawley alternatives and analogs
scrapy Sasila colly headless-chrome-crawler Lulu gopa newspaper isp-data-pollution webster cdp4j spidy stopstalk-deployment N2H4 memorious easy-scraping-tutorial antch pomp Harvester diffbot-php-client talospider corpuscrawler Python-Crawling-Tutorial learn.scrapinghub.com crawling-projects dig-etl-engine crawlkit scrapy-selenium spidyquotes zcrawl podcastcrawler