#

web-crawlers

Here are 45 public repositories matching this topic...

ScrapeGraphAI / Scrapegraph-ai

Python scraper based on AI

Updated Sep 25, 2026
Python

isbot

omrilotan / isbot

🤖/👨‍🦰 Detect bots/crawlers/spiders using the user agent string

user-agent user-agent-parser user-agent-analysis crawlers web-crawlers

Updated Aug 27, 2026
TypeScript

ScrapingAnt / amazon_scraper

Amazon products scraper with using of rotating proxies and headless Chrome from ScrapingAnt

data-mining scraper js amazon web-crawler scraping node-js scraping-websites web-crawling price-scraper amazon-scraper scraping-api scraping-python price-scraping scraping-web web-crawlers scraping-data amazon-scraping-library scrape-products

Updated Mar 15, 2024
JavaScript

crawler-commons / url-frontier

API definition, resources and reference implementation of URL Frontiers

grpc webcrawling web-crawlers url-frontier urlfrontier

Updated Sep 22, 2026
Java

fluoos / crawl2ai

一款强大的大模型微调数据集生成和管理工具。

dataset markdown-convertor dataset-generation large-models dataset-manager fine-tuning web-crawlers llm llm-training easy-dataset

Updated Sep 14, 2025
Python

ancatmara / learnpython2018

Python course for 2nd year NLP students at NRU HSE, 2018-2019

Updated Nov 17, 2023
Jupyter Notebook

dod-advana / gamechanger-crawlers

GAMECHANGER Policy Analytics Site Crawlers

dod policy-as-code web-crawlers

Updated Jul 24, 2024
Python

YiraBot-Crawler

OwenOrcan / YiraBot-Crawler

YiraBot: Simplifying Web Scraping for All. A user-friendly tool for developers and enthusiasts, offering command-line ease and Python integration. Ideal for research, SEO, and data collection.

open-source machine-learning data-mining scraping python3 text-extraction web-scraping html-parser robots-txt data-extraction seotools command-line-tool beginner-friendly contributions-welcome big-data-analytics seo-analysis good-first-issue sitemap-parser web-crawlers

Updated Jul 24, 2026
Python

ancatmara / learnpython2017

Python course for 2nd year NLP students at NRU HSE, 2017-2018

Updated Nov 17, 2023
Jupyter Notebook

romis2012 / is-bot

Detect bots/crawlers/spiders via user-agent string

python user-agent bots user-agent-parser crawlers bot-detection web-crawlers

Updated Aug 4, 2026
Python

mihaicodes / web-crawler

A Web Crawler developed in Python.

python crawler web script scripting web-crawler scripts python-script scripting-language python3 python-3 crawlers webcrawler web-crawling web-crawler-python webcrawling webcrawl crawler-python web-crawlers

Updated Aug 8, 2022
Python

awesome-web-crawler

ilovedevs / awesome-web-crawler

List of best web crawlers to extract data from the web. Find web crawling tools for different needs.

Updated Sep 19, 2025

AAlkiyumi / Senior-Design-Project

Web scraper for collecting product and review data from e-commerce sites using Scraping Bee, AWS, Selenium, and Pandas. Focuses on cost-effective solutions, user-friendly interfaces, and efficient data extraction and analysis.

aws data-storage selenium pandas web-scraping data-extraction uc data-analysis data-processing product-reviews web-crawlers university-of-cincinnati e-commerce-analytics scraping-bee senior-design-project e-commerce-data review-sentiment-analysis uc2026 cs5001

Updated Apr 4, 2025
Python

elmohq / awesome-answer-engine-optimization

Official documentation, crawler user agents, specifications, research, and tools for Answer Engine Optimization (AEO) and Generative Engine Optimization (GEO).

Updated Aug 20, 2026
Python

henu-wang / ai-robots-txt-generator

🤖 Generate optimized robots.txt files for AI search engine crawlers (GPTBot, PerplexityBot, ClaudeBot, and more)

python seo geo gemini web-scraping robots-txt web-optimization ai-search perplexity web-crawlers chatgpt gptbot ai-seo llms-txt generative-engine-optimization ai-crawlers geoscore claudebot

Updated Mar 9, 2026
Python

MiracleWc / Web-crawlers-travel-information

Web crawlers based on travel information, crawler data from Ctrip, tuniu, qunar.

web-crawlers travel-information crawler-data

Updated Sep 8, 2019
Python

codassassin / web-crawler-v2.0

This is an advanced version of the previously released version of web-crawler

python web-crawler python3 subdomain web-crawling web-crawler-python subdomain-scanner web-crawls subdomain-enumeration subdomain-scan web-crawlers subdomain-finder subdomain-scanners

Updated Aug 5, 2021
Python

dudgrayman / agent-ready-kit-template

Template statique pour rendre un petit site plus lisible par les agents IA, crawlers et outils d’extraction.

mcp seo static-site json-ld seo-optimization ai-agents seo-tools web-crawlers llms-txt agent-ready

Updated Jun 3, 2026
HTML

kinti / crawlward

Watchdog for AI crawlers at your own origin — access-log analyzer for Caddy/nginx. Zero dependencies.

nodejs nginx log-analysis robots-txt caddy log-analyzer web-crawlers gptbot ai-crawlers claudebot

Updated Sep 14, 2026
JavaScript

henu-wang / ai-crawlers-reference

Complete database of AI search engine crawler user-agents (GPTBot, PerplexityBot, ClaudeBot) with robots.txt configuration examples

database user-agent reference seo geo robots-txt ai-search user-agents web-crawlers chatgpt gptbot ai-seo llms-txt generative-engine-optimization ai-crawlers perplexitybot geoscore

Updated Mar 9, 2026
HTML

Add this topic to your repo

To associate your repository with the web-crawlers topic, visit your repo's landing page and select "manage topics."