Skip to content
@scrapy

Scrapy project

An open source and collaborative framework for extracting the data you need from websites. In a fast, simple, yet extensible way.

Pinned Loading

  1. scrapy scrapy Public

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    Python 64.5k 12k

  2. scrapyd scrapyd Public

    A service daemon to run Scrapy spiders

    Python 3.1k 579

  3. parsel parsel Public

    Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

    Python 1.4k 173

  4. w3lib w3lib Public

    Python library of web-related functions

    HTML 417 126

  5. protego protego Public

    A pure-Python robots.txt parser with support for modern conventions.

    DIGITAL Command Language 94 34

  6. itemadapter itemadapter Public

    Common interface for data container classes

    Python 69 17

Repositories

Showing 10 of 34 repositories
  • scrapy Public

    Scrapy, a fast high-level web crawling & scraping framework for Python.

    scrapy/scrapy's past year of commit activity
    Python 64,511 BSD-3-Clause 11,978 167 (9 issues need help) 158 Updated Sep 28, 2026
  • parsel Public

    Parsel lets you extract data from XML/HTML documents using XPath or CSS selectors

    scrapy/parsel's past year of commit activity
    Python 1,358 BSD-3-Clause 173 10 11 Updated Sep 28, 2026
  • scrapy-lint Public

    A linter for Scrapy projects.

    scrapy/scrapy-lint's past year of commit activity
    Python 22 MIT 8 20 22 Updated Sep 28, 2026
  • w3lib Public

    Python library of web-related functions

    scrapy/w3lib's past year of commit activity
    HTML 417 BSD-3-Clause 126 4 6 Updated Sep 27, 2026
  • xtractmime Public

    https://mimesniff.spec.whatwg.org/ implementation for Python

    scrapy/xtractmime's past year of commit activity
    Python 13 BSD-3-Clause 2 0 0 Updated Sep 26, 2026
  • form2request Public

    Python library to build HTTP requests out of HTML forms

    scrapy/form2request's past year of commit activity
    Python 5 Apache-2.0 5 2 2 Updated Sep 25, 2026
  • sphinx-scrapy Public

    Sphinx extension for documentation in the Scrapy ecosystem

    scrapy/sphinx-scrapy's past year of commit activity
    Python 2 BSD-3-Clause 2 0 1 Updated Sep 25, 2026
  • sphinx-llm-friendly Public

    Sphinx extension to make documentation LLM-friendly

    scrapy/sphinx-llm-friendly's past year of commit activity
    Python 2 MIT 1 0 0 Updated Sep 25, 2026
  • scrapy-mcp-official Public

    Scrapy MCP server

    scrapy/scrapy-mcp-official's past year of commit activity
    Python 10 Apache-2.0 1 0 0 Updated Sep 25, 2026
  • frostwork Public
    scrapy/frostwork's past year of commit activity
    Python 13 Apache-2.0 1 0 0 Updated Sep 23, 2026
View all repositories