Implementation:CrewAIInc CrewAI Spider Tool

Knowledge Sources	CrewAI
Domains	Tools, Web_Scraping, Web_Crawling
Last Updated	2026-02-11 00:00 GMT

Overview

SpiderTool scrapes and crawls websites using the Spider.cloud API, returning LLM-ready content in markdown format.

Description

The SpiderTool extends BaseTool and provides two modes of operation: scrape (extract content from a single page) and crawl (follow up to 5 links and extract content from multiple pages). It uses a SpiderToolConfig class that centralizes default settings including crawl limit of 5, return format "markdown", request mode "smart", and SVG filtering enabled. During initialization, it imports the Spider client from the spider-client package, prompting for installation if not found. The _validate_url method checks URL format and ensures only HTTP/HTTPS schemes, with URL decoding. The _run method builds request parameters, adds a crawl limit in crawl mode, merges custom parameters, and delegates to either spider.scrape_url or spider.crawl_url. Error handling covers ValueError, ImportError, and ConnectionError with configurable logging or re-raising based on the log_failures flag.

Usage

Use this tool when a CrewAI agent needs to scrape a single web page or crawl multiple linked pages for content extraction in agent workflows.

Code Reference

Source Location

• Repository: CrewAI
• File: lib/crewai-tools/src/crewai_tools/tools/spider_tool/spider_tool.py
• Lines: 1-218

Signature

class SpiderToolSchema(BaseModel):
    website_url: str = Field(..., description="Mandatory website URL to scrape or crawl")
    mode: Literal["scrape", "crawl"] = Field(default="scrape", ...)

class SpiderToolConfig(BaseModel):
    DEFAULT_CRAWL_LIMIT: int = 5
    DEFAULT_RETURN_FORMAT: str = "markdown"
    DEFAULT_REQUEST_MODE: str = "smart"
    FILTER_SVG: bool = True

class SpiderTool(BaseTool):
    name: str = "SpiderTool"
    description: str = "A tool to scrape or crawl a website and return LLM-ready content."
    args_schema: type[BaseModel] = SpiderToolSchema
    custom_params: dict[str, Any] | None = None
    website_url: str | None = None
    log_failures: bool = True
    env_vars: list[EnvVar]  # SPIDER_API_KEY

    def _run(self, website_url: str, mode: Literal["scrape", "crawl"] = "scrape") -> str | None:
        ...

Import

from crewai_tools import SpiderTool

I/O Contract

Inputs

Name	Type	Required	Description
website_url	str	Yes	The URL to scrape or crawl (must be HTTP/HTTPS)
mode	Literal["scrape", "crawl"]	No	Operation mode: "scrape" for single page, "crawl" for multiple pages (default "scrape")

Outputs

Name	Type	Description
_run() returns	str or None	Extracted content in markdown format, or None if extraction fails and log_failures is True

Usage Examples

Basic Usage

from crewai_tools import SpiderTool

tool = SpiderTool()
result = tool._run(website_url="https://example.com", mode="scrape")

Crawl Mode

from crewai_tools import SpiderTool

tool = SpiderTool(custom_params={"limit": 10})
result = tool._run(website_url="https://example.com", mode="crawl")

Related Pages

• Principle:CrewAIInc_CrewAI_Built_In_Tool_Selection