Implementation:CrewAIInc CrewAI Serper Scrape Website Tool

Knowledge Sources	CrewAI
Domains	Tools, Web_Scraping
Last Updated	2026-02-11 00:00 GMT

Overview

SerperScrapeWebsiteTool scrapes clean, readable content from any website URL using the Serper scraping API.

Description

The SerperScrapeWebsiteTool extends BaseTool to extract website content through the Serper scraping service at https://scrape.serper.dev. It accepts a URL and an optional flag to include markdown formatting, sends a POST request with the API key in an X-API-KEY header, and returns the text content from the JSON response. The tool includes comprehensive error handling for HTTP errors, JSON decode failures, and unexpected exceptions, returning descriptive error strings rather than raising exceptions.

Usage

Use this tool when a CrewAI agent needs to retrieve and read the textual content of a specific web page. It complements search-focused tools by enabling direct content retrieval from known URLs.

Code Reference

Source Location

• Repository: CrewAI
• File: lib/crewai-tools/src/crewai_tools/tools/serper_scrape_website_tool/serper_scrape_website_tool.py
• Lines: 1-83

Signature

class SerperScrapeWebsiteInput(BaseModel):
    url: str = Field(..., description="The URL of the website to scrape")
    include_markdown: bool = Field(default=True, ...)

class SerperScrapeWebsiteTool(BaseTool):
    name: str = "serper_scrape_website"
    description: str = "Scrapes website content using Serper's scraping API. ..."
    args_schema: type[BaseModel] = SerperScrapeWebsiteInput
    env_vars: list[EnvVar]  # SERPER_API_KEY

    def _run(self, url: str, include_markdown: bool = True) -> str:
        ...

Import

from crewai_tools import SerperScrapeWebsiteTool

I/O Contract

Inputs

Name	Type	Required	Description
url	str	Yes	The URL of the website to scrape
include_markdown	bool	No	Whether to include markdown formatting in the scraped content (default True)

Outputs

Name	Type	Description
_run() returns	str	Scraped website text content, or a descriptive error string on failure

Usage Examples

Basic Usage

from crewai_tools import SerperScrapeWebsiteTool

tool = SerperScrapeWebsiteTool()
result = tool._run(url="https://example.com", include_markdown=True)

Related Pages

• Principle:CrewAIInc_CrewAI_Built_In_Tool_Selection