Implementation:CrewAIInc CrewAI Selenium Scraping Tool
| Knowledge Sources | |
|---|---|
| Domains | Tools, Web_Scraping, Browser_Automation |
| Last Updated | 2026-02-11 00:00 GMT |
Overview
SeleniumScrapingTool scrapes web pages using Selenium WebDriver, enabling JavaScript rendering and content extraction from dynamic web pages.
Description
SeleniumScrapingTool extends BaseTool and uses the dual-schema pattern with SeleniumScrapingToolSchema (requiring website_url with thorough URL validation and css_element) and FixedSeleniumScrapingToolSchema (empty, for pre-configured URLs). URL validation checks for emptiness, length (max 2048), http/https prefix, valid format via urlparse, and whitespace. On initialization, the tool imports selenium (with interactive install fallback via click.confirm), creates a headless Chrome WebDriver (or accepts a custom driver/options), and stores cookie and CSS element settings. The _run() method navigates to the URL, waits for content to load (configurable wait_time), optionally sets cookies and reloads. Content extraction either grabs the full <body> element (when no CSS element is specified) or uses find_elements(By.CSS_SELECTOR) for targeted selection. The return_html flag controls whether outerHTML or plain text is returned. The driver is closed in a finally block.
Usage
Use this tool for scraping JavaScript-heavy single-page applications and dynamically rendered content that cannot be scraped with simple HTTP requests. It is the self-hosted alternative to API-based services like Scrapfly or Scrapegraph.
Code Reference
Source Location
- Repository: CrewAI
- File: lib/crewai-tools/src/crewai_tools/tools/selenium_scraping_tool/selenium_scraping_tool.py
- Lines: 1-198
Signature
class SeleniumScrapingToolSchema(FixedSeleniumScrapingToolSchema):
website_url: str = Field(..., description="Mandatory website url to read the file.")
css_element: str = Field(..., description="Mandatory css reference for element to scrape")
class SeleniumScrapingTool(BaseTool):
name: str = "Read a website content"
description: str = "A tool that can be used to read a website content."
args_schema: type[BaseModel] = SeleniumScrapingToolSchema
website_url: str | None = None
driver: Any | None = None
cookie: dict | None = None
wait_time: int | None = 3
css_element: str | None = None
return_html: bool | None = False
package_dependencies: list[str] # ["selenium", "webdriver-manager"]
def __init__(self, website_url=None, cookie=None, css_element=None, **kwargs)
def _run(self, **kwargs) -> Any
def close(self)
Import
from crewai_tools import SeleniumScrapingTool
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| website_url | str | Yes | URL of the website to scrape (optional if set at init). Must start with http:// or https:// |
| css_element | str | No | CSS selector for specific elements to extract. If empty, extracts full body |
| return_html | bool | No | If true, returns outerHTML instead of text content. Default false |
Outputs
| Name | Type | Description |
|---|---|---|
| _run() returns | str | Newline-joined content of matched elements (text or HTML), or error message string |
Usage Examples
Basic Usage
from crewai_tools import SeleniumScrapingTool
# Full page scraping
tool = SeleniumScrapingTool()
result = tool._run(website_url="https://example.com", css_element="")
# Pre-configured with CSS selector
tool = SeleniumScrapingTool(
website_url="https://example.com",
css_element="div.main-content",
cookie={"name": "session", "value": "SESSION_ENV_VAR"}
)
result = tool._run()