Implementation:CrewAIInc CrewAI BrightData Unlocker Tool
| Knowledge Sources | |
|---|---|
| Domains | Tools, Web_Scraping, BrightData |
| Last Updated | 2026-02-11 00:00 GMT |
Overview
Concrete tool for scraping website content using Bright Data's Web Unlocker API provided by CrewAI.
Description
The BrightDataWebUnlockerTool class extends BaseTool to perform web scraping by routing requests through Bright Data's unlocking and proxy infrastructure, which can bypass bot protection mechanisms like CAPTCHAs, geo-restrictions, and anti-bot detection. The tool sends POST requests to the BrightData API endpoint (https://api.brightdata.com/request) with the target URL, zone identifier, response format (raw), and optional data format conversion. It supports two data formats: html (default) and markdown. The tool validates the BRIGHT_DATA_API_KEY and BRIGHT_DATA_ZONE environment variables during initialization, and includes comprehensive error handling that includes response text in error messages for debugging.
Usage
Use this tool when CrewAI agents need to reliably extract content from protected websites that employ anti-scraping measures, CAPTCHAs, or geo-restrictions, without implementing complex bot evasion logic.
Code Reference
Source Location
- Repository: CrewAI
- File: lib/crewai-tools/src/crewai_tools/tools/brightdata_tool/brightdata_unlocker.py
- Lines: 1-146
Signature
class BrightDataWebUnlockerTool(BaseTool):
name: str = "Bright Data Web Unlocker Scraping"
description: str = "Tool to perform web scraping using Bright Data Web Unlocker"
args_schema: type[BaseModel] = BrightDataUnlockerToolSchema
base_url: str = ""
api_key: str = ""
zone: str = ""
url: str | None = None
format: str = "raw"
data_format: str = "markdown"
def __init__(self, url=None, format="raw", data_format="markdown", **kwargs): ...
def _run(self, url=None, format=None, data_format=None, **kwargs) -> Any: ...
Import
from crewai_tools import BrightDataWebUnlockerTool
I/O Contract
Inputs
| Name | Type | Required | Description |
|---|---|---|---|
| url | str | Yes | Target URL to scrape (constructor or runtime) |
| format | str or None | No | Response format from BrightData (default "raw") |
| data_format | str or None | No | Response data format: "html" or "markdown" (default "markdown") |
Outputs
| Name | Type | Description |
|---|---|---|
| _run() returns | str | Scraped web page content in the requested format (HTML or markdown) |
Usage Examples
Basic Usage
from crewai_tools import BrightDataWebUnlockerTool
tool = BrightDataWebUnlockerTool(data_format="markdown")
content = tool.run(url="https://example.com/protected-page")