Implementation:CrewAIInc CrewAI Hyperbrowser Load Tool

Knowledge Sources	CrewAI
Domains	Tools, Web_Scraping, Hyperbrowser
Last Updated	2026-02-11 00:00 GMT

Overview

Scrapes or crawls web pages using Hyperbrowser's scalable headless browser infrastructure and returns content as markdown or HTML.

Description

HyperbrowserLoadTool extends BaseTool and integrates with the hyperbrowser Python SDK. It requires an api_key from the constructor or the HYPERBROWSER_API_KEY environment variable to initialize a Hyperbrowser client. The tool supports two operations: scrape (single page extraction) and crawl (multi-page traversal), specified via the operation parameter. Optional params allow configuration of session options and scrape options including output format validation (only markdown and html are allowed). The _run() method delegates to scrape.start_and_wait or crawl.start_and_wait on the Hyperbrowser client, then extracts markdown or HTML content from the response. For crawl operations, content from multiple pages is concatenated with URL separators.

Usage

Use this tool when a CrewAI agent needs enterprise-grade web scraping with built-in anti-bot bypass capabilities, particularly for complex websites that block standard scrapers.

Code Reference

Source Location

Repository: CrewAI
File: lib/crewai-tools/src/crewai_tools/tools/hyperbrowser_load_tool/hyperbrowser_load_tool.py
Lines: 1-137

Signature

class HyperbrowserLoadTool(BaseTool):
    name: str = "Hyperbrowser web load tool"
    description: str = "Scrape or crawl a website using Hyperbrowser and return the contents ..."
    args_schema: type[BaseModel] = HyperbrowserLoadToolSchema
    api_key: str | None = None
    hyperbrowser: Any | None = None

    def __init__(self, api_key: str | None = None, **kwargs): ...
    def _run(self, url: str, operation: Literal["scrape", "crawl"] = "scrape",
             params: dict | None = None): ...

Import

from crewai_tools import HyperbrowserLoadTool

I/O Contract

Inputs

Name	Type	Required	Description
url	str	Yes	Website URL to scrape or crawl
operation	str	Yes	Operation type: `scrape` or `crawl`
params	dict	No	Optional parameters for session and scrape options
api_key	str	Yes (constructor)	Hyperbrowser API key (or HYPERBROWSER_API_KEY env var)

Outputs

Name	Type	Description
_run() returns	str	Extracted markdown or HTML content; for crawl operations, pages are concatenated with URL separators

Usage Examples

Scrape a Single Page

from crewai_tools import HyperbrowserLoadTool

tool = HyperbrowserLoadTool(api_key="your-hyperbrowser-key")
result = tool.run(url="https://example.com/article", operation="scrape")

Crawl with Custom Parameters

from crewai_tools import HyperbrowserLoadTool

tool = HyperbrowserLoadTool(api_key="your-hyperbrowser-key")
result = tool.run(
    url="https://docs.example.com",
    operation="crawl",
    params={
        "scrape_options": {"formats": ["markdown"]},
        "session_options": {"timeout": 30000},
    },
)

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment