Implementation:CrewAIInc CrewAI Oxylabs Universal Scraper Tool

Knowledge Sources	CrewAI
Domains	Tools, Web_Scraping, Data_Extraction
Last Updated	2026-02-11 00:00 GMT

Overview

OxylabsUniversalScraperTool scrapes any arbitrary website URL using the Oxylabs universal web scraper API.

Description

The module follows the standard Oxylabs tool family pattern with three classes: OxylabsUniversalScraperArgs (a Pydantic input schema accepting a website URL string, unlike the Amazon tools which take ASINs or search terms), OxylabsUniversalScraperConfig (a configuration model with options including geo_location, user_agent_type, render, callback_url, context, parse, and parsing_instructions), and OxylabsUniversalScraperTool (the main tool extending BaseTool). This is the most versatile tool in the Oxylabs scraper family, capable of scraping any website rather than being limited to specific platforms. Initialization handles credential resolution from constructor parameters or OXYLABS_USERNAME/OXYLABS_PASSWORD environment variables and creates an Oxylabs RealtimeClient with SDK version metadata. The _run method calls self.oxylabs_api.universal.scrape_url(url, **config) and returns the first result's content, serializing dict content as JSON. It supports JavaScript rendering, custom configurations, and structured parsing for any URL.

Usage

Use this tool when a CrewAI agent needs to scrape content from any website that is not covered by the specialized Oxylabs tools (Amazon, Google), serving as a general-purpose web content extraction tool.

Code Reference

Source Location

• Repository: CrewAI
• File: lib/crewai-tools/src/crewai_tools/tools/oxylabs_universal_scraper_tool/oxylabs_universal_scraper_tool.py
• Lines: 1-163

Signature

class OxylabsUniversalScraperArgs(BaseModel):
    url: str = Field(description="Website URL")

class OxylabsUniversalScraperConfig(BaseModel):
    geo_location: str | None = None
    user_agent_type: str | None = None
    render: str | None = None
    callback_url: str | None = None
    context: list | None = None
    parse: bool | None = None
    parsing_instructions: dict | None = None

class OxylabsUniversalScraperTool(BaseTool):
    name: str = "Oxylabs Universal Scraper tool"
    description: str = "Scrape any url with Oxylabs Universal Scraper"
    args_schema: type[BaseModel] = OxylabsUniversalScraperArgs
    oxylabs_api: RealtimeClient
    config: OxylabsUniversalScraperConfig

    def __init__(
        self,
        username: str | None = None,
        password: str | None = None,
        config: OxylabsUniversalScraperConfig | dict | None = None,
        **kwargs,
    ): ...

    def _run(self, url: str) -> str: ...

Import

from crewai_tools import OxylabsUniversalScraperTool

I/O Contract

Inputs

Name	Type	Required	Description
username	str or None	No	Oxylabs username; falls back to OXYLABS_USERNAME environment variable
password	str or None	No	Oxylabs password; falls back to OXYLABS_PASSWORD environment variable
config	OxylabsUniversalScraperConfig or dict or None	No	Configuration options for the scraper; defaults to empty config
url	str	Yes	Website URL to scrape

Outputs

Name	Type	Description
_run() returns	str	Scraped page content as string or JSON-serialized dict if structured parsing is enabled

Usage Examples

Basic Usage

from crewai_tools import OxylabsUniversalScraperTool

tool = OxylabsUniversalScraperTool(
    username="your_username",
    password="your_password",
)
result = tool._run(url="https://example.com/pricing")

With JavaScript Rendering

from crewai_tools import OxylabsUniversalScraperTool
from crewai_tools.tools.oxylabs_universal_scraper_tool.oxylabs_universal_scraper_tool import (
    OxylabsUniversalScraperConfig,
)

config = OxylabsUniversalScraperConfig(
    render="html",
    geo_location="United States",
    parse=True,
)
tool = OxylabsUniversalScraperTool(config=config)
result = tool._run(url="https://example.com/dynamic-page")

Related Pages

• Principle:CrewAIInc_CrewAI_Built_In_Tool_Selection