Implementation:Microsoft Autogen Studio Fetch Webpage Tool

Sources	python/packages/autogen-studio/autogenstudio/gallery/tools/fetch_webpage.py
Domains	Tools, Web Scraping, Content Extraction, Markdown, AutoGen Studio
Last Updated	2026-02-11

Overview

Description

The Fetch Webpage Tool is a utility within AutoGen Studio that enables agents to retrieve web pages and convert their HTML content into clean, readable markdown format. The tool provides a simple yet powerful interface for web content extraction with configurable options for image inclusion, content length limiting, and custom HTTP headers.

This implementation uses asynchronous HTTP requests for efficient operation and includes intelligent HTML parsing to remove unwanted elements (scripts, styles) while preserving content structure and converting relative URLs to absolute paths.

Key Features

• HTML to Markdown conversion: Automatic transformation of web content to markdown format
• Content cleaning: Removes scripts, styles, and other non-content elements
• URL normalization: Converts relative URLs to absolute paths for images and links
• Configurable output: Options for including/excluding images and limiting content length
• Custom headers: Support for custom HTTP headers for specialized requests
• Async operation: Non-blocking HTTP requests for better performance

Usage

The tool is designed to be used by AutoGen agents through the FunctionTool interface. It requires no API keys or special configuration, making it simple to deploy and use.

Basic Usage:

markdown_content = await fetch_webpage(
    url="https://example.com/article",
    include_images=True,
    max_length=5000
)

Code Reference

Source Location

• File: python/packages/autogen-studio/autogenstudio/gallery/tools/fetch_webpage.py
• Repository: https://github.com/microsoft/autogen
• Lines: 88 total

Function Signature

async def fetch_webpage(
    url: str,
    include_images: bool = True,
    max_length: Optional[int] = None,
    headers: Optional[Dict[str, str]] = None
) -> str

Import Statement

from autogenstudio.gallery.tools.fetch_webpage import fetch_webpage, fetch_webpage_tool

Dependencies

• Standard Library: typing, urllib.parse
• Third-party: httpx, html2text, beautifulsoup4
• AutoGen: autogen_core.code_executor, autogen_core.tools

I/O Contract

Inputs

Parameter	Type	Default	Description
url	str	(required)	The URL of the webpage to fetch
include_images	bool	True	Whether to include image references in the markdown output
max_length	Optional[int]	None	Maximum length of the output markdown (None for unlimited)
headers	Optional[Dict[str, str]]	None	Optional HTTP headers for the request (uses default User-Agent if None)

Outputs

Field	Type	Description
return	str	Markdown version of the webpage content, stripped of whitespace

Output Format:

• Clean markdown text with preserved document structure
• Headers, links, lists, tables, and formatting maintained
• Image references included (if include_images=True) as markdown image syntax
• Content truncated with "...(truncated)" suffix if max_length exceeded
• Whitespace trimmed from beginning and end

Exceptions

Exception	Condition
ValueError	Failed to fetch webpage (network error, timeout, invalid URL)
ValueError	Error processing webpage (parsing error, conversion error)

Implementation Details

Core Algorithm

1. HTTP Request Phase:
   1. Use default User-Agent header if none provided
   2. Create async HTTP client with httpx
   3. Fetch webpage with GET request (10-second timeout)
   4. Raise exception on HTTP errors (4xx, 5xx)
2. HTML Parsing Phase:
   1. Parse HTML content with BeautifulSoup
   2. Remove all <script> and <style> elements
   3. Find all anchor and image tags
   4. Convert relative URLs to absolute using urljoin()
3. Markdown Conversion Phase:
   1. Configure html2text converter with options:
      1. body_width=0 (no line wrapping)
      2. ignore_images based on include_images parameter
      3. Preserve emphasis, links, and tables
   2. Convert HTML to markdown
4. Post-processing Phase:
   1. Apply max_length truncation if specified
   2. Strip leading/trailing whitespace
   3. Return final markdown string

HTML2Text Configuration

The tool configures html2text with the following settings:

• body_width: 0 (disables line wrapping for clean output)
• ignore_images: Controlled by include_images parameter
• ignore_emphasis: False (preserves bold, italic formatting)
• ignore_links: False (preserves hyperlinks)
• ignore_tables: False (preserves table structures)

Default Headers

If no custom headers are provided, the tool uses:

{
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36"
}

This User-Agent string helps avoid blocking by websites that reject requests without proper headers.

Usage Examples

Example 1: Basic Webpage Fetch

# Fetch and convert a webpage to markdown
content = await fetch_webpage("https://en.wikipedia.org/wiki/Python_(programming_language)")

print(content)
# Output: Clean markdown with all content, images, and links

Example 2: Fetch Without Images

# Get text content only, excluding images
text_only = await fetch_webpage(
    url="https://news.example.com/article",
    include_images=False
)

# Useful for text analysis or when images aren't needed

Example 3: Limited Content Length

# Fetch only the first 2000 characters
summary = await fetch_webpage(
    url="https://blog.example.com/long-article",
    max_length=2000
)

# Content will end with "\n...(truncated)" if original was longer
print(f"Content length: {len(summary)} chars")

Example 4: Custom Headers

# Use custom headers for authenticated or specialized requests
custom_headers = {
    "User-Agent": "MyBot/1.0",
    "Accept-Language": "en-US",
    "Cookie": "session=abc123"
}

content = await fetch_webpage(
    url="https://api.example.com/documentation",
    headers=custom_headers
)

Example 5: Using the FunctionTool

from autogenstudio.gallery.tools.fetch_webpage import fetch_webpage_tool

# Add tool to an agent
agent = ConversableAgent(
    name="web_reader",
    tools=[fetch_webpage_tool]
)

# Agent can now process instructions like:
# "Fetch the webpage at https://example.com and summarize it"

Example 6: Error Handling

try:
    content = await fetch_webpage("https://invalid-url-example.com")
except ValueError as e:
    print(f"Error: {e}")
    # Handle error appropriately
    # e.g., "Failed to fetch webpage: Connection timeout"

Example 7: Batch Processing

import asyncio

urls = [
    "https://example.com/page1",
    "https://example.com/page2",
    "https://example.com/page3"
]

# Fetch multiple pages concurrently
tasks = [fetch_webpage(url, max_length=1000) for url in urls]
results = await asyncio.gather(*tasks, return_exceptions=True)

for url, result in zip(urls, results):
    if isinstance(result, Exception):
        print(f"Failed to fetch {url}: {result}")
    else:
        print(f"Fetched {url}: {len(result)} chars")

Implementation Notes

Performance Considerations

• Timeout: 10-second timeout prevents hanging on slow or unresponsive servers
• Async operation: Allows multiple concurrent fetches without blocking
• Memory efficiency: max_length parameter prevents excessive memory use on large pages

Content Extraction Quality

• Script removal: JavaScript code is removed to avoid clutter
• Style removal: CSS styles are removed as they don't translate to markdown
• URL preservation: All links and images maintain absolute URLs for portability
• Structure preservation: Headers, lists, tables, and formatting are maintained

Limitations

• JavaScript-rendered content: Cannot execute JavaScript; only fetches static HTML
• Authentication: Basic HTTP authentication only; complex auth flows not supported
• Rate limiting: No built-in rate limiting; use external mechanisms for high-volume scraping
• Encoding: Relies on httpx's automatic encoding detection

Related Pages

• Implementation: Studio Bing Search Tool - Search tool that uses fetch_webpage internally
• Implementation: Studio Google Search Tool - Another search tool with content fetching
• Microsoft Autogen Studio Tools - Overview of all AutoGen Studio gallery tools
• AutoGen Core Function Tools - Documentation on FunctionTool framework
• Web Scraping Best Practices - Guidelines for responsible web scraping

See Also

• html2text Documentation
• HTTPX Library Documentation
• BeautifulSoup Documentation
• HTML Parsing
• Async Programming
• Content Conversion