Workflow:Puppeteer Puppeteer Web Scraping And Interaction
| Knowledge Sources | |
|---|---|
| Domains | Browser_Automation, Web_Scraping, Data_Extraction |
| Last Updated | 2026-02-11 23:30 GMT |
Overview
End-to-end process for navigating web pages, interacting with UI elements, and extracting structured data using Puppeteer's automation APIs.
Description
This workflow covers the standard procedure for automated web scraping and interaction with Puppeteer. It demonstrates how to navigate to pages, simulate user input (keyboard, mouse, touch), wait for dynamic content to appear, interact with UI elements via selectors (CSS, ARIA, text, XPath), and extract structured data from the DOM using in-page evaluation. The workflow leverages Puppeteer's Locator API for resilient element interaction with automatic retries and visibility checks, as well as the traditional query-and-evaluate pattern for data extraction.
Usage
Execute this workflow when you need to automate multi-step user interactions on a web page and extract data from the resulting content. This includes searching websites, filling forms, navigating pagination, scraping search results, extracting product listings, and automating any sequence of UI interactions that produces data for collection.
Execution Steps
Step 1: Launch Browser And Create Page
Start a headless browser instance and open a new page. For scraping workflows, consider configuring the user agent string and viewport to match real browser behavior and avoid detection. The page provides all interaction and evaluation APIs needed for the subsequent steps.
Key considerations:
- Set a realistic viewport size to trigger proper responsive layouts
- Some sites require specific user agents; use page.setUserAgent() if needed
- Consider disabling images or other resources for faster scraping
Direct the page to load the starting URL. Use appropriate wait conditions to ensure the initial content is ready for interaction. For dynamic single-page applications, wait for specific selector elements rather than relying solely on network idle signals.
Key considerations:
- Choose wait conditions based on page behavior (load, networkidle0, networkidle2)
- Use waitForSelector() after navigation if the target content loads asynchronously
- Handle navigation errors (timeouts, DNS failures) with try/catch
Step 3: Interact With Page Elements
Simulate user interactions to reach the desired content. This includes typing into input fields, clicking buttons and links, pressing keyboard shortcuts, hovering over elements, and scrolling. Puppeteer offers two approaches: the modern Locator API (recommended) and the traditional selector-based methods.
Locator API approach:
- Use page.locator(selector) to create resilient element references
- Locators auto-retry, wait for visibility, and ensure elements are actionable
- Supports .click(), .fill(), .hover(), .scroll() with built-in waiting
- Composable with .filter(), .map(), and Locator.race() for complex scenarios
Traditional approach:
- Use page.type(selector, text) for typing
- Use page.click(selector) for clicking
- Use page.keyboard.press(key) for key presses
- Combine with page.waitForSelector() for explicit waiting
Step 4: Wait For Dynamic Content
After interactions, wait for the resulting content to appear in the DOM. Dynamic pages often load results asynchronously, requiring explicit waits before data extraction.
Key considerations:
- Use waitForSelector(selector) to wait for specific elements to appear
- Use waitForFunction(fn) for complex conditions (e.g., element count, text content)
- Use waitForNavigation() when interactions trigger full page navigations
- Configure timeouts appropriately for slow-loading content
Step 5: Extract Data From Page
Execute JavaScript within the page context using page.evaluate() to query the DOM and extract structured data. The evaluate function runs in the browser, has access to the full DOM API, and can return serializable data back to Node.js.
Key considerations:
- Arguments passed to evaluate() are serialized; pass selectors as strings
- Return values must be JSON-serializable (no DOM nodes, functions, or circular refs)
- Use page.$$eval(selector, fn) as shorthand for querySelectorAll + map
- For complex extraction, build data structures inside evaluate and return them
- Handle cases where expected elements are missing (defensive selectors)
Step 6: Process Results And Close Browser
Process the extracted data in Node.js (format, filter, save to file or database) and close the browser to release resources. For multi-page scraping, repeat steps 2-5 for each target page before closing.
Key considerations:
- Close the browser after all extraction is complete
- For pagination, navigate to next page and repeat extraction in a loop
- Use try/finally to ensure browser cleanup even on errors