Implementation:ArroyoSystems Arroyo Data Fetching

Knowledge Sources	ArroyoSystems_Arroyo
Domains	WebUI, React, API, DataFetching
Last Updated	2026-02-08 08:00 GMT

Overview

Central data fetching module that provides SWR-based React hooks and an openapi-fetch client for all Arroyo REST API interactions in the WebUI.

Description

This module serves as the single data access layer for the Arroyo WebUI. It creates a typed openapi-fetch client bound to the /api base URL (derived from window.__ARROYO_BASENAME) and exports get, post, patch, and del HTTP methods.

The module defines over 15 custom React hooks using SWR (stale-while-revalidate) and SWR Infinite for paginated resources:

usePing -- Health check with 1s polling
useConnectors -- Lists available connectors
useConnectionProfiles -- CRUD for connection profiles with mutation
useConnectionTables -- Paginated connection tables with infinite scroll
useConnectionProfileAutocomplete -- Autocomplete suggestions for connection profile fields
useJobs -- Lists all jobs
useJobMetrics -- Per-operator metric groups with 1s polling
useJobCheckpoints -- Checkpoint list with 5s polling
useCheckpointDetails -- Operator checkpoint group details
useQueryValidation -- Validates SQL queries with UDFs
useUdfValidation -- Validates UDF definitions with abort controller support
usePipelines -- Paginated pipeline list with 5s polling
usePipeline -- Single pipeline with update, restart, and delete operations
usePipelineJobs -- Jobs for a specific pipeline
useOperatorErrors -- Paginated job error messages
useGlobalUdfs -- Global UDF CRUD operations
useConnectionTableTest -- SSE-based connection table testing

All hooks follow a pattern of key-function + fetcher-function, with the key functions returning null to conditionally disable fetching.

The module also re-exports over 30 named type aliases from the generated API types.

Usage

Imported by virtually every page and component in the WebUI to fetch and mutate data from the Arroyo backend.

Code Reference

Source Location

Repository: ArroyoSystems_Arroyo
File: webui/src/lib/data_fetching.ts

Signature

// HTTP client
export const { get, post, patch, del } = createClient<paths>({ baseUrl: BASE_URL });

// Key hooks (subset)
export const usePing: () => { ping; pingLoading; pingError };
export const useConnectors: () => { connectors; connectorsLoading };
export const useConnectionProfiles: () => { connectionProfiles; connectionProfilesLoading; mutateConnectionProfiles };
export const usePipelines: () => { pipelinePages; pipelinesLoading; piplinesError; pipelineTotalPages; setPipelinesMaxPages };
export const usePipeline: (pipelineId?: string, refresh?: boolean) => { pipeline; pipelineError; pipelineLoading; updatePipeline; deletePipeline; restartPipeline };
export const useQueryValidation: (query?: string, localUdfs?: LocalUdf[]) => { queryValidation; queryValidationError; queryValidationLoading };
export const useGlobalUdfs: () => { globalUdfs; udfsLoading; udfError; createGlobalUdf; deleteGlobalUdf };

// Type exports
export type Pipeline = schemas['Pipeline'];
export type Job = schemas['Job'];
// ... 30+ type aliases

Import

import { usePipeline, usePipelineJobs, Pipeline, Job, post } from '../../lib/data_fetching';

I/O Contract

Inputs

Name	Type	Required	Description
window.__ARROYO_BASENAME	string	Yes	Global base path for API URL construction

Outputs

Name	Type	Description
SWR hooks	React hooks	Data fetching hooks returning data, loading, error, and mutation functions
HTTP client	{ get, post, patch, del }	Typed openapi-fetch client for direct API calls
Type aliases	TypeScript types	Re-exported schema types from api-types.ts

Usage Examples

// Fetch pipeline with auto-refresh
const { pipeline, updatePipeline, deletePipeline } = usePipeline(pipelineId, true);

// Validate a SQL query
const { queryValidation, queryValidationLoading } = useQueryValidation(queryInput, localUdfs);

// Direct API call
const { data, error } = await post('/v1/pipelines', {
  body: { name: 'my-pipeline', parallelism: 4, query: 'SELECT * FROM source' }
});

Related Pages

Page Connections

Double-click a node to navigate. Hold to expand connections.

Principle

Implementation

Heuristic

Environment