fetcher Agent

HTTP requests done right. Caching, retries, and error handling built in.

Basic Usage

agents:
  - name: fetch
    agent: fetcher
    inputs:
      url: https://api.example.com/data

Output:

{
  "status": 200,
  "headers": {...},
  "body": {...},
  "cached": false
}

Inputs

inputs:
  url:
    type: string
    required: true
    description: URL to fetch

  method:
    type: string
    default: GET
    description: HTTP method

  headers:
    type: object
    description: Request headers

  body:
    type: any
    description: Request body

  timeout:
    type: number
    default: 30000
    description: Request timeout (ms)

  cacheTTL:
    type: number
    description: Cache duration (seconds)

Configuration

GET Request

agents:
  - name: fetch
    agent: fetcher
    inputs:
      url: https://api.example.com/users/${input.user_id}

POST Request

agents:
  - name: create
    agent: fetcher
    inputs:
      url: https://api.example.com/users
      method: POST
      headers:
        Content-Type: application/json
        Authorization: Bearer ${env.API_KEY}
      body:
        name: ${input.name}
        email: ${input.email}

With Caching

agents:
  - name: fetch
    agent: fetcher
    inputs:
      url: https://api.example.com/data
      cacheTTL: 3600  # Cache for 1 hour

With Retries

agents:
  - name: fetch
    agent: fetcher
    inputs:
      url: https://api.example.com/data
    retry:
      maxAttempts: 5
      backoff: exponential
      retryOn: [500, 502, 503, 504]

Advanced Patterns

Multiple Endpoints

agents:
  # Fetch in parallel
  - name: fetch-users
    agent: fetcher
    inputs:
      url: https://api.example.com/users

  - name: fetch-posts
    agent: fetcher
    inputs:
      url: https://api.example.com/posts

  - name: fetch-comments
    agent: fetcher
    inputs:
      url: https://api.example.com/comments

  # Combine results
  - name: combine
    operation: code
    config:
      code: |
        return {
          users: ${fetch-users.output.body},
          posts: ${fetch-posts.output.body},
          comments: ${fetch-comments.output.body}
        };

Primary + Fallback

agents:
  - name: fetch-primary
    agent: fetcher
    inputs:
      url: https://primary-api.example.com/data
    retry:
      maxAttempts: 2

  - name: fetch-secondary
    condition: ${fetch-primary.failed}
    agent: fetcher
    inputs:
      url: https://secondary-api.example.com/data

  - name: result
    operation: code
    config:
      code: |
        return ${fetch-primary.output} || ${fetch-secondary.output};

Paginated Requests

agents:
  - name: fetch-page-1
    agent: fetcher
    inputs:
      url: https://api.example.com/items?page=1

  - name: fetch-page-2
    condition: ${fetch-page-1.output.body.hasMore}
    agent: fetcher
    inputs:
      url: https://api.example.com/items?page=2

  - name: fetch-page-3
    condition: ${fetch-page-2.output.body.hasMore}
    agent: fetcher
    inputs:
      url: https://api.example.com/items?page=3

  - name: combine
    operation: code
    config:
      code: |
        return {
          items: [
            ...${fetch-page-1.output.body.items},
            ...${fetch-page-2.output.body.items || []},
            ...${fetch-page-3.output.body.items || []}
          ]
        };

Authenticated Requests

agents:
  # Get auth token
  - name: auth
    agent: fetcher
    inputs:
      url: https://auth.example.com/token
      method: POST
      body:
        client_id: ${env.CLIENT_ID}
        client_secret: ${env.CLIENT_SECRET}
    cache:
      ttl: 3600
      key: auth-token

  # Use token
  - name: fetch-data
    agent: fetcher
    inputs:
      url: https://api.example.com/data
      headers:
        Authorization: Bearer ${auth.output.body.access_token}

Conditional Requests

agents:
  - name: check-cache
    operation: storage
    config:
      type: kv
      action: get
      key: data-${input.id}

  - name: fetch-if-stale
    condition: ${!check-cache.output}
    agent: fetcher
    inputs:
      url: https://api.example.com/data/${input.id}

  - name: update-cache
    condition: ${fetch-if-stale.executed}
    operation: storage
    config:
      type: kv
      action: put
      key: data-${input.id}
      value: ${fetch-if-stale.output.body}
      ttl: 3600

Output Schema

{
  status: number;       // HTTP status code
  headers: object;      // Response headers
  body: any;           // Response body (parsed JSON or text)
  cached: boolean;     // Whether response was cached
  took: number;        // Request duration (ms)
  url: string;         // Final URL (after redirects)
}

Error Handling

agents:
  - name: fetch
    agent: fetcher
    inputs:
      url: ${input.url}

  - name: handle-error
    condition: ${fetch.failed}
    operation: code
    config:
      code: |
        const error = ${fetch.error};

        if (error.status === 404) {
          return { notFound: true };
        } else if (error.status >= 500) {
          return { serverError: true, retryable: true };
        } else if (error.timeout) {
          return { timeout: true, retryable: true };
        } else {
          return { error: error.message };
        }

Best Practices

1. Use Caching for Slow APIs

inputs:
  cacheTTL: 3600  # 1 hour for slow/rate-limited APIs

2. Set Appropriate Timeouts

timeout: 5000   # 5s for fast APIs
timeout: 30000  # 30s for slow APIs
timeout: 60000  # 60s for webhooks

3. Handle Rate Limits

retry:
  maxAttempts: 5
  backoff: exponential
  initialDelay: 1000
  maxDelay: 30000
  retryOn: [429, 500, 502, 503]

4. Secure API Keys

headers:
  Authorization: Bearer ${env.API_KEY}  # Use env vars

5. Validate Responses

agents:
  - name: fetch
    agent: fetcher

  - name: validate
    agent: validator
    inputs:
      data: ${fetch.output.body}
      schema: ${component.response-schema@v1.0.0}

Common Use Cases

External API Integration

agents:
  - name: fetch-weather
    agent: fetcher
    inputs:
      url: https://api.weather.com/v1/current
      headers:
        X-API-Key: ${env.WEATHER_API_KEY}
      cacheTTL: 1800  # 30 minutes

Webhook Proxy

ensemble: webhook-proxy

agents:
  - name: validate
    agent: validator
    inputs:
      data: ${input}
      schema: ${component.webhook-schema@v1.0.0}

  - name: forward
    condition: ${validate.output.valid}
    agent: fetcher
    inputs:
      url: ${env.WEBHOOK_TARGET}
      method: POST
      body: ${validate.output.data}
      timeout: 60000

API Aggregator

ensemble: aggregate-data

agents:
  # Fetch from multiple sources
  - name: fetch-source-1
    agent: fetcher
    inputs:
      url: https://api1.example.com/data

  - name: fetch-source-2
    agent: fetcher
    inputs:
      url: https://api2.example.com/data

  - name: fetch-source-3
    agent: fetcher
    inputs:
      url: https://api3.example.com/data

  # Merge results
  - name: merge
    operation: code
    config:
      code: |
        return {
          combined: {
            source1: ${fetch-source-1.output.body},
            source2: ${fetch-source-2.output.body},
            source3: ${fetch-source-3.output.body}
          }
        };

GraphQL Query

agents:
  - name: query
    agent: fetcher
    inputs:
      url: https://api.example.com/graphql
      method: POST
      headers:
        Content-Type: application/json
      body:
        query: |
          query GetUser($id: ID!) {
            user(id: $id) {
              name
              email
              posts {
                title
                content
              }
            }
          }
        variables:
          id: ${input.user_id}

vs http Operation

Use fetcher agent for:

Automatic caching
Built-in retries
Common patterns

Use http operation for:

Custom retry logic
Streaming responses
Low-level control

Example comparison:

# fetcher (higher level)
agents:
  - name: fetch
    agent: fetcher
    inputs:
      url: ${url}
      cacheTTL: 3600

# http (lower level)
agents:
  - name: fetch
    operation: http
    config:
      url: ${url}
    cache:
      ttl: 3600
      key: fetch-${hash(url)}
    retry:
      maxAttempts: 3

Next Steps

http

HTTP operation reference

scraper

Web scraping

API Orchestration

Complete API playbook

Agents Overview

All pre-built agents

Conductor

Getting Started

Core Concepts

Building

Operations Reference

Pre-built Agents Reference

Playbooks

Reference

fetcher Agent

fetcher Agent

Basic Usage

Inputs

Configuration

GET Request

POST Request

With Caching

With Retries

Advanced Patterns

Multiple Endpoints

Primary + Fallback

Paginated Requests

Authenticated Requests

Conditional Requests

Output Schema

Error Handling

Best Practices

Common Use Cases

External API Integration

Webhook Proxy

API Aggregator

GraphQL Query

vs http Operation

Next Steps

http

scraper

API Orchestration

Agents Overview

Conductor

Getting Started

Core Concepts

Building

Operations Reference

Pre-built Agents Reference

Playbooks

Reference

​fetcher Agent

​Basic Usage

​Inputs

​Configuration

​GET Request

​POST Request

​With Caching

​With Retries

​Advanced Patterns

​Multiple Endpoints

​Primary + Fallback

​Paginated Requests

​Authenticated Requests

​Conditional Requests

​Output Schema

​Error Handling

​Best Practices

​Common Use Cases

​External API Integration

​Webhook Proxy

​API Aggregator

​GraphQL Query

​vs http Operation

​Next Steps

http

scraper

API Orchestration

Agents Overview

fetcher Agent

Basic Usage

Inputs

Configuration

GET Request

POST Request

With Caching

With Retries

Advanced Patterns

Multiple Endpoints

Primary + Fallback

Paginated Requests

Authenticated Requests

Conditional Requests

Output Schema

Error Handling

Best Practices

Common Use Cases

External API Integration

Webhook Proxy

API Aggregator

GraphQL Query

vs http Operation

Next Steps