Fetch Member - Ensemble

Overview

The Fetch member provides production-ready HTTP client functionality with automatic retries, timeout handling, header interpolation, and response parsing. Built for reliability and edge performance. Perfect for API integration, external data fetching, webhook calls, and microservices communication.

Quick Example

name: fetch-api-data
description: Fetch data from external API

flow:
  - member: fetch-data
    type: Fetch
    config:
      url: "https://api.example.com/data"
      method: GET
      headers:
        Authorization: "Bearer ${env.API_KEY}"
        Content-Type: "application/json"
      timeout: 30000
      retries: 3

output:
  data: ${fetch-data.output.data}
  status: ${fetch-data.output.status}

Configuration

Input Parameters

config:
  url: string              # Required: Target URL
  method: string           # Optional: GET, POST, PUT, DELETE, PATCH (default: GET)
  headers: object          # Optional: HTTP headers
  timeout: number          # Optional: Timeout in ms (default: 30000)
  retries: number          # Optional: Retry attempts (default: 0)

input:
  body: any               # Optional: Request body (auto-serialized)
  params: object          # Optional: Query parameters

Output Format

output:
  data: any               # Response body (parsed if JSON)
  status: number          # HTTP status code
  headers: object         # Response headers
  ok: boolean             # True if status 200-299

HTTP Methods

GET Request

- member: get-user
  type: Fetch
  config:
    url: "https://api.example.com/users/${input.userId}"
    method: GET
    headers:
      Authorization: "Bearer ${env.API_KEY}"

POST Request

- member: create-user
  type: Fetch
  config:
    url: "https://api.example.com/users"
    method: POST
    headers:
      Authorization: "Bearer ${env.API_KEY}"
      Content-Type: "application/json"
  input:
    body:
      name: "Alice"
      email: "alice@example.com"

PUT Request

- member: update-user
  type: Fetch
  config:
    url: "https://api.example.com/users/${input.userId}"
    method: PUT
  input:
    body:
      name: ${input.name}
      email: ${input.email}

DELETE Request

- member: delete-user
  type: Fetch
  config:
    url: "https://api.example.com/users/${input.userId}"
    method: DELETE

Common Patterns

API Client Pattern

name: api-client
description: Reusable API client with authentication

flow:
  - member: fetch-api
    type: Fetch
    config:
      url: "${env.API_BASE_URL}${input.endpoint}"
      method: ${input.method || 'GET'}
      headers:
        Authorization: "Bearer ${env.API_KEY}"
        Content-Type: "application/json"
        X-Request-ID: ${execution.id}
      timeout: 30000
      retries: 3
    input:
      body: ${input.body}

output:
  data: ${fetch-api.output.data}
  success: ${fetch-api.output.ok}

Paginated API Fetching

name: fetch-all-pages
description: Fetch all pages from paginated API

state:
  schema:
    allResults: array
    page: number
    hasMore: boolean

flow:
  - member: initialize
    type: Function
    state:
      set: [allResults, page, hasMore]
    input:
      allResults: []
      page: 1
      hasMore: true

  - member: fetch-page
    condition: ${state.hasMore}
    type: Fetch
    config:
      url: "https://api.example.com/items"
      method: GET
    input:
      params:
        page: ${state.page}
        limit: 100

  - member: accumulate
    condition: ${fetch-page.success}
    type: Function
    state:
      use: [allResults, page]
      set: [allResults, page, hasMore]
    input:
      currentResults: ${state.allResults}
      newResults: ${fetch-page.output.data.items}
      hasMore: ${fetch-page.output.data.hasMore}

output:
  results: ${state.allResults}
  totalPages: ${state.page}

Parallel API Calls

name: parallel-fetch
description: Fetch multiple APIs concurrently

flow:
  parallel:
    - member: fetch-users
      type: Fetch
      config:
        url: "https://api.example.com/users"

    - member: fetch-orders
      type: Fetch
      config:
        url: "https://api.example.com/orders"

    - member: fetch-products
      type: Fetch
      config:
        url: "https://api.example.com/products"

  - member: combine
    type: Transform
    input:
      data:
        users: ${fetch-users.output.data}
        orders: ${fetch-orders.output.data}
        products: ${fetch-products.output.data}

output:
  combined: ${combine.output}

Retry with Fallback

name: resilient-fetch
description: Primary API with fallback

flow:
  - member: fetch-primary
    type: Fetch
    config:
      url: "https://primary-api.example.com/data"
      retries: 3
    continue_on_error: true

  - member: fetch-fallback
    condition: ${!fetch-primary.success}
    type: Fetch
    config:
      url: "https://fallback-api.example.com/data"
      retries: 2

output:
  data: ${fetch-primary.success ? fetch-primary.output.data : fetch-fallback.output.data}
  source: ${fetch-primary.success ? 'primary' : 'fallback'}

Rate-Limited API

name: rate-limited-fetch
description: Respect API rate limits

flow:
  - member: check-rate-limit
    type: Data
    config:
      storage: kv
      operation: get
      binding: CACHE
    input:
      key: "rate-limit:api"

  - member: wait-if-needed
    condition: ${(check-rate-limit.output.value || 0) >= 100}
    type: Schedule
    config:
      delay: 60000  # Wait 1 minute

  - member: fetch-data
    type: Fetch
    config:
      url: "https://api.example.com/data"

  - member: increment-counter
    condition: ${fetch-data.success}
    type: Data
    config:
      storage: kv
      operation: put
      binding: CACHE
    input:
      key: "rate-limit:api"
      value: ${(check-rate-limit.output.value || 0) + 1}
      expirationTtl: 60  # Reset after 1 minute

output:
  data: ${fetch-data.output.data}

GraphQL Query

name: graphql-query
description: Query GraphQL API

flow:
  - member: graphql
    type: Fetch
    config:
      url: "https://api.example.com/graphql"
      method: POST
      headers:
        Authorization: "Bearer ${env.API_KEY}"
        Content-Type: "application/json"
    input:
      body:
        query: |
          query GetUser($id: ID!) {
            user(id: $id) {
              id
              name
              email
              posts {
                id
                title
              }
            }
          }
        variables:
          id: ${input.userId}

output:
  user: ${graphql.output.data.data.user}

Webhook Posting

name: post-webhook
description: Send webhook notification

flow:
  - member: send-webhook
    type: Fetch
    config:
      url: "${env.WEBHOOK_URL}"
      method: POST
      headers:
        Content-Type: "application/json"
        X-Webhook-Signature: "${env.WEBHOOK_SECRET}"
      retries: 3
    input:
      body:
        event: "workflow.completed"
        timestamp: ${Date.now()}
        data: ${input.data}

output:
  delivered: ${send-webhook.output.ok}

Authentication Patterns

Bearer Token

config:
  headers:
    Authorization: "Bearer ${env.API_KEY}"

Basic Auth

config:
  headers:
    Authorization: "Basic ${btoa(env.USERNAME + ':' + env.PASSWORD)}"

API Key in Header

config:
  headers:
    X-API-Key: "${env.API_KEY}"

API Key in Query

config:
  url: "https://api.example.com/data?api_key=${env.API_KEY}"

Error Handling

Automatic Retry

- member: fetch-with-retry
  type: Fetch
  config:
    url: "https://api.example.com/data"
    retries: 3  # Retry up to 3 times
    timeout: 10000

Conductor uses exponential backoff:

Attempt 1: immediate
Attempt 2: 1s delay
Attempt 3: 2s delay
Attempt 4: 4s delay

Handle Status Codes

flow:
  - member: fetch-data
    type: Fetch

  - member: handle-success
    condition: ${fetch-data.output.ok}

  - member: handle-client-error
    condition: ${fetch-data.output.status >= 400 && fetch-data.output.status < 500}

  - member: handle-server-error
    condition: ${fetch-data.output.status >= 500}

Timeout Handling

- member: fetch-with-timeout
  type: Fetch
  config:
    timeout: 5000  # 5 seconds
  continue_on_error: true

- member: handle-timeout
  condition: ${!fetch-with-timeout.success}
  type: Function
  input:
    error: "Request timed out"

Caching

Cache Responses

- member: fetch-data
  type: Fetch
  cache:
    ttl: 3600  # Cache for 1 hour
  config:
    url: "https://api.example.com/data"

Conditional Caching

- member: fetch-data
  type: Fetch
  cache:
    ttl: ${input.cacheable ? 3600 : 0}
  config:
    url: ${input.url}

Cache by Status

- member: fetch-data
  type: Fetch
  cache:
    ttl: ${fetch-data.output.ok ? 3600 : 0}  # Only cache successful responses
  config:
    url: "https://api.example.com/data"

Performance Optimization

Parallel Requests

parallel:
  - member: fetch-1
    type: Fetch
  - member: fetch-2
    type: Fetch
  - member: fetch-3
    type: Fetch

Set Appropriate Timeouts

# Fast API
- member: fetch-fast
  type: Fetch
  config:
    timeout: 2000  # 2 seconds

# Slow API
- member: fetch-slow
  type: Fetch
  config:
    timeout: 30000  # 30 seconds

Connection Reuse

Fetch member automatically reuses connections when possible.

Testing

import { describe, it, expect } from 'vitest';
import { TestConductor } from '@ensemble-edge/conductor/testing';

describe('fetch member', () => {
  it('should fetch API data', async () => {
    const conductor = await TestConductor.create({
      mocks: {
        http: {
          responses: {
            'https://api.example.com/users': {
              status: 200,
              data: { id: 1, name: 'Alice' }
            }
          }
        }
      }
    });

    const result = await conductor.executeMember('fetch-data', {
      url: 'https://api.example.com/users'
    });

    expect(result).toBeSuccessful();
    expect(result.output.status).toBe(200);
    expect(result.output.data.name).toBe('Alice');
  });

  it('should retry on failure', async () => {
    let attempts = 0;

    const conductor = await TestConductor.create({
      mocks: {
        http: {
          handler: async () => {
            attempts++;
            if (attempts < 3) {
              throw new Error('Network error');
            }
            return { status: 200, data: { success: true } };
          }
        }
      }
    });

    const result = await conductor.executeMember('fetch-with-retry', {
      retries: 3
    });

    expect(result).toBeSuccessful();
    expect(attempts).toBe(3);
  });

  it('should handle timeout', async () => {
    const conductor = await TestConductor.create({
      mocks: {
        http: {
          handler: async () => {
            await new Promise(resolve => setTimeout(resolve, 10000));
            return { status: 200, data: {} };
          }
        }
      }
    });

    const result = await conductor.executeMember('fetch-data', {
      timeout: 1000
    });

    expect(result).toHaveError(/timeout/i);
  });
});

Best Practices

Set timeouts - Prevent hanging requests
Use retries - Handle transient failures
Secure credentials - Use environment variables
Cache responses - Reduce API calls
Handle errors - Check status codes
Use HTTPS - Never use HTTP for sensitive data
Rate limit - Respect API provider limits
Parallel requests - Fetch concurrently when possible
Monitor failures - Track error rates
Test thoroughly - Mock HTTP responses

Built-In Members

All built-in members

API Member Type

Custom API members

Error Handling

Handle API failures

Webhooks Guide

Webhook patterns

Conductor

Core Concepts

Guides

Member Types

Built-In Members

Examples

API Reference

Deployment

​Overview

​Quick Example

​Configuration

​Input Parameters

​Output Format

​HTTP Methods

​GET Request

​POST Request

​PUT Request

​DELETE Request

​Common Patterns

​API Client Pattern

​Paginated API Fetching

​Parallel API Calls

​Retry with Fallback

​Rate-Limited API

​GraphQL Query

​Webhook Posting

​Authentication Patterns

​Bearer Token

​Basic Auth

​API Key in Header

​API Key in Query

​Error Handling

​Automatic Retry

​Handle Status Codes

​Timeout Handling

​Caching

​Cache Responses

​Conditional Caching

​Cache by Status

​Performance Optimization

​Parallel Requests

​Set Appropriate Timeouts

​Connection Reuse

​Testing

​Best Practices

​Related Documentation

Built-In Members

API Member Type

Error Handling

Webhooks Guide

Overview

Quick Example

Configuration

Input Parameters

Output Format

HTTP Methods

GET Request

POST Request

PUT Request

DELETE Request

Common Patterns

API Client Pattern

Paginated API Fetching

Parallel API Calls

Retry with Fallback

Rate-Limited API

GraphQL Query

Webhook Posting

Authentication Patterns

Bearer Token

Basic Auth

API Key in Header

API Key in Query

Error Handling

Automatic Retry

Handle Status Codes

Timeout Handling

Caching

Cache Responses

Conditional Caching

Cache by Status

Performance Optimization

Parallel Requests

Set Appropriate Timeouts

Connection Reuse

Testing

Best Practices

Related Documentation