Custom Matchers - Ensemble

Overview

Conductor provides custom Vitest matchers for testing ensemble executions with expressive assertions. These matchers make tests more readable and provide better error messages.

import { registerMatchers } from '@ensemble-edge/conductor/testing';

// Register matchers once in test setup
registerMatchers();

// Use in tests
expect(result).toBeSuccessful();
expect(result).toHaveExecutedMember('validate-input');
expect(result).toHaveCompletedIn(500);

Installation

npm install --save-dev @ensemble-edge/conductor

Setup

// vitest.setup.ts
import { registerMatchers } from '@ensemble-edge/conductor/testing';

registerMatchers();

Or in individual test files:

import { registerMatchers } from '@ensemble-edge/conductor/testing';
import { beforeAll } from 'vitest';

beforeAll(() => {
  registerMatchers();
});

Execution Matchers

toBeSuccessful()

Assert that execution completed successfully:

expect(result: TestExecutionResult).toBeSuccessful()

Example:

const result = await conductor.executeEnsemble('user-onboarding', {
  email: 'test@example.com'
});

expect(result).toBeSuccessful();

Error Message:

Expected execution to succeed but it failed with: Invalid email format

toHaveFailed()

Assert that execution failed:

expect(result: TestExecutionResult).toHaveFailed()

Example:

conductor.mockAI('validator', new Error('Validation failed'));

const result = await conductor.executeEnsemble('validate-content', {
  content: 'test'
});

expect(result).toHaveFailed();

Member Execution Matchers

toHaveExecutedMember()

Assert that a specific member was executed:

expect(result: TestExecutionResult).toHaveExecutedMember(memberName: string)

memberName

string

required

Name of the member to check

Example:

const result = await conductor.executeEnsemble('order-processing', {
  orderId: 'ORD-123'
});

expect(result).toHaveExecutedMember('validate-order');
expect(result).toHaveExecutedMember('process-payment');
expect(result).not.toHaveExecutedMember('send-refund-email');

Error Message:

Expected validate-order to be executed. Executed members: fetch-order, calculate-total

toHaveExecutedSteps()

Assert a specific number of steps were executed:

expect(result: TestExecutionResult).toHaveExecutedSteps(count: number)

count

number

required

Expected number of executed steps

Example:

const result = await conductor.executeEnsemble('simple-workflow', {});

expect(result).toHaveExecutedSteps(3);

Error Message:

Expected 3 steps to be executed, but 5 were executed

Performance Matchers

toHaveCompletedIn()

Assert execution completed within a time limit:

expect(result: TestExecutionResult).toHaveCompletedIn(ms: number)

number

required

Maximum execution time in milliseconds

Example:

const result = await conductor.executeEnsemble('quick-check', {
  data: 'test'
});

expect(result).toHaveCompletedIn(500); // 500ms
expect(result).toHaveCompletedIn(1000); // 1 second

Error Message:

Expected execution to complete in 500ms but it took 723ms

State Matchers

toHaveState()

Assert that execution state contains a key with optional value check:

expect(result: TestExecutionResult).toHaveState(key: string, value?: unknown)

key

string

required

State key to check

value

unknown

Expected value (optional)

Example:

const result = await conductor.executeEnsemble('stateful-workflow', {
  userId: '123'
});

// Check key exists
expect(result).toHaveState('currentStep');

// Check key and value
expect(result).toHaveState('userId', '123');
expect(result).toHaveState('status', 'completed');

Error Message:

Expected state to have key 'userId'
Expected state['status'] to be "completed" but got "pending"

AI Matchers

toHaveCalledAI()

Assert that AI was called, optionally for a specific member:

expect(result: TestExecutionResult).toHaveCalledAI(memberName?: string)

memberName

string

Name of the AI member (optional)

Example:

const result = await conductor.executeEnsemble('content-generation', {
  topic: 'testing'
});

// Check any AI call
expect(result).toHaveCalledAI();

// Check specific member
expect(result).toHaveCalledAI('generate-title');
expect(result).toHaveCalledAI('generate-summary');

Error Message:

Expected AI to be called for member 'generate-title' but it wasn't
Expected AI to be called but it wasn't

toHaveUsedTokens()

Assert that a minimum number of tokens were used:

expect(result: TestExecutionResult).toHaveUsedTokens(count: number)

count

number

required

Minimum number of tokens

Example:

const result = await conductor.executeEnsemble('content-analysis', {
  text: 'Long document to analyze...'
});

expect(result).toHaveUsedTokens(100);
expect(result).not.toHaveUsedTokens(10000); // Less than 10k

Error Message:

Expected to use at least 100 tokens but only used 45

toHaveCostLessThan()

Assert that execution cost is below a threshold:

expect(result: TestExecutionResult).toHaveCostLessThan(dollars: number)

dollars

number

required

Maximum cost in dollars

Example:

const result = await conductor.executeEnsemble('ai-workflow', {
  input: 'test'
});

expect(result).toHaveCostLessThan(0.01); // Less than 1 cent
expect(result).toHaveCostLessThan(0.001); // Less than 0.1 cent

Error Message:

Expected cost to be less than $0.01 but was $0.0234

Output Matchers

toHaveOutput()

Assert that output exactly matches expected value:

expect(result: TestExecutionResult).toHaveOutput(expected: unknown)

expected

unknown

required

Expected output value

Example:

const result = await conductor.executeEnsemble('calculator', {
  a: 2,
  b: 3,
  operation: 'add'
});

expect(result).toHaveOutput({ result: 5 });

Error Message:

Expected output to match:
{
  "result": 5
}

Received:
{
  "result": 8
}

toMatchOutputShape()

Assert that output has specific keys with correct types (partial match):

expect(result: TestExecutionResult).toMatchOutputShape(shape: Record<string, unknown>)

shape

object

required

Expected shape with type examples

Example:

const result = await conductor.executeEnsemble('user-creation', {
  email: 'test@example.com'
});

expect(result).toMatchOutputShape({
  userId: 'string',
  email: 'string',
  createdAt: 0,
  verified: false
});

Error Message:

Output is missing keys: userId, createdAt
Output has type mismatches: verified (expected boolean, got string)

Testing Patterns

Success Path

it('completes user onboarding', async () => {
  const result = await conductor.executeEnsemble('user-onboarding', {
    email: 'newuser@example.com',
    name: 'New User'
  });

  expect(result).toBeSuccessful();
  expect(result).toHaveExecutedSteps(4);
  expect(result).toHaveExecutedMember('send-welcome-email');
  expect(result).toMatchOutputShape({
    userId: 'string',
    email: 'string',
    status: 'string'
  });
});

Error Path

it('handles invalid input', async () => {
  const result = await conductor.executeEnsemble('validate-data', {
    email: 'invalid-email'
  });

  expect(result).toHaveFailed();
  expect(result).not.toHaveExecutedMember('save-to-database');
});

Performance Testing

it('executes quickly', async () => {
  const result = await conductor.executeEnsemble('health-check', {});

  expect(result).toBeSuccessful();
  expect(result).toHaveCompletedIn(100); // Must complete in 100ms
});

AI Cost Testing

it('stays within cost budget', async () => {
  const result = await conductor.executeEnsemble('content-generation', {
    topic: 'AI testing',
    length: 'short'
  });

  expect(result).toBeSuccessful();
  expect(result).toHaveCalledAI('generate-content');
  expect(result).toHaveCostLessThan(0.05); // Less than 5 cents
  expect(result).toHaveUsedTokens(50); // At least 50 tokens used
});

State Testing

it('maintains correct state', async () => {
  const result = await conductor.executeEnsemble('multi-step', {
    initialValue: 10
  });

  expect(result).toBeSuccessful();
  expect(result).toHaveState('currentValue', 20);
  expect(result).toHaveState('stepCount', 5);
  expect(result).toHaveState('status', 'completed');
});

Conditional Execution

it('skips optional steps when not needed', async () => {
  const result = await conductor.executeEnsemble('conditional-workflow', {
    premium: false
  });

  expect(result).toBeSuccessful();
  expect(result).toHaveExecutedMember('basic-processing');
  expect(result).not.toHaveExecutedMember('premium-features');
  expect(result).toHaveExecutedSteps(3); // Only 3 steps, not all 5
});

Combining Matchers

it('validates complete execution', async () => {
  const result = await conductor.executeEnsemble('complex-workflow', {
    userId: 'user_123',
    action: 'process'
  });

  // Execution success
  expect(result).toBeSuccessful();
  expect(result).toHaveCompletedIn(2000);

  // Member execution
  expect(result).toHaveExecutedSteps(7);
  expect(result).toHaveExecutedMember('fetch-user');
  expect(result).toHaveExecutedMember('validate-permissions');
  expect(result).toHaveExecutedMember('process-action');

  // AI usage
  expect(result).toHaveCalledAI('validate-content');
  expect(result).toHaveCostLessThan(0.02);

  // Output validation
  expect(result).toMatchOutputShape({
    success: true,
    processedAt: 0,
    result: {}
  });

  // State validation
  expect(result).toHaveState('userId', 'user_123');
  expect(result).toHaveState('status', 'completed');
});

Best Practices

Use specific matchers - More readable than generic assertions
Test both success and failure - Use toBeSuccessful() and toHaveFailed()
Check member execution - Verify workflow path with toHaveExecutedMember()
Monitor performance - Use toHaveCompletedIn() to catch slowdowns
Track AI costs - Use toHaveCostLessThan() to prevent budget overruns
Validate output shape - Use toMatchOutputShape() for flexible assertions
Check state - Verify state transitions with toHaveState()
Combine matchers - Build comprehensive test assertions
Use negative assertions - Test what shouldn’t happen with .not
Write descriptive tests - Matchers provide clear error messages

TypeScript Support

Add type declarations for custom matchers:

// vitest.d.ts
import type { CustomMatchers } from '@ensemble-edge/conductor/testing';

declare module 'vitest' {
  interface Assertion<T = any> extends CustomMatchers<T> {}
  interface AsymmetricMatchersContaining extends CustomMatchers {}
}

TestConductor

Testing helper class

Mocks

Mock utilities

Testing Guide

Complete testing guide

CI/CD

CI/CD setup

Conductor API

Core Classes

Member Types API

Built-In Members API

SDK API

Testing API

Durable Objects API

AI Providers API

HTTP API

​Overview

​Installation

​Setup

​Execution Matchers

​toBeSuccessful()

​toHaveFailed()

​Member Execution Matchers

​toHaveExecutedMember()

​toHaveExecutedSteps()

​Performance Matchers

​toHaveCompletedIn()

​State Matchers

​toHaveState()

​AI Matchers

​toHaveCalledAI()

​toHaveUsedTokens()

​toHaveCostLessThan()

​Output Matchers

​toHaveOutput()

​toMatchOutputShape()

​Testing Patterns

​Success Path

​Error Path

​Performance Testing

​AI Cost Testing

​State Testing

​Conditional Execution

​Combining Matchers

​Best Practices

​TypeScript Support

​Related Documentation

TestConductor

Mocks

Testing Guide

CI/CD

Overview

Installation

Setup

Execution Matchers

toBeSuccessful()

toHaveFailed()

Member Execution Matchers

toHaveExecutedMember()

toHaveExecutedSteps()

Performance Matchers

toHaveCompletedIn()

State Matchers

toHaveState()

AI Matchers

toHaveCalledAI()

toHaveUsedTokens()

toHaveCostLessThan()

Output Matchers

toHaveOutput()

toMatchOutputShape()

Testing Patterns

Success Path

Error Path

Performance Testing

AI Cost Testing

State Testing

Conditional Execution

Combining Matchers

Best Practices

TypeScript Support

Related Documentation