Validate Agent - Ensemble Edge

Starter Kit - Ships with your template. You own it - modify freely.

Basic Usage

agents:
  - name: validate
    agent: validate
    inputs:
      content: ${input.text}
    config:
      evalType: rule
      threshold: 0.7
      rules:
        - name: min_length
          check: "content.length >= 50"
          weight: 1

Output (success):

{
  "passed": true,
  "score": 1.0,
  "scores": {
    "min_length": 1.0
  },
  "evalType": "rule"
}

Output (failure):

{
  "passed": false,
  "score": 0.4,
  "scores": {
    "min_length": 0.4
  },
  "details": {
    "min_length": "Content too short"
  },
  "evalType": "rule"
}

Inputs

inputs:
  content:
    type: string
    required: true
    description: Content to validate

  reference:
    type: string
    required: false
    description: Reference text for comparison (NLP/embedding evaluators)

  expected:
    type: any
    required: false
    description: Expected value for exact comparison

Configuration

config:
  evalType:
    type: string
    enum: [rule, judge, nlp, embedding]
    default: rule
    description: Type of evaluator to use

  threshold:
    type: number
    default: 0.7
    minimum: 0
    maximum: 1
    description: Minimum score to pass

  rules:
    type: array
    description: Rules for rule evaluator (required if evalType=rule)

  criteria:
    type: array
    description: Criteria for judge evaluator (required if evalType=judge)

  metrics:
    type: array
    default: [bleu, rouge]
    description: NLP metrics to calculate (for evalType=nlp)

  model:
    type: string
    description: Model for judge/embedding evaluators

Evaluator Types

Rule-Based Validation

Validate using JavaScript expressions:

agents:
  - name: validate
    agent: validate
    inputs:
      content: ${input.text}
    config:
      evalType: rule
      threshold: 0.7
      rules:
        - name: min_length
          check: "content.length >= 50"
          weight: 1
          description: "Content must be at least 50 characters"

        - name: has_keyword
          check: "content.toLowerCase().includes('important')"
          weight: 0.5
          description: "Must mention 'important'"

        - name: no_profanity
          check: "!/(bad|word|list)/i.test(content)"
          weight: 2
          description: "No profanity allowed"

Available Variables:

content - Input content
reference - Reference text (if provided)
expected - Expected value (if provided)

Judge Evaluation (LLM)

Use AI to assess quality against criteria:

agents:
  - name: validate
    agent: validate
    inputs:
      content: ${input.essay}
    config:
      evalType: judge
      threshold: 0.8
      model: claude-3-5-sonnet-20241022
      criteria:
        - name: clarity
          description: "Is the writing clear and easy to understand?"
          weight: 1

        - name: structure
          description: "Does it have proper introduction, body, and conclusion?"
          weight: 1

        - name: grammar
          description: "Is grammar and spelling correct?"
          weight: 0.5

Requires: AI binding configured in your project

NLP Metrics

Compare content to reference using statistical metrics:

agents:
  - name: validate
    agent: validate
    inputs:
      content: ${input.translation}
      reference: ${input.source_text}
    config:
      evalType: nlp
      threshold: 0.6
      metrics: [bleu, rouge, length-ratio]

Metrics:

bleu - Precision-based similarity (0-1)
rouge - Recall-based similarity (0-1)
length-ratio - Length comparison (0-1)

Embedding Similarity

Semantic similarity via embeddings:

agents:
  - name: validate
    agent: validate
    inputs:
      content: ${input.response}
      reference: ${input.expected_response}
    config:
      evalType: embedding
      threshold: 0.85
      model: text-embedding-3-small

Requires: AI binding with embedding support

Configuration Examples

Basic Text Validation

agents:
  - name: validate-input
    agent: validate
    inputs:
      content: ${input.message}
    config:
      evalType: rule
      threshold: 0.6
      rules:
        - name: not_empty
          check: "content.trim().length > 0"
          weight: 1
        - name: max_length
          check: "content.length <= 1000"
          weight: 1

With Error Handling

agents:
  - name: validate
    agent: validate
    inputs:
      content: ${input.data}
    config:
      evalType: rule
      threshold: 0.8
      rules:
        - name: valid_format
          check: "/^[A-Z0-9-]+$/i.test(content)"
          weight: 1

  - name: handle-invalid
    condition: ${!validate.output.passed}
    operation: code
    config:
      script: scripts/handle-validation-errors
    input:
      errors: ${validate.output.details}

// scripts/handle-validation-errors.ts
import type { AgentExecutionContext } from '@ensemble-edge/conductor'

export default function handleValidationErrors(context: AgentExecutionContext) {
  const { errors } = context.input

  return {
    status: 400,
    message: 'Validation failed',
    errors
  }
}

Multi-Criteria Quality Check

agents:
  - name: validate-quality
    agent: validate
    inputs:
      content: ${scrape.output.text}
    config:
      evalType: judge
      threshold: 0.75
      model: claude-3-5-sonnet-20241022
      criteria:
        - name: relevance
          description: "Is the content relevant to the topic?"
          weight: 2

        - name: accuracy
          description: "Are the facts accurate and verifiable?"
          weight: 2

        - name: completeness
          description: "Does it cover all important aspects?"
          weight: 1

        - name: readability
          description: "Is it well-written and easy to read?"
          weight: 1

NLP Comparison

agents:
  - name: translate
    operation: think
    config:
      provider: anthropic
      model: claude-3-5-sonnet-20241022
      prompt: "Translate this to French: ${input.text}"

  - name: validate-translation
    agent: validate
    inputs:
      content: ${translate.output}
      reference: ${input.expected_translation}
    config:
      evalType: nlp
      threshold: 0.7
      metrics: [bleu, rouge]

Advanced Patterns

Multi-Step Validation

agents:
  - name: validate-structure
    agent: validate
    inputs:
      content: ${input.document}
    config:
      evalType: rule
      threshold: 1.0
      rules:
        - name: has_title
          check: "/^#\\s+.+/m.test(content)"
          weight: 1

  - name: validate-quality
    condition: ${validate-structure.output.passed}
    agent: validate
    inputs:
      content: ${input.document}
    config:
      evalType: judge
      threshold: 0.7
      criteria:
        - name: clarity
          description: "Is the content clear?"
          weight: 1

Conditional Validation

agents:
  - name: validate-basic
    agent: validate
    inputs:
      content: ${input.text}
    config:
      evalType: rule
      threshold: 0.5
      rules:
        - name: min_length
          check: "content.length >= 10"
          weight: 1

  - name: validate-premium
    condition: ${input.tier === 'premium'}
    agent: validate
    inputs:
      content: ${input.text}
    config:
      evalType: judge
      threshold: 0.8
      model: claude-3-5-sonnet-20241022
      criteria:
        - name: quality
          description: "High quality content"
          weight: 1

Validate Scraped Content

agents:
  - name: scrape
    agent: scrape
    inputs:
      url: ${input.url}
      selector: article

  - name: validate-content
    agent: validate
    inputs:
      content: ${scrape.output.text}
    config:
      evalType: rule
      threshold: 0.7
      rules:
        - name: sufficient_length
          check: "content.length >= 200"
          weight: 1
          description: "Article must have at least 200 characters"

        - name: has_paragraphs
          check: "content.split('\\n\\n').length >= 3"
          weight: 0.5
          description: "Must have multiple paragraphs"

Batch Validation

agents:
  - name: validate-batch
    operation: code
    config:
      script: scripts/validate-batch
    input:
      items: ${input.items}

// scripts/validate-batch.ts
import type { AgentExecutionContext } from '@ensemble-edge/conductor'

export default async function validateBatch(context: AgentExecutionContext) {
  const { items } = context.input

  const results = await Promise.all(
    items.map((item: any) =>
      context.conductor.executeAgent('validate', {
        content: item.text,
      }, {
        evalType: 'rule',
        threshold: 0.7,
        rules: [
          {
            name: 'valid',
            check: 'content.length > 0',
            weight: 1
          }
        ]
      })
    )
  )

  return {
    passed: results.every((r: any) => r.passed),
    results: results.map((r: any, i: number) => ({
      item: items[i],
      passed: r.passed,
      score: r.score
    }))
  }
}

Output Schema

{
  passed: boolean;          // Whether content passed threshold
  score: number;            // Average score (0-1)
  scores: {                 // Individual scores by criterion
    [name: string]: number;
  };
  details?: {               // Additional details per criterion
    [name: string]: string;
  };
  evalType: 'rule' | 'judge' | 'nlp' | 'embedding';
}

Best Practices

1. Use Appropriate Evaluator Type

# Structural checks → rule
evalType: rule
rules:
  - name: format
    check: "/^\\d{3}-\\d{3}-\\d{4}$/.test(content)"
    weight: 1

# Quality assessment → judge
evalType: judge
criteria:
  - name: quality
    description: "Is this well-written?"
    weight: 1

# Similarity comparison → nlp or embedding
evalType: nlp
metrics: [bleu, rouge]

2. Weight Important Rules Higher

rules:
  - name: critical_field
    check: "content.includes('required')"
    weight: 3  # 3x more important

  - name: optional_field
    check: "content.includes('optional')"
    weight: 1

3. Set Reasonable Thresholds

# Strict validation (must pass all rules)
threshold: 1.0

# Lenient validation (most rules pass)
threshold: 0.6

# Balanced validation
threshold: 0.7

4. Fail Fast

agents:
  - name: validate
    agent: validate

  - name: process
    condition: ${validate.output.passed}
    operation: code

5. Provide Clear Descriptions

rules:
  - name: email_format
    check: "/@/.test(content)"
    weight: 1
    description: "Must be a valid email address"  # Helps debugging

Common Validation Schemas

Email Validation

config:
  evalType: rule
  threshold: 1.0
  rules:
    - name: has_at
      check: "content.includes('@')"
      weight: 1
      description: "Must contain @ symbol"

    - name: has_domain
      check: "/\\.[a-z]{2,}$/i.test(content)"
      weight: 1
      description: "Must have valid domain extension"

    - name: format
      check: "/^[a-z0-9._%+-]+@[a-z0-9.-]+\\.[a-z]{2,}$/i.test(content)"
      weight: 1
      description: "Must match email format"

URL Validation

config:
  evalType: rule
  threshold: 1.0
  rules:
    - name: protocol
      check: "/^https?:\\/\\//i.test(content)"
      weight: 1
      description: "Must start with http:// or https://"

    - name: domain
      check: "/[a-z0-9.-]+\\.[a-z]{2,}/i.test(content)"
      weight: 1
      description: "Must have valid domain"

Content Quality

config:
  evalType: judge
  threshold: 0.75
  model: claude-3-5-sonnet-20241022
  criteria:
    - name: grammar
      description: "Correct grammar and spelling"
      weight: 1

    - name: coherence
      description: "Logical flow and structure"
      weight: 1

    - name: tone
      description: "Professional and appropriate tone"
      weight: 0.5

API Response

config:
  evalType: rule
  threshold: 1.0
  rules:
    - name: has_id
      check: "content.id !== undefined"
      weight: 1

    - name: has_data
      check: "content.data !== null"
      weight: 1

    - name: valid_status
      check: "['success', 'error'].includes(content.status)"
      weight: 1

Next Steps

Scrape Agent

Scrape then validate

Starter Kit Overview

All starter kit agents

Creating Agents

Build custom validators

YAML Schema

Full schema reference

Conductor

Getting Started

Core Concepts

Building

Components

Operations Reference

Plugins

Starter Kit

Playbooks

Reference

​Basic Usage

​Inputs

​Configuration

​Evaluator Types

​Rule-Based Validation

​Judge Evaluation (LLM)

​NLP Metrics

​Embedding Similarity

​Configuration Examples

​Basic Text Validation

​With Error Handling

​Multi-Criteria Quality Check

​NLP Comparison

​Advanced Patterns

​Multi-Step Validation

​Conditional Validation

​Validate Scraped Content

​Batch Validation

​Output Schema

​Best Practices

​Common Validation Schemas

​Email Validation

​URL Validation

​Content Quality

​API Response

​Next Steps

Scrape Agent

Starter Kit Overview

Creating Agents

YAML Schema

Basic Usage

Inputs

Configuration

Evaluator Types

Rule-Based Validation

Judge Evaluation (LLM)

NLP Metrics

Embedding Similarity

Configuration Examples

Basic Text Validation

With Error Handling

Multi-Criteria Quality Check

NLP Comparison

Advanced Patterns

Multi-Step Validation

Conditional Validation

Validate Scraped Content

Batch Validation

Output Schema

Best Practices

Common Validation Schemas

Email Validation

URL Validation

Content Quality

API Response

Next Steps