• Inside-the-Loop Control. Influence decisions at every agent step — model call, tool call, sub-agent handoff — where most cost, delay, and failure actually happen. External proxies only see request boundaries; cascadeflow sees decision boundaries.
• Multi-Dimensional Optimization. Optimize across cost, latency, quality, budget, compliance/risk, and energy simultaneously — relevant to engineering, finance, security, operations, and sustainability stakeholders.
• Business Logic Injection. Embed KPI weights and policy intent directly into agent behavior at runtime. Shift AI control from static prompt design to live business governance.
• Runtime Enforcement. Directly steer outcomes with four actions: allow, switch_model, deny_tool, stop — based on current context and policy state. Closes the gap between analytics and execution.
• Auditability & Transparency. Every runtime decision is traceable and attributable. Supports audit requirements, faster tuning cycles, and trust in regulated or high-stakes workflows.
• Measurable Value. Prove impact with reproducible metrics on realistic agent workflows — better economics and latency while preserving quality thresholds.
• Latency Advantage. Proxy-based optimization adds 40-60ms per call. In a 10-step agent loop, that is 400-600ms of avoidable overhead. cascadeflow runs in-process with sub-5ms overhead — critical for real-time UX, task throughput, and enterprise SLAs.
• Framework & Provider Neutral. Works with LangChain, OpenAI Agents SDK, CrewAI, PydanticAI, Google ADK, Vercel AI SDK, n8n, Hermes Agent, and custom frameworks. Unified API across OpenAI, Anthropic, Groq, Ollama, vLLM, Together, and more.
• Self-Improving Agent Intelligence. Because cascadeflow runs inside the agent loop, it accumulates deep insight into every model call, tool result, quality score, and routing decision over time. This enables cascadeflow to learn which models perform best for which tasks, adapt routing strategies, and continuously improve cost-quality tradeoffs — without manual tuning. The agent gets smarter the more it runs.
• Edge & Local-Hosted AI. Handle most queries with local models (vLLM, Ollama), automatically escalate complex queries to cloud providers only when needed.

ℹ️ Note: SLMs (under 10B parameters) are sufficiently powerful for 60-70% of agentic AI tasks. Research paper

For advanced use cases, you can add ML-based semantic similarity checking to validate that responses align with queries.

Step 1: Install the optional ML package:

pip install cascadeflow[semantic]  # Adds semantic similarity via FastEmbed (~80MB model)

Step 2: Use semantic quality validation:

from cascadeflow.quality.semantic import SemanticQualityChecker

# Initialize semantic checker (downloads model on first use)
checker = SemanticQualityChecker(
    similarity_threshold=0.5,  # Minimum similarity score (0-1)
    toxicity_threshold=0.7     # Maximum toxicity score (0-1)
)

# Validate query-response alignment
query = "Explain Python decorators"
response = "Decorators are a way to modify functions using @syntax..."

result = checker.validate(query, response, check_toxicity=True)

print(f"Similarity: {result.similarity:.2%}")
print(f"Passed: {result.passed}")
print(f"Toxic: {result.is_toxic}")

What you get:

• 🎯 Semantic similarity scoring (query ↔ response alignment)
• 🛡️ Optional toxicity detection
• 🔄 Automatic model download and caching
• 🚀 Fast inference (~100ms per check)

Full example: See semantic_quality_domain_detection.py

For advanced quality validation, enable ML-based semantic similarity checking to ensure responses align with queries.

Step 1: Install the optional ML packages:

npm install @cascadeflow/ml @huggingface/transformers

Step 2: Enable semantic validation in your cascade:

import { CascadeAgent, SemanticQualityChecker } from '@cascadeflow/core';

const agent = new CascadeAgent({
  models: [
    { name: 'nous/hermes-flash', provider: 'openai', cost: 0.000375 },
    { name: 'gpt-4o', provider: 'openai', cost: 0.00625 },
  ],
  quality: {
    threshold: 0.40,                    // Traditional confidence threshold
    requireMinimumTokens: 5,            // Minimum response length
    useSemanticValidation: true,        // Enable ML validation
    semanticThreshold: 0.5,             // 50% minimum similarity
  },
});

// Responses now validated for semantic alignment
const result = await agent.run('Explain TypeScript generics');

Step 3: Or use semantic validation directly:

import { SemanticQualityChecker } from '@cascadeflow/core';

const checker = new SemanticQualityChecker();

if (await checker.isAvailable()) {
  const result = await checker.checkSimilarity(
    'What is TypeScript?',
    'TypeScript is a typed superset of JavaScript.'
  );

  console.log(`Similarity: ${(result.similarity * 100).toFixed(1)}%`);
  console.log(`Passed: ${result.passed}`);
}

What you get:

• 🎯 Query-response semantic alignment detection
• 🚫 Off-topic response filtering
• 📦 BGE-small-en-v1.5 embeddings (~40MB, auto-downloads)
• ⚡ Fast CPU inference (~50-100ms with caching)
• 🔄 Request-scoped caching (50% latency reduction)
• 🌐 Works in Node.js, Browser, and Edge Functions

Example: semantic-quality.ts

If you already have an app using the OpenAI or Anthropic APIs and want the fastest integration, run the gateway and point your existing client at it:

python -m cascadeflow.server --mode auto --port 8084

n8n Integration

Use cascadeflow in n8n workflows for no-code AI automation with automatic cost optimization!

Installation

1. Open n8n
2. Go to Settings → Community Nodes
3. Search for: @cascadeflow/n8n-nodes-cascadeflow
4. Click Install

Two Nodes

Quick Start (Model):

1. Add two AI Chat Model nodes (cheap drafter + powerful verifier)
2. Add CascadeFlow (Model) and connect both models
3. Connect to Basic LLM Chain or Chain node
4. Check Logs tab on the Chain node to see cascade decisions

Quick Start (Agent):

1. Add a Chat Trigger node
2. Add CascadeFlow Agent and connect it to the trigger
3. Connect Drafter, Verifier, optional Memory and Tools
4. Check the Agent Output tab for cascade metadata and trace

Result: 40-85% cost savings in your n8n workflows!

Features:

• Works with any AI Chat Model node (OpenAI, Anthropic, Ollama, Azure, etc.)
• Mix providers (e.g., Ollama drafter + GPT-4o verifier)
• Agent node: tool calling, memory, per-tool routing, tool call validation
• 16-domain cascading for specialized model routing
• Real-time flow visualization in Logs/Output tabs

🔌 Learn more: n8n Integration Guide | n8n Package

Use CascadeFlow as an optional Hermes Agent delegation router for subagents. Hermes keeps provider credentials, base URLs, fallback chains, and API modes; CascadeFlow returns a structured routing decision before Hermes spawns a child agent.

This works as a released CascadeFlow module even before a native Hermes PR is accepted. Users can call the router from a local wrapper, local Hermes fork, or small hook script and keep Hermes' current provider configuration as the final source of truth.

from cascadeflow.integrations.hermes import (
    HermesDelegationRequest,
    HermesDelegationRouter,
)

router = HermesDelegationRouter.from_dict({
    "enabled": True,
    "mode": "observe",
    "routes": {
        "code": {
            "provider": "nous",
            "model": "nous/hermes-4.1",
            "reasoning_effort": "high",
        },
        "simple": {
            "provider": "openai",
            "model": "gpt-4.1-mini",
            "reasoning_effort": "low",
        },
    },
})

decision = router.route_delegation(HermesDelegationRequest(
    goal="Debug the failing unit test and propose a patch",
    toolsets=("terminal", "git"),
    loaded_skills=("python", "debugging"),
))
print(decision.to_dict())

What Hermes gets:

• Per-skill model routing: coding, research, legal/finance, and lightweight utility skills can receive different model and reasoning profiles instead of inheriting one global default.
• Task-complexity routing: simple delegated tasks can use cheaper/faster models, while hard debugging, architecture, research, or code-generation tasks route to stronger models.
• Topic-aware subagent routing: subagents can route differently for code, research, data, creative, ops, medical, legal, finance, and other domains.
• Better subagent economics: Hermes avoids paying flagship-model prices for simple worker tasks.
• Better quality for hard tasks: difficult subagent work no longer has to inherit a weak or cheap default model.
• Dry-run/observe mode: Hermes users can see what CascadeFlow would route without changing runtime behavior.
• Auditability: routing decisions include reason, confidence, domain, complexity, and selected model.
• Safer rollout: missing CascadeFlow, disabled config, low confidence, high-stakes gaps, or bad routing inputs fall back to Hermes' current behavior.
• No credential rewrite: Hermes still owns provider credentials, base URLs, fallback chains, and API modes.

Learn more: Hermes Agent Integration Guide

Standalone example: examples/integrations/hermes_delegation_router.py

LangChain Integration

Use cascadeflow with LangChain for intelligent model cascading with full LCEL, streaming, and tools support!

Installation

TypeScript

npm install @cascadeflow/langchain @langchain/core @langchain/openai

Python

pip install cascadeflow langchain-openai

Quick Start

TypeScript - Drop-in replacement for any LangChain chat model

hljs language-typescript

Copy

import { ChatOpenAI } from '@langchain/openai';
import { ChatAnthropic } from '@langchain/anthropic';
import { withCascade } from '@cascadeflow/langchain';

const cascade = withCascade({
  drafter: new ChatOpenAI({ model: 'nous/hermes-flash' }),      // $0.15/$0.60 per 1M tokens
  verifier: new ChatAnthropic({ model: 'claude-sonnet-4-5' }),  // $3/$15 per 1M tokens
  qualityThreshold: 0.8, // 80% queries use drafter
});

// Use like any LangChain chat model
const result = await cascade.invoke('Explain quantum computing');

// Optional: Enable LangSmith tracing (see https://smith.langchain.com)
// Set LANGSMITH_API_KEY, LANGSMITH_PROJECT, LANGSMITH_TRACING=true

// Or with LCEL chains
const chain = prompt.pipe(cascade).pipe(new StringOutputParser());

Python - Drop-in replacement for any LangChain chat model

hljs language-python

Copy

from langchain_openai import ChatOpenAI
from langchain_anthropic import ChatAnthropic
from cascadeflow.integrations.langchain import CascadeFlow

cascade = CascadeFlow(
    drafter=ChatOpenAI(model="nous/hermes-flash"),      # $0.15/$0.60 per 1M tokens
    verifier=ChatAnthropic(model="claude-sonnet-4-5"),  # $3/$15 per 1M tokens
    quality_threshold=0.8,  # 80% queries use drafter
)

# Use like any LangChain chat model
result = await cascade.ainvoke("Explain quantum computing")

# Optional: Enable LangSmith tracing (see https://smith.langchain.com)
# Set LANGSMITH_API_KEY, LANGSMITH_PROJECT, LANGSMITH_TRACING=true

# Or with LCEL chains
chain = prompt | cascade | StrOutputParser()

from cascadeflow.integrations.langchain.langchain_callbacks import get_cascade_callback

# Track costs similar to get_openai_callback()
with get_cascade_callback() as cb:
    response = await cascade.ainvoke("What is Python?")

    print(f"Total cost: ${cb.total_cost:.6f}")
    print(f"Drafter cost: ${cb.drafter_cost:.6f}")
    print(f"Verifier cost: ${cb.verifier_cost:.6f}")
    print(f"Total tokens: {cb.total_tokens}")
    print(f"Successful requests: {cb.successful_requests}")

import {
  discoverCascadePairs,
  findBestCascadePair,
  analyzeModel,
  validateCascadePair
} from '@cascadeflow/langchain';

// Your existing LangChain models (configured with YOUR API keys)
const myModels = [
  new ChatOpenAI({ model: 'gpt-3.5-turbo' }),
  new ChatOpenAI({ model: 'nous/hermes-flash' }),
  new ChatOpenAI({ model: 'gpt-4o' }),
  new ChatAnthropic({ model: 'claude-3-haiku' }),
  // ... any LangChain chat models
];

// Quick: Find best cascade pair
const best = findBestCascadePair(myModels);
console.log(`Best pair: ${best.analysis.drafterModel} → ${best.analysis.verifierModel}`);
console.log(`Estimated savings: ${best.estimatedSavings}%`);

// Use it immediately
const cascade = withCascade({
  drafter: best.drafter,
  verifier: best.verifier,
});

// Advanced: Discover all valid pairs
const pairs = discoverCascadePairs(myModels, {
  minSavings: 50,              // Only pairs with ≥50% savings
  requireSameProvider: false,  // Allow cross-provider cascades
});

// Validate specific pair
const validation = validateCascadePair(drafter, verifier);
console.log(`Valid: ${validation.valid}`);
console.log(`Warnings: ${validation.warnings}`);

Features:

• ✅ Full LCEL support (pipes, sequences, batch)
• ✅ Streaming with pre-routing
• ✅ Tool calling and structured output
• ✅ LangSmith cost tracking metadata
• ✅ Cost tracking callbacks (Python)
• ✅ Works with all LangChain features

🦜 Learn more: LangChain Integration Guide | TypeScript Package | Python Examples