Discovery Engine

Overview
Architecture
Discovery Objectives
Obligation Model
Session Management
Completeness Scoring
Research System
Negotiation Protocol
Working Memory
Database Schema
Service Layer
API Endpoints
Frontend Integration

Overview

Status: ✅ Production (Migrations 180-187 - February 2026)

The Discovery Engine is Praetor's conversational requirements gathering system that replaces rigid question forms with adaptive, context-aware conversations. It tracks obligations (information that must be collected), manages working memory across turns, and triggers background research when needed.

Key Features

✅ Obligation-driven conversations — Track what needs to be collected with confidence scores ✅ Session phases — Opening, exploration, validation, closing ✅ Completeness scoring — Real-time 0-1 score of discovery completeness ✅ Working memory — Agent maintains conversation context across turns ✅ Research triggers — Automatically spawn research tasks for market/tech feasibility ✅ Negotiation protocol — Handle user disagreement with extracted values ✅ Audit trail — Full history of all obligation lifecycle events ✅ Re-evaluation engine — Periodically reassess obligations based on new context

Architecture

┌─────────────────────────────────────────────────────────────────────────────┐
│                         DISCOVERY ENGINE ARCHITECTURE                       │
│                                                                             │
│  ┌──────────────────────┐                                                  │
│  │ DISCOVERY OBJECTIVES │                                                  │
│  │  (Configuration)     │                                                  │
│  │                      │                                                  │
│  │ - business_profile   │                                                  │
│  │ - idea_discovery     │                                                  │
│  │ - project_discovery  │                                                  │
│  └──────────┬───────────┘                                                  │
│             │ defines                                                      │
│             ▼                                                              │
│  ┌──────────────────────┐                                                  │
│  │ ELICITATION_SESSIONS │                                                  │
│  │                      │                                                  │
│  │ - objective_id       │                                                  │
│  │ - session_phase      │ ◄────────── Session Manager                     │
│  │ - working_memory     │                                                  │
│  │ - completeness_score │ ◄────────── Completeness Engine                 │
│  │ - turn_count         │                                                  │
│  └──────────┬───────────┘                                                  │
│             │ 1:N                                                          │
│             ▼                                                              │
│  ┌────────────────────────────────────────┐                               │
│  │   ELICITATION_OBLIGATIONS              │                               │
│  │   (21 new fields)                      │                               │
│  │                                        │                               │
│  │ Core:                                  │                               │
│  │ - obligation_key                       │                               │
│  │ - field_path                           │                               │
│  │ - status (pending → satisfied)         │                               │
│  │                                        │                               │
│  │ Collection:                            │                               │
│  │ - collection_type                      │ ◄────── Obligation Tracker    │
│  │ - widget_type                          │                               │
│  │ - attempts / max_attempts              │                               │
│  │                                        │                               │
│  │ Extraction:                            │                               │
│  │ - confidence (0.00-1.00)               │ ◄────── Re-evaluation Engine  │
│  │ - extracted_value (JSONB)              │                               │
│  │ - satisfied_at_turn                    │                               │
│  │                                        │                               │
│  │ Advanced:                              │                               │
│  │ - dependencies (JSONB array)           │                               │
│  │ - negotiation_state (JSONB)            │ ◄────── Negotiation Protocol  │
│  │ - task_priority                        │                               │
│  │ - embedding (vector 1536)              │                               │
│  └────────────┬───────────────────────────┘                               │
│               │ events                                                    │
│               ▼                                                           │
│  ┌────────────────────────┐                                               │
│  │  OBLIGATION_EVENTS     │                                               │
│  │  (Audit Trail)         │                                               │
│  │                        │                                               │
│  │ - created              │                                               │
│  │ - status_changed       │                                               │
│  │ - confidence_updated   │                                               │
│  │ - value_extracted      │                                               │
│  │ - negotiation          │                                               │
│  │ - dependency_resolved  │                                               │
│  └────────────────────────┘                                               │
│                                                                           │
│  ┌────────────────────────┐        ┌─────────────────────────┐           │
│  │ DISCOVERY_RESEARCH     │        │ DISCOVERY_SESSION       │           │
│  │ _TASKS                 │        │ _SUMMARIES              │           │
│  │                        │        │                         │           │
│  │ - competitive_analysis │        │ - summary_text          │           │
│  │ - tech_feasibility     │        │ - obligation_snapshot   │           │
│  │ - market_research      │        │ - research_summary      │           │
│  │ - domain_research      │        │ - turn_range            │           │
│  │ - integration_research │        └─────────────────────────┘           │
│  └────────────────────────┘                                               │
│         ▲                                                                 │
│         │ triggers                                                        │
│         │                                                                 │
│  ┌──────────────────────────┐                                            │
│  │ RESEARCH_TRIGGER         │                                            │
│  │ _EVALUATOR               │                                            │
│  │                          │                                            │
│  │ Checks if obligation     │                                            │
│  │ needs research:          │                                            │
│  │ - Low confidence         │                                            │
│  │ - Complex domain         │                                            │
│  │ - User request           │                                            │
│  └──────────────────────────┘                                            │
│                                                                           │
└─────────────────────────────────────────────────────────────────────────────┘

Discovery Objectives

What Are Discovery Objectives?

Discovery objectives define what the discovery engine should collect for a given context type. They're configurable templates that specify:

Required obligations (fields to collect)
Session phases and their goals
Completion criteria
Widget types for UI rendering

Three Context Types

business_profile — Business model discovery
idea_discovery — Product idea exploration
project_discovery — Detailed requirements gathering

Database Schema

CREATE TABLE discovery_objectives (
  id UUID PRIMARY KEY,
  context_type TEXT CHECK (context_type IN (
    'business_profile', 'idea_discovery', 'project_discovery'
  )),
  version TEXT DEFAULT '1.0.0',
  display_name TEXT,
  config JSONB NOT NULL,  -- DiscoveryObjectiveDefinition
  active BOOLEAN DEFAULT true,
  created_at TIMESTAMPTZ,
  updated_at TIMESTAMPTZ
);

-- Only one active objective per context_type
CREATE UNIQUE INDEX idx_discovery_objectives_active_context
  ON discovery_objectives(context_type) WHERE active = true;

Config Structure (DiscoveryObjectiveDefinition)

interface DiscoveryObjectiveDefinition {
  displayName: string;
  description: string;

  // Obligations to collect
  obligations: ObligationTemplate[];

  // Session phases
  phases: {
    opening: PhaseConfig;
    exploration: PhaseConfig;
    validation: PhaseConfig;
    closing: PhaseConfig;
  };

  // Completion criteria
  completion: {
    minObligationsSatisfied: number;
    minConfidenceScore: number;
    requiredObligations: string[];  // Obligation keys that MUST be satisfied
  };
}

interface ObligationTemplate {
  key: string;                    // Unique obligation identifier
  fieldPath: string;              // Dot-notation path (e.g., "constraints.timeline")
  prompt: string;                 // Question to ask
  collectionType: 'direct_question' | 'inferred' | 'research';
  widgetType: 'text' | 'choice' | 'multiselect' | 'date' | 'number';
  dependencies?: string[];        // Other obligation keys this depends on
  priority: number;               // 1-10 (higher = more important)
  satisfiedWhen?: string;         // Condition for satisfaction
}

Example Objective

{
  "displayName": "Business Profile Discovery",
  "description": "Understand the user's business model and goals",
  "obligations": [
    {
      "key": "industry",
      "fieldPath": "businessProfile.industry",
      "prompt": "What industry is your business in?",
      "collectionType": "direct_question",
      "widgetType": "text",
      "priority": 10,
      "satisfiedWhen": "confidence >= 0.8"
    },
    {
      "key": "target_market",
      "fieldPath": "businessProfile.targetMarket",
      "prompt": "Who is your target market?",
      "collectionType": "direct_question",
      "widgetType": "text",
      "dependencies": ["industry"],
      "priority": 9
    }
  ],
  "completion": {
    "minObligationsSatisfied": 8,
    "minConfidenceScore": 0.75,
    "requiredObligations": ["industry", "target_market", "primary_goal"]
  }
}

Obligation Model

The 21 New Fields

Migration 182 added 21 new columns to elicitation_obligations:

Column	Type	Purpose
`collection_type`	TEXT	How to collect: direct_question, inferred, research
`widget_type`	TEXT	UI widget: text, choice, multiselect, date, number
`dependencies`	JSONB	Array of obligation keys this depends on
`source`	TEXT	Where this obligation came from
`origin_layer`	TEXT	Layer in objective hierarchy
`origin_source_id`	UUID	Source entity ID
`pool_id`	TEXT	Question pool ID
`task_priority`	INTEGER	1-10 priority
`negotiation_state`	JSONB	State of value negotiation
`confidence`	NUMERIC(3,2)	0.00-1.00 confidence score
`extracted_value`	JSONB	The extracted value
`attempts`	INTEGER	Collection attempts count
`max_attempts`	INTEGER	Max attempts before giving up
`last_attempt_turn`	INTEGER	Turn number of last attempt
`satisfied_at_turn`	INTEGER	Turn when satisfied
`category`	TEXT	Obligation category
`field_path`	VARCHAR(500)	Dot-notation path
`agent_notes`	TEXT	Agent's internal notes
`follow_up_strategy`	TEXT	How to follow up if unsatisfied
`satisfied_when`	TEXT	Condition for satisfaction
`expertise_required`	TEXT	Domain expertise needed
`embedding`	vector(1536)	Semantic search vector

Obligation Lifecycle

┌──────────┐
│ PENDING  │  Initial state
└────┬─────┘
     │
     │ Agent attempts collection
     ▼
┌──────────┐
│IN_PROGRESS│  Being actively collected
└────┬─────┘
     │
     ├──► ┌─────────┐
     │    │ PARTIAL │  Some info collected, low confidence
     │    └─────────┘
     │
     ├──► ┌──────────┐
     │    │SATISFIED │  Collected with high confidence
     │    └──────────┘
     │
     ├──► ┌─────────┐
     │    │DEFERRED │  User wants to answer later
     │    └─────────┘
     │
     └──► ┌────────┐
          │SKIPPED │  Not applicable / user declined
          └────────┘

Confidence Scoring

Confidence ranges:

Range	Meaning	Action
0.00-0.40	Low — Unreliable	Re-ask, trigger research
0.41-0.70	Medium — Acceptable	Mark partial, may follow up
0.71-0.90	High — Reliable	Mark satisfied
0.91-1.00	Very High — Explicit statement	Mark satisfied, high priority

Dependencies

Obligations can depend on other obligations:

{
  "key": "target_market_size",
  "dependencies": ["target_market"],  // Can't ask size until we know the market
  "dependenciesResolved": false
}

Dependency Resolution:

Agent checks dependencies array
Loads dependent obligations
If all dependencies have status = 'satisfied', mark dependenciesResolved = true
Only then can this obligation be attempted

Session Management

Session Phases

Sessions progress through 4 phases:

type SessionPhase = 'opening' | 'exploration' | 'validation' | 'closing';

Phase Transitions:

opening
  │
  │ Initial rapport, set expectations
  │ Completeness: 0-20%
  ▼
exploration
  │
  │ Main discovery, obligation collection
  │ Completeness: 20-80%
  ▼
validation
  │
  │ Confirm extracted values, negotiate disagreements
  │ Completeness: 80-95%
  ▼
closing
  │
  │ Summary, next steps
  │ Completeness: 95-100%
  ▼
[Session Complete]

Session Manager

File: src/workspaces/discovery/session-manager.ts

Functions:

// Create new discovery session
async function createSession(params: {
  tenantId: string;
  contextType: 'business_profile' | 'idea_discovery' | 'project_discovery';
  userId: string;
}): Promise<ElicitationSession>

// Get active session
async function getActiveSession(
  tenantId: string,
  contextType: string
): Promise<ElicitationSession | null>

// Update session phase
async function updateSessionPhase(
  sessionId: string,
  newPhase: SessionPhase
): Promise<void>

// Update completeness score
async function updateCompletenessScore(
  sessionId: string,
  score: number
): Promise<void>

// Increment turn count
async function incrementTurnCount(sessionId: string): Promise<void>

Completeness Scoring

Completeness Engine

File: src/workspaces/discovery/completeness-engine.ts

Algorithm:

function calculateCompleteness(
  obligations: Obligation[],
  objective: DiscoveryObjective
): number {
  const total = obligations.length;
  let satisfied = 0;
  let weightedSum = 0;

  obligations.forEach(ob => {
    if (ob.status === 'satisfied') {
      satisfied++;
      weightedSum += ob.confidence * ob.task_priority;
    } else if (ob.status === 'partial') {
      weightedSum += (ob.confidence * 0.5) * ob.task_priority;
    }
  });

  // Normalize
  const maxPossibleWeight = obligations.reduce(
    (sum, ob) => sum + ob.task_priority, 0
  );
  const normalizedScore = weightedSum / maxPossibleWeight;

  // Required obligations check
  const requiredSatisfied = objective.completion.requiredObligations.every(
    key => obligations.find(o => o.obligation_key === key && o.status === 'satisfied')
  );

  if (!requiredSatisfied) {
    return Math.min(normalizedScore, 0.85);  // Cap at 85% if required missing
  }

  return normalizedScore;
}

Example:

10 obligations total
7 satisfied with avg confidence 0.9
2 partial with avg confidence 0.6
1 pending

weightedSum = (7 * 0.9) + (2 * 0.6 * 0.5) + 0
            = 6.3 + 0.6 + 0
            = 6.9

normalizedScore = 6.9 / 10 = 0.69 (69%)

Research System

Research Trigger Evaluator

File: src/workspaces/discovery/research-trigger-evaluator.ts

When to Trigger Research:

Low Confidence — confidence < 0.6 after 2 attempts
Complex Domain — expertise_required is set
User Request — User explicitly asks "Can you research this?"
Ambiguous Answer — Agent detects vague/conflicting info

Task Types:

type ResearchTaskType =
  | 'competitive_analysis'    // Research competitors
  | 'tech_feasibility'        // Assess technical viability
  | 'market_research'         // Market size, trends
  | 'domain_research'         // Industry-specific knowledge
  | 'integration_research';   // Third-party integrations

Example Trigger:

// Obligation: "target_market"
// Extracted value: "small businesses"
// Confidence: 0.45 (low!)

const shouldResearch = await evaluator.evaluate({
  obligation: {
    key: 'target_market',
    extracted_value: { market: 'small businesses' },
    confidence: 0.45,
    attempts: 2
  }
});

if (shouldResearch) {
  await createResearchTask({
    sessionId,
    obligationKey: 'target_market',
    taskType: 'market_research',
    query: 'What is the size and characteristics of the small business market?',
    priority: 8
  });
}

Discovery Research Tasks

Database:

CREATE TABLE discovery_research_tasks (
  id UUID PRIMARY KEY,
  session_id UUID NOT NULL,
  tenant_id UUID NOT NULL,
  obligation_key TEXT,
  task_type TEXT CHECK (task_type IN (
    'competitive_analysis', 'tech_feasibility',
    'market_research', 'domain_research', 'integration_research'
  )),
  query TEXT NOT NULL,
  status TEXT CHECK (status IN ('pending', 'running', 'completed', 'failed')),
  result JSONB,
  error_message TEXT,
  priority INTEGER DEFAULT 5,
  created_at TIMESTAMPTZ,
  started_at TIMESTAMPTZ,
  completed_at TIMESTAMPTZ
);

Research Execution:

Inngest function: src/inngest/discovery-research/execute-research.ts

export const executeResearchTask = inngest.createFunction(
  { id: 'discovery-research.execute' },
  { event: 'discovery/research.triggered' },
  async ({ event, step }) => {
    const { taskId, query, taskType } = event.data;

    // Step 1: Web search
    const searchResults = await step.run('web-search', async () => {
      return await performWebSearch(query);
    });

    // Step 2: AI summarization
    const summary = await step.run('summarize', async () => {
      return await summarizeResearchResults(searchResults, taskType);
    });

    // Step 3: Store result
    await step.run('store-result', async () => {
      await updateResearchTask(taskId, {
        status: 'completed',
        result: summary
      });
    });

    // Step 4: Update obligation confidence
    await step.run('update-obligation', async () => {
      await updateObligationFromResearch(obligationKey, summary);
    });
  }
);

Negotiation Protocol

When Negotiation Occurs

User disagrees with an extracted value:

Agent: "Based on what you said, I understand your target market is 'enterprise businesses'."
User: "No, actually it's small businesses, not enterprise."

Negotiation State

JSONB Structure:

interface NegotiationState {
  status: 'active' | 'resolved' | 'deferred';
  rounds: NegotiationRound[];
  finalValue?: any;
}

interface NegotiationRound {
  turnNumber: number;
  agentProposal: any;
  userResponse: 'accept' | 'reject' | 'modify';
  userValue?: any;
  agentReasoning: string;
}

Example:

{
  "status": "resolved",
  "rounds": [
    {
      "turnNumber": 5,
      "agentProposal": { "market": "enterprise businesses" },
      "userResponse": "reject",
      "userValue": { "market": "small businesses" },
      "agentReasoning": "User explicitly stated 'not enterprise'"
    }
  ],
  "finalValue": { "market": "small businesses" }
}

Negotiation Protocol Service

File: src/workspaces/discovery/negotiation-protocol.ts

Functions:

// Start negotiation
async function startNegotiation(params: {
  sessionId: string;
  obligationKey: string;
  agentProposal: any;
  userRejection: string;
}): Promise<NegotiationState>

// User provides alternative value
async function submitUserValue(params: {
  sessionId: string;
  obligationKey: string;
  userValue: any;
}): Promise<void>

// Resolve negotiation
async function resolveNegotiation(params: {
  sessionId: string;
  obligationKey: string;
  finalValue: any;
  resolution: 'user_accepted' | 'agent_accepted' | 'compromise';
}): Promise<void>

Working Memory

What Is Working Memory?

The agent's internal state maintained across conversation turns. Includes:

Recently discussed topics
Pending follow-ups
Inferred relationships between obligations
User preferences and patterns

Structure

interface WorkingMemory {
  recentTopics: string[];                 // Last 5 topics discussed
  pendingFollowUps: FollowUp[];           // Questions to revisit
  inferredRelationships: Relationship[];  // Connections between obligations
  userPreferences: Record<string, any>;   // Learned preferences
  conversationStyle: 'formal' | 'casual'; // Detected tone
}

interface FollowUp {
  obligationKey: string;
  reason: string;         // Why following up
  priority: number;
  scheduledTurn?: number; // Defer to specific turn
}

Example

{
  "recentTopics": ["target_market", "product_pricing", "competition"],
  "pendingFollowUps": [
    {
      "obligationKey": "timeline",
      "reason": "User mentioned 'tight deadline' but didn't specify",
      "priority": 8
    }
  ],
  "inferredRelationships": [
    {
      "from": "target_market",
      "to": "pricing_strategy",
      "type": "influences",
      "confidence": 0.85
    }
  ],
  "userPreferences": {
    "prefers_examples": true,
    "technical_depth": "medium"
  },
  "conversationStyle": "casual"
}

Working Memory Updates

Updated after each turn in post-turn-processor.ts:

async function updateWorkingMemory(
  sessionId: string,
  turnData: TurnData
): Promise<void> {
  const memory = await getWorkingMemory(sessionId);

  // Add new topic
  memory.recentTopics.push(turnData.topic);
  if (memory.recentTopics.length > 5) {
    memory.recentTopics.shift();  // Keep last 5
  }

  // Add follow-ups
  if (turnData.followUpsNeeded) {
    memory.pendingFollowUps.push(...turnData.followUpsNeeded);
  }

  // Update preferences
  if (turnData.preferenceSignals) {
    Object.assign(memory.userPreferences, turnData.preferenceSignals);
  }

  // Save
  await saveWorkingMemory(sessionId, memory);
}

Database Schema

Extended Tables

elicitation_sessions:

ALTER TABLE elicitation_sessions
  ADD COLUMN objective_id UUID REFERENCES discovery_objectives(id),
  ADD COLUMN session_phase VARCHAR DEFAULT 'opening',
  ADD COLUMN working_memory_snapshot JSONB,
  ADD COLUMN completeness_score NUMERIC(3,2) DEFAULT 0,
  ADD COLUMN turn_count INTEGER DEFAULT 0;

elicitation_obligations:

See Obligation Model section for all 21 new columns

New Tables

obligation_events:

CREATE TABLE obligation_events (
  id UUID PRIMARY KEY,
  session_id UUID NOT NULL,
  obligation_key TEXT NOT NULL,
  event_type TEXT CHECK (event_type IN (
    'created', 'status_changed', 'confidence_updated',
    'value_extracted', 'negotiation', 'dependency_resolved',
    'merged', 'split'
  )),
  old_status TEXT,
  new_status TEXT,
  old_confidence NUMERIC(3,2),
  new_confidence NUMERIC(3,2),
  source TEXT,
  metadata JSONB DEFAULT '{}',
  turn_number INTEGER,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

discovery_session_summaries:

CREATE TABLE discovery_session_summaries (
  id UUID PRIMARY KEY,
  session_id UUID NOT NULL,
  tenant_id UUID NOT NULL,
  summary_text TEXT,
  obligation_snapshot JSONB,
  research_summary JSONB,
  turn_range_start INTEGER,
  turn_range_end INTEGER,
  created_at TIMESTAMPTZ DEFAULT NOW()
);

discovery_research_tasks:

CREATE TABLE discovery_research_tasks (
  id UUID PRIMARY KEY,
  session_id UUID NOT NULL,
  tenant_id UUID NOT NULL,
  obligation_key TEXT,
  task_type TEXT CHECK (task_type IN (
    'competitive_analysis', 'tech_feasibility',
    'market_research', 'domain_research', 'integration_research'
  )),
  query TEXT NOT NULL,
  status TEXT CHECK (status IN ('pending', 'running', 'completed', 'failed')),
  result JSONB,
  error_message TEXT,
  priority INTEGER DEFAULT 5,
  created_at TIMESTAMPTZ,
  started_at TIMESTAMPTZ,
  completed_at TIMESTAMPTZ
);

Service Layer

All services in src/workspaces/discovery/:

Obligation Tracker

File: obligation-tracker.ts

Functions:

createObligation(params) — Create new obligation
updateObligationStatus(obligationKey, status) — Update status
updateConfidence(obligationKey, confidence) — Update confidence
extractValue(obligationKey, value, confidence) — Store extracted value
checkDependencies(obligationKey) — Check if dependencies satisfied

Obligation Loader

File: obligation-loader.ts

Functions:

loadObligationsFromObjective(objectiveId) — Initialize obligations from objective
getObligationsBySession(sessionId) — Get all obligations for session
getUnresolvedObligations(sessionId) — Get pending/partial obligations
getPrioritizedObligations(sessionId) — Get obligations sorted by priority

Completeness Engine

File: completeness-engine.ts

Functions:

calculateCompleteness(sessionId) — Calculate current completeness score
getCompletenessBreakdown(sessionId) — Detailed breakdown by category
checkCompletionCriteria(sessionId) — Check if session can be completed

Negotiation Protocol

File: negotiation-protocol.ts

Functions:

startNegotiation(params) — Start negotiation for obligation
submitUserValue(params) — User provides alternative value
resolveNegotiation(params) — Finalize negotiation

Research Trigger Evaluator

File: research-trigger-evaluator.ts

Functions:

evaluate(obligation) — Determine if research needed
getRecommendedResearchType(obligation) — Suggest research type
triggerResearch(params) — Create research task

Re-evaluation Engine

File: re-evaluation-engine.ts

Functions:

reevaluateObligation(obligationKey) — Reassess obligation based on new context
reevaluateSession(sessionId) — Reassess all obligations
detectConflicts(sessionId) — Find conflicting values

Session Manager

File: session-manager.ts

Functions:

createSession(params) — Create new session
getActiveSession(tenantId, contextType) — Get active session
updateSessionPhase(sessionId, phase) — Update phase
completeSession(sessionId) — Mark session complete

API Endpoints

Discovery Chat

POST /api/discovery-chat

Request:

{
  "sessionId": "session-uuid",
  "contextType": "business_profile",
  "message": "My target market is small businesses in healthcare",
  "userId": "user-uuid",
  "tenantId": "tenant-uuid"
}

Response (SSE stream):

data: {"type":"message","content":"I understand..."}
data: {"type":"obligation","key":"target_market","status":"satisfied","confidence":0.9}
data: {"type":"completeness","score":0.75}
data: {"type":"phase","phase":"validation"}
data: {"type":"done"}