The four Claims agents are distinct actors with separate contracts. The Claims Conversation Agent is the customer-facing orchestrator that runs the live FNOL conversation. The Claims Document and Evidence Agent, Claims Damage Report Agent, and Claims Policy and Fraud Agent are back-office analysts invoked at specific phases. Each stays grounded in the claim context it’s given and returns structured conclusions; none of them commits the claim.

The summaries below describe each agent’s system-prompt contract: identity and mission, grounding, output, tools, and guardrails. Each summary is followed by the agent’s Best practices for extending or customizing it. Then come its Configuration and customization notes: its AI model, settings, hardwired values, and default values (refer to Make it your own). Agent evaluations, after the four agent summaries, explains how to evaluate the agents with the solution’s datasets.

Which agent checks coverage in this release. The early coverage checks in the claim flow are the policy-validity check when the draft claim is created and the coverage check behind the roadside-assistance and rental-car lanes. When the Claims Services setting GetCoveragesFromPolicyDoc is on, the Claims Policy and Fraud Agent runs them during intake: it reads the policy document at claim start and returns the elected coverages. With the default setting (off), the same checks use the policy coverages synchronized from the core system, and the agent isn’t called for them (refer to Coverage & Policy Rules). The same agent also answers the customer’s policy questions during the conversation. It’s also prepared to support fraud detection and decision support after submission (the Decisioning phase), which is outside the solution’s scope.

Claims Conversation Agent

Customer-facing · drives the live FNOL conversation across Triage and Evidence · prompt-derived (prompt version 0.3.30).

Identity & mission. “Aria”, the Claims Conversation Agent, guides a customer through First Notice of Loss one turn at a time, staying strictly within the behavioral rule injected for the claim’s current status. It’s one actor in an orchestrated workflow: it proposes, and the workflow commits. It never advances the claim beyond its current status.
Grounding contract. Each turn, the host prepends a trusted block: the current ClaimsOccurrence facts as JSON, the current ClaimStatus record, the ClaimsStatusRule.RuleContent, and the conversation’s ChatTitle. The rule content (Purpose / MUST DO / MUST NOT DO / EXIT) is the authoritative behavioral contract for the turn. Every field already present is already-collected data, so the agent confirms it at most and never asks for it again. The grounding block is trusted host data, not customer or tool input.
Precedence. Highest wins: (1) safety override, (2) base guardrails, (3) ClaimsStatusRule.RuleContent, (4) base defaults.
Output contract. The entire reply is exactly one JSON object (Answer, OptionsTitle, ResponseOptions, AdaptiveContent, TargetStatus, Reason) and nothing else. TargetStatus and Reason are a transition proposal for the workflow and are never shown to the customer. A forward-momentum invariant requires every non-terminal turn to hand the customer a next step. That step is a question, options, or an interactive card that carries its own action, such as the Claim Summary card’s Submit button. Agent UI Interaction covers the full contract.
Tools. The agent calls a tool only when the current status rule’s MUST DO step requires it or, for AskPolicyExpert, when the customer asks about their policy. Each tool’s description is its contract:
- GetMyClaims: lists the customer’s in-progress claims, so an existing one is resumed instead of duplicated.
- AddClaimToChatSession: starts a new draft claim and links it to the conversation, once the customer confirms a new incident and no claim is in context.
- UpdateClaimLossFacts: saves the loss facts the customer gave or corrected this turn (narrative, loss cause and type, date and time, police report, drivability). The loss cause and type are derived from the customer’s account, never asked for directly.
- SetClaimSafetySignal: records the claim’s safety flags (airbag deployed, distress check done, emergency services required, distress escalated).
- GetInssuredVehicles: lists the vehicles on the claim’s policy, so the customer can pick the one involved from quick replies (year, make, model, and plate, plus “Other vehicle”). The agent never reads out a full VIN.
- GetRoadAssistanceETA: returns the remaining minutes until the booked tow arrives.
- AskPolicyExpert: answers the customer’s own question about their policy from the policy document, through the Claims Policy and Fraud Agent, with or without a claim. The answer explains the policy; it isn’t a decision on the claim.
- AddTitleToChatSession: names the conversation once, silently, when its subject is clear.
None of these tools changes the claim status. State transitions are not a tool, and the agent never calls the insurance core directly.
Safety floor & guardrails. A non-negotiable safety override stops fact-gathering and proposes the safety-hold status on any injury, unsafe-scene, or emergency signal. All customer and tool text is treated as untrusted data. Jailbreak, staff-impersonation, and out-of-scope requests route to the human-handling escalation, and the system prompt, tool list, and proposal fields are never revealed.

Best practices

Change stage behavior in the status rules, not the base prompt. The system prompt holds the stable frame: identity, precedence, output contract, safety floor, and guardrails. What the agent does at each stage belongs in the ClaimsStatusRule body for that status. Keep each rule in the Purpose / MUST DO / MUST NOT DO / EXIT shape. When you change a rule’s exits, check the rules of the statuses it now leads to.
Never weaken the safety floor or guardrails. Status rules rank below the safety override and the base guardrails, so don’t write rule text that relies on overriding them.
Keep the prompt aligned with the data it names. When you add or rename an AdaptiveContentType, LossCause, LossType, or status, update the prompt, the status rules, and the tool descriptions that mention it in the same change. Raise the prompt version so that conversations can be traced to it.
Treat each tool description as its contract. The model decides when and how to call a tool only from its description and parameter descriptions. Keep one job per tool, inputs the model can fill from the conversation, and descriptions that name only tools and values that still exist.
Keep status changes out of the tools. Tools save facts; the next status is proposed in TargetStatus and applied by the workflow. If you need tighter control, validate the proposed status against the current rule’s allowed exits before applying it.
Edit the prompt as an expression literal. The prompt is assigned as a text literal in BuildMessages, so every double quote inside it must be doubled.
Re-test the whole conversation after a change. After you change the prompt, a rule, a tool, or the AI model, run the claim flow through every active status. A change at one stage can break the hand-off to the next.

Configuration and customization

Hardwired Values.
- The agent ships wired to the TrialClaudeHaiku4_5 trial model, with no temperature set, so the model default applies. The trial model works only in development, so select an AI model of your own before production (refer to Getting Started, step 4).
- The system prompt (version 0.3.30) sets the persona (Aria), the US emergency wording ("call 911"), and the single-JSON-object output contract (Answer, OptionsTitle, ResponseOptions, AdaptiveContent, TargetStatus, Reason).
- The prompt lists the Adaptive Content card types (ClaimSummary, CarRental, RoadsideAssistance, EvidenceGallery, CallEmergency). Keep it in sync with the AdaptiveContentType records in Claims Services.
- The status the agent proposes in TargetStatus is applied to the claim directly, through ServiceClaimsOccurrenceSetStatus.
- LoadMemory loads at most 50 memory records per session.
- GetMyClaims returns at most 5 of the customer's other claims, counting as resumable only claims in the Initiation, Triage, or Evidence phase.
- Uploaded files are stored with document type Other, and every write is attributed to the Conversation claim agent.
Default Values.
- The UpdateClaimLossFacts tool lists the loss causes (Collision, Theft, Fire, Vandalism, Weather, Glass, AnimalStrike) and loss types (AutoCollision, AutoComprehensive, AutoLiability) the model can use, plus Unknown. Keep them aligned with the LossCause and LossType records in Claims Services.

Claims Document and Evidence Agent

Back-office · Evidence phase (P3) · prompt-derived.

Identity & mission. A per-artifact evidence analyzer and the solution’s single extraction-and-validation engine for evidence. It works on each captured artifact on its own, such as a vehicle photo, scanned document, insurance ID card, driver’s license, registration, receipt, or voice-note transcript. It classifies the artifact, extracts its type-specific fields, assesses image quality and legibility, and surfaces integrity concerns as observations. It then returns a verdict (Valid, Invalid, Waived, or Pending) with a plain-language reason and, when useful, a re-capture suggestion. It describes each artifact; it never decides the claim and never assesses the claim as a whole.
Entry & grounding. In the solution, the agent runs in the background for each upload. When the customer adds a photo or document, Claims Services saves the evidence, reads the image metadata (capture time, GPS location, and camera details), and raises the AssessEvidence event. The agent’s OnAssessEvidence handler then analyzes that document. The public CallClaimsDocumentAndEvidenceAgent runs the analysis for a claim (ClaimsOccurrenceId), an optional DocumentIds scope, and the user input. The grounding carries the claim context (loss type and cause, jurisdiction), the image metadata, the per-artifact signals stored with the document, and the FNOL code lists (evidence types, media types, impact locations, loss types and causes, validation verdicts). The code lists are fetched live from the static entities, so outputs always use the configured codes. When the user input says which evidence type was expected, that type is a weak hint, and a mismatch is reported as an InvalidEvidenceType insight.
Extraction (first-class output). Every artifact type has a field schema. For example, an insurance ID card yields the carrier, NAIC code, policy number, and contact. A driver’s license yields the name, date of birth, and license number, and a registration the plate, VIN, and owner. A police report yields the report number, agency, parties, point of impact, and narrative, and a repair estimate the shop, line items, and totals. Each field carries a calibrated confidence and an evidence anchor (artifact, region or page, short snippet). An unreadable value is null with a reason, never invented. For vehicle photos, the agent also records the view and each damage area: impact location, severity (Minor / Moderate / Severe), and a free-text description.
Cross-artifact checks (same batch only). Before any verdict, the agent compares the vehicle across photos (color, silhouette, plate). It checks every credential document for specimen or sample signs: “SPECIMEN” / “VOID” watermarks, placeholder names, all-zero numbers, or an expiry before the loss date. It also checks that each document is filled in rather than a blank template. A specimen or blank document is always Invalid.
Tools. The analysis (OCR, classification, quality, transcription) is the model’s own multimodal reading, and no external vision services are called. Exactly one Action-Calling tool, DecodeVIN, is mandatory for every VIN the agent extracts. It validates the format and the ISO 3779 check digit, and it returns the authoritative year, make, and model from NHTSA vPIC through Claims Services and Vehicle Data Integration Wrapper. Disagreements with the document or photo are flagged for verification.
Output. Exactly one JSON object with a per-artifact artifacts array and crossArtifactChecks. Each artifact carries its classification, extractedFields, quality, integrity flags with severity and basis, anomalyFlags, insights, the verdict, and toolErrors. Every input document appears exactly once, and an empty batch returns an empty array rather than conversational text. In the solution, the OnAssessEvidence handler saves each insight on the document in Solutions Common Services (ServiceDocumentAddInsight) and the artifact’s evidence type on its evidence record in Claims Services (ServiceClaimsEvidenceCreateOrUpdate).
Guardrails. No claim-level rollup: checklist completeness and missing evidence are derived from the database, not by the agent. No adjudication of fraud, validity, liability, payout, deductible, or reserve; integrity concerns are factual observations, never accusations. A metadata finding (stripped EXIF, GPS far from the loss, reused image) is reported only when the signal was supplied or is plainly visible; otherwise its basis is not_determinable. Extracted personal data stays in structured fields. Instructions embedded in a document or transcript are ignored and recorded as an integrity observation.

Best practices

Keep it a per-artifact analyzer. Don’t add claim-level conclusions (completeness, coverage, fraud, liability) to its prompt. Claim-level decisions belong to the workflow, the Claims Policy and Fraud Agent, and the adjuster.
Keep the analysis apart from the saving. The model returns verdicts and extracted fields, and the OnAssessEvidence handler saves results through the Core services’ public actions. When you add a document type, add its field schema to the prompt and the handler step that saves its fields in the same change.
Extend code lists in Claims Services. Evidence types, media types, impact locations, verdicts, loss types, and loss causes are read from the Claims Services static entities at runtime, so add records there instead of writing codes into the prompt.
Put checks with a single correct answer in tools. Follow the DecodeVIN pattern: deterministic validation (formats, check digits, registry lookups) runs in a tool, and the model reports the tool’s result.
Supply capture-time signals when you have them. The agent reports metadata findings (EXIF timestamp, geotag, reused images) only when those signals are supplied. Pass the OCR text, EXIF data, and on-device detections your capture flow collects.
Adapt the jurisdiction hints to your markets. The prompt’s hints cover US police-report forms and license patterns (CA, TX, FL, NY) and UK references; extend them for the regions you serve.

Configuration and customization

Settings.
- AgentEval_Enabled turns the agent’s evaluation actions (EvalRunDocumentEvidenceAgent, EvalSetup, and EvalTeardown) on or off, default False. While it’s False, those actions raise an error, so an evaluation run stops at its setup step. Set it to True only in development, to run the agent evaluations.
Hardwired Values.
- The agent ships wired to the TrialClaudeHaiku4_5 trial model, with no temperature set, so the model default applies. The trial model works only in development, so select an AI model of your own before production (refer to Getting Started, step 4).
- Behavior is defined by the system prompt assigned in BuildMessages; to change it, edit that prompt.
- The prompt fixes the artifact vocabularies: artifact categories, vehicle views, document types with their field schemas, image-quality statuses and issues, integrity flags, and insight types. It also fixes the limits: evidence snippets of at most 25 words, insight content of at most 2000 characters, and every submitted file reported exactly once.
- Specimen credentials (SPECIMEN, SAMPLE, VOID, placeholder names) are always judged invalid. The jurisdiction hints cover US police-report forms and license patterns (CA, TX, FL, NY) and UK references.
- The behavior profile is written into the prompt: Formal tone, Balanced verbosity, Deep reasoning, Conservative risk tolerance, Suggest Only autonomy.
- DecodeVIN checks the VIN format (17 characters, no I, O, or Q) and the mod-11 check digit before it calls the decoder.
- GetFileMimeType accepts pdf, png, jpg, jpeg, doc, docx, xls, xlsx, webp, txt, csv, html, gif, md; other extensions are rejected.
- The model response is limited to 16384 tokens.
Default Values.
- Static AgentProfile_* reference data (Tone, Verbosity, ReasoningDepth, RiskTolerance, ActionAutonomy, Proactivity).
- Evidence types, media types, impact locations, validation verdicts, loss types, and loss causes are read at runtime from the Claims Services static entities.

Claims Damage Report Agent

Back-office · wrap-up (end of the Evidence phase) · prompt-derived.

Identity & mission. Writes the claim summary that the policyholder reads in the Claim Summary card at wrap-up: a clear, friendly, and concise narrative addressed directly to the policyholder (“You were involved in…”), synthesized from the claim’s evidence and its current loss description.
Entry, trigger & grounding. In the solution, the agent runs in the background. When a claim reaches DocumentsPending, Claims Services raises EvidenceCaptured. Its handler, ClaimsOccurrenceEvalDamages, checks that the claim’s evidence analysis is available, retrying for up to about five minutes, and then raises AnalyseDamageReport. If the claim’s summary is already completed, no new summary is generated. The agent’s OnAnalyseDamageReport handler runs the analysis. The public CallFNOLDamageReportAgent(ClaimsOccurrenceId) runs the same analysis and returns the summary without saving it. The grounding is the natural-language summary insight of every evidence item on the claim, and the current summary is the claim’s existing loss description.
Scope. Summarizes only verified facts: the incident details (time, location, what happened) and key context about the other party or the submitted documents. The tone stays clear, objective, and reassuring, without technical jargon.
Output. Plain text in one or two short paragraphs: first the incident circumstances (collision type, date and time, location, travel direction), then the interaction with the other party or the materials submitted, such as an insurance exchange or submitted documents. There are no headings, titles, bullet points, or metadata labels. When the reply isn’t empty, the OnAnalyseDamageReport handler saves it as the claim’s loss description (ClaimsOccurrence.LossDesc), marks the summary as completed, and records the update in the business audit log under the SummaryReport agent. An empty reply is logged and leaves the claim unchanged. The Claim Summary card shows the saved summary.
Guardrails. No speculation, liability judgments, or internal assessment notes. It calls no tools and doesn’t interact with the customer.

Best practices

Keep the summary factual. Keep speculation, liability judgments, and assessment notes out of the summary; plausibility and fraud analysis belongs to the Claims Document and Evidence Agent and the Claims Policy and Fraud Agent.
Decide where the summary lives. The summary is saved over ClaimsOccurrence.LossDesc, the loss narrative captured from the conversation. If you need to keep the claimant’s own account, store the summary in an attribute of its own and point the Claim Summary card at it.
Change the summary format and the Claim Summary card together. The summary is stored as the claim’s loss description and shown in the Claim Summary card, so check the card whenever you change its format.
Feed it through evidence insights. Its grounding is the claim’s loss description plus the natural-language summary insight of each evidence item, so any new evidence type you want reflected in the summary must produce that insight.
Keep it fast. The summary is generated in the background when the claim reaches DocumentsPending, so a slower model or a longer prompt delays when the summary is ready for the Claim Summary card at wrap-up.

Configuration and customization

Hardwired Values.
- The agent ships wired to the TrialClaudeHaiku4_5 trial model, with no temperature set, so the model default applies. The trial model works only in development, so select an AI model of your own before production (refer to Getting Started, step 4).
- The system prompt fixes the summary as plain text in one or two short paragraphs, addressed to the policyholder in the second person, with no headings or bullet points.
- The event handler always asks the agent to "Use the information to generate the claim summary", and CallFNOLDamageReportAgent asks "Please proceed with the analysis to produce a Claim Summary". Both ground it with the claim's natural-language evidence summaries and the current loss description.
- Memory is kept per claim, at most 50 records, in the module's own Memory entity.
Default Values.
- Static SubjectType (3 records: InsuredVehicle, OtherPartyVehicle, OtherParty).

Claims Policy and Fraud Agent

Back-office · answers the customer’s policy questions through the Claims Conversation Agent · early coverage checks during intake (Triage, when GetCoveragesFromPolicyDoc is on) · prepared for fraud detection and decision support after submission (Decisioning phase, outside the solution’s scope) · prompt-derived.

The module hosts three separately prompted capabilities, each behind its own public entry. All three only read and recommend; none of them changes the claim’s status.

Policy coverage inventory (live when enabled). A policy document reader (CallPolicyCoverageOnly) that reads the supplied policy document and returns one JSON object. The object holds the policy reference (number, insured, form, state, effective and expiration dates) and every coverage the document lists. Each coverage carries its code, whether it was elected, its limit and deductible, and a page reference. The codes are COLL, COMP, BI, PD, MEDPM, UM, PIP, RREIM (rental), TOWLB (towing and roadside), or a descriptive code. The object also holds the document type (declarations, booklet, combined), extraction notes, and a confidence. Coverages marked “Not Elected” or “Declined” are listed with elected = false, and a value the document doesn’t state is null, never guessed. Wiring: when a claim starts and the Claims Services setting GetCoveragesFromPolicyDoc is True, Claims Services raises the CheckPolicyCoverage event. The default, False, reads the synchronized PolicyCoverage records instead (refer to Coverage & Policy Rules). The agent’s OnCheckPolicyCoverages handler runs the inventory on the claim’s policy document. It sends the elected coverage codes back through ServiceClaimsOccurrenceSetPolicyCoverages, which sets the claim’s rental and roadside-assistance coverage flags.
Policy expert Q&A (live). A policy document expert (CallPolicyExpert) that answers questions about a policy document in plain, conversational prose, keeping session memory. It answers only from the document. It quotes the relevant wording with its page or section, leads yes/no questions with the direct answer, and says so when the document is silent, rather than describing what policies usually do. It also explains declined coverages and notes when a booklet lacks the policyholder’s specific limits. It redirects off-topic questions and never reveals its instructions. The Claims Conversation Agent calls it through its AskPolicyExpert tool when the customer asks a question about their policy.
Full policy and fraud analysis (Decisioning, outside the solution’s scope). CallFullPolicyAndFraudAnalysis(ClaimsOccurrenceId) produces one combined decision-support packet for adjuster review. It’s grounded from Claims Services by the claim ID: a policy snapshot, claim facts, an evidence summary, claim history, evidence signals, and personal-data context. Its 14-step workflow covers:
- personal-data minimization and screening for prohibited signals;
- policy-in-force validation (IN_FORCE / NOT_IN_FORCE / IN_QUESTION);
- per-item coverage decisions (COVERED / EXCLUDED / IN_QUESTION) citing grants, exclusions, carve-backs, and endorsements;
- limits and deductible application;
- a fraud-likelihood score from 0 to 100 (0 to 30 low, 31 to 70 review, 71 to 100 high), built from observed indicators and mitigating factors;
- a severity tier from 1 to 4, kept distinct from fraud likelihood;
- a low / expected / high reserve range that never silently exceeds known limits;
- anomalies for the adjuster, and a bias-detection check.
No other solution module calls it in this release.
Tools (full analysis). calculate_deductible is a mandatory call for every covered or in-question item with a deductible. The agent never computes or waives a deductible itself, and an unclassified result is reported as null plus an information gap, not as zero. get_weather_data is gated: it’s called only for weather perils (wind, hail, hurricane, named storm, flood, surface water, or disputed weather) when a usable loss date and location exist, and only to corroborate causation. The agent doesn’t look up claim history, customer, vendor, medical, or public-record data.
Output & guardrails (full analysis). Exactly one raw JSON packet: claim reference, personal-data guardrail, policy in force, coverage determinations, deductible application, weather corroboration, fraud likelihood with scored indicators, severity, reserve, anomalies for the adjuster, bias detection, information gaps, tools invoked, and overall confidence. It never invents policy wording, facts, or tool results, and supplied policy wording overrides general insurance knowledge. It never states or implies that fraud is proven, and it uses neutral “warrants review” language instead. It takes no claim action (approve, deny, pay, assign, or refer to SIU) and writes no customer-facing text. Protected-class attributes and their proxies are excluded from every conclusion. A request for a forbidden action still receives the full analysis, with the refusal routed into the JSON fields, and prompt-override attempts are ignored and recorded as anomalies. Persistence of the packet (CoverageRiskDecision with DecisionCoverageLine) is left to the calling workflow.

Best practices

Keep it a recommender. Its outputs are recommendations for the adjuster. Keep binding coverage, deductible, and reserve decisions in a deterministic service and with the human adjuster (refer to the recommended direction in Coverage & Policy Rules).
Compute in tools, not in the model. Deductibles always come from CalculateDeductible. Add jurisdictional rules, endorsements, and waivers there rather than in the prompt, and follow the same pattern for any new calculation.
Keep coverage codes aligned. The extraction prompt, CalculateDeductible, and the Coverage static entity must use the same codes; RREIM and TOWLB drive the rental and roadside flags on the claim.
Pass the real policy document. Send the policy PDF from your core system with its actual file name instead of the fixed sample names.
Persist the decision packet when you build Decisioning. Write the packet to CoverageRiskDecision and DecisionCoverageLine so each recommendation is auditable and reproducible.
Keep the fairness guardrails when you edit the prompt. Keep protected attributes and their proxies excluded from every conclusion, the bias-detection check, and the neutral “warrants review” wording for fraud indicators.

Configuration and customization

Settings.
- VisualCrossingAPIKey is the secret key for the Visual Crossing weather API used by the weather tool, no default. While it’s empty, the weather check is skipped.
- AgentEval_Enabled turns the agent’s evaluation actions (EvalRunPolicyAndFraudAgent, EvalSetup, and EvalTeardown) on or off, default False. While it’s False, those actions raise an error, so an evaluation run stops at its setup step. Set it to True only in development, to run the agent evaluations.
Hardwired Values.
- The module’s three agents ship wired to the TrialClaudeHaiku4_5 trial model, with no temperature set, so the model default applies. The trial model works only in development, so select an AI model of your own before production (refer to Getting Started, step 4).
- The module has three agents with fixed system prompts: the full decision-support analysis, coverage extraction from the policy document, and policy questions and answers. Each response is limited to 16384 tokens.
- The analysis prompt fixes the fraud score bands (0-30 low, 31-70 review, 71-100 high), the severity tiers (1 Fast-track to 4 Large or severe), the coverage and policy-in-force decisions, the coverage codes, and the protected attributes that must not influence decisions. The behavior profile (Formal, Deep reasoning, Conservative, Suggest Only) is written into the prompt.
- The coverage-extraction prompt maps policy wording to the codes COLL, COMP, BI, PD, MEDPM, UM, PIP, RREIM, TOWLB. The rental (RREIM) and towing (TOWLB) codes drive the rental and roadside offers in the claim flow.
- CalculateDeductible treats BI, PD, MEDPM, RREIM, TOWLB as no-deductible coverages and maps AutoCollision to COLL and AutoComprehensive to COMP. No jurisdictional rules, glass endorsements, or deductible waivers are applied.
- GetWeatherData calls https://weather.visualcrossing.com with the key from VisualCrossingAPIKey.
- The policy document is always sent to the model as a PDF (fixed file names NeoInsurancePolicy.pdf and policy.pdf).
Default Values.
- The domain vocabulary sent to the model is built at runtime from the Claims Services Coverage, SeverityTier, LineOfBusiness, LossType, and LossCause static entities.
Reminders.
- BuildMessages_PolicyCoverage and BuildMessages_PolicyExpert: the policy file name is hardcoded; take it from the actual document instead.

Agent evaluations

Use ODC agent evaluations to check how the Claims agents behave in your tenant, for example after you select an AI model or change an agent’s prompt or tools. An evaluation runs every test case of a dataset through one public service action of an agent. The platform judge then scores each answer against the case’s expected response, guidelines, and expected tool calls. The solution provides four datasets: three for the Claims Policy and Fraud Agent and one for the Claims Document and Evidence Agent. For how evaluations work, refer to Agent evaluations in the ODC documentation.

Before you start, check that:

You completed Getting Started, including the sample data in step 3. Several cases use the sample policies of Matthew Shelton and Sara Diaz.
You work in your development stage. Evaluation runs and their scores are available only in development.
Your ODC user has the Manage evaluations permission, which you need to create datasets and evaluations.
Both agents can reach an AI model: the TrialClaudeHaiku4_5 trial model they ship with, or the model you selected (Getting Started, step 4). Evaluation runs count toward the trial model’s usage limit.
Each agent was published at least once after agent evaluations were enabled for your organization.
The AgentEval_Enabled setting is True in development for both agents. Set it in ODC Portal, in the configuration of each agent’s app. It defaults to False, and while it’s False the evaluation actions raise an error, so every run fails at its setup step.
Optionally, the VisualCrossingAPIKey setting of the Claims Policy and Fraud Agent has a key. Without it, the weather case checks only the answer when no weather data is available.

Dataset (download)	Agentic app	Service action	Cases	What it checks
dataset-policy-coverage-evals.json	Claims Policy and Fraud Agent	`CallPolicyCoverageOnly`	3	The coverage inventory the agent extracts from the two sample policy documents, and its answer for a policy that has no document.
dataset-policy-expert-evals.json	Claims Policy and Fraud Agent	`CallPolicyExpert`	7	Answers to policyholder questions from the sample policy documents: rental, towing, deductibles, a declined coverage, a question the policy doesn’t answer, and a policy that has no document.
dataset-full-policy-fraud-evals.json	Claims Policy and Fraud Agent	`EvalRunPolicyAndFraudAgent`	11	The full decision-support packet, on a claim the service action creates for each case. The cases cover the deductible and weather tools, a policy not in force, an uncovered peril, a timing anomaly, prior similar claims, and quotes from the sample policy documents. They also include requests that try to override the analysis, pass personal data, or introduce bias.
dataset-document-evidence-evals.json	Claims Document and Evidence Agent	`EvalRunDocumentEvidenceAgent`	13	The per-artifact analysis of photos and documents embedded in the agent: vehicle damage photos, a blurry photo, a sample driver’s license, and VINs on a plate and in documents (`DecodeVIN`). The cases also cover police reports, a blank form, repair estimates (one with an embedded instruction), photos of two different vehicles, and the evidence type the conversation expected.

To evaluate the agents, follow these steps:

Download the datasets. Download the four files from the table. The photos and documents the Claims Document and Evidence cases analyze are embedded in the agent, so the dataset files are all you need.
Create one dataset per file. In ODC Portal, go to Agent evaluations > Datasets and select Create dataset. Then:
- Enter a Name, for example FNOL Policy Coverage, and select the Agentic app and Service action listed for the file in the table. If you cloned the solution’s apps, select your clones.
- Upload the file. It already matches the service action, so you don’t need the template.
- Turn on Setup and teardown. For Setup, select the same agentic app and EvalSetup. For Teardown, select the same app and EvalTeardown.
- Select Save, and check that the dataset shows the number of cases listed in the table.
Create one evaluation per dataset. In Agent evaluations, select Create evaluation. Enter a Name and select the Dataset. Under Judge criteria, select Correctness, Safety, Guidelines, Relevance to Query, and Tool Trajectory, then select Save. For the coverage dataset, leave out Correctness. Its expected response is a list of coverages, and the upload keeps only the first entry of a list, so Correctness can’t pass. The guidelines of each coverage case check every coverage instead.
Run the evaluations one at a time. Open an evaluation and select Run evaluation. ODC runs one evaluation at a time per agent. The three Claims Policy and Fraud Agent evaluations also share the same setup and teardown, so wait until a run finishes before you start the next one.
Review the results. The Runs tab of an evaluation shows each run’s status, its overall score, and its score per judge criterion. Select a run to see each case’s result. On a case, the Overview tab holds the judges’ reasoning, and the Outcomes tab shows the agent’s actual response and tool calls next to the expected ones.

For the Portal forms, the judge criteria, and the run statuses, refer to Create and manage datasets and Run your first evaluation.

Always configure both hooks. The full-analysis and Claims Document and Evidence cases create their own test records, marked with the EVAL- prefix. EvalSetup removes what earlier runs left behind, and EvalTeardown deletes the run’s test records and agent memory. Configure the hooks on the coverage and policy expert datasets too. Their cases keep agent memory under fixed session IDs, and the hooks clear it, so no run reads the answers of an earlier one. The first time it runs, the Claims Document and Evidence Agent’s EvalSetup also creates the customer EVAL-DE-0001 in Solutions Common Services. That customer owns the evaluation documents and is kept between runs.
Expect some Claims Document and Evidence cases to fail. The dataset is stricter than the agent’s current behavior:
- The agent can report an evidence type’s label instead of its code, for example Front Of Vehicle instead of FrontOfVehicle.
- It can place a damage on the wrong side of the vehicle.
- It reports an unexpected evidence type as InvalidEvidenceType, where the dataset expects UnexpectedEvidenceType.
- For some PDF documents, its answer can’t be read, and the case shows “The service action returned an unexpected response.”
Keep your first run as the baseline and compare later runs with it.
If a run fails at its setup step, check that AgentEval_Enabled is True in development for that agent, then check the logs of the agent’s app.
Re-run after every change. Run the evaluations again after you publish the agent, for example with another AI model, and compare the runs. ODC keeps the summary of every run, and each case’s details for 30 days. Change a dataset’s expected results only when the correct behavior changes on purpose.
Don’t publish during a run. Avoid publishing the agent or Claims Services while an evaluation of that agent runs.
Keep the evaluation actions off outside development. The evaluation actions are public service actions that create and delete test data, so keep AgentEval_Enabled set to False in every other stage.

Source & verification. Each agent summary is distilled from the system prompts in its agent module and checked against how the solution calls the module. The Claims Conversation Agent is at prompt version 0.3.30. The Claims Policy and Fraud Agent module holds three prompts. Its coverage inventory (when GetCoveragesFromPolicyDoc is on) and its policy expert (through the Claims Conversation Agent) are part of the live claim flow. The full policy and fraud analysis is reserved for the Decisioning phase, which is outside the solution’s scope.