Skip to content
March 30, 2026

Building a Prompt Evaluation System with Spring AI & Claude— Part 2

rajeshsgr · 7 min read · Containers

In this blog we will build a prompt evaluation system using Spring AI and Claude.The system automatically generates a dataset of Java functions, runs an explainer prompt against each one, grades the responses using an LLM-as-judge, and produces a formatted Excel report with scores and reasoning.

The goal is simple: measure prompt quality with real numbers and improve it through iteration.Architecture and Component Flow

The application is a Spring Boot command-line runner. There is no web server and no database.

source code on GitHub:

Tech stack

  • Runtime: Java 17
  • Framework: Spring Boot 3.2.5
  • AI client: Spring AI 1.0.0-M6 (Anthropic starter)
  • LLM: Claude Haiku — fast and cheap, ideal for eval loops
  • Excel output: Apache POI 5.2.5
  • Build: Maven 3.8+

Components and their roles

  • EvalRunner (runner) — CommandLineRunner that orchestrates the full evaluation pipeline
  • DatasetGenerator (service) — calls Claude to generate Java functions and ideal reference answers
  • ExplainerService (service) — the prompt under test; the only file you edit when iterating
  • GraderService (service) — LLM-as-judge; scores each explanation on four dimensions
  • ReportService (service) — aggregates scores and prints console summary
  • ExcelReportService (service) — writes the formatted .xlsx report
  • JavaFunction (model) — record holding name, code, difficulty, and ideal explanation
  • ExplanationResult (model) — record pairing a JavaFunction with the actual LLM response
  • GradeResult (model) — record with four dimension scores, reasons, overall score, and pass/fail
  • EvalReport (model) — aggregated averages, pass rate, and weakest dimension

The Evaluation Pipeline

The key design principle: the dataset and grader remain constant between runs — only the prompt changes.

  • Step 1 — Generate dataset: use Claude to create diverse test inputs (simple, medium, and complex Java functions) with ideal reference answers baked in.
  • Step 2 — Run the prompt under test: execute the explainer prompt on every dataset item and collect the raw LLM responses.
  • Step 3 — Grade each response: use Claude as an independent judge, scoring each dimension separately and returning structured JSON.
  • Step 4 — Aggregate and report: compute averages per dimension, identify the weakest, write the Excel report.

The four grading dimensions

  • Accuracy
  • Score 5: fully correct, no mistakes.
  • Score 3: one minor inaccuracy.
  • Score 1: wrong or misleading.
  • Simplicity
  • Score 5: zero jargon, a non-developer would understand it.
  • Score 3: one or two technical words used.
  • Score 1: heavy jargon throughout.
  • Completeness
  • Score 5: all inputs, outputs, and purpose covered.
  • Score 3: main point covered but misses a detail.
  • Score 1: vague or only partial.
  • Conciseness
  • Score 5: 1–3 tight sentences
  • Score 3: 4–6 sentences, could be shorter.
  • Score 1: seven or more sentences or heavily padded.

Scoring dimensions independently is the key design decision. A single overall score hides where the prompt fails. If simplicity averages 2.8 but accuracy averages 4.6, you know exactly what to fix — the prompt needs stronger jargon prevention, not accuracy improvements.

The LLM-as-judge pattern

The grader is itself a prompt sent to Claude with a strict rubric and a JSON output requirement. It receives three inputs:

  • The Java code — the original input the explainer received
  • The ideal explanation — generated alongside the dataset, used as a reference not the only acceptable answer
  • The actual explanation — what the explainer produced, this is what gets scored

The grader returns structured JSON:

{
  "accuracy":     { "score": 4, "reason": "Correct but omits the null check behaviour" },
  "simplicity":   { "score": 3, "reason": "Uses the word 'iterate' which is jargon" },
  "completeness": { "score": 5, "reason": "Covers input, output, and edge case" },
  "conciseness":  { "score": 5, "reason": "Two tight sentences, no padding" }
}

The grader uses the ideal explanation as a reference, not as the only correct answer. The actual explanation can use completely different wording and still score 5/5 as long as it meets the rubric criteria. This prevents the grader from penalising valid paraphrases.

Java File Walkthrough

DatasetGenerator.java

Sends a single prompt to Claude asking it to generate N Java functions spread across difficulty levels, each paired with an ideal explanation. Returns List<JavaFunction>.

  • Synthetic data first. Real production data is ideal long-term, but synthetic lets you start immediately, control difficulty spread precisely, and cover edge cases that rarely appear naturally.
  • Ideal explanation generated alongside the code. The reference answer is always semantically aligned with the function — no manual labelling needed.
  • Strict JSON output instruction. The system prompt ends with “Always respond with ONLY a valid JSON array. No markdown, no preamble.” This is the single most important instruction for reliable parsing.
  • stripFences() handles LLM markdown wrapping. Despite the instruction, models sometimes wrap JSON in code fences. The method detects the opening fence, removes it with replaceFirst("```[a-z]*\\n?"), then removes the closing fence using lastIndexOf to avoid accidentally removing content that contains backticks.
  • Error logging shows first and last 500 chars on parse failure. This immediately tells you whether the issue is truncation (last chars show mid-sentence) or malformed JSON.

ExplainerService.java

The only file you edit when iterating. Takes a JavaFunction, sends system prompt plus raw Java code to Claude, returns an ExplanationResult.

  • User message is just the raw code — no wrapper text. This mirrors real-world usage where a developer pastes code directly. All instruction lives in the system message.
  • Two prompt versions are provided. V1 is the active baseline with a basic instruction. V2 is commented out and adds explicit forbidden words, a one-to-two sentence limit, and a rule against starting with “This function” or “This method”. Uncommenting V2 and re-running shows you exactly what the stricter prompt buys in score terms.
  • The iteration rule: change only this file between runs. Keep the dataset, grader, and threshold identical. If both change at the same time you cannot know which change moved the score.

GraderService.java

Sends the original code, ideal explanation, and actual explanation to Claude with a rubric-based grading prompt. Parses the JSON response into a GradeResult with four independent dimension scores.

  • Explicit 1/3/5 anchors per dimension. Without them the model interpolates what each score means and introduces inconsistency across runs. The anchors are the most important part of the grader prompt.
  • JSON parsed with Jackson’s tree model. Uses root.path("accuracy").path("score").asInt(3) rather than strict object mapping. If a field is missing, asInt(3) returns a safe default of 3 rather than throwing an exception.
  • Fail-safe on parse error. If the response cannot be parsed at all, the method returns a GradeResult with all scores set to 1 rather than aborting the run. One bad API response does not kill the entire eval.
  • Overall score computed automatically. GradeResult.of() takes the four raw scores and computes the arithmetic mean. Pass/fail is determined by comparing the overall against the configurable threshold in application.yml.

ReportService.java

Takes the full List<GradeResult>, computes averages per dimension using Java streams, identifies the weakest dimension, and builds an EvalReport.

  • Weakest dimension found via stream min. Builds a Map<String, Double> of dimension name to average, then uses Comparator.comparingDouble to find the minimum entry. This name flows through to the Excel Summary sheet where it is highlighted in red — the action item for the next prompt iteration.
  • Per-function console output. Every function prints its actual explanation, each dimension score with reason, and overall score. This gives a quick read during development without opening the Excel file.

ExcelReportService.java

Creates a two-sheet Excel workbook. Sheet 1 is a summary. Sheet 2 is one row per function with all prompt, response, and grade detail side by side.

  • Sheet 2 columns (Detail): function name, difficulty, full prompt sent to LLM (system message and Java code — blue background), LLM response that was graded (amber background), ideal explanation used as reference (green background), then accuracy/simplicity/completeness/conciseness score and reason each in their own column, overall score, and pass/fail.
  • Score cells are colour coded. Green for score 4 or above (passes), amber for score 3 (borderline), red for score 2 or below (fails). The same scoreStyle() method is reused for all four dimension cells and the overall cell.
  • Full prompt shown in column C. The column shows both the system message and the Java code — not just the code. Anyone reading the report can see exactly what was sent to the LLM, making the scores auditable without needing to open the codebase.
  • Auto-filter enabled on all columns. You can sort by overall score, filter to show only failures, or filter by difficulty to see how scores vary between simple and complex functions.

Running It

# Set your API key — never stored in any file
export ANTHROPIC_API_KEY=sk-ant-...

# Run — Spring Boot starts, runs pipeline, exits
mvn spring-boot:run

After the run you get eval-report.xlsx in the project root. The Summary sheet shows the pass rate and highlights the weakest dimension. The Detail sheet shows the full prompt, LLM response, and scores for every function.

To iterate: open ExplainerService.java, uncomment V2 (or write V3), re-run, compare the average score for the weakest dimension against the previous run. The number tells you whether the change helped — not your intuition.

Summary

  • Use case: explain Java functions in plain English — reliably, at scale
  • Pipeline: generate dataset → run prompt → grade → report
  • Key pattern: LLM-as-judge with explicit rubrics and structured JSON output
  • Iteration: change only ExplainerService.java, keep everything else constant
  • Cost: approximately $0.001 per full run at Haiku pricing
  • Output: formatted Excel with full prompt, LLM response, and graded scores

Tags: , , ,

Leave a Reply