Skip to content
January 6, 2025

Structured outputs in LLM

· 3 min read · GenAI, Spring AI
  • LLMs are proficient at generating free-form and natural language text but for real world applications we will need consistent, and structured information.
  • An unstructured output from LLM might look : “There seems to be an issue with the order id 987654 placed by Customer ID 12345. The shipping address is not complete, and the product selected, ‘Book X’, is currently out of stock. We need to contact the customer to clarify the shipping information and confirm if they want a refund or wait for a week.“
  • Now imagine the same information getting produced in a structured format every time
{
  "order_id": "987654",
  "customer_id": "12345",
  "issues": [
    {
      "type": "shipping_address",
      "description": "Incomplete shipping address",
      "action_required": "Contact customer to clarify address"
    },
    {
      "type": "inventory",
      "description": "Book X out of stock",
      "action_required": "Contact customer to confirm a refund or wait"
    }
  ],
  "status": "pending_resolution"
}
  • This is where the concept of structured outputs from LLMs play an important role.
  • Structured outputs are responses produced by LLMs that follow a defined, predetermined format. This structure can vary and include forms such as: JSON, XML, Tabular Data etc.
  • Generating results in a structured format ensures that the information has a proper structure every time , make it easy for the consumer APIs, reporting tools or applications to process it
  • For an LLM to produce output in a structured format, a ‘predefined’ agreement or specification must be in place. This defines the expected format and structure of the output, which ensures a clear and consistent outcome
  • To understand how it works, lets see the steps involved in the process
  • Large Language Models (LLMs) typically generate text through a sequential process. They consider a series of potential ‘tokens’ (which can be words, parts of words, or punctuation) and assign probabilities to each token based on the likelihood of its occurrence in the given context. The LLM then selects the most probable token to generate the next part of the text, and this process repeats iteratively.
  • Generating text in a specific format, such as JSON, requires a more constrained approach. The LLM cannot simply select any token; it must adhere to the strict rules defining the target format. To achieve this, a Finite State Machine (FSM) framework is employed
  • A Finite State Machine (FSM) is a computational model used to guide Large Language Models (LLMs) in generating text that adheres to a specific structure, such as JSON or XML.

How FSMs work:

  • LLMs generate text token by token.
  • The FSM defines states representing different stages in the generation process.
  • Each state determines which tokens are valid based on the desired structure.
  • The FSM uses logit bias to adjust token probabilities, making invalid tokens less likely.
  • This ensures the output aligns perfectly with the expected format.

Building an FSM:

  • Define the desired output format using a JSON schema.
  • Construct a directed graph where nodes represent valid token sequences and edges represent valid transitions.

Ensuring Structured Output in LLMs

  • Filtering Tokens: The FSM checks if each token is valid for the current state.
  • Invalid tokens are assigned zero probability.
  • Efficiency:
  • Optimizations are crucial for fast and efficient processing, especially for long outputs.
  • Tools and techniques are used to speed up FSM construction and token filtering.
  • Generating structured outputs with LLMs can be achieved through various techniques like Prompting, Function Calling, Tool Calling and JSON Mode, each offering different levels of precision and flexibility. These methods help produce data in specific formats, ensuring that outputs are well-organized and consistent.

In the next blog, i will explain how it is done in Spring AI.

Thank You !

Tags: , , , ,

Leave a Reply