Skip to content
January 5, 2025

Optimizing AI Interactions in Spring AI using ChatResponseMetadata

· 5 min read · Containers, GenAI, Spring AI
  • As we build applications which interact with LLMs, it’s very important to understand the nuances of LLM interaction .
  • One of the essential aspects of such applications is monitoring and controlling the resource usage of the AI models, ensuring that the system operates within predefined limits.
  • Spring AI has ChatResponseMetadata class. This metadata offers valuable insights into your AI model’s usage and performance, enabling you to optimize your applications and ensure efficient resource utilization.
  • It provides following information
    • Rate Limits: Information about the rate limits imposed by the AI service, including the number of requests allowed, the remaining requests, and token limits.
    • Usage: Details about token consumption, including the number of tokens used for the prompt and the generated response.
    • Model Information: The specific AI model used for the interaction.
  • Why is ChatResponseMetadata Important?
    • Cost Optimization: By understanding the token consumption, you can optimize your prompts to be more concise, reducing costs associated with AI usage.
    • Performance Monitoring: Tracking rate limits allows you to proactively monitor and adjust your application’s interaction frequency with the AI model to avoid exceeding limits.
    • Application Improvement: Analyzing usage data can help you refine your AI interactions, leading to better user experiences and more efficient application performance.
    • Resource Planning: The metadata provides valuable data for planning and scaling your AI infrastructure based on anticipated usage patterns.

The following code snippet demonstrates how to access and utilize ChatResponseMetadata within a Spring application:

@GetMapping("/storyGen")
    public MetadataResponse chat(@RequestParam(defaultValue = "Tell a story in less than 100 words") String message) {

        // Step 1: Send the user-provided message to the AI model through the chatClient
        // If no message is provided, the default message "Tell a story in less than 100 words" is used.
        ChatResponse chatResponse = chatClient
                .prompt() // Start the prompt request
                .user(message)// Add the user's message to the prompt
                .call() // Make the API call to generate a response
                .chatResponse(); // Retrieve the response from the AI model

        // Step 2: Extract metadata from the chat response
        // The metadata includes important information such as rate limit status and token usage
        ChatResponseMetadata metaData = chatResponse.getMetadata();

        // Step 3: Retrieve rate limit information from metadata
        RateLimit rateLimit = metaData.getRateLimit();

        // Step 4: Retrieve usage information (tokens consumed by the prompt and generation)
        Usage usage = metaData.getUsage();

        // Step 5: Extract the generated response content from the AI model's output
        AssistantMessage output = chatResponse.getResult().getOutput();

        // Step 6: Create and return a MetadataResponse object containing the following data:
        // - Model used for the request
        // - Rate limit information (requests and tokens limits, remaining counts)
        // - Token usage details (prompt and generation tokens)
        // - The AI-generated content (output)
        MetadataResponse response = new MetadataResponse(
                metaData.getModel(),                   // AI model used to generate the response
                rateLimit.getRequestsLimit(),          // Maximum number of requests allowed
                rateLimit.getRequestsRemaining(),      // Number of requests remaining in the current limit window
                rateLimit.getTokensReset(),           // Time when the token limits will reset
                rateLimit.getTokensLimit(),           // Total number of tokens allowed
                rateLimit.getTokensRemaining(),       // Tokens remaining for use
                usage.getPromptTokens(),              // Tokens consumed by the input prompt
                usage.getGenerationTokens(),          // Tokens consumed by the AI model's response
                output.getContent()                   // AI-generated content (the output response)
        );

        // Return the MetadataResponse to the client with detailed usage information
        return response;

    }
  • The code interacts with an AI model using the chatClient.
  • Once the AI model has processed the user’s input, we extract the metadata from the ChatResponse object:
ChatResponseMetadata metaData = chatResponse.getMetadata();
  • ChatResponseMetadata contains key information about the AI response and the system’s current state, including important details about rate limiting, usage statistics, and model details.
  • Rate Limit Information
  • Rate limits are crucial for controlling how many requests can be made to the AI model within a certain time period. They prevent overuse of the model and ensure fair usage across users.
RateLimit rateLimit = metaData.getRateLimit();
  • The RateLimit object provides the following details:
  • Requests Limit (getRequestsLimit()): The maximum number of requests allowed within a specific time period.
  • Requests Remaining (getRequestsRemaining()): The number of requests remaining within the current rate limit window.
  • Tokens Reset (getTokensReset()): The time (timestamp) when the token limit will reset.
  • Tokens Limit (getTokensLimit()): The total number of tokens allowed within a given period.
  • Tokens Remaining (getTokensRemaining()): The number of tokens left to use before hitting the token limit.
  • Usage Information
  • Usage data provides insights into how many tokens have been consumed by the user’s request. Tokens are the fundamental units used by the AI models to process and generate text.
Usage usage = metaData.getUsage();
  • The Usage object includes:
  • Prompt Tokens (getPromptTokens()): The number of tokens used by the input prompt.
  • Generation Tokens (getGenerationTokens()): The number of tokens used by the model to generate the output.

These values are helpful for tracking how much of the allocated tokens have been consumed, and whether your usage is within the allowed limits. They are also useful for users who may be concerned about token consumption and costs.

  • AI Model Information
  • In some use cases, you may need to know which version or type of AI model generated the response. This is included in the metadata as:
metaData.getModel();

This allows you to track which AI model is responding to requests, which can be useful when you have multiple models deployed or are working with different versions of a model.

  • Once we’ve extracted the necessary metadata, we package it into a MetadataResponse object. This is the final output of the /storyGen endpoint:

The ChatResponseMetadata class in Spring AI offers key insights into the operational aspects of your AI-powered application. By tracking rate limits, token usage, and model information, you can ensure that your system is running efficiently, stay within resource limits, and optimize costs. Additionally, offering this metadata to end-users can increase transparency and improve the overall user experience.

By incorporating ChatResponseMetadata into your Spring Boot application, you are not just interacting with AI models; you are managing them effectively, ensuring they provide maximum value with optimal resource usage.

Source code – https://github.com/rajeshsgr/chatmetadata-spring-ai/tree/main

Tags: , , ,

Leave a Reply