Skip to content
January 6, 2025

Streaming LLM response in Spring AI

  • In the context of AI-powered applications, streaming refers to the ability to receive and process data incrementally, without waiting for the entire response to be generated
  • This is especially useful when you’re interacting with AI services where responses can be long, or when you want to start displaying or using data as it’s being generated.
  • With Spring AI, the stream() method enables you to receive a continuous flow of data from the AI service.
  • Instead of waiting for the entire response to be ready, you can begin processing the incoming data as it arrives.
  • This makes it especially suitable for chatbots, real-time applications, or any other scenario where you need to interact with AI in an asynchronous manner.
  • When using stream(), you don’t wait for the entire response at once. Instead, you receive chunks of data as the AI model generates it.
  • The method stream() is available on the ChatClient and offers two primary ways to handle streaming responses:
  • Flux<String> content(): This method returns a Flux of Strings, which represents the content being generated by the AI model. It allows you to stream just the generated content (such as a message, response, or text output) directly.
Flux<String> output = chatClient.prompt()
    .user("Tell me a joke")
    .stream()
    .content();
  • Flux<ChatResponse> chatResponse(): Returns a Flux of the ChatResponse object, which contains additional metadata about the response
Flux<ChatResponse> response = chatClient.prompt()
    .user("Tell me a joke")
    .stream()
    .chatResponse();
  • The primary benefit of using the stream() method with Spring AI is asynchronous processing. This opens up multiple possibilities for enhancing the responsiveness and performance of your application:
    • Real-Time Responses: Instead of waiting for the AI to generate an entire response, you get chunks of data in real-time, allowing you to provide instant feedback to users.
    • Efficient Data Handling: Streaming is more efficient than waiting for large data sets to be returned in full. This minimizes memory usage and reduces the time the application spends waiting on the network or the AI model to process large requests.
    • Improved User Experience: By enabling real-time feedback, streaming makes the interaction feel faster and more dynamic. Users don’t have to wait for the entire response to load before they see results.
  • Flux<String> and Flux<ChatResponse> approach ensures that both your response and system metrics (rate limits, token usage) are efficiently handled in real-time using Spring AI’s reactive streaming capabilities.
  • The below given code sample has 2 methods, the first streams the output of the LLM. The second method has an example which streams the content generated and prints the metadata on the system console.

 @GetMapping("/storyGen")
    public Flux<String> storytWithStream(@RequestParam(defaultValue = "Tell a story in less than 100 words") String message) {
        // 1. Get a prompt object from the chat client
        return chatClient.prompt()
                // 2. Set the user's input as the prompt
                .user(message)
                // 3. Stream the response from the chat client
                .stream()
                // 4. Extract and return the content of the streamed response
                .content();
    }

    @GetMapping("/storyNmetaData")
    public Flux<String> storyWithStreamMetaData(@RequestParam(defaultValue = "Tell a story in less than 100 words") String message) {
        return chatClient.prompt()
                .user(message)
                .stream()
                .chatResponse()
                .doOnNext(chatResponse -> {
                    // Extract metadata from the ChatResponse
                    ChatResponseMetadata metadata = chatResponse.getMetadata();
                    RateLimit rateLimit = metadata.getRateLimit();
                    Usage usage = metadata.getUsage();

                    // Print metadata to the console
                    System.out.println("Model: " + metadata.getModel());
                    System.out.println("Requests Limit: " + rateLimit.getRequestsLimit());
                    System.out.println("Requests Remaining: " + rateLimit.getRequestsRemaining());
                    System.out.println("Tokens Reset: " + rateLimit.getTokensReset());
                    System.out.println("Tokens Limit: " + rateLimit.getTokensLimit());
                    System.out.println("Tokens Remaining: " + rateLimit.getTokensRemaining());
                    System.out.println("Prompt Tokens: " + usage.getPromptTokens());
                    System.out.println("Generation Tokens: " + usage.getGenerationTokens());
                })
                // Map the Flux<ChatResponse> to Flux<String> to return only the content
                .map(chatResponse -> chatResponse.getResult().getOutput().getContent());


    }

Source Code : https://github.com/rajeshsgr/stream-demo-spring-ai/tree/main

Thanks

Tags: , , , , ,

Leave a Reply