Docs

Docs

LLM Providers

Connect the AIOrchestrator to Spring AI, LangChain4j, or a custom LLM framework using the LLMProvider interface.

An AI framework — such as Spring AI or LangChain4j — is a Java library that handles the protocol details of talking to LLM services like OpenAI or Anthropic. The orchestrator plugs into your chosen framework through the LLMProvider interface. Create a provider by instantiating the appropriate implementation directly. Two implementations are provided: one for Spring AI and one for LangChain4j.

Important
Memory Window Limit
Both built-in providers maintain a 30-message memory window. Older messages are evicted from the provider’s working memory. The orchestrator’s getHistory() retains the full conversation, but the LLM only sees the most recent 30 messages.

Spring AI

SpringAILLMProvider supports both streaming and synchronous Spring AI models.

Source code
Java
// From ChatModel - use an implementation of Spring AI ChatModel
ChatModel chatModel = OpenAiChatModel.builder()
    .openAiClient(...).options(...).build();
SpringAILLMProvider provider = new SpringAILLMProvider(chatModel);

// From ChatClient - use a Spring AI ChatClient
ChatClient chatClient = ChatClient.builder(...)
    .defaultAdvisors(...).build();
SpringAILLMProvider provider = new SpringAILLMProvider(chatClient);

When created from a ChatModel, the provider manages its own conversation memory using a 30-message window. When created from a ChatClient, memory must be configured externally on the client.

Streaming is enabled by default. To disable it, call setStreaming(false):

Source code
Java
provider.setStreaming(false);

In synchronous mode, the whole exchange runs in the request that triggered it and blocks the UI until the response is complete. See Background Execution for keeping the UI responsive during long prompts.

Note
History Restoration with ChatClient
A provider created from a ChatModel restores the conversation into its own memory, so withHistory() and reconnect() need no extra work. A provider created from a ChatClient cannot do that — the application owns that client’s memory — so its setHistory() does nothing beyond logging what it observed: a warning when the client carries no chat memory advisor or no default conversation id, since a restored conversation then never reaches the LLM. Load the conversation into the client’s own ChatMemory before passing the client to the provider, or use new SpringAILLMProvider(chatModel) and let the provider handle it. The orchestrator’s own conversation history and the Message List are restored either way.

LangChain4j

LangChain4JLLMProvider supports both streaming and synchronous LangChain4j models. The mode is determined by the model type passed to the constructor:

Source code
Java
// Streaming mode - use an implementation of LangChain4j StreamingChatModel
StreamingChatModel streamingChatModel = OpenAiStreamingChatModel.builder()
    .apiKey(...).modelName(...).build();
LangChain4JLLMProvider provider = new LangChain4JLLMProvider(streamingChatModel);

// Synchronous mode - use an implementation of LangChain4j ChatModel
ChatModel chatModel = OpenAiChatModel.builder()
    .apiKey(...).modelName(...).build();
LangChain4JLLMProvider provider = new LangChain4JLLMProvider(chatModel);

The provider manages its own conversation memory using a 30-message window.

Synchronous mode blocks the UI for the duration of each exchange; see Background Execution.

Background Execution

A synchronous provider produces the response on the thread that asks for it. For a prompt sent from the browser, that is the request thread: the request does not return until the model has produced the complete response, including any tool calls along the way. The interface freezes for the whole wait, and since the request holds the session lock, other views in the same session wait too.

Background execution moves the exchange to a background thread instead. Enable it on either built-in provider:

Source code
Java
provider.setBackgroundExecution(true);

The user’s message and an empty assistant message then appear immediately, the UI stays responsive, and the response is filled in when the model finishes. The setting is off by default. It’s read for each prompt, so it can be changed at any time; the next prompt uses the new mode. It has no effect with a streaming model, whose response already arrives on the LLM client’s own threads.

The response is now produced outside any request, so it reaches the browser through server push or polling. Enable push by annotating the application shell with @Push (see Server Push), or enable polling with UI.setPollInterval(). Without either, the response only shows up with the next request the browser happens to make — the page looks stuck even though the turn completed on the server. The provider logs a warning, once per provider instance, when neither is active. Manual push mode is not enough on its own, because nothing in the framework calls ui.push() for the application.

Everything that happens before the model is called still runs in the request thread: the request interceptor, adding the user’s message and the empty assistant message to the Message List, AIController.onRequest(), the request listener, and the session context supplier. The model calls, every tool execution, and the ResponseListener run on the background thread, where UI.getCurrent() and other Vaadin thread locals return null and components must not be touched directly. Thread-bound framework state, such as Spring Security’s SecurityContext, is absent there for the same reason.

Wrap component access in ui.access(), or capture what a tool needs in AIController.onRequest() while the request thread is still current — see Tool Calling & Programmatic Prompts and Controllers. The built-in controllers already handle this. AIController.onResponse() is the exception: the orchestrator calls it through ui.access(), so it can update components directly.

Note
One Prompt at a Time
The orchestrator processes one prompt at a time. In the default synchronous mode, a message submitted while a turn is running waits for the session lock and is processed when the turn ends. With background execution the lock is free, so the same message is rejected and dropped with a server-side warning — and the Message Input has already cleared its text.

If the user closes or reloads the browser tab while a turn is running, the turn still completes on the server: the response is recorded in the conversation history and the ResponseListener fires as usual. Only the UI updates are skipped, along with AIController.onResponse(), which needs an attached UI. The setting itself lives on the provider and is not serialized with the session — apply it again to the recreated provider after a session restore, before passing it to reconnect(). See Conversation History & Session Persistence.

Custom LLM Providers

Implement the LLMProvider interface to connect to any LLM framework:

Source code
Java
public class MyLLMProvider implements LLMProvider {

    @Override
    public Flux<String> stream(LLMRequest request) {
        // Return a reactive stream of response tokens
        // request.userMessage()  -- the user's prompt
        // request.attachments()  -- any file attachments
        // request.systemPrompt() -- the system prompt
        // request.tools()        -- registered tool objects
        // request.metadataSink() -- consumer for response metadata
    }

    @Override
    public void setHistory(List<ChatMessage> history,
            Map<String, List<AIAttachment>> attachmentsByMessageId) {
        // Restore conversation context
    }
}

The response stream carries text only. The provider can also publish the finish reason and token usage of the turn through the metadataSink() consumer on the request. Each call carries everything observed so far and replaces the value of any earlier call, so publish whenever the provider learns more — a turn that fails midway has then still reported what was observed. Pass null for any value the framework doesn’t report; a provider that observes no metadata never calls the consumer. See Response Metadata for how applications read it.

The orchestrator calls stream() on the thread that triggers the prompt and subscribes to the returned stream on that same thread — whether a turn runs in the background is decided entirely by the implementation. An implementation whose LLM call blocks should schedule that call itself; otherwise it occupies the request thread and holds the session lock for the whole turn. See Background Execution for how the built-in providers expose this as a setting.

Updated –