How to log raw calls to LLM?

[JP-droidic]

First of all, amazing library! Our app has been working with SK, but due to how slow things are moving, we are looking for a substitute. So far, LlmTornado has been fantastic.

A question: is there a way to log raw calls to an LLM - we want to see exactly what has been sent and received?

[lofcz]

Hi, there sure is. Which APIs are you using? We have two layers for logging.

[JP-droidic]

Gemini and Anthropic. The issue we facing with is as soon as tool is called it stops streaming and we want to see what got sent to LLM and what was received

[lofcz]

For streaming, I suppose you are using Conversation.StreamResponseRich? If not, please do that. There is very little overhead compared to using raw endpoints. Create your conversation, I'm calling it chat here:

await chat.StreamResponseRich(new ChatStreamEventHandler
{
    MessageTokenHandler = (token) =>
    {
        // whatever, optional
        Console.Write(token);
        return ValueTask.CompletedTask;
    },
    OnSse = (evt) =>
    {
        // when streaming, you don't get one response but a stream of server-sent events
        // this enables you to observe the raw incoming data
        return ValueTask.CompletedTask;
    },
    OutboundHttpRequestHandler = (request) =>
    {
        // this enables observing the outbound data, request has URL, body, and headers.
        return ValueTask.CompletedTask;
    }
    // note: add other handlers here
});

[lofcz]

Also, for the sake of completeness - the other way to log outbound data is constructing your own EndpointProvider, which is really simple:

TornadoApi api = new TornadoApi(new AnthropicEndpointProvider // or GoogleEndpointProvider for Gemini
{
    Auth = new ProviderAuthentication("YOUR_API_KEY"),
    RequestResolver = (request, data, streaming) =>
    {
         // log here, "request" has the payload serialized, "data" references the underlying object
    },
    RequestSerializer = (data, ctx) =>
    {
        // can be used to transform outbound requests if needed
    }
});

[JP-droidic]

Thank you so much such a great support. Highly appreciated