How does conversation history work?

[jimgoo]

Very fun project :)

I couldn't quite figure out from a quick first read of the code how to use conversation history. Like if I run this, it doesn't know what I just asked:

r = lf.query('Who is Larry Page', lm=lm)
print(r)
# Larry Page is a prominent American computer scientist ...

r2 = lf.query('What did I just ask?', lm=lm)
print(r2)
# I'm sorry, but I can't recall or access previous interactions. 

[jimgoo]

Might have answered my own question, but let me know if there is a more preferred method:

import langfun as lf
from langfun.core.templates.conversation import Conversation

class Answer(pg.Object):
  result: int

lm = lf.llms.Gpt4o()

c = Conversation(
  preamble='You are a helpful and joyful chat bot. Let us chat.',
)

input_message = 'The result of one plus two is'
r = lf.query(c.prompt.render(input_message=input_message), Answer, lm=lm, returns_message=True)
# Answer(result=3)

# update memory
c.add(input_message, r)

input_message = 'What did I just say?'
r2 = lf.query(c.prompt.render(input_message=input_message), lm=lm, returns_message=True)
# "you just asked me the result of one plus two..."

It seems like under the hood, when making the call to the LLM provider, this Conversation template puts all of the turns into one single message with the user role instead of multiple messages with alternating roles?

[daiyip]

Hi Jimmie, excellent question!

You're absolutely right that Conversation is currently the closest option for achieving what you're looking for. At present, Langfun doesn’t provide a way for users to attach historical messages in the style of LLM-specific chat APIs (such as OpenAI's message list with roles). This aligns with our initial focus on low-level LLM interactions, which are common in research projects.

That said, I’ve been considering designs for offering high-level LLM access, especially as more users are utilizing Langfun to develop agents. Certain features, like applying the prefix cache, rely on having such access, so it's definitely a priority for us.

One early idea is to support special markup in templates, such as {{markup.Role('system')}} or {{markup.PrefixCache(ttl=100)}}. These markups could then be translated into the appropriate representations for different LLM providers, making the system more flexible and user-friendly.

Stay tuned and I will be updating this thread once it's ready.