Optional chat completion mode

[tisnik]

Description

Optional chat completion mode

Type of change

• Refactor
• New feature
• Bug fix
• CVE fix
• Optimization
• Documentation Update
• Configuration Update
• Bump-up dependent library
• Bump-up library or tool used for development (does not change the final image)
• CI configuration change
• Konflux configuration change

[manstis]

IMO all interactions with llama-stack will be through an agent.

Here you're calling the inference provider directly.

"Chat" can be implemented like this:

agent = Agent(client, ...)
# [session/conversation persistence but this is something else for the future...]
# session_id = agent.create_session("test-session")

prompt = input("Enter a prompt: ")
response = agent.create_turn(
  messages=[UserMessage(
    role="user",
    content=prompt,
  )],
  session_id=session_id,
)

for log in EventLogger().log(response):
  log.print()

[manstis]

Using the inference provider on it's own means you're skipping the Agent's support for "safety/shields (Question validity, Answer redaction)" and RAG and other things. You'd be left to implement support for these things yourself - which was OK in road-core but is supported "out of the box" with llama-stack.