Model Roles

LLM’s are predicting machines, and in order to help them predict the next token better, they are fine-tuned on patterns to ensure a certain type of behaviour. Essentially what this means is that models will respond better when they are given input that matches those patterns.

In the case of conversational LLMs, this takes the form of “Roles”, essentially qualifiers of “Who is speaking”, these can help determine in the model where the model should be providing input vs. where in the prompt the user is providing input, and in some cases even where the system is pre-prompted with instructions.

For example in OpenAI the roles are:

  • user
  • assistant
  • system

And in Vicuna-based LLM’s it’s:

  • USER
  • ASSISTANT

While for Vicuna versions pre 1.1 it’s: ### USER ### ASSISTANT

And in SuperHOT models it might be:

  • Instruction
  • User

How does Montag use Roles?

In Montag all prompts are constructed of templates, and one of the variables that is injected into templates is the roles. By storing roles separately form the prompts it is possible to re-use prompts on models that have different role tags and encourages re-use and makes experimentation less error-prone.

For example this is a standard Q&A prompt:

{{.Instructions}}
{{.ModelRoles.User}} {{ if .ContextToRender }}Use the following information to answer my question concisely:{{ range $ctx := .ContextToRender }}
{{$ctx}}
{{ end }}{{ end }}

{{.ModelRoles.User}} Anwer this question based on the information provided above, do not generate code: {{.Body}}
{{.ModelRoles.AI}}

This prompt is generic because it uses the model roles template variables, and so we could quickly switch between models to find the one that responds best to benchmarking questions.

What are the roles available to template?

Each role type in Montag has a corresponding RoleEnd tag e.g. .SystemEnd, for cases where the underlying model requires additional syntactic sugar to close relevant sections of the prompt

  • System: A system instruction to tell the bot how to behave
  • Instruction: Similar to system, but could be used to further specialize the prompt
  • User: The User input
  • AI: Where the AI should respond

In the Q&A example above you’ll notice that the {{ModelRoles.AI}} variables sits on it’s own at the end, this is to prompt the LLM into a more accurate completion, as some open-source LLMs perform better when instructed where to respond.