Skip to main content
Version: 0.0.0

Output log probabilities

When using logprobs the LLM outputs the log probabilities of each output token during generation.

There are two relevant parameters for this mode:

  • logprobs: default=False - When set to True the LLM outputs the log probabilities of each output token during generation.
  • top_logprobs: default=None - When set to an integer value, the LLM outputs the log probabilities of the top top_logprobs most likely tokens at each token position during generation. top_logprobs must be between 0 and 5.
warning

Anyscale doesn't support meta-llama/Llama-2-70b-chat-hf and meta-llama/Llama-2-13b-chat-hf.

Example

curl "$OPENAI_BASE_URL/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
"messages": [{"role": "system", "content": "You are a helpful assistant."}, {"role": "user", "content": "Say 'Test'."}],
"temperature": 0.7,
"logprobs": true,
"top_logprobs": 1
}'

Example output:

{"id":"mistralai/Mixtral-8x7B-Instruct-v0.1",
"object":"text_completion",
"created":1705450393,
"model":"mistralai/Mixtral-8x7B-Instruct-v0.1",
"choices": [{"message":
{"role":"assistant",
"content":"Test.",
"tool_calls":null,
"tool_call_id":null},
"index":0,
"finish_reason":"stop",
"logprobs":{"content":
[
{"token":"Test",
"logprob":-0.12771208584308624,
"bytes":[84,101,115,116],
"top_logprobs": [
{"logprob":-0.12771208584308624,
"token":"Test",
"bytes":[84,101,115,116]
}
]
},
{"token":".",
"logprob":-0.0008685392094776034,
"bytes":[46],
"top_logprobs": [
{"logprob":-0.0008685392094776034,
"token":".",
"bytes":[46]
}
]
},
{"token":"",
"logprob":0.0,
"bytes":[],
"top_logprobs":[
{"logprob":0.0,
"token":"",
"bytes":[]
}
]
}
]
}
}],
"usage": {"prompt_tokens": 26,
"completion_tokens": 3,
"total_tokens":29
}
}