Skip to main content
Version: 0.0.0

JSON mode

JSON Mode enables the LLM to produce a JSON-formatted responses, useful when you want to integrate the LLM with other systems that expect a specific JSON schema.

You can set the JSON schema by setting the response_format = {"type": "json_object"} parameter in the POST request to the LLM. You can optionally provide a schema by setting the response_format = {"type": "json_object", "schema": <schema>} parameter. If you don't provide a schema, the LLM outputs any JSON object that fits.

Supported models

The following models support JSON mode:

warning

top_p and top_k sampling parameters are incompatible with JSON mode.

Example

curl "$OPENAI_BASE_URL/chat/completions" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $OPENAI_API_KEY" \
-d '{
"model": "mistralai/Mixtral-8x7B-Instruct-v0.1",
"messages": [{"role": "system", "content": "You are a helpful assistant that outputs in JSON."}, {"role": "user", "content": "Who won the world series in 2020"}],
"response_format": {"type": "json_object", "schema": {"type": "object", "properties": {"team_name": {"type": "string"}}, "required": ["team_name"]}},
"temperature": 0.7
}'

The output will look like:

{'choices': [{'finish_reason': 'stop',
'index': 0,
'message': {'content': ' {\n'
'"team_name": " Los Angeles Dodgers "\n'
'}',
'function_call': None,
'role': 'assistant',
'tool_calls': None}}],
'created': 1701933511,
'id': 'mistralai/Mixtral-8x7B-Instruct-v0.1-4-7sX-l4EqKMl0FxZu7f5HQ8DXIKMIQgCkT1sWGLwn8',
'model': 'mistralai/Mixtral-8x7B-Instruct-v0.1',
'object': 'text_completion',
'system_fingerprint': None,
'usage': {'completion_tokens': 16, 'prompt_tokens': 30, 'total_tokens': 46}}
tip

Not explicitly instructing the model to output in JSON is prone to outputting a sting of whitespaces before and after the JSON object, costing more tokens and time. To prevent this issue Anyscale raises an error if the key json isn't found in your message, insensitive to case.

Streaming JSON mode completions

You can also use the streaming API to get JSON mode completions. Here's an example using the Python SDK:

import openai

client = openai.OpenAI(
base_url = "https://api.endpoints.anyscale.com/v1",
api_key = "esecret_YOUR_API_KEY")

# Note: not all arguments are currently supported and will be ignored by the backend.
chat_completion = client.chat.completions.create(
model="mistralai/Mixtral-8x7B-Instruct-v0.1",
messages=[
{"role": "system", "content": "You are a helpful assistant that outputs in JSON."},
{"role": "user", "content": "Who won the world series in 2020"}
],
response_format={
"type": "json_object",
"schema": {
"type": "object",
"properties": {
"team_name": {"type": "string"}
},
"required": ["team_name"]
},
},
temperature=0.7,
stream=True
)

for chunk in chat_completion:
print(chunk)

Example output:

ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, role='assistant', tool_calls=None), finish_reason=None, index=0, logprobs=ChoiceLogprobs(content=[]))], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content=' {', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='"', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='team', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='_', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='name', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='":', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content=' "', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='Los', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content=' Angeles', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content=' Dod', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='gers', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='"', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809316, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='}', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809317, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809318, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='\n', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809318, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content='', function_call=None, role=None, tool_calls=None), finish_reason=None, index=0, logprobs=None)], created=1706809318, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage=None)
ChatCompletionChunk(id='mistralai/Mixtral-8x7B-Instruct-v0.1-ssaOTZXxxoS44tiPeIhJWcYKDnKLy3Lnxk20OzxAuwM', choices=[Choice(delta=ChoiceDelta(content=None, function_call=None, role=None, tool_calls=None), finish_reason='stop', index=0, logprobs=None)], created=1706809318, model='mistralai/Mixtral-8x7B-Instruct-v0.1', object='text_completion', system_fingerprint=None, usage={'prompt_tokens': 30, 'completion_tokens': 28, 'total_tokens': 58})