How to get log probabilities
Prerequisites
This guide assumes familiarity with the following concepts:
Certain chat models can be configured to return token-level log probabilities representing the likelihood of a given token. This guide walks through how to get this information in LangChain.
OpenAI
Install the @langchain/openai package and set your API key:
- npm
- yarn
- pnpm
npm i @langchain/openai @langchain/core
yarn add @langchain/openai @langchain/core
pnpm add @langchain/openai @langchain/core
For the OpenAI API to return log probabilities, we need to set the
logprobs param to true. Then, the logprobs are included on each
output
AIMessage
as part of the response_metadata:
import { ChatOpenAI } from "@langchain/openai";
const model = new ChatOpenAI({
model: "gpt-4o",
logprobs: true,
});
const responseMessage = await model.invoke("how are you today?");
responseMessage.response_metadata.logprobs.content.slice(0, 5);
[
{
token: "Thank",
logprob: -0.70174205,
bytes: [ 84, 104, 97, 110, 107 ],
top_logprobs: []
},
{
token: " you",
logprob: 0,
bytes: [ 32, 121, 111, 117 ],
top_logprobs: []
},
{
token: " for",
logprob: -0.000004723352,
bytes: [ 32, 102, 111, 114 ],
top_logprobs: []
},
{
token: " asking",
logprob: -0.0000013856493,
bytes: [
32, 97, 115,
107, 105, 110,
103
],
top_logprobs: []
},
{
token: "!",
logprob: -0.00030102333,
bytes: [ 33 ],
top_logprobs: []
}
]
And are part of streamed Message chunks as well:
let count = 0;
const stream = await model.stream("How are you today?");
let aggregateResponse;
for await (const chunk of stream) {
if (count > 5) {
break;
}
if (aggregateResponse === undefined) {
aggregateResponse = chunk;
} else {
aggregateResponse = aggregateResponse.concat(chunk);
}
console.log(aggregateResponse.response_metadata.logprobs?.content);
count++;
}
[]
[
{
token: "Thank",
logprob: -0.23375113,
bytes: [ 84, 104, 97, 110, 107 ],
top_logprobs: []
}
]
[
{
token: "Thank",
logprob: -0.23375113,
bytes: [ 84, 104, 97, 110, 107 ],
top_logprobs: []
},
{
token: " you",
logprob: 0,
bytes: [ 32, 121, 111, 117 ],
top_logprobs: []
}
]
[
{
token: "Thank",
logprob: -0.23375113,
bytes: [ 84, 104, 97, 110, 107 ],
top_logprobs: []
},
{
token: " you",
logprob: 0,
bytes: [ 32, 121, 111, 117 ],
top_logprobs: []
},
{
token: " for",
logprob: -0.000004723352,
bytes: [ 32, 102, 111, 114 ],
top_logprobs: []
}
]
[
{
token: "Thank",
logprob: -0.23375113,
bytes: [ 84, 104, 97, 110, 107 ],
top_logprobs: []
},
{
token: " you",
logprob: 0,
bytes: [ 32, 121, 111, 117 ],
top_logprobs: []
},
{
token: " for",
logprob: -0.000004723352,
bytes: [ 32, 102, 111, 114 ],
top_logprobs: []
},
{
token: " asking",
logprob: -0.0000029352968,
bytes: [
32, 97, 115,
107, 105, 110,
103
],
top_logprobs: []
}
]
[
{
token: "Thank",
logprob: -0.23375113,
bytes: [ 84, 104, 97, 110, 107 ],
top_logprobs: []
},
{
token: " you",
logprob: 0,
bytes: [ 32, 121, 111, 117 ],
top_logprobs: []
},
{
token: " for",
logprob: -0.000004723352,
bytes: [ 32, 102, 111, 114 ],
top_logprobs: []
},
{
token: " asking",
logprob: -0.0000029352968,
bytes: [
32, 97, 115,
107, 105, 110,
103
],
top_logprobs: []
},
{
token: "!",
logprob: -0.00039694557,
bytes: [ 33 ],
top_logprobs: []
}
]
topLogprobs
To see alternate potential generations at each step, you can use the
topLogprobs parameter:
const modelWithTopLogprobs = new ChatOpenAI({
model: "gpt-4o",
logprobs: true,
topLogprobs: 3,
});
const res = await modelWithTopLogprobs.invoke("how are you today?");
res.response_metadata.logprobs.content.slice(0, 5);
[
{
token: "I'm",
logprob: -2.2864406,
bytes: [ 73, 39, 109 ],
top_logprobs: [
{
token: "Thank",
logprob: -0.28644064,
bytes: [ 84, 104, 97, 110, 107 ]
},
{
token: "Hello",
logprob: -2.0364406,
bytes: [ 72, 101, 108, 108, 111 ]
},
{ token: "I'm", logprob: -2.2864406, bytes: [ 73, 39, 109 ] }
]
},
{
token: " just",
logprob: -0.14442946,
bytes: [ 32, 106, 117, 115, 116 ],
top_logprobs: [
{
token: " just",
logprob: -0.14442946,
bytes: [ 32, 106, 117, 115, 116 ]
},
{ token: " an", logprob: -2.2694294, bytes: [ 32, 97, 110 ] },
{
token: " here",
logprob: -4.0194297,
bytes: [ 32, 104, 101, 114, 101 ]
}
]
},
{
token: " a",
logprob: -0.00066632946,
bytes: [ 32, 97 ],
top_logprobs: [
{ token: " a", logprob: -0.00066632946, bytes: [ 32, 97 ] },
{
token: " lines",
logprob: -7.750666,
bytes: [ 32, 108, 105, 110, 101, 115 ]
},
{ token: " an", logprob: -9.250667, bytes: [ 32, 97, 110 ] }
]
},
{
token: " computer",
logprob: -0.015423919,
bytes: [
32, 99, 111, 109,
112, 117, 116, 101,
114
],
top_logprobs: [
{
token: " computer",
logprob: -0.015423919,
bytes: [
32, 99, 111, 109,
112, 117, 116, 101,
114
]
},
{
token: " program",
logprob: -5.265424,
bytes: [
32, 112, 114, 111,
103, 114, 97, 109
]
},
{
token: " machine",
logprob: -5.390424,
bytes: [
32, 109, 97, 99,
104, 105, 110, 101
]
}
]
},
{
token: " program",
logprob: -0.0010724656,
bytes: [
32, 112, 114, 111,
103, 114, 97, 109
],
top_logprobs: [
{
token: " program",
logprob: -0.0010724656,
bytes: [
32, 112, 114, 111,
103, 114, 97, 109
]
},
{
token: "-based",
logprob: -6.8760724,
bytes: [ 45, 98, 97, 115, 101, 100 ]
},
{
token: " algorithm",
logprob: -10.626073,
bytes: [
32, 97, 108, 103,
111, 114, 105, 116,
104, 109
]
}
]
}
]
Next steps
You’ve now learned how to get logprobs from OpenAI models in LangChain.
Next, check out the other how-to guides chat models in this section, like how to get a model to return structured output or how to track token usage.