artefactual.scoring.uncertainty_detector#

Classes

LogProbUncertaintyDetector([k])

A base class for uncertainty detection methods based on logprobs.

SentenceProbabilityDetector()

UncertaintyDetector()

A base class for uncertainty detection methods.
class artefactual.scoring.uncertainty_detector.LogProbUncertaintyDetector(k=15)[source]#

Bases: UncertaintyDetector

A base class for uncertainty detection methods based on logprobs.

__init__(k=15)[source]#

Initialize the uncertainty detector.

Args:
k: Number of top log probabilities to consider per token.

Must be positive. Default is 15.

Raises:

ValueError: If k is not positive

abstract compute_token_scores(inputs)[source]#

Compute token-level uncertainty scores from inputs.

Return type:

list[ndarray[tuple[int, ...], dtype[floating]]]

Args:

inputs: The inputs to process (e.g. completions or model outputs).

Returns:

The computed token-level scores.

class artefactual.scoring.uncertainty_detector.SentenceProbabilityDetector[source]#

Bases: UncertaintyDetector

abstract compute_token_scores(inputs)[source]#

Compute token-level uncertainty scores from sampled logprobs.

Return type:

list[ndarray[tuple[int, ...], dtype[floating]]]

Args:

inputs: The inputs to process (e.g. completions or model outputs).

Returns:

The computed token-level scores.

class artefactual.scoring.uncertainty_detector.UncertaintyDetector[source]#

Bases: ABC

A base class for uncertainty detection methods.

abstract compute(inputs)[source]#

Compute sequence-level uncertainty scores from inputs.

Return type:

list[float]

Args:

inputs: The inputs to process (e.g. completions or model outputs).

Returns:

The computed sequence-level scores.