artefactual.scoring.entropy_methods.epr

Classes

EPR([pretrained_model_name_or_path, k])

Computes Entropy Production Rate (EPR) from model log probabilities.

class artefactual.scoring.entropy_methods.epr.EPR(pretrained_model_name_or_path=None, k=15)

Bases: UncertaintyDetector

Computes Entropy Production Rate (EPR) from model log probabilities. EPR quantifies uncertainty based on the entropy of the model’s predicted token distributions. It calculates the entropy contributions of the top K predicted tokens at each position and averages these contributions over the sequence to produce a sequence-level uncertainty score. You can parse raw model outputs using the parse_model_outputs method from artefactual.preprocessing.

__init__(pretrained_model_name_or_path=None, k=15)

Initialize the EPR scorer.

Args:

pretrained_model_name_or_path: Model name or path to load calibration coefficients.

If not provided, the scorer returns raw uncalibrated scores and issues a warning.

k: Number of top log probabilities to consider (default: 15).

Raises:

ValueError: If calibration cannot be loaded from the provided valid path.

compute(parsed_logprobs)

Compute EPR-based uncertainty scores from parsed log probabilities. You can parse raw model outputs using the parse_model_outputs`method from `artefactual.preprocessing.

Return type:

list[float]

Args:

parsed_logprobs: Parsed log probabilities.

Returns:

List of sequence-level EPR scores.

compute_token_scores(parsed_logprobs)

Compute token-level EPR scores from parsed logprobs. You can parse raw model outputs using the parse_model_outputs`method from `artefactual.preprocessing.

Return type:

list[ndarray[tuple[int, ...], dtype[floating]]]

Args:

parsed_logprobs: Parsed log probabilities.

Returns:

List of token-level EPR scores (numpy arrays).