artefactual.scoring.entropy_methods

Entropy methods package exports.

Expose commonly used functions and classes at the package level so users don’t need to import from deep submodules.

class artefactual.scoring.entropy_methods.EPR(pretrained_model_name_or_path=None, k=15)

Bases: LogProbUncertaintyDetector

Computes Entropy Production Rate (EPR) from model log probabilities. EPR quantifies uncertainty based on the entropy of the model’s predicted token distributions. It calculates the entropy contributions of the top K predicted tokens at each position and averages these contributions over the sequence to produce a sequence-level uncertainty score. You can parse raw model outputs using the parse_top_logprobs method from artefactual.preprocessing.

__init__(pretrained_model_name_or_path=None, k=15)

Initialize the EPR scorer.

Args:

pretrained_model_name_or_path: Model name or path to load calibration coefficients.

If not provided, the scorer returns raw uncalibrated scores and issues a warning.

k: Number of top log probabilities to consider (default: 15).

Raises:

ValueError: If calibration cannot be loaded from the provided valid path.

compute(parsed_logprobs)

Compute EPR-based uncertainty scores from parsed log probabilities. You can parse raw model outputs using the parse_top_logprobs method from artefactual.preprocessing.

Return type:

list[float]

Args:

parsed_logprobs: Parsed log probabilities.

Returns:

List of sequence-level EPR scores.

compute_token_scores(parsed_logprobs)

Compute token-level EPR scores from parsed logprobs. You can parse raw model outputs using the parse_top_logprobs method from artefactual.preprocessing.

Return type:

list[ndarray[tuple[int, ...], dtype[floating]]]

Args:

parsed_logprobs: Parsed log probabilities.

Returns:

List of token-level EPR scores (numpy arrays).

class artefactual.scoring.entropy_methods.WEPR(pretrained_model_name_or_path)

Bases: LogProbUncertaintyDetector

Computes Weighted Entropy Production Rate (WEPR) from model log probabilities. WEPR extends EPR by applying learned weights to the entropy contributions based on their ranks. It computes both mean-weighted and max-weighted contributions to produce a sequence-level uncertainty score. Token-level WEPR scores are also provided. You can parse raw model outputs using the parse_top_logprobs method from artefactual.preprocessing.

__init__(pretrained_model_name_or_path)

Initialize the WEPR scorer with weights loaded from the specified source.

Args:

pretrained_model_name_or_path: Either a built-in model name or a local file path to load weights from.

compute(parsed_logprobs)

Compute WEPR-based uncertainty scores from parsed log probabilities. You can parse raw model outputs using the parse_top_logprobs method from artefactual.preprocessing.

Return type:

list[float]

Args:

parsed_logprobs: Parsed log probabilities.

Returns:

List of sequence-level WEPR scores.

compute_token_scores(parsed_logprobs)

Compute token-level WEPR scores from parsed logprobs. You can parse raw model outputs using the parse_top_logprobs method from artefactual.preprocessing.

Return type:

list[ndarray[tuple[int, ...], dtype[floating]]]

Args:

parsed_logprobs: Parsed log probabilities.

Returns:

List of token-level WEPR scores (numpy arrays).

artefactual.scoring.entropy_methods.compute_entropy_contributions(logprobs, k)

Return type:

ndarray[tuple[int, ...], dtype[floating]]

Compute entropic contributions s_kj = -p_k log(p_k) for top-K logprobs using vectorized operations. Args:

logprobs: A 2D array of shape (num_tokens, num_logprobs) containing log probabilities. k: Number of top log probabilities to consider per token.

Returns:

A 2D array of shape (num_tokens, K) containing entropy contributions.

Modules

entropy_contributions
epr
wepr