artefactual.calibration

Calibration module for artefactual library.

This module exports configuration classes and training functions for calibration.

class artefactual.calibration.GenerationConfig(**data)

Bases: BaseModel

Configuration for entropy dataset generation.

iterations: int

log_to_file: bool

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

model_path: str

n_queries: int

number_logprobs: int

temperature: float

top_k_sampling: int

class artefactual.calibration.RatingConfig(**data)

Bases: BaseModel

Configuration for answer rating.

input_file: str | Path

judge_model_path: str

max_tokens: int

model_config: ClassVar[ConfigDict] = {}

Configuration for the model, should be a dictionary conforming to [ConfigDict][pydantic.config.ConfigDict].

seed: int

temperature: float

artefactual.calibration.clear_gpu_memory(llm)

Clears GPU memory by destroying the model parallel and distributed environment, deleting the LLM object, and clearing the CUDA cache.

This function is intended to be called after using a vLLM model to free up GPU resources. It handles the destruction of various components created by vLLM and PyTorch for distributed processing.

Return type:

None

Args:

llm (LLM): The vLLM LLM object to be deleted.

artefactual.calibration.get_model_name(model_path)

Return type:

str

artefactual.calibration.init_llm(model_path, seed)

Initialize a vLLM LLM instance.

Return type:

LLM

Args:

model_path: Path to the model or model identifier. seed: Random seed for initialization.

Returns:

An initialized LLM object.

artefactual.calibration.load_tqa_from_json(input_file)

Load the pack data from a JSON file.

Return type:

list[tuple[str, str, str, list[str]]]

Args:

input_file (str): Path to the JSON file

Returns:

List of (question, question_id, short_answer, answer_aliases) tuples.

artefactual.calibration.save_to_json(data, output_file)

Save data to a JSON file.

Supports two common use cases: - Dataset-level outputs: A single dict with metadata and results (e.g., entropy datasets) - Item-level outputs: A list of individual item dicts

Return type:

None

Args:

data: Data to save (dict or list of dicts). output_file: Path to the output JSON file.

artefactual.calibration.train_calibration(input_file, output_file)

Train a logistic regression model to calibrate uncertainty scores.

Return type:

None

Args:

input_file: Path to the CSV file containing ‘uncertainty_score’ and ‘judgment’. output_file: Path to save the calibration weights (JSON).

Modules

outputs_entropy	Generate a dataset with entropy scores for model outputs.
rates_answers
train_calibration(input_file, output_file)	Train a logistic regression model to calibrate uncertainty scores.