gen_ai_hub.evaluations.helpers package

Submodules

gen_ai_hub.evaluations.helpers.collector module

class ValidationCollector

Bases: object

A class to collect and manage validation errors encountered during the orchestration configuration validation process.

__init__()
add_error(internal_code, message)

Adds an error to the collector. Args:

internal_code (str): The internal error code. message (str): The detailed error message.

has_error_code(code)

checks if error code exists in the list of errors

has_errors()

checks if the collector collected any errors

raise_if_errors()

Raises a ValidationError if there are any collected errors.

gen_ai_hub.evaluations.helpers.config_data module

build_accumulated_config(evaluation_configs_data, has_mixed_config_types=False)

Build accumulated configuration data and determine execution flow strategy.

Analyzes evaluation configurations to determine whether they can be executed as a single job (if datasets and metrics match) or require multiple executions. Also determines if artifacts can be reused across executions.

Parameters:

  • evaluation_configs_data (List[_EvaluationConfigData]) -- List of extracted evaluation configuration data objects

  • has_mixed_config_types (bool) -- Whether evaluation configs have mixed types (llm+template and orchestration_registry)

Returns:

Tuple containing: - accumulated_config_data: Either a single accumulated config or list of configs - single_execution_flow: True if all configs can be executed as one job - reusable_artifact: True if dataset artifact can be reused across executions

Return type:

Tuple[Union[List[_EvaluationConfigData], _EvaluationConfigData], bool, bool]

extract_config_data(evaluation_configs, ai_core_client, object_store_credentials, resource_group, gen_ai_hub_proxy_client, error_collector)

Extract configuration data from user-provided evaluation configs.

This function processes evaluation configurations to extract orchestration config, dataset data, metric templates, and variable mappings for each configuration.

Parameters:

  • evaluation_configs (List[EvaluationConfig]) -- List of evaluation configuration objects to process

  • ai_core_client (AICoreV2Client) -- AI Core V2 client for API interactions

  • object_store_credentials (_AWSObjectStoreData) -- Credentials for accessing object storage (AWS S3)

  • resource_group (str) -- AI Core resource group name

  • gen_ai_hub_proxy_client (GenAIHubProxyClient) -- GenAI Hub proxy client for orchestration operations

  • error_collector (ValidationCollector) -- Collector for validation errors

Returns:

List of extracted evaluation configuration data objects

Return type:

List[_EvaluationConfigData]

gen_ai_hub.evaluations.helpers.evaluation_optimization_flow module

configure_dataset_artifact(evaluation_configs, accumulated_config_data, input_object_store_secret_name, object_store_credentials, ai_core_client, resource_group, error_collector)
Parameters:

  • evaluation_configs (List[EvaluationConfig])

  • accumulated_config_data (_EvaluationConfigData)

  • input_object_store_secret_name (str)

  • object_store_credentials (_AWSObjectStoreData)

  • ai_core_client (AICoreV2Client)

  • resource_group (str)

  • error_collector (ValidationCollector)

configure_orchestration_config_for_simplified_executable(evaluation_configs)
Parameters:

evaluation_configs (List[EvaluationConfig])

multiple_evaluation_jobs_flow(evaluation_configs, accumulated_config_data, input_object_store_secret_name, object_store_credentials, ai_core_client, resource_group, orchestration_url, reusable_artifact, error_collector)
Parameters:

  • evaluation_configs (List[EvaluationConfig])

  • accumulated_config_data (List[_EvaluationConfigData])

  • input_object_store_secret_name (str)

  • object_store_credentials (_AWSObjectStoreData)

  • ai_core_client (AICoreV2Client)

  • resource_group (str)

  • orchestration_url (str)

  • reusable_artifact (bool)

  • error_collector (ValidationCollector)

single_evaluation_job_flow(evaluation_configs, accumulated_config_data, input_object_store_secret_name, object_store_credentials, ai_core_client, resource_group, orchestration_url, error_collector)
Parameters:

  • evaluation_configs (List[EvaluationConfig])

  • accumulated_config_data (_EvaluationConfigData)

  • input_object_store_secret_name (str)

  • object_store_credentials (_AWSObjectStoreData)

  • ai_core_client (AICoreV2Client)

  • resource_group (str)

  • orchestration_url (str)

  • error_collector (ValidationCollector)

upload_dataset_data_and_register_aicore_artifact(input_object_store_secret_name, accumulated_config_data, object_store_credentials, ai_core_client, resource_group, error_collector)
Parameters:

  • input_object_store_secret_name (str)

  • accumulated_config_data (_EvaluationConfigData)

  • object_store_credentials (_AWSObjectStoreData)

  • ai_core_client (AICoreV2Client)

  • resource_group (str)

  • error_collector (ValidationCollector)

gen_ai_hub.evaluations.helpers.logging module

get_logger(name=None)
Parameters:

name (str)

set_log_level(logger, default_level=20)
Parameters:

logger (Logger)

gen_ai_hub.evaluations.helpers.s3_file_client module

class S3FileClient

Bases: object

S3 client for read/write file operations with format-specific parsing.

__init__(bucket_name, region=None, aws_access_key_id=None, aws_secret_access_key=None, error_collector=None)

Initialize S3 client with flexible authentication options.

Parameters:

  • bucket_name (str) -- S3 bucket name

  • region (str, optional) -- AWS region (defaults to boto3 default), defaults to None

  • aws_access_key_id (str, optional) -- AWS access key (optional if using IAM/profile), defaults to None

  • aws_secret_access_key (str, optional) -- AWS secret key (optional if using IAM/profile), defaults to None

  • error_collector (ValidationCollector, optional) -- Validation error collector, defaults to None

get_sqlitedb_tables_data_from_s3(s3_key, tables_list)

Download SQLite DB from S3, load given tables into memory, return dict of lists.

Parameters:

  • s3_key (str) -- S3 object key for the SQLite database file

  • tables_list (List[str]) -- List of table names to extract from the database

Returns:

Dictionary mapping table names to lists of row dictionaries

Return type:

Dict[str, List[Dict]]

Raises:

RuntimeError -- If database operations fail

read_csv(s3_key, encoding='utf-8')

Read CSV file from S3.

Parameters:

  • s3_key (str) -- S3 object key

  • encoding (str, optional) -- File encoding, defaults to "utf-8"

Returns:

List of dictionaries, one per row

Return type:

List[Dict[str, Any]]

read_json(s3_key, encoding='utf-8')

Read JSON file from S3.

Parameters:

  • s3_key (str) -- S3 object key

  • encoding (str, optional) -- File encoding, defaults to "utf-8"

Returns:

Dictionary containing JSON data, or empty list if file is empty or error occurs

Return type:

Dict[str, Any]

read_jsonl(s3_key, encoding='utf-8')

Read JSONL (JSON Lines) file from S3.

Parameters:

  • s3_key (str) -- S3 object key

  • encoding (str, optional) -- File encoding, defaults to "utf-8"

Returns:

List of dictionaries, one per line

Return type:

List[Dict[str, Any]]

upload_csv(data, s3_key, **kwargs)

Upload CSV data to S3.

Parameters:

  • data (List[Dict]) -- List of dictionaries to upload as CSV

  • s3_key (str) -- S3 key path

  • kwargs (dict) -- Additional S3 put_object parameters

Returns:

True if upload succeeded, False otherwise

Return type:

bool

upload_json(data, s3_key, **kwargs)

Upload JSON data to S3.

Parameters:

  • data (Any) -- Data to upload (will be JSON-serialized)

  • s3_key (str) -- S3 key path

  • kwargs (dict) -- Additional S3 put_object parameters

Returns:

True if upload succeeded, False otherwise

Return type:

bool

upload_jsonl(data, s3_key, **kwargs)

Upload JSONL data to S3.

Parameters:

  • data (List[Dict]) -- List of dictionaries to upload as JSONL

  • s3_key (str) -- S3 key path

  • kwargs (dict) -- Additional S3 put_object parameters

Returns:

True if upload succeeded, False otherwise

Return type:

bool