gen_ai_hub.evaluations.utils package

Submodules

gen_ai_hub.evaluations.utils.aicore_utils module

build_s3_file_key(object_store_secret_metadata_details, artifact_url_relative_path, artifact_source)
Parameters:

  • object_store_secret_metadata_details (Dict[str, str])

  • artifact_url_relative_path (str)

  • artifact_source (ArtifactSource)

call_orchestration_service_with_v2_config(test_orch_config, ai_core_client, orchestration_deployment_url, resource_group, error_collector, proxy_client=None)
Parameters:

  • test_orch_config (dict)

  • ai_core_client (AICoreV2Client)

  • orchestration_deployment_url (str)

  • resource_group (str)

  • error_collector (ValidationCollector)

create_deployment_by_configuration_id(ai_core_client, configuration_id, resource_group)
Parameters:

  • ai_core_client (AICoreV2Client)

  • configuration_id (str)

  • resource_group (str)

create_llm_orchestration_deployment_url(ai_core_client, resource_group)

creates the llm-orchestration configuration based on orchestration global scenario and then creates a deployment using that configuration

Parameters:

  • ai_core_client (AICoreV2Client)

  • resource_group (str)

fetch_configuration_by_id(configuration_id, ai_core_client, resource_group, error_collector)
Parameters:

  • configuration_id (str)

  • ai_core_client (AICoreV2Client)

  • resource_group (str)

  • error_collector (ValidationCollector)

fetch_deployment_config(deployment_id, ai_core_client, resource_group, error_collector)
Parameters:

  • deployment_id (str)

  • ai_core_client (AICoreV2Client)

  • resource_group (str)

  • error_collector (ValidationCollector)

fetch_orchestration_config_from_registry(orchestration_registry_reference, ai_core_client, error_collector)
Parameters:

  • orchestration_registry_reference (str)

  • ai_core_client (AICoreV2Client)

  • error_collector (ValidationCollector)

find_configuration_id_by_name(configurations_list, target_name)
Parameters:

  • configurations_list (List[Configuration])

  • target_name (str)

generate_random_id()

generates and returns a random uuid everytime

get_all_configurations(ai_core_client, resource_group, scenario_id)
Parameters:

  • ai_core_client (AICoreV2Client)

  • resource_group (str)

  • scenario_id (str)

Return type:

List[Configuration]

get_running_deployments_by_configuration_id(ai_core_client, configuration_id, resource_group)
Parameters:

  • ai_core_client (AICoreV2Client)

  • configuration_id (str)

  • resource_group (str)

Return type:

List[Deployment]

list_available_llm_models(ai_core_client, resource_group)
Parameters:

  • ai_core_client (AICoreV2Client)

  • resource_group (str)

read_data_from_artifact(object_store_credentials, object_store_secret_metadata_details, s3_file_key, file_type, error_collector)
Parameters:

  • object_store_credentials (_AWSObjectStoreData)

  • object_store_secret_metadata_details (Dict[str, str])

  • s3_file_key (str)

  • file_type (str)

  • error_collector (ValidationCollector)

register_aicore_artifact(artifact_folder_path, ai_core_client, resource_group, object_store_secret_name, error_collector)
Parameters:

  • artifact_folder_path (str)

  • ai_core_client (AICoreV2Client)

  • resource_group (str)

  • object_store_secret_name (str)

  • error_collector (ValidationCollector)

register_aicore_configuration(aicore_artifact_id, ai_core_client, resource_group, accumulated_config_data, orchestration_url, dataset_file_key, run_ids_list, llm_model_config, template_config, orchestration_registry_config, error_collector)
Parameters:

  • aicore_artifact_id (str)

  • ai_core_client (AICoreV2Client)

  • resource_group (str)

  • accumulated_config_data (_EvaluationConfigData)

  • orchestration_url (str)

  • dataset_file_key (str)

  • run_ids_list (List[str])

  • llm_model_config (str)

  • template_config (List)

  • orchestration_registry_config (str)

  • error_collector (ValidationCollector)

register_aicore_execution(ai_core_client, configuration_id, resource_group, error_collector)
Parameters:

  • ai_core_client (AICoreV2Client)

  • configuration_id (str)

  • resource_group (str)

  • error_collector (ValidationCollector)

resolve_artifact_path(artifact_source, ai_core_client, object_store_credentials, resource_group, error_collector)
Parameters:

  • artifact_source (ArtifactSource)

  • ai_core_client (AICoreV2Client)

  • object_store_credentials (_AWSObjectStoreData)

  • resource_group (str)

  • error_collector (ValidationCollector)

resolve_metric_identifiers(metrics, ai_core_client, resource_group, error_collector)

Resolves metric identifiers to metric template metadata.

Parameters:
Return type:

List[Dict]

resolve_metric_names(metric_configs_list, error_collector)
Parameters:
upload_evaluation_dataset_data(evaluation_config_data, object_store_credentials, object_store_secret_name, ai_core_client, resource_group, error_collector)

Method to upload the evaluation config data using the object store secrets data passed

Parameters:

  • evaluation_config_data (_EvaluationConfigData)

  • object_store_credentials (_AWSObjectStoreData)

  • object_store_secret_name (str)

  • ai_core_client (AICoreV2Client)

  • resource_group (str)

  • error_collector (ValidationCollector)

upload_file_to_aws_s3(object_store_credentials, object_store_secret_metadata_details, file_data, file_key, file_type, error_collector)
Parameters:

  • object_store_credentials (_AWSObjectStoreData)

  • object_store_secret_metadata_details (Dict[str, str])

  • file_data (Any)

  • file_key (str)

  • file_type (str)

  • error_collector (ValidationCollector)

wait_for_target_status(status_fetcher, target_status, extract_url=None, timeout=1200, initial_interval=120, pending_interval=40)

Reusable polling function to wait until a resource reaches target_status.

Parameters:

  • status_fetcher (Callable[[], Any]) -- Function to get current status response

  • target_status (Status) -- Target status to wait for (Status enum)

  • extract_url (Optional[Callable[[Any], str]]) -- Optional function to extract URL from response, defaults to None

  • timeout (int) -- Maximum time to wait in seconds, defaults to 1200

  • initial_interval (int) -- Initial polling interval in seconds, defaults to 120

  • pending_interval (int) -- Polling interval for pending/running status in seconds, defaults to 40

Returns:

Extracted URL if extract_url is provided and status reached, None otherwise

Return type:

Optional[str]

gen_ai_hub.evaluations.utils.config_data_utils module

get_dataset_data(dataset_config, ai_core_client, object_store_credentials, resource_group, error_collector)
Parameters:

  • dataset_config (Dataset)

  • ai_core_client (AICoreV2Client)

  • object_store_credentials (_AWSObjectStoreData)

  • resource_group (str)

  • error_collector (ValidationCollector)

get_orch_config_data(evaluation_config, ai_core_client, gen_ai_hub_proxy_client, error_collector)
Parameters:

gen_ai_hub.evaluations.utils.file_utils module

load_config_file(file_path, error_collector)

Load config from local file path or Path object

Parameters:
Return type:

Dict[str, Any] | List[Dict[str, Any]] | List

read_local_csv_file(file_path, error_collector)

reads csv and returns the data as df

Parameters:
Return type:

List[Dict[str, Any]]

gen_ai_hub.evaluations.utils.gen_utils module

build_model_versions_map(model_list)

Builds a map of model names to their versions.

Return type:

Dict[str, List[str]]

check_if_content_filter_provider_supported(orch_config, error_collector)

Validates if all filters in the filtering module configuration are of supported types.

Parameters:

  • orch_config (dict) -- Orchestration configuration dictionary.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

True if all filters are supported or no filtering is configured, False otherwise.

Return type:

bool

check_if_metric_is_defined(metrics, metric_templates, error_collector)
Parameters:
Return type:

None

count_user_prompts_from_template_list(template_list)
Return type:

int

create_custom_metric_name(custom_metric_config, error_collector)

Creates a custom metric name based on the provided custom metric configuration.

Parameters:

  • custom_metric_config (dict) -- Dictionary containing metric configuration.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

A string representing the custom metric name.

Raises:

ValidationError -- If required fields are missing or invalid.

Return type:

str

create_model_versions_map_from_configuration_param_bindings(param_bindings, error_collector)
Parameters:

error_collector (ValidationCollector)

Return type:

Tuple[Dict[str, List[str]], str | None]

create_model_versions_map_from_custom_metric_config(custom_metric_config_data)
Return type:

Dict[str, List[str]]

create_model_versions_map_from_orch_configs(orchestration_configs_data, error_collector)
Parameters:
Return type:

Dict[str, List[str]] | None

extract_dataset_columns(template_variables)

Extracts column names from the template variables provided. If the value is a list, extracts column names from the first rows; else, extracts from the template_variables directly.

Return type:

List[str]

extract_metrics_variables(metric_templates, metric_name=None)

Extracts unique set of variables from the 'variables' key.

Parameters:

metric_name (str)

Return type:

Set

flatten_prompt_configuration(prompt_config)

Flatten a nested prompt configuration dictionary into a readable string.

Parameters:

prompt_config (dict)

Return type:

str

get_accumulated_config_data(evaluation_configs_data)
Parameters:

evaluation_configs_data (List[_EvaluationConfigData])

Return type:

_EvaluationConfigData

get_custom_metric_ids_from_input(custom_metric_config_data, error_collector)

Retrieve custom metric ids from the file data provided by the user.

Parameters:

  • custom_metric_config_data (List[dict]) -- List of dictionaries containing custom metric definitions.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

List of custom metric ids.

Return type:

list[str]

get_defaults(orchestration_configuration)

Returns the default field from the orchestration configuration.

Parameters:

orchestration_configuration (dict)

Return type:

dict

get_filter_config(orch_config)

Extract the filtering configuration from the orchestration config.

Parameters:

orch_config (dict) -- Orchestration configuration dictionary.

Returns:

Filtering configuration dictionary, or empty dict if not present.

Return type:

dict

get_grounding_config_from_orch_config(orch_config)

Extracts the grounding configuration from the orchestration configuration. Args:

orch_config (dict): Orchestration configuration.

Returns:

dict: Grounding configuration if present, otherwise an empty dictionary.

Return type:

dict

get_grounding_output_param_key(orch_config)

Determines the correct key to extract the grounding output parameter based on the API version. Args:

orch_config (dict): Orchestration configuration.

Returns:

str: The key to extract the grounding output parameter.

Parameters:

orch_config (dict)

Return type:

str

get_mapped_value_if_exists(key, mapping_keys, variable_mapping, dataset_columns)

Gets the first valid mapped value from the list of keys if it exists in variable mapping, else returns the first key

Return type:

str

get_prompt_variables_from_orch_config(orch_config)
Parameters:

orch_config (dict)

Return type:

Set[str]

get_template_list_from_orch_config(orch_config)
Return type:

List

handle_json_schema_match(metrics, dataset_data, variable_mapping, error_collector)
Parameters:

  • metrics (list[str])

  • dataset_data (list)

  • variable_mapping (dict)

  • error_collector (ValidationCollector)

Return type:

None

handle_language_match(metrics, dataset_data, variable_mapping, error_collector)
Parameters:

  • metrics (list)

  • dataset_data (list)

  • variable_mapping (dict)

  • error_collector (ValidationCollector)

Return type:

None

handle_missing_dependent_variables_in_dataset(dataset_data, metrics, metric_templates, variable_mapping, error_collector)

validates whether all the dependent variables for the metrics list are either directly present as columns in dataset or a variable mapping is provided Args:

dataset_data (List[dict]): Dataset rows to validate (list of row dictionaries) metrics (list[str]): List of metrics provided in the input configuration. metric_templates (list[dict]): Metric templates information resolved from Metric Management Service variable_mapping (dict): The variable mapping provided in the input configuration.

Raises:

ValidationError: If any dependent variable is missing in the dataset and the variable mapping is invalid for that metric.

Parameters:

  • dataset_data (List[dict])

  • metrics (list[str])

  • metric_templates (list[dict])

  • variable_mapping (dict)

  • error_collector (ValidationCollector)

Return type:

None

handle_reference_missing_rows(dataset_data, variable_mapping, metrics, error_collector)

validates whether the reference columns in the rows are missing in the dataset for all metrics and for each individual metrics

Args:

dataset_data (List[dict]): Dataset rows to validate (list of row dictionaries) variable_mapping (dict): The variable mapping provided in the input configuration. metrics (list): List of metrics provided in the input configuration.

Raises:

ValidationError: If any required variable mapping is invalid or the default column does not exist in the dataset.

Parameters:

  • dataset_data (List[dict])

  • variable_mapping (dict)

  • metrics (list)

  • error_collector (ValidationCollector)

Return type:

None

has_filter_key(orch_config)

Check if the 'filtering' key is present in the orchestration config.

Parameters:

orch_config (dict) -- Orchestration configuration dictionary.

Returns:

True if filtering key is present, False otherwise.

Return type:

bool

is_value_in_json(value, name, data)

Check if a value matches a name or exists in a mapping dictionary.

Parameters:

  • value (Any) -- The value to search for.

  • name (str) -- The name to compare against.

  • data (dict[str, str]) -- Dictionary to search in (keys or values).

Returns:

True if value matches name or is found in data, False otherwise.

Return type:

bool

list_prompt_variables(format_string)

Get all fields (parameters) of the form {{ ?param_name }} from the template. Optionally return the raw field names without stripping spaces and '?'.

Parameters:

format_string (str)

Return type:

list[str]

parse_model_filter_list(param, error_collector)

Parse and return model filter list from a param.

Parameters:

error_collector (ValidationCollector)

Return type:

List

populate_dataset_data_if_data_missing(dataset_data, variable_mapped_key, error_collector)

Validates and Populates the dataset_data with missing data fields if golden truth is present and throws error if dataset is partially filled

Parameters:
Return type:

None

populate_dataset_data_if_individual_metric_reference_provided(dataset_data, variable_mapping, metrics, error_collector)

populates reference value across all rows of dataset_data if individual metric reference is provided and is different than all metrics reference provided. This population happens if the provided reference is a golden instance

Args:

dataset_data (List[dict]): Dataset rows to validate (list of row dictionaries) variable_mapping (dict): The variable mapping provided in the input configuration. metrics (list): List of metrics provided in the input configuration.

Parameters:

  • dataset_data (List[dict])

  • variable_mapping (dict)

  • metrics (list)

  • error_collector (ValidationCollector)

Return type:

None

populate_dataset_data_if_single_reference_provided(dataset_data, variable_mapping, collector)
Populates the dataset_data with missing reference column entries across rows of

dataset_data for only all metrics case where a golden reference is present

Args:

dataset_data (List[dict]): Dataset rows to validate (list of row dictionaries) variable_mapping (dict): The variable mapping provided in the input configuration.

Parameters:

  • dataset_data (List[dict])

  • variable_mapping (dict)

Return type:

None

populate_dataset_data_if_single_schema_provided(dataset_data, variable_mapping, collector)

Populates the dataset_data with missing json schema column entries across rows of dataset_data

Return type:

None

remove_filter_metrics_if_provider_not_supported(orchestration_config_data, metrics, error_collector)

Removes content filter-related metric IDs from the metrics list if the content filter provider is not supported for any of the runs in orchestration_config_data.

Parameters:

  • orchestration_config_data (List[dict])

  • metrics (List[str])

  • error_collector (ValidationCollector)

Return type:

None

resolve_orchestration_config_v2(template_data, llm)
Parameters:
Return type:

dict

select_model_details_randomly(orchestration_config_data, error_collector)

Selects at random, model name and version from the list of model names and versions provided by the users run data.

Parameters:
Return type:

Tuple[str, str] | None

set_model_details_from_run_configs(orch_config)

Sets the model name and version from the run data if available. If not available, it returns None.

Return type:

Tuple[str, str] | None

update_artifact_dict(artifact_reference, artifact_dict_count)
Parameters:
Return type:

None

update_test_orch_config(model_name, model_version, error_collector)
Parameters:

error_collector (ValidationCollector)

Return type:

dict | None

update_variable_mapping(variable_mapping, prefix_key, variable_mapping_dict)
Parameters:

  • variable_mapping (dict)

  • prefix_key (str)

  • variable_mapping_dict (dict)

Return type:

dict

validate_all_metrics_mapping(variable_mapping, dataset_columns, error_collector)

Validates the variable mapping for 'all_metrics' with a zero-tolerance failure threshold.

Args:

variable_mapping (dict): The variable mapping provided in the input configuration. dataset_columns (list): List of column names in the dataset.

Raises:

ValidationError: If any 'all_metrics' mapping is invalid or the direct column does not exist in the dataset.

Parameters:
Return type:

None

validate_individual_custom_metrics(variable_mapping, dataset_columns, custom_metric_ids, custom_metric_variables, error_collector)

Validates the variable mapping for any metric mapping with a zero-tolerance failure threshold.

Parameters:

  • variable_mapping (dict)

  • dataset_columns (list)

  • custom_metric_ids (list)

  • custom_metric_variables (set)

  • error_collector (ValidationCollector)

Return type:

None

validate_individual_metrics(metrics, variable_mapping, dataset_columns, metric_dependent_variables, error_collector)

Validates the variable mapping for any metric mapping with a zero-tolerance failure threshold.

Args:

metrics (list): List of metrics provided in the input configuration. variable_mapping (dict): The variable mapping provided in the input configuration. dataset_columns (list): List of column names in the dataset. metric_dependent_variables (set): Set of dependent variables for all metrics

Raises:

ValidationError: If any metric mapping is invalid or the direct column does not exist in the dataset.

Parameters:

  • metrics (list[str])

  • variable_mapping (dict)

  • dataset_columns (list)

  • metric_dependent_variables (set)

  • error_collector (ValidationCollector)

Return type:

None

validate_language_code_and_data_population(dataset_data, variable_mapping, error_collector)

Populates the dataset_data with missing language column entries across rows of dataset_data

Parameters:
Return type:

None

validate_metric_name(metric, all_supported_metrics, error_collector)

Validates if metrics name is not empty and the value actually exists in the list of supported metrics

Parameters:
Return type:

None

validate_metrics(metrics, metric_templates, orchestration_config_data, error_collector)

Validates if metrics list is empty or metric name is invalid

Parameters:

  • metrics (List[str])

  • metric_templates (List[dict])

  • orchestration_config_data (List[dict])

  • error_collector (ValidationCollector)

Return type:

None

validate_prompts_in_templating_module(orchestration_config_data, metric, error_collector)

Checks whether the templating config provided in the Orchestration Config has exactly one user prompt

Parameters:
Return type:

None

validate_variable_mapping_of_metrics(metrics, metric_templates, dataset_data, variable_mapping, error_collector)

Validates variable mapping of metrics with tolerance to zero failure threshold

Args:

metrics: List of metrics provided in the input config metric_templates (list[dict]): Metric templates information resolved from Metric Management Service dataset_data: Dataset rows to validate variable_mapping: variable mapping provided in input config

Returns:

Validates and throws validation error even if one variable mapping related to metrics is invalid.

Parameters:

  • metrics (list[str])

  • metric_templates (list[dict])

  • dataset_data (List[dict])

  • variable_mapping (dict)

  • error_collector (ValidationCollector)

Return type:

None

validate_variable_mapping_of_prompts(orchestration_config_data, dataset_data, variable_mapping, error_collector)

Validates the variable mapping for prompts with a zero-tolerance failure threshold.

Args:

orchestration_config_data (list): Orchestration run configuration dataset_data (dict): Dataset rows to validate variable_mapping (dict): The variable mapping provided in the input configuration.

Raises:

ValidationError: If any prompts variable mapping is invalid or does not exist in the dataset.

Parameters:

  • orchestration_config_data (list)

  • dataset_data (List[dict])

  • variable_mapping (dict)

  • error_collector (ValidationCollector)

Return type:

None

gen_ai_hub.evaluations.utils.language_match_utils module

class LanguageMapper

Bases: object

Uses the langcodes library to convert language strings to normalized ISO 639-1 strings, handling regional variants like zh-CN.

classmethod get_iso_code_639_1(code)

Converts a language code string to its ISO 639-1 two-letter code.

Parameters:

code (str) -- Language code string (e.g., 'zh-CN', 'en_US', 'en')

Returns:

Two-letter ISO 639-1 code (e.g., 'zh', 'en'), or None if the code is invalid, excluded, or empty

Return type:

Optional[str]

EXCLUDED_CODES = {'bs', 'ms'}

gen_ai_hub.evaluations.utils.metric_client_utils module

fetch_all_system_predefined_metrics(ai_core_client, resource_group, error_collector)

Fetches all system-predefined metrics from the GenAI metrics server.

Parameters:

  • ai_core_client (AICoreV2Client) -- AI Core client instance for API access.

  • resource_group (str) -- The resource group name.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

List of system-predefined metric templates.

Return type:

list

Raises:

RuntimeError -- If fetching system predefined metrics fails.

get_custom_metric_by_id(metric_id, ai_core_client, resource_group, error_collector)

Fetches a specific custom metric by its ID from the GenAI metrics server.

Note: Not using rest_client.get() because it converts camelCase to snake_case, but the Metric Management Service requires exact camelCase field names like 'additionalProperties' (would become 'additional_properties' if using rest_client).

Parameters:

  • metric_id (str) -- The unique ID of the metric to retrieve.

  • ai_core_client (AICoreV2Client) -- AI Core client instance for API access.

  • resource_group (str) -- The resource group name.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

Parsed JSON response as a dictionary, or None if not found.

Return type:

dict | None

get_metric_template_info_from_server(metric, ai_core_client, resource_group, error_collector)

Retrieves metric template information from the server by metric name.

Parameters:

  • metric (str) -- The name of the metric to retrieve.

  • ai_core_client (AICoreV2Client) -- AI Core client instance for API access.

  • resource_group (str) -- The resource group name.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

Metric information dictionary, or None if not found.

Return type:

dict | None

get_metric_version_history(scenario, metric_id, version, ai_core_client, resource_group, error_collector)

Fetches the version history for a specific evaluation metric in a scenario.

Note: Not using rest_client.get() because it converts camelCase to snake_case, but the Metric Management Service requires exact camelCase field names.

Parameters:

  • scenario (str) -- The name of the scenario.

  • metric_id (str) -- The unique ID of the evaluation metric.

  • version (str) -- The version of the metric.

  • ai_core_client (AICoreV2Client) -- AI Core client instance for API access.

  • resource_group (str) -- The resource group name.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

Parsed JSON response as a dictionary, or None if not found.

Return type:

dict | None

gen_ai_hub.evaluations.utils.orch_config_utils module

get_model_name(orch_config, keys, error_collector)

Retrieves the value from orch_config using the list of keys provided. If any key in the path is missing, logs an error and returns None.

Args:

orch_config (dict): The orchestration configuration JSON object. keys (list): List of keys representing the path to the desired field. error_collector (ValidationCollector): The error collector to log errors.

Returns:

The value at the specified path in orch_config, or None if any key is missing.

Parameters:
Return type:

Any

get_prompt_templating_config(orch_config, error_collector)

Returns the prompt_templating configuration from the orchestration config.

Parameters:

  • orch_config (dict) -- The orchestration configuration dictionary.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

get_template_key(orch_config, keys, error_collector)

Retrieves the value from orch_config using the list of keys provided.

If any key in the path is missing, logs an error and returns None.

Parameters:

  • orch_config (dict) -- The orchestration configuration dictionary.

  • keys (list[str]) -- List of keys representing the path to the desired field.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

The value at the specified path in orch_config, or None if any key is missing.

Return type:

Any | None

get_template_list_from_orch_config(orch_config)

Returns the template list from the orchestration config.

Parameters:

orch_config (dict) -- The orchestration configuration dictionary.

Returns:

The template list from the orchestration config.

Return type:

list

to_comparable(x)

Converts an object to a comparable format (dict or primitive).

Handles Pydantic v1, Pydantic v2, custom classes, and primitives.

Parameters:

x (Any) -- The object to convert.

Returns:

Dictionary representation or the primitive value.

Return type:

dict | Any

validate_if_all_grounding_input_params_present_in_prompt_variables(orch_config, error_collector)

Validates if all the input params of the grounding module exist in the prompt variables for v2 configuration.

Parameters:

  • orch_config (dict) -- The orchestration configuration dictionary.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

validate_if_content_inside_template_is_empty_in_templating_module_config(orch_config, error_collector)

Validates if the template list is an array and is a valid dict and content exists in template.

Parameters:

  • orch_config (dict) -- The orchestration configuration dictionary.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

validate_if_grounding_output_present_in_prompt_variables(orch_config, error_collector)

Validates if the grounding output parameter is present in prompt variables.

Parameters:

  • orch_config (dict) -- The orchestration configuration dictionary.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

validate_if_image_url_is_provided_in_content_type_inside_templating_module_config(orch_config, error_collector)

Validates if inside the template list of templating_module_config if it has image_url type inside the content.

Parameters:

  • orch_config (dict) -- The orchestration configuration dictionary.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

validate_if_template_list_is_empty(orch_config, keys, error_collector)

Validates if template list exists and is not empty.

Parameters:

  • orch_config (dict) -- The orchestration configuration dictionary.

  • keys (list[str]) -- List of keys representing the path to the template list.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

validate_if_template_list_is_empty_in_templating_module_config(orch_config, error_collector)

Validates if template list exists and is not empty in the templating module config.

Parameters:

  • orch_config (dict) -- The orchestration configuration dictionary.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

validate_mandatory_modules(orch_config, error_collector, module_key, required_keys, config_keys)

Validates the presence of mandatory modules and their structure.

Parameters:

  • orch_config (dict) -- The orchestration configuration dictionary.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

  • module_key (str) -- The key of the module to validate.

  • required_keys (list[str]) -- List of required keys in the module.

  • config_keys (list[str]) -- List of configuration keys to validate as dictionaries.

Returns:

None

Return type:

None

validate_model_name(orch_config, keys, error_collector)

Validates if llm_module_config is a dict and the model name exists.

Parameters:

  • orch_config (dict) -- The orchestration configuration dictionary.

  • keys (list[str]) -- List of keys representing the path to the model name.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

validate_model_name_in_llm_module_config(orch_config, error_collector)

Validates if the model configuration in llm_module_config is a dict and the model name exists.

Parameters:

  • orch_config (dict) -- The orchestration configuration dictionary.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

validate_orch_config_mandatory_modules(orch_config, error_collector)

Validates if the outer structure of the orchestration config is valid and exists.

Parameters:

  • orch_config (dict) -- The orchestration configuration dictionary.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

validate_orchestration_params_from_evaluation_config(evaluation_configs, error_collector)

Validates orchestration parameters from evaluation configuration.

Ensures that either orchestration_registry_reference is provided alone, or both template and llm are provided together.

Parameters:

  • evaluation_configs (List[EvaluationConfig]) -- List of evaluation configuration objects.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

validate_template_ref_absent_in_config(orch_config, error_collector)

Validates if template_ref is given in templating module config and raises an error.

Parameters:

  • orch_config (dict) -- The orchestration configuration dictionary.

  • error_collector (ValidationCollector) -- ValidationCollector instance for collecting validation errors.

Returns:

None

Return type:

None

gen_ai_hub.evaluations.utils.oss_secret_utils module

class ObjectStoreData

Bases: object

ObjectStoreData(provider_name: str, aws_access_key_id: str, aws_secret_access_key: str)

__init__(provider_name, aws_access_key_id, aws_secret_access_key)
Parameters:

  • provider_name (str)

  • aws_access_key_id (str)

  • aws_secret_access_key (str)

Return type:

None

aws_access_key_id: str
aws_secret_access_key: str
provider_name: str
create_aws_object_store_secret(aws_access_key_id, aws_secret_access_key, ai_core_client, resource_group, secret_body, is_default_secret)

creates the s3 based object store secrets in aicore environment

Parameters:

  • aws_access_key_id (str)

  • aws_secret_access_key (str)

  • ai_core_client (AICoreV2Client)

  • resource_group (str)

  • secret_body (dict)

  • is_default_secret (bool)

delete_object_store_secret(ai_core_client, name, resource_group)

Delete an object store secret. Returns None if secret doesn't exist (404 error).

Parameters:

  • ai_core_client (AICoreV2Client)

  • name (str)

  • resource_group (str)

fetch_object_store_secret_by_name(ai_core_client, name, resource_group, collector)
Parameters:

  • ai_core_client (AICoreV2Client)

  • name (str)

  • resource_group (str)

  • collector (ValidationCollector)

gen_ai_hub.evaluations.utils.validation_utils module

extract_deployment_id(orch_url)
Return type:

str

fetch_and_validate_orchestration_config(ai_core_client, configuration_id, orchestration_config_data, resource_group, error_collector)
Parameters:

  • ai_core_client (AICoreV2Client)

  • configuration_id (str)

  • orchestration_config_data (List[dict])

  • resource_group (str)

  • error_collector (ValidationCollector)

validate_config_data_collection(accumulated_config_data, error_collector)

wrapper function to perform validation of fetched config data in case of single vs multiple executions flow

Parameters:

  • accumulated_config_data (List[_EvaluationConfigData] | _EvaluationConfigData)

  • error_collector (ValidationCollector)

validate_filtered_models(configuration_param_bindings, orchestration_config_data, error_collector)
Parameters:
validate_input_config(orchestration_config_data, metrics, metric_templates, error_collector)

Validates the input parameters of run data and metrics

Parameters:

  • orchestration_config_data (List[dict])

  • metrics (List[str])

  • metric_templates (List[dict])

  • error_collector (ValidationCollector)

validate_merged_config_data(evaluation_config_data, error_collector)

handles the validation of config provided from the user

Parameters:
validate_orchestration_configuration(orchestration_config_data, error_collector)

Validates the Orchestration configuration provided by user

Parameters:
validate_orchestration_url(evaluation_config_data, orchestration_url, ai_core_client, resource_group, error_collector, proxy_client=None)

Validates if the orchestration deployment url provided via config resides in same resourceGroup as workload or not. Also validates if url is valid and orchestration deployment is not in terminal state

Parameters:

  • evaluation_config_data (_EvaluationConfigData)

  • orchestration_url (str)

  • ai_core_client (AICoreV2Client)

  • resource_group (str)

  • error_collector (ValidationCollector)

validate_orchestration_url_across_configs(accumulated_config_data, orchestration_url, ai_core_client, resource_group, error_collector, proxy_client=None)

wrapper function to perform validation of fetched config data in case of single vs multiple executions flow

Parameters:

  • accumulated_config_data (List[_EvaluationConfigData] | _EvaluationConfigData)

  • orchestration_url (str)

  • ai_core_client (AICoreV2Client)

  • resource_group (str)

  • error_collector (ValidationCollector)

validate_variable_mapping_with_input_config(orchestration_config_data, dataset_data, variable_mapping, metrics, metric_templates, error_collector)

Validates all the required variable mappings provided in input config with a zero-tolerance failure threshold.

Args:

orchestration_config_data(list): Orchestration run configuration dataset_data (dict): Dataset rows to validate variable_mapping (dict): The variable mapping provided in the input configuration. metrics (list[str]): List of metrics provided in the input configuration. metric_templates (list[dict]): Metric templates information resolved from Metric Management Service error_collector (ValidationCollector): To accumulate the errors occurred during the process

Raises:

ValidationError: If any required variable mapping is invalid or the default column does not exist in the dataset.

Parameters:

  • orchestration_config_data (List[dict])

  • dataset_data (dict)

  • variable_mapping (dict)

  • metrics (List[str])

  • metric_templates (List[dict])

  • error_collector (ValidationCollector)