gen_ai_hub.evaluations package

Subpackages

Submodules

gen_ai_hub.evaluations.client module

class EvaluationClient

Bases: object

Base Client for the Evaluations service

static from_env(profile_name=None, **kwargs)

Alternative way to create an EvaluationClient object.

Parameter resolution precedence: 1. Explicit keyword arguments 2. Environment variables 3. Configuration file 4. VCAP_SERVICES environment variable

Parameters:

  • profile_name (str, optional) -- Profile name defined in configuration.

  • kwargs -- Additional parameters passed to constructor.

Returns:

Configured EvaluationClient instance.

Return type:

EvaluationClient

__init__(base_url, auth_url=None, client_id=None, client_secret=None, cert_str=None, key_str=None, cert_file_path=None, key_file_path=None, resource_group=None, aws_access_key_id=None, aws_secret_access_key=None, ai_core_client=None, orchestration_url=None, input_object_store_secret_name=None, provider_name='aws')

EvaluationsClient root object to be used for Evaluations.

Parameters:

  • base_url (str) -- Base URL of the AI Core instance (must include /v2 suffix).

  • auth_url (str, optional) -- Authentication URL used to retrieve access tokens.

  • client_id (str, optional) -- OAuth client ID.

  • client_secret (str, optional) -- OAuth client secret.

  • cert_str (str, optional) -- X.509 certificate content as a string.

  • key_str (str, optional) -- X.509 private key content as a string.

  • cert_file_path (str, optional) -- File path to X.509 certificate.

  • key_file_path (str, optional) -- File path to X.509 private key.

  • resource_group (str, optional) -- Resource group name within the AI Core instance.

  • aws_access_key_id (str, optional) -- AWS access key ID.

  • aws_secret_access_key (str, optional) -- AWS secret access key.

  • ai_core_client (AICoreV2Client, optional) -- Pre-configured AI Core client instance.

  • orchestration_url (str, optional) -- Pre-existing orchestration deployment URL.

  • input_object_store_secret_name (str, optional) -- Name of input object store secret.

  • provider_name (str, optional) -- Hyperscaler provider name (e.g., "aws").

Raises:

ValueError -- If required hyperscaler provider parameters are missing.

create_or_update_object_store_secret(*, context, secret_body, is_default, result_key, attr_name, creator_mapping, replace_existing, result)
Parameters:

  • secret_body (dict)

  • is_default (bool)

  • result_key (str)

  • attr_name (str)

  • creator_mapping (dict)

  • replace_existing (bool)

  • result (dict)

evaluate(evaluation_configs)

Main evaluate function to create the Evaluation job

Parameters:

evaluation_configs(List[EvaluationConfig]): A list of one or more of the EvaluationConfig objects

Returns:

List[EvaluationRun]: A list of EvaluationRun objects, one for each EvaluationConfig provided.

Parameters:

evaluation_configs (List[EvaluationConfig])

Return type:

List[EvaluationRun]

get_system_supported_metrics()

helper method to get the list of all supported metric ids

Return type:

List[str]

list_available_models()

Method to list all the available llm models

resolve_orchestration_deployment_url()

Resolves the orchestration deployment URL.

For non-default resource groups, creates a new deployment. For default resource group, attempts to discover existing deployment with the default config name using the orchestration service, or creates one if not found.

Returns:

The orchestration deployment URL.

Return type:

str

setup(input_secret_body=None, default_secret_body=None, replace_existing=False)

One time setup function which does object store secrets creation and orchestration deployment url creation if not provided.

Parameters:

  • input_secret_body (dict | None)

  • default_secret_body (dict | None)

  • replace_existing (bool)

validate_secret_type(secret_type, creator_mapping)
Parameters:

  • secret_type (str)

  • creator_mapping (dict)

gen_ai_hub.evaluations.constants module

gen_ai_hub.evaluations.credentials module

class CredentialsValue

Bases: object

CredentialsValue(name: 'str', vcap_key: 'Optional[Tuple[str, ...]]' = None, transform_fn: 'Optional[Callable]' = None)

__init__(name, vcap_key=None, transform_fn=None)
Parameters:

  • name (str)

  • vcap_key (Tuple[str, ...] | None)

  • transform_fn (Callable | None)

Return type:

None

name: str
transform_fn: Callable | None = None
vcap_key: Tuple[str, ...] | None = None
class Service

Bases: object

__init__(env)
Parameters:

env (Dict[str, Any])

get(key, default=<object object>)
property label: str | None
property name: str | None
class Source

Bases: object

Source(name: 'str', get: 'Callable[[CredentialsValue], Optional[str]]')

__init__(name, get)
Parameters:
Return type:

None

get: Callable[[CredentialsValue], str | None]
name: str
class VCAPEnvironment

Bases: object

VCAPEnvironment(services: 'List[Service]')

classmethod from_dict(env)
Parameters:

env (Dict[str, Any])

classmethod from_env(env_var=None)
Parameters:

env_var (str | None)

__init__(services)
Parameters:

services (List[Service])

Return type:

None

get_service(label, exactly_one=True)
Parameters:

exactly_one (bool)

Return type:

Service

get_service_by_name(name, exactly_one=True)
Parameters:

exactly_one (bool)

Return type:

Service

services: List[Service]
extract_credentials(source, exclude=None)

Extract all credentials from a source.

Parameters:

  • source (Source)

  • exclude (List[str])

Return type:

Dict[str, str]

fetch_credentials(profile=None, **kwargs)

Fetch credentials from a single source based on precedence.

Precedence order: kwargs > environment variables > config file > VCAP service

Once a source is selected (first one with any credential), all credentials come from that source only. Resource group is an exception and follows precedence independently.

Parameters:

profile (str)

Return type:

Dict[str, str]

get_home()
Return type:

str

get_nested_value(data_dict, keys)

Retrieve a nested value from a dictionary using a list of strings.

Parameters:

  • data_dict -- The dictionary to search.

  • keys (List[str]) -- A list of strings representing nested keys.

Returns:

The value associated with the nested keys, or None if not found.

init_conf(profile=None)
Parameters:

profile (str)

resolve_credentials(sources)

Extract credentials from the first source that has any defined.

Parameters:

sources (List[Source])

Return type:

Dict[str, str]

resolve_resource_group(sources)

Find resource_group from the first source that defines it.

Parameters:

sources (List[Source])

Return type:

str | None

validate_credentials(credentials)

Validate that we have a complete authentication method.

Parameters:

credentials (Dict[str, str])

Return type:

None