service.llm.base module

class service.llm.base.BASE_LLM_CACHE

Bases: object

A base class for implementing caching of queries and responses for a Language Model (LLM). This class is intended to be inherited and implemented by a subclass.

check_cache(query)

Checks if a cached response exists for the provided query.

Parameters:

query – The query to check in the cache.

Returns:

The cached response if available, otherwise a constant (NO_CACHE_YET). Also updates the internal usage cost for cached responses.

clear_usage()

Clears the stored usage statistics.

print_usage()

Prints the cumulative token usage statistics.

write_cache(query, response, usage)

Writes the provided query, response, and usage information to the cache.

Parameters:

• query – The query to be cached.

• response – The response to be cached.

• usage – The token usage information.