Google has launched LMEval, an open-source system designed to standardize the evaluation of large AI models from various tech companies. Capable of benchmarking text, image, and code iterations, Google's new offering aims to simplify comparisons between different AI models and streamline the safety analysis process. LMEval runs on the LiteLLM framework, eliminating API differences across platforms, while its visualization tool, LMEvalboard, allows for detailed comparison and drill-down views.
Google has launched LMEval, an open-source system designed to standardize the evaluation of large AI models from various tech companies. Capable of benchmarking text, image, and code iterations, Google's new offering aims to simplify comparisons between different AI models and streamline the safety analysis process. LMEval runs on the LiteLLM framework, eliminating API differences across platforms, while its visualization tool, LMEvalboard, allows for detailed comparison and drill-down views.