EPRI has recently released first-of-its-kind, domain-specific benchmarking results for the electric power sector. This initial application includes multiple-choice and open-ended questions rooted in real-world utility topics, providing a more realistic view of how Large Language Models (LLM) perform. Results indicate expert oversight remains imperative, especially with open-ended questions, which could result in less than 50% accuracy in some cases.
Many existing benchmarks assess broad academic knowledge, such as math, science, and coding, and may not capture the operational and contextual complexity of real-world utility environments. Benchmarking with electric power-specific questions, such as generation and transmission and distribution asset-related inquiries, helps assess how well LLMs understand and respond to technical, regulatory and operational questions that utilities face.
