Cognite, a global leader in artificial intelligence for industry, has announced the release of its innovative Cognite Atlas AI™ LLM & SLM Benchmark Report for Industrial Agents. This report, described as the first of its kind, aims to address the limitations of general benchmark datasets by focusing specifically on large language models (LLM) and small language models (SLM) that cater to specialized industrial tasks. The objective is to enhance the reliability, accuracy, and overall effectiveness of industrial AI solutions.
Speaking at IMPACT 2024, Knut Vidvei, Head of Product Management at Cognite, highlighted the inadequacies of existing benchmark reports. He noted that general assessments fail to capture the intricacies of industrial environments and do not align with the unique demands of industrial operations, where factors such as precision, safety, and domain expertise are paramount. “With the Cognite Atlas AI™ Benchmark Report, we’ve tailored an evaluation framework to real-world industrial tasks, ensuring AI agents are reliable and effective, driving the advancement of industrial AI,” Vidvei stated.
The Cognite Atlas AI™ platform builds upon Cognite Data Fusion®, recognised as the leading industrial data and AI platform. The company has received accolades such as Frost & Sullivan’s Global Company of the Year Award in the digital industrial platforms sector, along with Frost Radar™: Digital Industrial Platforms market powerhouse status, reinforcing its reputation as a trusted authority in the field.
The benchmark report will initially concentrate on natural language search capabilities, a critical tool for data retrieval within industrial AI applications. It features a comprehensive test set that includes a diverse array of data models tailored for sectors such as oil and gas, as well as manufacturing. The evaluation process utilises real-life question-answer pairs to assess performance in various scenarios. Responses are measured against multiple evaluation metrics, allowing for systematic analysis of the system’s ability to answer complex inquiries, such as tracking open safety-critical work orders in industrial facilities.
Cognite plans to expand the scope of future benchmark reports to include evaluations of additional AI tools designed for summarising, analysing, and reasoning with industrial data. This will provide a more holistic view of the performance capabilities of industrial AI agents.
The inaugural Cognite Atlas AI™ LLM & SLM Benchmark Report for Industrial Agents will be available for free download on October 28, 2024. Following its initial release, Cognite aims to regularly publish updated versions of the report, enabling leaders in digital transformation to leverage generative AI technologies to execute more complex operations with enhanced accuracy.
Cognite’s commitment to advancing industrial AI through rigorous evaluation frameworks underscores its dedication to supporting industries in navigating the complexities of modern operations, ensuring safety and efficiency in their processes. Visit us at www.cognite.ai
