Server Energy Efficiency tool released

Today SPEC released a major update to its server energy efficiency rating tool, SERT. See press release.

Created initially for the requirements of the US EPA Energy Star program, it also takes into account the needs of regulatory agencies worldwide. SPEC also has considered the requirements of the
European Union’s DG ENTR Lot 9 Ecodesign Directive for enterprise servers, the Chinese National Institute of Standardization (CNIS), and Japan’s Ministry of Economy, Trade & Industry’s (METI) Top Runner energy efficiency program. The SERT user guide is available in Chinese as well as English: 服务器功效评估工具(SERT) -用户指南

Figure 2: Memory channel impact on CryptoAES.
From “The SERT Metric and the Impact of Server Configuration“
Copyright © 1988‐2017 SPEC

Among the news in SERT 2 is the addition of a single-value composite metric to quantify server efficiency across different workloads. An excellent description of SERT (referring specifically to the 1.1.0 version) was given at the Beijing 2016 SPEC Summit. See slides.

As SERT adoption spreads, computer vendors will be able to run a single test to produce data to establish compliance with energy regulations. Even when different nations set different threshold criteria and other requirements, the benefit to vendors remains of having to run a single test to sell worldwide instead of 195 different tests for every nation. For consumers the benefit is in having energy regulations tied to a measure that accurately reflects the energy usage of the computing equipment.

A Tool Not a Benchmark

SPEC is clear that SERT is a tool, not a benchmark. That seems like a distinction without a difference because it’s also clear that any benchmark can be used more generally as a tool to investigate performance characteristics, and that SERT scores could be used as benchmark results. The differences are in intent and in license restrictions.

Regulatory programs like US EPA Energy Star like to say if your product meets this value it is compliant with our low energy standards and it may be sold in certain realms, and if it fails to meet that value it is not compliant and other restrictions apply. They want a measure of the power and performance of the server “out of the box.” In traditional benchmarks by contrast we want to show how well the server can perform, typically including expert performance tuning. There are usually restrictions on what tuning may be performed in order that the benchmark result remains representative of the performance ordinary users will see in the real world. But the emphasis is on comparability between systems, hardware platforms and/or software systems.

Fair Use

SERT is run with a limited set of reasonable tuning options reviewed and agreed ahead of time by the SPEC committee. So for the tester it’s simple. Just install SERT and run the test. The scores are meant to determine compliance with energy regulations, not to compare one server or processor or software stack against another. For that purpose you do much better to use an energy efficiency benchmark such as SPECpower_ssj2008 or one of many other benchmarks from SPEC and other organizations which have adopted the SPEC Power Methodology.

The Fair Use Rules are the reason you don’t see lists of SERT results. The software license prohibits making such competitive comparisons. The combination of the legal restriction plus the ready availability of superior measures for marketing comparisons and for academic research has worked very well to keep SERT in its proper place in the regulatory domain.