{"id":18,"date":"2017-03-22T00:24:47","date_gmt":"2017-03-22T00:24:47","guid":{"rendered":"http:\/\/amberdot.com\/walterbays\/?p=18"},"modified":"2017-03-22T00:28:32","modified_gmt":"2017-03-22T00:28:32","slug":"performance-trajectories-speed-and-throughput","status":"publish","type":"post","link":"http:\/\/amberdot.com\/walterbays\/index.php\/2017\/03\/22\/performance-trajectories-speed-and-throughput\/","title":{"rendered":"Performance Trajectories &#8211; speed and throughput"},"content":{"rendered":"<p>Among the challenges SPEC faces in addressing change in the computing industry is the very trend which SPEC has done more than anyone to bring about. That is the rapid growth in computing power &#8211; close to 10,000 times the single thread performance today as the venerable VAX-11\/780.<\/p>\n<p><a href=\"http:\/\/www.cs.columbia.edu\/~sedwards\/\">Dr. Stephen Edwards<\/a> teaches performance history at Columbia. See on this (log scale) relative performance chart the rate of increase taking off (+52% per year) in the early 80&#8217;s as the RISC vs. CISC wars ignited, sometimes called the benchMarketing wars as exaggerated and loosely substantiated claims were common, which in turn led to the creation of SPEC to standardize the metrics and test protocols. Edwards shows the rate of improvement slowing from 2002 down to +20% per <a href=\"http:\/\/www.cs.columbia.edu\/~sedwards\/classes\/2012\/3827-spring\/advanced-arch-2011.pdf\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-19 alignleft\" src=\"https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/columbia.png?resize=300%2C185\" alt=\"\" width=\"300\" height=\"185\" srcset=\"https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/columbia.png?resize=300%2C185 300w, https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/columbia.png?resize=768%2C472 768w, https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/columbia.png?w=969 969w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a>year.<\/p>\n<p>Click on this chart through to his course handouts to see detail on the chart of processor names (remember Alpha?) and discussion of where the performance improvements came from. He discusses thermal and power issues driving the shift to multi-core, and the memory bottleneck.<\/p>\n<p><a href=\"http:\/\/preshing.com\/20120208\/a-look-back-at-single-threaded-cpu-performance\/\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"size-medium wp-image-22 alignleft\" src=\"https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/preshing.png?resize=300%2C245\" alt=\"\" width=\"300\" height=\"245\" srcset=\"https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/preshing.png?resize=300%2C245 300w, https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/preshing.png?w=556 556w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a>Now read Jeff Preshing&#8217;s analysis showing +52% increase per year slowing from 2004 to +21% per year. Preshing marks the inflection point with publication of Herb Sutter&#8217;s paper <a href=\"http:\/\/www.gotw.ca\/publications\/concurrency-ddj.htm\">&#8220;The Free Lunch Is Over: A Fundamental Turn Toward Concurrency in Software.&#8221;<\/a> Today&#8217;s processors are multi-core and those cores are mostly multi-threaded. Preshing also observes that the rate of increase for non-Intel processors has been only +13% per year.<\/p>\n<p>Now let&#8217;s look not just at single thread speed but at the total computational power available to applications running on the processor. I considered all compliant CPU 2000 and CPU 2006 results published on spec.org of the &#8220;rate&#8221; or computational throughput metric, for systems with a single processor. I used the &#8220;peak&#8221; or highly tuned metric and for a small number of results with no peak metric I used their &#8220;base&#8221; metric. This gave me about 800 each integer and floating point results for CPU 2000, and 3,000 each for CPU 2006.<\/p>\n<p>Here are the integer results plotted on a linear scale with a trend line plotted as a logarithmic least squares regression. In 2006 you <a href=\"https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/int.png\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-21\" src=\"https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/int.png?resize=474%2C216\" alt=\"\" width=\"474\" height=\"216\" \/><\/a>see the transition with a brief overlap when SPEC published results from both the old and the new benchmarks. I did not attempt to normalize the two benchmarks to a common scale as some have done because there is no simple &#8220;conversion factor&#8221; from CPU 2000 to CPU 2006. If there were, of course SPEC would have had no reason to develop the latter benchmark.\u00a0You can see generally how such a normalization would work by moving the CPU 2006 graph up so that its trend line starts where the CPU 2000 trend line ends.<\/p>\n<p>But that will also show you something important: the high outliers, the red points so far above the trend line, are gone with CPU 2006. That is essentially because the newer benchmark is &#8220;harder&#8221; with fewer easy performance wins which might not translate into real world performance improvements. It&#8217;s not that the processor developers and compiler writers of 2005 were &#8220;cheating&#8221; an old vulnerable benchmark. Rather they were exploiting more and more opportunities for speedups, and the dated applications in CPU 2000 did not respond to these optimizations in the same way that more complex applicaitons of 2005 responded. One big goal of SPEC benchmark design is to give no easy wins, but force higher benchmark numbers to be tied to\u00a0solid improvements that benefit real users.<\/p>\n<p>The systems which so far outstripped the SPECint_rate2000 trend line at the end of CPU 2000&#8217;s life were mostly Intel Xeons with a few Intel Core 2 processors at the top end. The world was learning what happens when a processor with a very large on-chip cache meets a relatively small problem size. CPU 2006 problem sizes were much larger, as were typical real world applications which had come to rely on all the increased computing power delivered over the past years.<\/p>\n<p>As CPU 2006 has aged you again see more high outliers with performance far above the trend line. To some extent this is normal progress. These are the best performing processors, and in this way the trend line is self correcting via capitalism. That is, processors well below the trend line are taken off the market. The three highest points on the chart represent an Oracle SPARC T7-1 with 32 cores published November 2015, and two Intel Xeon Phi&#8217;s with 72 cores published December 2016 and March 2017.<\/p>\n<p>Those systems are outliers because they deliver massive parallelism above that of the single and dual core processors typical at the debut of CPU 2006. But it&#8217;s also the case that CPU 2006 is ageing and some of that potential computing power does not translate well to real world applications. I&#8217;ll write more about parallelism and usable capacity in a later posting.<\/p>\n<p><a href=\"https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/fp.png\"><img data-recalc-dims=\"1\" loading=\"lazy\" decoding=\"async\" class=\"aligncenter wp-image-20\" src=\"https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/fp.png?resize=453%2C236\" alt=\"\" width=\"453\" height=\"236\" srcset=\"https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/fp.png?resize=300%2C155 300w, https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/fp.png?resize=768%2C398 768w, https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/fp.png?w=1576 1576w, https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/fp.png?w=948 948w, https:\/\/i0.wp.com\/amberdot.com\/walterbays\/wp-content\/uploads\/2017\/03\/fp.png?w=1422 1422w\" sizes=\"auto, (max-width: 453px) 100vw, 453px\" \/><\/a>The story with floating point is much the same. Clearly it is time for a new SPEC CPU benchmark suite. The\u00a0larger question for SPEC is, with such improvements does performance even matter any more? Spoiler alert: my answer is yes, and that will be the subject of a future posting.<\/p>\n<p><strong>Note<\/strong>: <em>These are my ideas and opinions, with no SPEC &#8220;inside information.&#8221; I&#8217;m bound by the same confidentiality rules as any SPEC participant. These rules are not to conceal what the vendors are doing. Actually SPEC is quite an open organization. Rather there are two main points: 1) I cannot state SPEC (consensus) positions nor commit to product deliverables and dates. Only SPEC can speak for SPEC. And 2) I cannot relate positions of member companies, whether correctly or incorrectly related, as SPEC participants must feel free to argue and agree with one another.<\/em><\/p>\n","protected":false},"excerpt":{"rendered":"<p>Among the challenges SPEC faces in addressing change in the computing industry is the very trend which SPEC has done more than anyone to bring about. That is the rapid growth in computing power &#8211; close to 10,000 times the single thread performance today as the venerable VAX-11\/780. Dr. Stephen Edwards teaches performance history at &hellip; <a href=\"http:\/\/amberdot.com\/walterbays\/index.php\/2017\/03\/22\/performance-trajectories-speed-and-throughput\/\" class=\"more-link\">Continue reading <span class=\"screen-reader-text\">Performance Trajectories &#8211; speed and throughput<\/span> <span class=\"meta-nav\">&rarr;<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"nf_dc_page":"","om_disable_all_campaigns":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"nfd_image_url":"","nfd_onboarding_generated":"","footnotes":"","jetpack_post_was_ever_published":false},"categories":[2],"tags":[],"class_list":["post-18","post","type-post","status-publish","format-standard","hentry","category-spec"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"","_links":{"self":[{"href":"http:\/\/amberdot.com\/walterbays\/index.php\/wp-json\/wp\/v2\/posts\/18","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/amberdot.com\/walterbays\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/amberdot.com\/walterbays\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/amberdot.com\/walterbays\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"http:\/\/amberdot.com\/walterbays\/index.php\/wp-json\/wp\/v2\/comments?post=18"}],"version-history":[{"count":3,"href":"http:\/\/amberdot.com\/walterbays\/index.php\/wp-json\/wp\/v2\/posts\/18\/revisions"}],"predecessor-version":[{"id":26,"href":"http:\/\/amberdot.com\/walterbays\/index.php\/wp-json\/wp\/v2\/posts\/18\/revisions\/26"}],"wp:attachment":[{"href":"http:\/\/amberdot.com\/walterbays\/index.php\/wp-json\/wp\/v2\/media?parent=18"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/amberdot.com\/walterbays\/index.php\/wp-json\/wp\/v2\/categories?post=18"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/amberdot.com\/walterbays\/index.php\/wp-json\/wp\/v2\/tags?post=18"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}