<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
@font-face
{font-family:"Courier New \,serif";
panose-1:0 0 0 0 0 0 0 0 0 0;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
code
{mso-style-priority:99;
font-family:"Courier New";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.line
{mso-style-name:line;}
span.EmailStyle26
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:351609561;
mso-list-template-ids:1271054596;}
@list l0:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level2
{mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1
{mso-list-id:451285117;
mso-list-template-ids:-2081275220;}
@list l1:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level2
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level3
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level5
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level6
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level8
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l1:level9
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2
{mso-list-id:581990874;
mso-list-template-ids:82503306;}
@list l2:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l2:level3
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level5
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level6
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level8
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l2:level9
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l3
{mso-list-id:616914144;
mso-list-type:hybrid;
mso-list-template-ids:2105163810 134807553 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l3:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l3:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l3:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l3:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l3:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l3:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l3:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l4
{mso-list-id:675301323;
mso-list-template-ids:-681260820;}
@list l4:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level2
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level3
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level5
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level6
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level8
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l4:level9
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5
{mso-list-id:949318904;
mso-list-template-ids:1197664566;}
@list l5:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level2
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level3
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level5
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level6
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level8
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l5:level9
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l6
{mso-list-id:1055272184;
mso-list-template-ids:-377160358;}
@list l6:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l6:level2
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l6:level3
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l6:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l6:level5
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l6:level6
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l6:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l6:level8
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l6:level9
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l7
{mso-list-id:1203447298;
mso-list-template-ids:-1401414062;}
@list l7:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l7:level2
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l7:level3
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l7:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l7:level5
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l7:level6
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l7:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l7:level8
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l7:level9
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l8
{mso-list-id:1228569700;
mso-list-template-ids:-833207604;}
@list l8:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l8:level2
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l8:level3
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l8:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l8:level5
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l8:level6
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l8:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l8:level8
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l8:level9
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l9
{mso-list-id:1728723836;
mso-list-template-ids:1422162502;}
@list l9:level1
{mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l9:level2
{mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l9:level3
{mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l9:level4
{mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l9:level5
{mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l9:level6
{mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l9:level7
{mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l9:level8
{mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l9:level9
{mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l10
{mso-list-id:1873882282;
mso-list-template-ids:1422162502;}
@list l10:level1
{mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l10:level2
{mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l10:level3
{mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l10:level4
{mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l10:level5
{mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l10:level6
{mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l10:level7
{mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l10:level8
{mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l10:level9
{mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l11
{mso-list-id:1971669979;
mso-list-template-ids:-1264965908;}
@list l11:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l11:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l11:level3
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l11:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l11:level5
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l11:level6
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l11:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l11:level8
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l11:level9
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l12
{mso-list-id:2051954400;
mso-list-template-ids:584598960;}
@list l12:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l12:level2
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l12:level3
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l12:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l12:level5
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l12:level6
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l12:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l12:level8
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l12:level9
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l13
{mso-list-id:2059430332;
mso-list-template-ids:1281681506;}
@list l13:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l13:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l13:level3
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l13:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l13:level5
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l13:level6
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l13:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l13:level8
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l13:level9
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l14
{mso-list-id:2084179046;
mso-list-template-ids:-864887618;}
@list l14:level1
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:36.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l14:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:72.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:"Courier New";
mso-bidi-font-family:"Times New Roman";}
@list l14:level3
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:108.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l14:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:144.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l14:level5
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:180.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l14:level6
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:216.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l14:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:252.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l14:level8
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:288.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
@list l14:level9
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:324.0pt;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-font-size:10.0pt;
font-family:Symbol;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-GB" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal" style="margin-left:36.0pt">This information is a bit spread out over the wiki page.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><i>Which</i> wiki page? Yes, it’d be fantastic to write this out clearly. Thanks!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<pre style="margin-left:36.0pt">$ git checkout a12b34c56 && git submodule update --init<o:p></o:p></pre>
<pre style="margin-left:36.0pt">$ ./hadrian/build.sh test --only-perf<o:p></o:p></pre>
<pre style="margin-left:36.0pt">$ git checkout x98y76z54 && git submodule update --init<o:p></o:p></pre>
<pre style="margin-left:36.0pt">$ ./hadrian/build.sh test --only-perf<o:p></o:p></pre>
<pre style="margin-left:36.0pt">$ <code>python3 testsuite/driver/perf_notes.py --chart chart.html --test-env local </code>a12b34c56 x98y76z54<o:p></o:p></pre>
<pre style="margin-left:36.0pt">$ firefox <code>chart.html</code><o:p></o:p></pre>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Ah. Now I’m lost. Somehow the second and fourth line must be recording info, locally in my tree, but two distinct batches of information. Perhaps kept distinct by the current commit? Where is the info actually stored?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">OK, suppose I start from commit XX, and make some local changes. Then I do the –only-perf thing. presumably that’ll be recorded tagged with XX. That’s fine; just want it to be clear. Worth adding this info to the wiki page, so we have
a clear mental model.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Simon<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> David Eichmann <davide@well-typed.com>
<br>
<b>Sent:</b> 23 January 2020 11:19<br>
<b>To:</b> Simon Peyton Jones <simonpj@microsoft.com><br>
<b>Subject:</b> Re: GHC perf<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p>Simon<o:p></o:p></p>
<ul type="disc">
<li class="MsoListParagraph" style="mso-list:l12 level1 lfo1">This compares two local builds<o:p></o:p></li></ul>
<p>Yes<o:p></o:p></p>
<ul type="disc">
<li class="MsoListParagraph" style="mso-list:l8 level1 lfo2">It does not require fetching CI perf data; in fact it 100% independent of the CI system<o:p></o:p></li></ul>
<p>Yes<o:p></o:p></p>
<ul type="disc">
<li class="MsoListParagraph" style="mso-list:l5 level1 lfo3">It does require two separate build trees (that is fine)<o:p></o:p></li></ul>
<p>No, this does not require different build trees, <baseline> and <target> are git commits (or similar e.g. branch name). The actual process might look like:
<o:p></o:p></p>
<pre>$ git checkout a12b34c56 && git submodule update --init<o:p></o:p></pre>
<pre>$ ./hadrian/build.sh test --only-perf<o:p></o:p></pre>
<pre>$ git checkout x98y76z54 && git submodule update --init<o:p></o:p></pre>
<pre>$ ./hadrian/build.sh test --only-perf<o:p></o:p></pre>
<pre>$ <code>python3 testsuite/driver/perf_notes.py --chart chart.html --test-env local </code>a12b34c56 x98y76z54<o:p></o:p></pre>
<pre>$ firefox <code>chart.html</code><o:p></o:p></pre>
<p>This information is a bit spread out over the wiki page. Perhaps a "quick start" section describing this use case would be helpful.<o:p></o:p></p>
<div>
<p class="MsoNormal">On 1/22/20 10:54 AM, Simon Peyton Jones wrote:<o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">David<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Thanks. Concerning this:<o:p></o:p></p>
<ol start="1" type="1">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l10 level1 lfo4">
Checkout an the <baseline> commit.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l10 level1 lfo4">
Use `git status` to double check git sees a clean working tree.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l10 level1 lfo4">
Run the performance tests.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l10 level1 lfo4">
Check out your <target> branch.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l10 level1 lfo4">
Use `git status` to double check git sees a clean working tree (else commit any changes)<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l10 level1 lfo4">
Run the performance tests.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l10 level1 lfo4">
Compare metrics (filtering for `local` metrics and outputting a chart):<o:p></o:p></li></ol>
<p> <code><span style="font-size:10.0pt">python3 testsuite/driver/perf_notes.py --chart chart.html --test-env local <baseline> <target></span></code><o:p></o:p></p>
<p class="MsoNormal">I believe that<o:p></o:p></p>
<ul type="disc">
<li class="MsoListParagraph" style="mso-list:l3 level1 lfo5">This compares two local builds<o:p></o:p></li><li class="MsoListParagraph" style="mso-list:l3 level1 lfo5">It does not require fetching CI perf data; in fact it 100% independent of the CI system<o:p></o:p></li><li class="MsoListParagraph" style="mso-list:l3 level1 lfo5">It does require two separate build trees (that is fine)<o:p></o:p></li></ul>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Is that right? If so, two questions<o:p></o:p></p>
<ul type="disc">
<li class="MsoListParagraph" style="mso-list:l3 level1 lfo5">In that Python command line (step 7) is “<baseline>” the path to the root of the baseline tree, or to some file within that tree?<o:p></o:p></li><li class="MsoListParagraph" style="mso-list:l3 level1 lfo5">Is this process (and what it does) written up on some wiki page somewhere? Where? Rather than replying to me individually, it’d be better to use this conversation to produce better guidance for everyone.<o:p></o:p></li></ul>
<p class="MsoNormal">Thanks<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Simon<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt">
<div>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> David Eichmann
<a href="mailto:davide@well-typed.com"><davide@well-typed.com></a> <br>
<b>Sent:</b> 20 January 2020 10:37<br>
<b>To:</b> Simon Peyton Jones <a href="mailto:simonpj@microsoft.com"><simonpj@microsoft.com></a>; Ben Gamari
<a href="mailto:ben@well-typed.com"><ben@well-typed.com></a><br>
<b>Cc:</b> ghc-devs <a href="mailto:ghc-devs@haskell.org"><ghc-devs@haskell.org></a><br>
<b>Subject:</b> Re: GHC perf</span><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">Hi Simon, <o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<ul type="disc">
<li class="MsoListParagraph" style="mso-list:l6 level1 lfo6">There are two things going on:<o:p></o:p></li></ul>
<ul type="disc">
<ol start="1" type="1">
<li class="MsoListParagraph" style="mso-list:l0 level2 lfo7">CI perf measurements<o:p></o:p></li><li class="MsoListParagraph" style="mso-list:l0 level2 lfo7">Local machine perf measurements<o:p></o:p></li></ol>
</ul>
<p class="MsoListParagraph">I think that they are somehow handled differently (why?) but they are all muddled up on the wiki page.<o:p></o:p></p>
</div>
</blockquote>
<p>They are handled differently because we do not want to compare local metrics with CI metrics. The exception is when local metrics don't exist, then we fall back to CI metrics as a baseline (see
<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2Fwikis%2Fbuilding%2Frunning-tests%2Fperformance-tests%23how-baselines-are-calculated&data=02%7C01%7Csimonpj%40microsoft.com%7Cc4ed72566411438ee43208d79ff60518%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637153751314078539&sdata=s%2BNW5fv9ot2PMytFO%2FdGznCgVViJOafOnR3F4HJVSvs%3D&reserved=0">
How baseline metrics are calculated</a>).<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<ul type="disc">
<li class="MsoListParagraph" style="mso-list:l7 level1 lfo8">My goal is this: <o:p>
</o:p></li></ul>
<ul type="disc">
<ul type="circle">
<li class="MsoListParagraph" style="mso-list:l2 level2 lfo9">Start with a master commit, say from Dec 2019.<o:p></o:p></li><li class="MsoListParagraph" style="mso-list:l2 level2 lfo9">Implement some change, on a branch.<o:p></o:p></li><li class="MsoListParagraph" style="mso-list:l2 level2 lfo9">sh validate –legacy (or something else if you like)<o:p></o:p></li><li class="MsoListParagraph" style="mso-list:l2 level2 lfo9">Look at perf regressions.<o:p></o:p></li></ul>
</ul>
</div>
</blockquote>
<p class="MsoNormal">Getting to the *raw data* should be easy: <o:p></o:p></p>
<ol start="1" type="1">
<li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l9 level1 lfo10">
Checkout an the <baseline> commit.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l9 level1 lfo10">
Use `git status` to double check git sees a clean working tree.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l9 level1 lfo10">
Run the performance tests.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l9 level1 lfo10">
Check out your <target> branch.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l9 level1 lfo10">
Use `git status` to double check git sees a clean working tree (else commit any changes)<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l9 level1 lfo10">
Run the performance tests.<o:p></o:p></li><li class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;mso-list:l9 level1 lfo10">
Compare metrics (filtering for `local` metrics and outputting a chart):<o:p></o:p></li></ol>
<p> <code><span style="font-size:10.0pt">python3 testsuite/driver/perf_notes.py --chart chart.html --test-env local <baseline> <target></span></code><o:p></o:p></p>
<p class="MsoNormal">see `<code><span style="font-size:10.0pt">python3 testsuite/driver/perf_notes.py --help`</span></code> for more filtering options. This doesn't detect regressions automatically, it only shows you the raw data. Ideally we'd add an option
to the testrunner to let you specify a baseline commit manually. I suspect that would be close to what you're looking for.<br>
<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<ul type="disc">
<li class="MsoListParagraph" style="mso-list:l4 level1 lfo11">I believe I have first to utter the incantation<o:p></o:p></li></ul>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto;margin-left:72.0pt">
<span class="line"><span style="font-size:10.0pt;font-family:"Courier New ,serif",serif">$ git fetch
<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc-performance-notes.git&data=02%7C01%7Csimonpj%40microsoft.com%7Cc4ed72566411438ee43208d79ff60518%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637153751314078539&sdata=4%2FpItW9dgtS4n1Mw6Nc%2FLwk4zBTm6Hs%2BUFY2nmax%2Fe8%3D&reserved=0">
https://gitlab.haskell.org/ghc/ghc-performance-notes.git</a> refs/notes/perf:refs/notes/ci/perf</span></span><o:p></o:p></p>
</div>
</blockquote>
<p class="MsoNormal">Yes, this fetches the latest CI metrics into your git notes.<br>
<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<ul type="disc">
<li class="MsoListParagraph" style="mso-list:l13 level1 lfo12">But then:<o:p></o:p></li></ul>
<ul type="disc">
<ul type="circle">
<li class="MsoListParagraph" style="mso-list:l13 level2 lfo12">How do I ensure that the baseline perf numbers I get relate to the master commit I started from, back in Dec 2019? I don’t want numbers from Jan 2020.<o:p></o:p></li></ul>
</ul>
</div>
</blockquote>
<p class="MsoNormal">see above.<br>
<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<ul type="disc">
<ul type="circle">
<li class="MsoListParagraph" style="mso-list:l14 level2 lfo13">If I rebase my branch on top of HEAD, say, how do I update the perf baseline numbers to be for HEAD<o:p></o:p></li></ul>
</ul>
</div>
</blockquote>
<p class="MsoNormal">The test runner should use HEAD's metrics automatically (see
<a href="https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.haskell.org%2Fghc%2Fghc%2Fwikis%2Fbuilding%2Frunning-tests%2Fperformance-tests%23how-baselines-are-calculated&data=02%7C01%7Csimonpj%40microsoft.com%7Cc4ed72566411438ee43208d79ff60518%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637153751314088533&sdata=QwciHCg3%2FkBrddbKxTRR04kyYVUQkACuRdlpitqSMrU%3D&reserved=0">
How baseline metrics are calculated</a>), though you will need to fetch CI metrics or run the perf tests locally on HEAD to get the relevant metrics.<br>
<br>
<br>
<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<ul type="disc">
<ul type="circle">
<li class="MsoListParagraph" style="mso-list:l11 level2 lfo14">Generally, how can I tell the commit to which the baseline numbers relate?<o:p></o:p></li></ul>
</ul>
</div>
</blockquote>
<p class="MsoNormal">The test runner will output (per test) which baseline commit is used e.g. "... from local
<o:p></o:p></p>
<div>
<p class="MsoNormal"><span style="color:black">baseline @ HEAD~2" says the baseline was a local run from 2 commits ago.</span><o:p></o:p></p>
</div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<ul type="disc">
<li class="MsoListParagraph" style="mso-list:l1 level1 lfo15">Also, in my tree I have a series of incremental changes; I want to see if any of them have perf regressions. How do I do that?<o:p></o:p></li></ul>
</div>
</blockquote>
<p>You can run the perf tests on each commit *in commit order*, and the previous commit will always be used as the baseline. You can also then chart the results:<o:p></o:p></p>
<p> <code><span style="font-size:10.0pt">python3 testsuite/driver/perf_notes.py --chart chart.html --test-env local <oldestCommit>..<newestCommit></span></code><o:p></o:p></p>
<p>Sorry if this is a bit unoptimal, but I Hope that helps<o:p></o:p></p>
<p>- David E<o:p></o:p></p>
<p> <o:p></o:p></p>
<pre> <o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>-- <o:p></o:p></pre>
<pre>David Eichmann, Haskell Consultant<o:p></o:p></pre>
<pre>Well-Typed LLP, <a href="https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.well-typed.com%2F&data=02%7C01%7Csimonpj%40microsoft.com%7Cc4ed72566411438ee43208d79ff60518%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637153751314088533&sdata=McXFFQ5Ln4exv4mfPhL742%2BbmGbcodP1PgLNBi%2BUWmI%3D&reserved=0">http://www.well-typed.com</a><o:p></o:p></pre>
<pre> <o:p></o:p></pre>
<pre>Registered in England & Wales, OC335890<o:p></o:p></pre>
<pre>118 Wymering Mansions, Wymering Road, London W9 2NF, England <o:p></o:p></pre>
</div>
</blockquote>
<pre>-- <o:p></o:p></pre>
<pre>David Eichmann, Haskell Consultant<o:p></o:p></pre>
<pre>Well-Typed LLP, <a href="https://nam06.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.well-typed.com%2F&data=02%7C01%7Csimonpj%40microsoft.com%7Cc4ed72566411438ee43208d79ff60518%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637153751314098526&sdata=5jcfMZMpIvQ2UbF8sfkcT7ESqtSVDTKeehTMG5%2BhPFU%3D&reserved=0">http://www.well-typed.com</a><o:p></o:p></pre>
<pre><o:p> </o:p></pre>
<pre>Registered in England & Wales, OC335890<o:p></o:p></pre>
<pre>118 Wymering Mansions, Wymering Road, London W9 2NF, England <o:p></o:p></pre>
</div>
</div>
</body>
</html>