criterion performance measurements
overview
want to understand this report?
Int/IntMap/sorted
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.819502838635092e-3 | 8.937630044753272e-3 | 9.11687814107912e-3 |
| Standard deviation | 2.961206051453886e-4 | 3.8811768961642034e-4 | 5.209891461490966e-4 |
Outlying measurements have moderate (0.1783968758696835%) effect on estimated standard deviation.
Int/IntMap/random
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.8542157501053912e-2 | 2.900743929111707e-2 | 2.958125243996035e-2 |
| Standard deviation | 8.657166917576245e-4 | 1.0721292520478878e-3 | 1.363882252593622e-3 |
Outlying measurements have moderate (0.10833474193374858%) effect on estimated standard deviation.
Int/IntMap/revsorted
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.112766790888372e-3 | 9.362502704743527e-3 | 9.835610852580709e-3 |
| Standard deviation | 5.352201403618318e-4 | 8.624425557441545e-4 | 1.2370142915980008e-3 |
Outlying measurements have severe (0.5003056810937044%) effect on estimated standard deviation.
Int/Map/sorted
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.074059258295756e-2 | 3.211011289529789e-2 | 3.527622608186352e-2 |
| Standard deviation | 1.864156077219858e-3 | 4.2796965648887655e-3 | 7.382183902703018e-3 |
Outlying measurements have severe (0.5703320814503217%) effect on estimated standard deviation.
Int/Map/random
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.256863169781285e-2 | 5.423701345317711e-2 | 5.75263665915807e-2 |
| Standard deviation | 1.8884040663696014e-3 | 4.018496861800647e-3 | 6.914612452206933e-3 |
Outlying measurements have moderate (0.2191092792162958%) effect on estimated standard deviation.
Int/Map/revsorted
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 2.893885738311336e-2 | 2.9700954738630478e-2 | 3.0814787788813195e-2 |
| Standard deviation | 1.2367503170643201e-3 | 1.9677833953441904e-3 | 3.1655852527274785e-3 |
Outlying measurements have moderate (0.2237015542046663%) effect on estimated standard deviation.
Int/HashMap/sorted
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.4908814983483784e-2 | 3.609431558970996e-2 | 3.836960118282639e-2 |
| Standard deviation | 1.8750127434045843e-3 | 3.3717688192625266e-3 | 5.852156692756491e-3 |
Outlying measurements have moderate (0.3592629030261099%) effect on estimated standard deviation.
Int/HashMap/random
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.484697359895852e-2 | 3.704957245731288e-2 | 4.314680605467057e-2 |
| Standard deviation | 2.109599986174092e-3 | 6.49252372752462e-3 | 1.0967241520954466e-2 |
Outlying measurements have severe (0.6515906864355713%) effect on estimated standard deviation.
Int/HashMap/revsorted
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 3.438757486779517e-2 | 3.6006875840224564e-2 | 3.8605071212944604e-2 |
| Standard deviation | 2.3146055359431917e-3 | 3.86867675559429e-3 | 5.443213496982174e-3 |
Outlying measurements have moderate (0.42130793821828655%) effect on estimated standard deviation.
ByteString/Map/sorted
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.07748582599183e-3 | 8.32904866130215e-3 | 8.806077418534286e-3 |
| Standard deviation | 5.211793934928915e-4 | 9.312595677147134e-4 | 1.4390411593630463e-3 |
Outlying measurements have severe (0.6210257293950999%) effect on estimated standard deviation.
ByteString/Map/random
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 1.0110133435932062e-2 | 1.0387725441922272e-2 | 1.0842328111211384e-2 |
| Standard deviation | 5.911127933420502e-4 | 9.711546553142731e-4 | 1.592031821668909e-3 |
Outlying measurements have severe (0.5163836979954514%) effect on estimated standard deviation.
ByteString/Map/revsorted
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 8.028422157830457e-3 | 8.286984315863834e-3 | 8.759854503977735e-3 |
| Standard deviation | 5.086836504735383e-4 | 9.26637169575568e-4 | 1.7093393975807325e-3 |
Outlying measurements have severe (0.6210284027587997%) effect on estimated standard deviation.
ByteString/HashMap/sorted
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.59492287832012e-3 | 5.715304733736597e-3 | 5.854873905308174e-3 |
| Standard deviation | 2.897995995948311e-4 | 3.6805494396342846e-4 | 5.351977853987202e-4 |
Outlying measurements have moderate (0.3703243985993333%) effect on estimated standard deviation.
ByteString/HashMap/random
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.719008056297191e-3 | 6.011378247359856e-3 | 6.4127589615828925e-3 |
| Standard deviation | 6.413363439360069e-4 | 9.996367294972932e-4 | 1.4127482949420139e-3 |
Outlying measurements have severe (0.8007711032151971%) effect on estimated standard deviation.
ByteString/HashMap/revsorted
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 5.624547446996474e-3 | 5.687942787530952e-3 | 5.770744183837958e-3 |
| Standard deviation | 1.6570136451117353e-4 | 2.341919953104207e-4 | 3.860014545750548e-4 |
Outlying measurements have moderate (0.19518394390669644%) effect on estimated standard deviation.
ByteString/CritBit/sorted
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.230125738946211e-3 | 4.303448916215421e-3 | 4.397927873894102e-3 |
| Standard deviation | 2.0581719155502942e-4 | 2.643838981947279e-4 | 3.476473011416311e-4 |
Outlying measurements have moderate (0.3933449069031089%) effect on estimated standard deviation.
ByteString/CritBit/random
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 9.041657318955057e-3 | 9.351051711024927e-3 | 9.996201118714297e-3 |
| Standard deviation | 6.022697620875256e-4 | 1.1937269103816997e-3 | 2.142053000129604e-3 |
Outlying measurements have severe (0.6632260188850402%) effect on estimated standard deviation.
ByteString/CritBit/revsorted
| lower bound | estimate | upper bound | |
|---|---|---|---|
| OLS regression | xxx | xxx | xxx |
| R² goodness-of-fit | xxx | xxx | xxx |
| Mean execution time | 4.291530142015521e-3 | 4.520433177048569e-3 | 4.95343113631778e-3 |
| Standard deviation | 6.692430182547295e-4 | 1.0286480378572111e-3 | 1.695181647792266e-3 |
Outlying measurements have severe (0.9034231878411371%) effect on estimated standard deviation.
understanding this report
In this report, each function benchmarked by criterion is assigned a section of its own. The charts in each section are active; if you hover your mouse over data points and annotations, you will see more details.
- The chart on the left is a kernel density estimate (also known as a KDE) of time measurements. This graphs the probability of any given time measurement occurring. A spike indicates that a measurement of a particular time occurred; its height indicates how often that measurement was repeated.
- The chart on the right is the raw data from which the kernel density estimate is built. The x axis indicates the number of loop iterations, while the y axis shows measured execution time for the given number of loop iterations. The line behind the values is the linear regression prediction of execution time for a given number of iterations. Ideally, all measurements will be on (or very near) this line.
Under the charts is a small table. The first two rows are the results of a linear regression run on the measurements displayed in the right-hand chart.
- OLS regression indicates the time estimated for a single loop iteration using an ordinary least-squares regression model. This number is more accurate than the mean estimate below it, as it more effectively eliminates measurement overhead and other constant factors.
- R² goodness-of-fit is a measure of how accurately the linear regression model fits the observed measurements. If the measurements are not too noisy, R² should lie between 0.99 and 1, indicating an excellent fit. If the number is below 0.99, something is confounding the accuracy of the linear model.
- Mean execution time and standard deviation are statistics calculated from execution time divided by number of iterations.
We use a statistical technique called the bootstrap to provide confidence intervals on our estimates. The bootstrap-derived upper and lower bounds on estimates let you see how accurate we believe those estimates to be. (Hover the mouse over the table headers to see the confidence levels.)
A noisy benchmarking environment can cause some or many measurements to fall far from the mean. These outlying measurements can have a significant inflationary effect on the estimate of the standard deviation. We calculate and display an estimate of the extent to which the standard deviation has been inflated by outliers.