PERF EVAL (CONT’D)
description
Transcript of PERF EVAL (CONT’D)
![Page 1: PERF EVAL (CONT’D)](https://reader036.fdocumente.com/reader036/viewer/2022083012/56815d34550346895dcb305b/html5/thumbnails/1.jpg)
1
PERF EVAL (CONT’D)PERF EVAL (CONT’D)There are many other “tools of the
trade” used in performance evaluationOnly a few will be mentioned here:
– queueing theory– verification and validation– statistical analysis– multi-variate analysis– presentation of results
![Page 2: PERF EVAL (CONT’D)](https://reader036.fdocumente.com/reader036/viewer/2022083012/56815d34550346895dcb305b/html5/thumbnails/2.jpg)
2
Queueing TheoryQueueing Theory
A mathematical technique that specializes in the analysis of queues (e.g., customer arrivals at a bank, jobs arriving at CPU, I/O requests arriving at a disk subsystem)
General diagram:CustomerArrivals Departures
Buffer Server
![Page 3: PERF EVAL (CONT’D)](https://reader036.fdocumente.com/reader036/viewer/2022083012/56815d34550346895dcb305b/html5/thumbnails/3.jpg)
3
Queueing Theory (cont’d)Queueing Theory (cont’d)
The queueing system is characterized by:– Arrival process (M, G)– Service time (M, D, G)– Number of servers (1 to infinity)– Number of buffers (infinite or finite)
Example notation: M/M/1, M/D/1Example notation: M/M/ , M/G/1/k8
![Page 4: PERF EVAL (CONT’D)](https://reader036.fdocumente.com/reader036/viewer/2022083012/56815d34550346895dcb305b/html5/thumbnails/4.jpg)
4
Queueing Theory (cont’d)Queueing Theory (cont’d)
There are well-known mathematical results for the mean waiting time and the number of customers in the system for several simple queueing models
E.g., M/M/1, M/D/1, M/G/1Example: M/M/1
– q = rho/ (1 - rho) where rho = lambda/mu < 1
![Page 5: PERF EVAL (CONT’D)](https://reader036.fdocumente.com/reader036/viewer/2022083012/56815d34550346895dcb305b/html5/thumbnails/5.jpg)
5
Queueing Theory (cont’d)Queueing Theory (cont’d)These simple models can be cascaded in
series and in parallel to create arbitrarily large complicated queueing network models
Two main types:– closed queueing network model (finite pop.)– open queueing network model (infinite pop.)
Software packages exist for solving these types of models to determine steady-state performance (e.g., delay, throughput, util.)
![Page 6: PERF EVAL (CONT’D)](https://reader036.fdocumente.com/reader036/viewer/2022083012/56815d34550346895dcb305b/html5/thumbnails/6.jpg)
6
Verification and ValidationVerification and ValidationAn important step in any modeling work
(simulation or analytical) is convincing others that the model is “correct”
Verification: develop simple test cases with known inputs; compare to expected outputs
Validation: the “reality check” to see if model predictions agree with real world
Sanity checks (e.g., Little’s Law: N = T)This V&V process is often overlooked!!!
![Page 7: PERF EVAL (CONT’D)](https://reader036.fdocumente.com/reader036/viewer/2022083012/56815d34550346895dcb305b/html5/thumbnails/7.jpg)
7
Statistical AnalysisStatistical Analysis “Math and stats can be your friends!!!” CWThere are lots of “standard” techniques from
mathematics, probability, and statistics that are of immense value in performance work:– confidence intervals, null hypotheses, F-tests,
T-tests, linear regression, least-squares fit, maximum likelihood estimation, correlation, time series analysis, transforms, Q-Q, EM...
– working knowledge of commonly-observed statistical distributions
![Page 8: PERF EVAL (CONT’D)](https://reader036.fdocumente.com/reader036/viewer/2022083012/56815d34550346895dcb305b/html5/thumbnails/8.jpg)
8
Multi-Variate AnalysisMulti-Variate Analysis For in-depth and really messy data analysis,
there are multi-variate techniques that can be immensely helpful
In many cases, good data visualization tools will tell you a lot (e.g., plotting graphs), but in other cases you might try things like:– multi-variate regression: find out which
parameters are relevant or not for curve fitting– ANOVA: analysis of variance can show the
parameters with greatest impact on results
![Page 9: PERF EVAL (CONT’D)](https://reader036.fdocumente.com/reader036/viewer/2022083012/56815d34550346895dcb305b/html5/thumbnails/9.jpg)
9
Presentation of ResultsPresentation of ResultsGraphs and tables are the two most common
ways of illustrating and/or summarizing data– graphs can show you the trends– tables provide the details
There are good ways and bad ways to do each of these
Again, it is a bit of an “art”, but there are lots of good tips and guidelines as well
![Page 10: PERF EVAL (CONT’D)](https://reader036.fdocumente.com/reader036/viewer/2022083012/56815d34550346895dcb305b/html5/thumbnails/10.jpg)
10
Table TipsTable TipsDecide if a table is really needed; if so, should it
be part of main paper, or just an appendix?Choose formatting software with which you are
familiar; easy to import data, export tablesTable caption goes at the topClearly delineate rows and columns (lines)Logically organize rows and columnsReport results to several significant digitsBe consistent in formatting wherever possible
![Page 11: PERF EVAL (CONT’D)](https://reader036.fdocumente.com/reader036/viewer/2022083012/56815d34550346895dcb305b/html5/thumbnails/11.jpg)
11
Graphing TipsGraphing TipsChoose a good software package, preferably
one with which you are familiar, and one for which it is easy to import data, export graphs
Title at top; caption below (informative)Labels on each axis, including unitsLogical step sizes along axes (10’s, 100’s…)Make sure choice of scale is clear for each
axis (linear, log-linear, log-log)Graph should start from origin (zero) unless
there is a compelling reason not to do so
![Page 12: PERF EVAL (CONT’D)](https://reader036.fdocumente.com/reader036/viewer/2022083012/56815d34550346895dcb305b/html5/thumbnails/12.jpg)
12
Graphing Tips (cont’d)Graphing Tips (cont’d)Make judicious choice of type of plot
– scatter plot, line graph, bar chart, histogramMake judicious choice of line types
– solid, dashed, dotted, lines and points, colours If multiple lines on a plot, then use a key,
which should be well-placed and informative If graph is “well-behaved”, then organize the
key to match the lines on the graph (try it!)Be consistent from one graph to the next
wherever possible (size, scale, key, colours)