Minor driver update to swingbench and dbtimemonitor

I’ve posted a minor update to swingbench and dbtimemonitor to bring them up to the latest versions of the jdbc drivers and fix a “few” niggling bugs. Please refresh to these builds particularly if you plan to do any 12.1.0.2 testing in the future.

You can as always download them from
here
Comments

New jdbc drivers for swingbench

With the launch of Oracle Database 12c Release 1 (12.1.0.2) I’ve taken the opportunity to update the jdbc drivers that swingbench uses. If you’re testing UCP or FAN I strongly recommend using these new drivers.

You can download it
here
Comments

New build of swingbench with improved stats

I’ve uploaded a new build of swingbench that fixes a number of bugs but also improves on the stats being produced.

One problem with using averages when analysing results is that they hide a multitude of evils. This is particularly true for response times where there are likely to be big skews hidden if just the average is considered. You can see this in the chart below where response times are mapped into 100 buckets. The response time range from 2 to 6274 milliseconds. The average is 257ms.

Numbers

It might be that in many instances the average is adequate for your needs but if your interested in the spread of results and the impact metrics like user counts, IO, memory etc have on the responsiveness of your system it might also be useful to model the spread of response so you can look for outliers.

In the latest build of swingbench when you now specify full stats collection you end up with a more complete set of results as shown below

<Result id="Process Orders">
        <AverageResponse>257.86816475601097</AverageResponse>
        <MinimumResponse>2</MinimumResponse>
        <MaximumResponse>6274</MaximumResponse>
        <GeometricMean>117.80819459762236</GeometricMean>
        <StdDeviation>276.6820016426486</StdDeviation>
        <Variance>76552.9300329826</Variance>
        <Kurtosis>7.96379726468569</Kurtosis>
        <Skewness>1.7844202125604447</Skewness>
        <TenthPrecentile>14</TenthPrecentile>
        <TwentiethPercentile>25</TwentiethPercentile>
        <ThirtiethPercentile>43</ThirtiethPercentile>
        <FortiethPrecentile>87</FortiethPrecentile>
        <FiftythPercentile>170</FiftythPercentile>
        <SixtiethPercentile>257</SixtiethPercentile>
        <SeventiethPercentile>354</SeventiethPercentile>
        <EightiethPercentile>466</EightiethPercentile>
        <NinetiethPercentile>636</NinetiethPercentile>
        <ResultsHistogram>21495, 5260, 4380, 4084, 3929, 3798, 3266, 2756, 2303, 1801, 1465, 1138, 889, 632, 499, 381, 259, 201, 140, 105, 102, 55, 43, 21, 21, 21, 17, 12, 9, 4, 5, 3, 12, 1, 6, 3, 7, 5, 1, 1, 6, 1, 2, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1</ResultsHistogram>
        <TransactionCount>59142</TransactionCount>
        <FailedTransactionCount>0</FailedTransactionCount>
        <RollbackCount>0</RollbackCount>
</Result>


I’ve included a complete set of percentiles and some additional metrics for consideration (variance, Kurtosis, Skewness, Geometric Mean). Over the coming weeks I’ll be attempting to process a results file into a more useful document.

You can enable stats collection in the UI from the preferences tab i.e.

Preferences

You can also set it from the command line. i.e.

./charbench -bg -s -stats full -rt 0:04 -cs //oracle12c2/orcl -a -uc 25 -com "Test of full stats collection" -intermin 2 -intermax 2 -bs 0:01 -be 0:04 &

One thing to watch out for is that you may need to change the metric that transactions are measure in i.e. from milliseconds to microseconds to model a better spread of response times.

Along side the improvements to stats I’ve also fixed the following
  • Fixed windowed stats collection (-be -bs) and full stats (-stats full) working together
  • Fixed the -bg (background) option so it works on Solaris
  • Numerous stability fixes

You can download the code front he usual place
here
Comments

New build of swingbench

I’ve just finished a new build of swingbench and the command line got a little bit of love. I’ve added the following new functionality

  • Added new transactions to sh benchmark
  • Percentiles now report 10th to 90th percentiles instead of just 25th,50th and 75th percentile
  • Added a new command line option to allow users to change stats collection target.
  • Fixed the command line option “-bg” so it’s possible to background charbench
  • Added new commands to coordinator to make it simpler to use
    • Changed -stop to -kill to better indicate what it does
    • Changed -halt to -stop to indicate what it does
    • Added -stopall to stop all attached clients
    • Added -runall to start all attached clients
    • Added -stats to enable all display aggregated transaction rates of all attached clients
  • Included python script to parse the one or more results files into csv format (located in $SWINGHOME/utils)
  • forced users to specify either -create, -drop or -generate when specify character mode (-c)

The functionality enabling you to background and the changes to the coordinator might need a little more explaining. The idea is that you might want to run up more than one load generator either because you want to create a truly tremendous load you want to run different load generators at different databases. To do this you might want to use a combination of backgrounding the load generators and using the coordinator to start,stop and report on them. For example

$ ./coordinator &
$ ./charbench -cs //oracle12c/pdb1  -bg -s -uc 25 -co localhost &
$ ./charbench -cs //oracle12c/pdb2  -bg -s -uc 25 -co localhost &
$ ./charbench -cs //oracle12c/pdb3  -bg -s -uc 25 -co localhost &
$ ./coordinator -runall
$ ./coordinator -stats
Aggregated results for 3 load generators
Use Ctrl-C to halt stats collection
      Time     Users       TPM       TPS
  17:52:03        75      3898       460
  17:52:06        75      5263       483
  17:52:09        75      6669       489
. . .
. . .
$./coordinator -stopall
$./coordinator -kill


Note : to background charbench you must specify -bg (it indicates the client will no longer be taking input from stdin)

You can download it as usual from
here

Comments

Fixes come in thick and fast...

Yet another fix and some more minor UI changes.

In fixing some code I regressed some basic functionality. In the last build if you restarted a benchmark from within the swingbench and minibench GUI it gave you an error and you had to restart swingbench to get it going again. This is now fixed in this new build.

I also took the time to add some functionality to enable you to specify Inter and Intra sleep times. You can do this in the “Load” tab as shown below

Inter/Intra Sleep Time Dialogue

It gives me the opportunity to explain the difference between inter and intra sleep times. As the name implies intra sleep times occur “inside” of a transaction. Inter sleep times occur between transactions. Many of the transactions inside of the swingbench “SOE” have sleep times between DML operations (select, insert, update). In some situations this better emulates what happens in some legacy form based systems, this is what is controlled by intra sleep times. However most systems these days tend to utilise web based front ends where DML operations tend to be fired as a single operation when the user submits a form. This approach results in a more scalable architecture with fewer locks being held and for shorter periods of time. Hopefully the following diagram will explain the differences in a clearer fashion.

6db6b24e689710900f82c65055a214b1

You can also set the intra and inter sleep time from the command line with the -min (intra min) -max (intramax) -intermin (inter min) -intermax (inter max).
Comments