How to read the tables?
Meaning of the model-checker's results| Model-checker on the instrumented program | Observed state in Litmus | Picture |
| VERIFICATION FAILED | YES | ![]() |
| VERIFICATION SUCCESSFUL | NO | ![]() |
Reading the results in the experiments tables
The table below is an excerpt of former results we obtained for x86/podwr experiments.
- In the first column, we can read the name of the original Litmus test (with a link to the file), and the expected violation cycle. The second column gathers the results of the model-checkers for SC. As we are interested in WMM bugs, the verification for SC should be
. - The three next columns concern TSO model. The last column tells which result we should get for this program under TSO, namely
. We see in the third column that Satabs indeed finds the correct result when checking the instrumented program. ESBMC in this example is missing the counterexample. Threader is either returning an error, or killed by timeout (here, it is an error).
- Note that there are two columns for TSO results: in the first one, we instrument all the delays; in the second one, we instrument only one delay per cycle.
- We also experiment under PSO, RMO and Power, and with additional model-checkers: additional columns of results (and expected results) appear after TSO's columns.
| Name | SC | TSO (all delay/cyc) | TSO (1 delay/cyc) | Expected (TSO) | ... |
|
x86/podwr000 Fre PodWR Fre PodWR |
SatAbs: ![]() 0.60sec ESBMC:
0.30secThreader:
7.92sec... |
Cycles SatAbs:
3.94sec (x6.56) ESBMC:
8.34secThreader: ERROR/TIMEOUT[1] 491.99sec ... |
Cycles SatAbs:
2.43sec (x4.05) ESBMC:
7.58sec
Threader: ERROR/TIMEOUT[1] 182.60sec ... |
|
... |
x86/podwr (tooling date Sun Jan 22 02:23:23 GMT 2012)
This family corresponds to litmus tests involving write-read pairs, to exercise typical x86 reordering of write-read pairs scenarios. Therefore they should answer Yes in litmus terms, or Verification Failed in verification terms, as soon as the model is x86/TSO or a weaker one.
| Name | SC | CAV'11 | TSO (all delay/cyc) | TSO (1 delay/cyc) | TSO (opt delay/cyc) | Expected (TSO) | PSO (all delay/cyc) | PSO (1 delay/cyc) | PSO (opt delay/cyc) | Expected (PSO) | RMO (all delay/cyc) | RMO (1 delay/cyc) | RMO (opt delay/cyc) | Expected (RMO) | Power (all delay/cyc) | Power (1 delay/cyc) | Power (opt delay/cyc) | Expected (Power) |
|
x86/podwr000 |
SatAbs: ![]() 0.57sec Satabs-prefix-first:
0.36sec
CBMC-WMM:
0.05sec
CBMC-SC:
0.05sec
CImpact:
0.11sec
Checkfence: 0.01sec ESBMC:
0.07sec
Threader-monolithic:
0.12sec
Threader-owicki-gries:
0.10sec
Threader-rely-guarantee:
0.09sec
Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: sc):
0.55sec
|
Cycles
SatAbs: ![]() 7.51sec (x13.17) Satabs-prefix-first:
15.27sec
CBMC-WMM:
0.07sec
CBMC-SC:
5.69sec
CImpact:
0.36sec
ESBMC:
0.10sec
Threader-monolithic: ERROR/TIMEOUT[130] 602.21sec Threader-owicki-gries: ERROR/TIMEOUT[130] 635.04sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 620.18sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
0.40sec
|
Cycles
SatAbs: ![]() 7.97sec (x13.98) Satabs-prefix-first:
13.02sec
CBMC-WMM:
0.07sec
CBMC-SC:
5.50sec
CImpact:
0.38sec
Checkfence: 0.01sec ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[130] 645.51sec Threader-owicki-gries: ERROR/TIMEOUT[130] 654.92sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 589.78sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
0.40sec
|
Cycles
SatAbs: ![]() 2.31sec (x4.05) Satabs-prefix-first:
6.90sec
CBMC-WMM:
0.06sec
CBMC-SC:
0.86sec
CImpact:
0.25sec
ESBMC:
0.07sec
Threader-monolithic: ERROR/TIMEOUT[1] 220.89sec Threader-owicki-gries: ERROR/TIMEOUT[1] 232.29sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 229.49sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
0.59sec
|
Cycles
SatAbs: ![]() 2.45sec (x4.29) Satabs-prefix-first:
6.54sec
CBMC-WMM:
0.07sec
CBMC-SC:
0.88sec
CImpact:
0.27sec
ESBMC:
0.10sec
Threader-monolithic: ERROR/TIMEOUT[1] 216.65sec Threader-owicki-gries: ERROR/TIMEOUT[1] 236.62sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 226.56sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
0.37sec
|
|
Cycles
SatAbs: ![]() 8.10sec (x14.21) Satabs-prefix-first:
12.91sec
CBMC-WMM:
0.07sec
CBMC-SC:
5.67sec
CImpact:
0.35sec
ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[130] 600.25sec Threader-owicki-gries: ERROR/TIMEOUT[130] 596.85sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 601.18sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender:
|
Cycles
SatAbs: ![]() 2.59sec (x4.54) Satabs-prefix-first:
7.25sec
CBMC-WMM:
0.04sec
CBMC-SC:
0.88sec
CImpact:
0.27sec
ESBMC:
0.09sec
Threader-monolithic: ERROR/TIMEOUT[1] 234.56sec Threader-owicki-gries: ERROR/TIMEOUT[1] 218.74sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 217.38sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender:
|
Cycles
SatAbs: ![]() 2.31sec (x4.05) Satabs-prefix-first:
6.58sec
CBMC-WMM:
0.07sec
CBMC-SC:
0.86sec
CImpact:
0.26sec
ESBMC:
0.07sec
Threader-monolithic: ERROR/TIMEOUT[1] 215.07sec Threader-owicki-gries: ERROR/TIMEOUT[1] 224.18sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 234.22sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender:
|
|
Cycles
SatAbs: ![]() 7.80sec (x13.68) Satabs-prefix-first:
13.09sec
CBMC-WMM:
0.07sec
CBMC-SC:
5.32sec
CImpact:
0.38sec
ESBMC:
0.10sec
Threader-monolithic: ERROR/TIMEOUT[130] 795.32sec Threader-owicki-gries: ERROR/TIMEOUT[130] 694.18sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 730.50sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 3.57sec (x6.26) Satabs-prefix-first:
9.17sec
CBMC-WMM:
0.08sec
CBMC-SC:
1.00sec
CImpact:
0.27sec
ESBMC:
0.10sec
Threader-monolithic: ERROR/TIMEOUT[1] 218.40sec Threader-owicki-gries: ERROR/TIMEOUT[1] 228.64sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 221.15sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 2.52sec (x4.42) Satabs-prefix-first:
9.03sec
CBMC-WMM:
0.09sec
CBMC-SC:
1.13sec
CImpact:
0.28sec
ESBMC:
0.10sec
Threader-monolithic: ERROR/TIMEOUT[1] 221.46sec Threader-owicki-gries: ERROR/TIMEOUT[1] 225.54sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 222.32sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
|
Cycles
SatAbs: ![]() 8.01sec (x14.05) Satabs-prefix-first:
13.02sec
CBMC-WMM:
0.06sec
CBMC-SC:
5.48sec
CImpact:
0.39sec
ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[130] 676.45sec Threader-owicki-gries: ERROR/TIMEOUT[130] 776.29sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 770.68sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 2.37sec (x4.15) Satabs-prefix-first:
6.49sec
CBMC-WMM:
0.08sec
CBMC-SC:
0.83sec
CImpact:
0.24sec
ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[1] 221.41sec Threader-owicki-gries: ERROR/TIMEOUT[1] 217.81sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 218.29sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 2.42sec (x4.24) Satabs-prefix-first:
7.11sec
CBMC-WMM:
0.08sec
CBMC-SC:
0.85sec
CImpact:
0.29sec
ESBMC:
0.09sec
Threader-monolithic: ERROR/TIMEOUT[1] 222.60sec Threader-owicki-gries: ERROR/TIMEOUT[1] 216.85sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 238.74sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
|
|
x86/podwr001 |
SatAbs: ![]() 2.23sec Satabs-prefix-first:
0.94sec
CBMC-WMM:
0.07sec
CBMC-SC:
0.06sec
CImpact:
242.47sec
Checkfence: 0.01sec ESBMC:
0.08sec
Threader-monolithic:
0.13sec
Threader-owicki-gries:
0.11sec
Threader-rely-guarantee:
0.14sec
Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: sc):
0.50sec
|
Cycles
SatAbs: ![]() 24.52sec (x10.99) Satabs-prefix-first:
59.02sec
CBMC-WMM:
0.12sec
CBMC-SC:
13.96sec
CImpact: ERROR/TIMEOUT[8] 451.20sec ESBMC:
0.11sec
Threader-monolithic: ERROR/TIMEOUT[130] 897.17sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.10sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.36sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
0.55sec
|
Cycles
SatAbs: ![]() 29.20sec (x13.09) Satabs-prefix-first:
66.56sec
CBMC-WMM:
0.10sec
CBMC-SC:
10.15sec
CImpact: ERROR/TIMEOUT[8] 423.36sec Checkfence: 0.01sec ESBMC:
0.10sec
Threader-monolithic: ERROR/TIMEOUT[130] 896.77sec Threader-owicki-gries: ERROR/TIMEOUT[130] 896.25sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.24sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
0.48sec
|
Cycles
SatAbs: ![]() 7.06sec (x3.16) Satabs-prefix-first:
8.80sec
CBMC-WMM:
0.11sec
CBMC-SC:
1.08sec
CImpact: ERROR/TIMEOUT[8] 331.69sec ESBMC:
0.09sec
Threader-monolithic: ERROR/TIMEOUT[1] 197.76sec Threader-owicki-gries: ERROR/TIMEOUT[1] 255.52sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 256.25sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
0.46sec
|
Cycles
SatAbs: ![]() 6.22sec (x2.78) Satabs-prefix-first:
8.33sec
CBMC-WMM:
0.12sec
CBMC-SC:
1.04sec
CImpact: ERROR/TIMEOUT[8] 325.55sec ESBMC:
0.09sec
Threader-monolithic: ERROR/TIMEOUT[1] 256.57sec Threader-owicki-gries: ERROR/TIMEOUT[1] 256.22sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 227.18sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
0.53sec
|
|
Cycles
SatAbs: ![]() 29.06sec (x13.03) Satabs-prefix-first:
63.37sec
CBMC-WMM:
0.12sec
CBMC-SC:
15.96sec
CImpact: ERROR/TIMEOUT[8] 464.42sec ESBMC:
0.11sec
Threader-monolithic: ERROR/TIMEOUT[130] 897.28sec Threader-owicki-gries: ERROR/TIMEOUT[130] 896.96sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 896.80sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender:
|
Cycles
SatAbs: ![]() 7.98sec (x3.57) Satabs-prefix-first:
8.30sec
CBMC-WMM:
0.11sec
CBMC-SC:
1.12sec
CImpact: ERROR/TIMEOUT[8] 306.55sec ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[1] 245.43sec Threader-owicki-gries: ERROR/TIMEOUT[1] 258.20sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 256.38sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender:
|
Cycles
SatAbs: ![]() 6.57sec (x2.94) Satabs-prefix-first:
8.12sec
CBMC-WMM:
0.11sec
CBMC-SC:
1.00sec
CImpact: ERROR/TIMEOUT[8] 333.16sec ESBMC:
0.09sec
Threader-monolithic: ERROR/TIMEOUT[1] 257.93sec Threader-owicki-gries: ERROR/TIMEOUT[1] 239.83sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 234.72sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender:
|
|
Cycles
SatAbs: ![]() 29.26sec (x13.12) Satabs-prefix-first:
72.09sec
CBMC-WMM:
0.14sec
CBMC-SC:
14.05sec
CImpact: ERROR/TIMEOUT[8] 329.89sec ESBMC:
0.11sec
Threader-monolithic: ERROR/TIMEOUT[130] 861.08sec Threader-owicki-gries: ERROR/TIMEOUT[130] 874.04sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 896.40sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 8.01sec (x3.59) Satabs-prefix-first:
8.37sec
CBMC-WMM:
0.13sec
CBMC-SC:
1.22sec
CImpact: ERROR/TIMEOUT[8] 317.91sec ESBMC:
0.06sec
Threader-monolithic: ERROR/TIMEOUT[1] 234.36sec Threader-owicki-gries: ERROR/TIMEOUT[1] 192.15sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 253.84sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 6.59sec (x2.95) Satabs-prefix-first:
8.51sec
CBMC-WMM:
0.11sec
CBMC-SC:
1.11sec
CImpact: ERROR/TIMEOUT[8] 235.61sec ESBMC:
0.09sec
Threader-monolithic: ERROR/TIMEOUT[1] 234.48sec Threader-owicki-gries: ERROR/TIMEOUT[1] 216.57sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 257.84sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
|
Cycles
SatAbs: ![]() 27.32sec (x12.25) Satabs-prefix-first:
67.16sec
CBMC-WMM:
0.12sec
CBMC-SC:
13.26sec
CImpact: ERROR/TIMEOUT[8] 376.41sec ESBMC:
0.10sec
Threader-monolithic: ERROR/TIMEOUT[130] 895.72sec Threader-owicki-gries: ERROR/TIMEOUT[130] 896.75sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.50sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 6.42sec (x2.87) Satabs-prefix-first:
8.45sec
CBMC-WMM:
0.15sec
CBMC-SC:
1.09sec
CImpact: ERROR/TIMEOUT[8] 255.58sec ESBMC:
0.09sec
Threader-monolithic: ERROR/TIMEOUT[1] 257.70sec Threader-owicki-gries: ERROR/TIMEOUT[1] 237.76sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 254.91sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 6.38sec (x2.86) Satabs-prefix-first:
8.12sec
CBMC-WMM:
0.12sec
CBMC-SC:
1.05sec
CImpact: ERROR/TIMEOUT[8] 339.55sec ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[1] 246.25sec Threader-owicki-gries: ERROR/TIMEOUT[1] 220.12sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 241.22sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
|