How to read the tables?
Meaning of the model-checker's results| Model-checker on the instrumented program | Observed state in Litmus | Picture |
| VERIFICATION FAILED | YES | ![]() |
| VERIFICATION SUCCESSFUL | NO | ![]() |
Reading the results in the experiments tables
The table below is an excerpt of former results we obtained for x86/podwr experiments.
- In the first column, we can read the name of the original Litmus test (with a link to the file), and the expected violation cycle. The second column gathers the results of the model-checkers for SC. As we are interested in WMM bugs, the verification for SC should be
. - The three next columns concern TSO model. The last column tells which result we should get for this program under TSO, namely
. We see in the third column that Satabs indeed finds the correct result when checking the instrumented program. ESBMC in this example is missing the counterexample. Threader is either returning an error, or killed by timeout (here, it is an error).
- Note that there are two columns for TSO results: in the first one, we instrument all the delays; in the second one, we instrument only one delay per cycle.
- We also experiment under PSO, RMO and Power, and with additional model-checkers: additional columns of results (and expected results) appear after TSO's columns.
| Name | SC | TSO (all delay/cyc) | TSO (1 delay/cyc) | Expected (TSO) | ... |
|
x86/podwr000 Fre PodWR Fre PodWR |
SatAbs: ![]() 0.60sec ESBMC:
0.30secThreader:
7.92sec... |
Cycles SatAbs:
3.94sec (x6.56) ESBMC:
8.34secThreader: ERROR/TIMEOUT[1] 491.99sec ... |
Cycles SatAbs:
2.43sec (x4.05) ESBMC:
7.58sec
Threader: ERROR/TIMEOUT[1] 182.60sec ... |
|
... |
ppc/podww (tooling date Sun Jan 22 02:23:23 GMT 2012)
This family corresponds to litmus tests involving pairs showing that Power can reorder write-write pairs. Therefore they should answer Yes in litmus terms, or Verification Failed in verification terms as soon as the model is Power.
| Name | SC | CAV'11 | TSO (all delay/cyc) | TSO (1 delay/cyc) | TSO (opt delay/cyc) | Expected (TSO) | PSO (all delay/cyc) | PSO (1 delay/cyc) | PSO (opt delay/cyc) | Expected (PSO) | RMO (all delay/cyc) | RMO (1 delay/cyc) | RMO (opt delay/cyc) | Expected (RMO) | Power (all delay/cyc) | Power (1 delay/cyc) | Power (opt delay/cyc) | Expected (Power) |
|
ppc/podww000 |
SatAbs: ![]() 0.56sec Satabs-prefix-first:
0.35sec
CBMC-WMM:
0.05sec
CBMC-SC:
0.10sec
CImpact:
3.19sec
Checkfence: 0.01sec ESBMC:
0.07sec
Threader-monolithic:
0.17sec
Threader-owicki-gries:
0.13sec
Threader-rely-guarantee:
0.14sec
Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: sc):
0.46sec
|
Cycles
SatAbs: ![]() 3.02sec (x5.39) Satabs-prefix-first:
3.08sec
CBMC-WMM:
0.09sec
CBMC-SC:
11.68sec
CImpact:
1.59sec
ESBMC:
0.09sec
Threader-monolithic: ERROR/TIMEOUT[130] 872.56sec Threader-owicki-gries: ERROR/TIMEOUT[130] 892.31sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 895.98sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
0.19sec
|
Cycles
SatAbs: ![]() 1.19sec (x2.12) Satabs-prefix-first:
1.28sec
CBMC-WMM:
0.10sec
CBMC-SC:
0.06sec
CImpact:
82.10sec
Checkfence: 0.01sec ESBMC:
0.07sec
Threader-monolithic: ERROR/TIMEOUT[1] 0.10sec Threader-owicki-gries: ERROR/TIMEOUT[1] 0.11sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 0.11sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
0.18sec
|
Cycles
SatAbs: ![]() 1.23sec (x2.19) Satabs-prefix-first:
1.28sec
CBMC-WMM:
0.09sec
CBMC-SC:
0.05sec
CImpact:
74.09sec
ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[1] 0.10sec Threader-owicki-gries: ERROR/TIMEOUT[1] 0.08sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 0.10sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
0.20sec
|
Cycles
SatAbs: ![]() 1.25sec (x2.23) Satabs-prefix-first:
1.24sec
CBMC-WMM:
0.10sec
CBMC-SC:
0.05sec
CImpact:
75.56sec
ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[1] 0.11sec Threader-owicki-gries: ERROR/TIMEOUT[1] 0.10sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 0.10sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
0.19sec
|
|
Cycles
SatAbs: ![]() 18.01sec (x32.16) Satabs-prefix-first:
10.18sec
CBMC-WMM:
0.09sec
CBMC-SC:
7.78sec
CImpact:
1.20sec
ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[130] 896.98sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.65sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.29sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender:
|
Cycles
SatAbs: ![]() 9.01sec (x16.08) Satabs-prefix-first:
2.91sec
CBMC-WMM:
0.10sec
CBMC-SC:
0.73sec
CImpact:
0.71sec
ESBMC:
0.10sec
Threader-monolithic: ERROR/TIMEOUT[130] 897.12sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.17sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.30sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender:
|
Cycles
SatAbs: ![]() 9.33sec (x16.66) Satabs-prefix-first:
9.01sec
CBMC-WMM:
0.07sec
CBMC-SC:
0.70sec
CImpact:
27.14sec
ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[130] 896.82sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.30sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.21sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender:
|
|
Cycles
SatAbs: ![]() 16.09sec (x28.73) Satabs-prefix-first:
8.15sec
CBMC-WMM:
0.10sec
CBMC-SC:
6.99sec
CImpact:
1.18sec
ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[130] 897.73sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.58sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.69sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 7.08sec (x12.64) Satabs-prefix-first:
2.40sec
CBMC-WMM:
0.10sec
CBMC-SC:
0.69sec
CImpact:
0.75sec
ESBMC:
0.10sec
Threader-monolithic: ERROR/TIMEOUT[130] 897.27sec Threader-owicki-gries: ERROR/TIMEOUT[130] 896.59sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 896.78sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 9.71sec (x17.33) Satabs-prefix-first:
8.75sec
CBMC-WMM:
0.09sec
CBMC-SC:
0.70sec
CImpact:
27.67sec
ESBMC:
0.09sec
Threader-monolithic: ERROR/TIMEOUT[130] 897.35sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.29sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.04sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
|
Cycles
SatAbs: ![]() 3.57sec (x6.37) Satabs-prefix-first:
4.69sec
CBMC-WMM:
0.09sec
CBMC-SC:
19.36sec
CImpact:
1.54sec
ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[130] 897.82sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.50sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.58sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 7.83sec (x13.98) Satabs-prefix-first:
2.37sec
CBMC-WMM:
0.09sec
CBMC-SC:
0.66sec
CImpact:
0.68sec
ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[130] 896.82sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.41sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.36sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 9.55sec (x17.05) Satabs-prefix-first:
9.23sec
CBMC-WMM:
0.10sec
CBMC-SC:
0.70sec
CImpact:
28.05sec
ESBMC:
0.07sec
Threader-monolithic: ERROR/TIMEOUT[130] 897.20sec Threader-owicki-gries: ERROR/TIMEOUT[130] 895.92sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 895.99sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
|
|
ppc/podww001 |
SatAbs: ![]() 2.26sec Satabs-prefix-first:
1.18sec
CBMC-WMM:
0.10sec
CBMC-SC:
0.10sec
CImpact: ERROR/TIMEOUT[8] 292.27sec Checkfence: 0.01sec ESBMC:
0.05sec
Threader-monolithic:
0.12sec
Threader-owicki-gries:
0.13sec
Threader-rely-guarantee:
0.14sec
Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: sc):
0.37sec
|
Cycles
SatAbs: ![]() 2.96sec (x1.30) Satabs-prefix-first:
3.55sec
CBMC-WMM:
0.14sec
CBMC-SC:
33.90sec
CImpact:
1.75sec
ESBMC:
0.11sec
Threader-monolithic: ERROR/TIMEOUT[130] 897.45sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.49sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.74sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
1.80sec
|
Cycles
SatAbs: ![]() 6.08sec (x2.69) Satabs-prefix-first:
4.10sec
CBMC-WMM:
0.17sec
CBMC-SC:
0.09sec
CImpact: ERROR/TIMEOUT[8] 317.40sec Checkfence: 0.01sec ESBMC:
0.06sec
Threader-monolithic: ERROR/TIMEOUT[1] 0.12sec Threader-owicki-gries: ERROR/TIMEOUT[1] 0.11sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 0.12sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
1.71sec
|
Cycles
SatAbs: ![]() 5.86sec (x2.59) Satabs-prefix-first:
3.83sec
CBMC-WMM:
0.16sec
CBMC-SC:
0.07sec
CImpact: ERROR/TIMEOUT[8] 309.21sec ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[1] 0.11sec Threader-owicki-gries: ERROR/TIMEOUT[1] 0.11sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 0.11sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
1.68sec
|
Cycles
SatAbs: ![]() 5.87sec (x2.59) Satabs-prefix-first:
3.69sec
CBMC-WMM:
0.16sec
CBMC-SC:
0.09sec
CImpact: ERROR/TIMEOUT[8] 351.19sec ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[1] 0.11sec Threader-owicki-gries: ERROR/TIMEOUT[1] 0.11sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 0.10sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp):
1.33sec
|
|
Cycles
SatAbs: ![]() 68.13sec (x30.14) Satabs-prefix-first:
41.73sec
CBMC-WMM:
0.13sec
CBMC-SC:
24.56sec
CImpact: ERROR/TIMEOUT[8] 407.01sec ESBMC:
0.11sec
Threader-monolithic: ERROR/TIMEOUT[130] 897.57sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.73sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.56sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender:
|
Cycles
SatAbs: ![]() 26.30sec (x11.63) Satabs-prefix-first:
14.97sec
CBMC-WMM:
0.12sec
CBMC-SC:
1.03sec
CImpact: ERROR/TIMEOUT[8] 351.99sec ESBMC:
0.06sec
Threader-monolithic: ERROR/TIMEOUT[130] 897.20sec Threader-owicki-gries: ERROR/TIMEOUT[130] 896.58sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.24sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender:
|
Cycles
SatAbs: ![]() 22.17sec (x9.80) Satabs-prefix-first:
15.88sec
CBMC-WMM:
0.14sec
CBMC-SC:
0.83sec
CImpact: ERROR/TIMEOUT[8] 326.20sec ESBMC:
0.10sec
Threader-monolithic: ERROR/TIMEOUT[130] 896.62sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.15sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.36sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender:
|
|
Cycles
SatAbs: ![]() 62.48sec (x27.64) Satabs-prefix-first:
36.36sec
CBMC-WMM:
0.13sec
CBMC-SC:
23.61sec
CImpact: ERROR/TIMEOUT[8] 307.92sec ESBMC:
0.10sec
Threader-monolithic: ERROR/TIMEOUT[130] 896.91sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.57sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.37sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 32.26sec (x14.27) Satabs-prefix-first:
17.69sec
CBMC-WMM:
0.13sec
CBMC-SC:
1.16sec
CImpact: ERROR/TIMEOUT[8] 343.74sec ESBMC:
0.11sec
Threader-monolithic: ERROR/TIMEOUT[130] 897.49sec Threader-owicki-gries: ERROR/TIMEOUT[130] 896.81sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.16sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 25.21sec (x11.15) Satabs-prefix-first:
15.71sec
CBMC-WMM:
0.15sec
CBMC-SC:
0.86sec
CImpact: ERROR/TIMEOUT[8] 356.64sec ESBMC:
0.07sec
Threader-monolithic: ERROR/TIMEOUT[130] 896.81sec Threader-owicki-gries: ERROR/TIMEOUT[130] 896.86sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.17sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
|
Cycles
SatAbs: ![]() 5.19sec (x2.29) Satabs-prefix-first:
6.43sec
CBMC-WMM:
0.15sec
CBMC-SC:
23.34sec
CImpact:
2.17sec
ESBMC:
0.12sec
Threader-monolithic: ERROR/TIMEOUT[130] 897.58sec Threader-owicki-gries: ERROR/TIMEOUT[130] 896.89sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.71sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 27.48sec (x12.15) Satabs-prefix-first:
15.85sec
CBMC-WMM:
0.16sec
CBMC-SC:
1.06sec
CImpact: ERROR/TIMEOUT[8] 351.62sec ESBMC:
0.07sec
Threader-monolithic: ERROR/TIMEOUT[130] 897.22sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.07sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.41sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 22.42sec (x9.92) Satabs-prefix-first:
15.89sec
CBMC-WMM:
0.14sec
CBMC-SC:
0.82sec
CImpact: ERROR/TIMEOUT[8] 314.65sec ESBMC:
0.08sec
Threader-monolithic: ERROR/TIMEOUT[130] 897.01sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.42sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.37sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
|