How to read the tables?
Meaning of the model-checker's resultsModel-checker on the instrumented program | Observed state in Litmus | Picture |
VERIFICATION FAILED | YES | ![]() |
VERIFICATION SUCCESSFUL | NO | ![]() |
Reading the results in the experiments tables
The table below is an excerpt of former results we obtained for x86/podwr experiments.
- In the first column, we can read the name of the original Litmus test (with a link to the file), and the expected violation cycle. The second column gathers the results of the model-checkers for SC. As we are interested in WMM bugs, the verification for SC should be
.
- The three next columns concern TSO model. The last column tells which result we should get for this program under TSO, namely
. We see in the third column that Satabs indeed finds the correct result when checking the instrumented program. ESBMC in this example is missing the counterexample. Threader is either returning an error, or killed by timeout (here, it is an error).
- Note that there are two columns for TSO results: in the first one, we instrument all the delays; in the second one, we instrument only one delay per cycle.
- We also experiment under PSO, RMO and Power, and with additional model-checkers: additional columns of results (and expected results) appear after TSO's columns.
Name | SC | TSO (all delay/cyc) | TSO (1 delay/cyc) | Expected (TSO) | ... |
x86/podwr000 Fre PodWR Fre PodWR |
SatAbs: ![]() 0.60sec ESBMC: ![]() Threader: ![]() ... |
Cycles SatAbs: ![]() 3.94sec (x6.56) ESBMC: ![]() Threader: ERROR/TIMEOUT[1] 491.99sec ... |
Cycles SatAbs: ![]() 2.43sec (x4.05) ESBMC: ![]() Threader: ERROR/TIMEOUT[1] 182.60sec ... |
![]() |
... |
x86/podwr (tooling date Sun Jan 22 02:23:23 GMT 2012)
This family corresponds to litmus tests involving write-read pairs, to exercise typical x86 reordering of write-read pairs scenarios. Therefore they should answer Yes in litmus terms, or Verification Failed in verification terms, as soon as the model is x86/TSO or a weaker one.
Name | SC | CAV'11 | TSO (all delay/cyc) | TSO (1 delay/cyc) | TSO (opt delay/cyc) | Expected (TSO) | PSO (all delay/cyc) | PSO (1 delay/cyc) | PSO (opt delay/cyc) | Expected (PSO) | RMO (all delay/cyc) | RMO (1 delay/cyc) | RMO (opt delay/cyc) | Expected (RMO) | Power (all delay/cyc) | Power (1 delay/cyc) | Power (opt delay/cyc) | Expected (Power) |
x86/podwr000 |
SatAbs: ![]() 0.57sec Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ![]() Checkfence: 0.01sec ESBMC: ![]() Threader-monolithic: ![]() Threader-owicki-gries: ![]() Threader-rely-guarantee: ![]() Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: sc): ![]() |
Cycles
SatAbs: ![]() 7.51sec (x13.17) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ![]() ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[130] 602.21sec Threader-owicki-gries: ERROR/TIMEOUT[130] 635.04sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 620.18sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): ![]() |
Cycles
SatAbs: ![]() 7.97sec (x13.98) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ![]() Checkfence: 0.01sec ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[130] 645.51sec Threader-owicki-gries: ERROR/TIMEOUT[130] 654.92sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 589.78sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): ![]() |
Cycles
SatAbs: ![]() 2.31sec (x4.05) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ![]() ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 220.89sec Threader-owicki-gries: ERROR/TIMEOUT[1] 232.29sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 229.49sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): ![]() |
Cycles
SatAbs: ![]() 2.45sec (x4.29) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ![]() ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 216.65sec Threader-owicki-gries: ERROR/TIMEOUT[1] 236.62sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 226.56sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): ![]() |
![]() |
Cycles
SatAbs: ![]() 8.10sec (x14.21) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ![]() ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[130] 600.25sec Threader-owicki-gries: ERROR/TIMEOUT[130] 596.85sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 601.18sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender: ![]() |
Cycles
SatAbs: ![]() 2.59sec (x4.54) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ![]() ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 234.56sec Threader-owicki-gries: ERROR/TIMEOUT[1] 218.74sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 217.38sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender: ![]() |
Cycles
SatAbs: ![]() 2.31sec (x4.05) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ![]() ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 215.07sec Threader-owicki-gries: ERROR/TIMEOUT[1] 224.18sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 234.22sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender: ![]() |
![]() |
Cycles
SatAbs: ![]() 7.80sec (x13.68) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ![]() ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[130] 795.32sec Threader-owicki-gries: ERROR/TIMEOUT[130] 694.18sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 730.50sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 3.57sec (x6.26) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ![]() ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 218.40sec Threader-owicki-gries: ERROR/TIMEOUT[1] 228.64sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 221.15sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 2.52sec (x4.42) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ![]() ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 221.46sec Threader-owicki-gries: ERROR/TIMEOUT[1] 225.54sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 222.32sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
![]() |
Cycles
SatAbs: ![]() 8.01sec (x14.05) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ![]() ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[130] 676.45sec Threader-owicki-gries: ERROR/TIMEOUT[130] 776.29sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 770.68sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 2.37sec (x4.15) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ![]() ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 221.41sec Threader-owicki-gries: ERROR/TIMEOUT[1] 217.81sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 218.29sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 2.42sec (x4.24) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ![]() ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 222.60sec Threader-owicki-gries: ERROR/TIMEOUT[1] 216.85sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 238.74sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
![]() |
x86/podwr001 |
SatAbs: ![]() 2.23sec Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ![]() Checkfence: 0.01sec ESBMC: ![]() Threader-monolithic: ![]() Threader-owicki-gries: ![]() Threader-rely-guarantee: ![]() Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: sc): ![]() |
Cycles
SatAbs: ![]() 24.52sec (x10.99) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ERROR/TIMEOUT[8] 451.20sec ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[130] 897.17sec Threader-owicki-gries: ERROR/TIMEOUT[130] 897.10sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.36sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): ![]() |
Cycles
SatAbs: ![]() 29.20sec (x13.09) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ERROR/TIMEOUT[8] 423.36sec Checkfence: 0.01sec ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[130] 896.77sec Threader-owicki-gries: ERROR/TIMEOUT[130] 896.25sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.24sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): ![]() |
Cycles
SatAbs: ![]() 7.06sec (x3.16) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ERROR/TIMEOUT[8] 331.69sec ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 197.76sec Threader-owicki-gries: ERROR/TIMEOUT[1] 255.52sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 256.25sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): ![]() |
Cycles
SatAbs: ![]() 6.22sec (x2.78) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ERROR/TIMEOUT[8] 325.55sec ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 256.57sec Threader-owicki-gries: ERROR/TIMEOUT[1] 256.22sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 227.18sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT MMChecker (MM: csharp): ![]() |
![]() |
Cycles
SatAbs: ![]() 29.06sec (x13.03) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ERROR/TIMEOUT[8] 464.42sec ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[130] 897.28sec Threader-owicki-gries: ERROR/TIMEOUT[130] 896.96sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 896.80sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender: ![]() |
Cycles
SatAbs: ![]() 7.98sec (x3.57) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ERROR/TIMEOUT[8] 306.55sec ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 245.43sec Threader-owicki-gries: ERROR/TIMEOUT[1] 258.20sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 256.38sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender: ![]() |
Cycles
SatAbs: ![]() 6.57sec (x2.94) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ERROR/TIMEOUT[8] 333.16sec ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 257.93sec Threader-owicki-gries: ERROR/TIMEOUT[1] 239.83sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 234.72sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT Blender: ![]() |
![]() |
Cycles
SatAbs: ![]() 29.26sec (x13.12) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ERROR/TIMEOUT[8] 329.89sec ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[130] 861.08sec Threader-owicki-gries: ERROR/TIMEOUT[130] 874.04sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 896.40sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 8.01sec (x3.59) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ERROR/TIMEOUT[8] 317.91sec ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 234.36sec Threader-owicki-gries: ERROR/TIMEOUT[1] 192.15sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 253.84sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 6.59sec (x2.95) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ERROR/TIMEOUT[8] 235.61sec ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 234.48sec Threader-owicki-gries: ERROR/TIMEOUT[1] 216.57sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 257.84sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
![]() |
Cycles
SatAbs: ![]() 27.32sec (x12.25) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ERROR/TIMEOUT[8] 376.41sec ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[130] 895.72sec Threader-owicki-gries: ERROR/TIMEOUT[130] 896.75sec Threader-rely-guarantee: ERROR/TIMEOUT[130] 897.50sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 6.42sec (x2.87) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ERROR/TIMEOUT[8] 255.58sec ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 257.70sec Threader-owicki-gries: ERROR/TIMEOUT[1] 237.76sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 254.91sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
Cycles
SatAbs: ![]() 6.38sec (x2.86) Satabs-prefix-first: ![]() CBMC-WMM: ![]() CBMC-SC: ![]() CImpact: ERROR/TIMEOUT[8] 339.55sec ESBMC: ![]() Threader-monolithic: ERROR/TIMEOUT[1] 246.25sec Threader-owicki-gries: ERROR/TIMEOUT[1] 220.12sec Threader-rely-guarantee: ERROR/TIMEOUT[1] 241.22sec Poirot-Integer: ERROR/TIMEOUT Poirot-BitVector: ERROR/TIMEOUT |
![]() |