Internal Cache blows up with many constrain's

[doyougnu]

Hey Levent,

I have a computation that exists entirely in the Query monad. I recently did some profiling and it looks like the internal Cache is blowing up in size leading to about 95% GC use. So I'm looking for help in understanding why this behavior is occurring and how to avoid it.

The basic computation is a fold over a custom abstract syntax tree where I accumulate on SBools. I've been able to reproduce this with the following tiny program that folds a list of SBools in the query monad:

import qualified Data.SBV                as S
import qualified Data.SBV.Control        as SC
import qualified Data.SBV.Internals      as SI

-- | generate an infinite list of unique strings and take n of them dropping the
-- empty string
stringList :: Int -> [String]
stringList n = tail . take (n+1) $ concatMap (flip replicateM "abc") [0..]


-- | the test runner, takes a computation that runs in the query monad and an
-- int that dictates size of the list of sbools
test :: ([S.SBool] -> SC.Query (Map String Bool)) -> Int -> IO (Map String Bool)
test f n = S.runSMT $
           do
             prop' <- S.sBools $! stringList n
             SC.query $ f prop'


-- | I fold over the string of SBools here constraining at each accumulation,
-- this seems to blow up the internal cache severely leading to about 95% GC
bad :: [S.SBool] -> SC.Query (Map String Bool)
bad prop' = do b <- foldM (helper) S.true prop'
               S.constrain b
               fmap (fmap SI.cwToBool) $ S.getModelDictionary <$> SC.getSMTResult
  -- | combine the current sbool with the accumulated sbool, constrain the
  -- two and then return the accumulated result
  where helper x acc = do let b = x S.&&& acc
                          S.constrain b
                          return b


-- | identical to the bad version but I do not constrain for each accumulation
good :: [S.SBool] -> SC.Query (Map String Bool)
good prop' = do b <- foldM (helper) S.true prop'
                S.constrain b
                fmap (fmap SI.cwToBool) $ S.getModelDictionary <$> SC.getSMTResult
  -- | this helper is equivalent to just foldr' (S.&&&)
  where helper x acc = do let b = x S.&&& acc
                          return b

main = do
  putStrLn "Running Good:\n"
  goodRes <- test good 1000

  putStrLn "Running Bad:\n"
  badRes <- test bad  1000

  -- just ensuring evaluation
  print (size goodRes)
  print (size badRes)

I just commented out the lines for each test and ran with stack bench --profile cache-test --benchmark-arguments='+RTS -hc -s -RTS'; I get the following results:

for good:

Running Good:

     278,823,496 bytes allocated in the heap
1000
      15,291,392 bytes copied during GC
Benchmark auto: FINISH
       2,165,352 bytes maximum residency (7 sample(s))
         110,480 bytes maximum slop
              10 MB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0       165 colls,   165 par    0.064s   0.017s     0.0001s    0.0010s
  Gen  1         7 colls,     6 par    0.051s   0.014s     0.0021s    0.0042s

  Parallel GC work balance: 30.47% (serial 0%, perfect 100%)

  TASKS: 10 (1 bound, 9 peak workers (9 total), using -N4)

  SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)

  INIT    time    0.004s  (  0.003s elapsed)
  MUT     time    0.332s  (  0.316s elapsed)
  GC      time    0.102s  (  0.027s elapsed)
  RP      time    0.000s  (  0.000s elapsed)
  PROF    time    0.013s  (  0.004s elapsed)
  EXIT    time    0.001s  (  0.000s elapsed)
  Total   time    0.452s  (  0.350s elapsed)

  Alloc rate    840,844,689 bytes per MUT second

  Productivity  73.5% of total user, 90.3% of total elapsed

gc_alloc_block_sync: 3526
whitehole_spin: 0
gen[0].sync: 1
gen[1].sync: 35
Completed 2 action(s).

with heap profile.

and for bad:

Running Bad:

1000
   2,131,924,376 bytes allocated in the heap
   2,973,614,912 bytes copied during GC
     239,523,080 bytes maximum residency (27 sample(s))
         920,312 bytes maximum slop
             472 MB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0      1497 colls,  1497 par   48.666s  12.479s     0.0083s    0.0225s
  Gen  1        27 colls,    26 par   12.980s   3.371s     0.1248s    0.3105s

  Parallel GC work balance: 82.55% (serial 0%, perfect 100%)

  TASKS: 10 (1 bound, 9 peak workers (9 total), using -N4)

  SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)

  INIT    time    0.003s  (  0.003s elapsed)
  MUT     time    2.904s  (  2.630s elapsed)
  GC      time   53.022s  ( 13.638s elapsed)
  RP      time    0.000s  (  0.000s elapsed)
  PROF    time    8.624s  (  2.212s elapsed)
  EXIT    time    0.001s  (  0.001s elapsed)
  Total   time   64.553s  ( 18.483s elapsed)

  Alloc rate    734,259,598 bytes per MUT second

  Productivity   4.5% of total user, 14.2% of total elapsed

gc_alloc_block_sync: 498638
whitehole_spin: 0
gen[0].sync: 2
gen[1].sync: 689287
Benchmark auto: FINISH
Completed 2 action(s).

with heap profile

Notice the discrepancy in the GC time and calculated Productivity and the difference in the y-axis of the heap profiles. So I assume it is the case that (constrain $ a and b and c) /= ((constrain (a and b) >> return (a and b) >>= constrain . (and c)). Any suggestions for whats happening here? I've already reduced the amount of constrains in my computation to a maximum, but I'm dealing with about 16000 variables which easily blows up the cache.

Let me know if I can further assist in any way and thanks for the help.

[LeventErkok]

Thanks for the detailed write-up! I've never looked into performance characteristics, and there might very well be a memory leak in there. Let me take some time to investigate.

[LeventErkok]

Is there a reason why you can't collect all the "must-be-constraint" SBools in a Haskell data-structure, collect them all, and then assert their conjunction in the end? I wonder if that would address the issue.

Additionally, I can also imagine adding a variant of constrain that took a whole bunch of variables and instead of anding them, which will cost you the construction of the tree of binary ands, assert all of them in one go. That should also give some savings. But I'd first explore the first idea above to see if that'd handle the issue.

What do you think?

[doyougnu]

In the test example all the sBools are and'd but in my application that is not strictly true. If it were then I think having a separate data structure that stores, and then conjoins the SBools would work. Essentially I have a AST like:

data Lang a
   = Lit Bool
   | Ref a
   | Neg (Lang a)
   | Or (Lang a)
   | And (Lang a)
   | Foo !String (Lang a) (Lang a)

The semantics of which are identical to normal propositional logic except for the Foo constructor, which describes an interaction with the assertion stack of the underlying solver. For example if I have something like this (pretty printing for readability):

myFormula :: Lang String
myFormula = a /\ (~b => (a \/ c)) /\ (Foo a e) \/ ((a => b) \/ (c => a))

Then i do a left fold (or prefix traversal) of the AST. i end up with accumulatedSBool /\ (Foo a e) \/ (a => b \/ c => a). When I see a Foo I push onto the assertion stack, and combine the accumulatedSBool with Foos left child and recur to get a model representing the left child out, I then pop and repeat for the right side. For example with myFormula the recursion into the left side would become accumulatedSBool /\ a \/ ((a => b) \/ (c => a)) which would then be converted into a single SBool via recursion, at the end of the recursive call it would be constrain'd, used to get a model, then popped to reset the assertion stack back before the evaluation of the left side of Foo. So in this way a Foo describes a point where I manipulate the assertion stack directly to solve the two related formulas that are represented by Foos children. This is essentially why I need to keep a handle on the accumulated SBool, i.e., so that I can continue the recursion for a child of Foo for any binary logical connective such as implication, and/or etc..

This leads to having 2^|Foo| constraints because I only constrain when I evaluate on a Foo, other than that I just accumulate the SBools. I was under the impression that constraining an SBool adds the assertions represented by the SBool onto the assertion stack. If that is the case then I am already accumulating as much as possible and constraining as minimally as possible. Furthermore, if that is true then I would want to constrain as much as possible to keep the assertions on the assertion stack small and fine grained so that when I got to a Foo constructor I could reuse the state of the assertion stack as much as possible.

Hopefully that makes sense. I can mock up a more worked example if necessary.

[LeventErkok]

Yes; something that I can run an experiment with would be awesome!

Do you know if this is also an issue when you don't use query features at all? Looks like your use case does call for queries, but I'm curious if the culprit is in the query-layer or if this sort of bad behavior can be triggered from regular Symbolic code.

[doyougnu]

Interesting I observe the same behavior with only Symbolic code:

-- | the test runner, this time f returns a predicate and we have no query
-- | operations whatsoever
testS :: ([S.SBool] -> S.Predicate) -> Int -> IO S.SatResult
testS f n = S.sat $ (S.sBools $! stringList n) >>= f


-- | I fold over the string of SBools here constraining at each accumulation,
-- this seems to blow up the internal cache severely leading to about 95% GC
-- time
badS :: [S.SBool] -> S.Predicate
badS prop' = do b <- foldM (helper) S.true prop'
                S.constrain b
                return b
  -- | combine the current sbool with the accumulated sbool, constrain the
  -- two and then return the accumulated result
  where helper x acc = do let b = x S.&&& acc
                          S.constrain b
                          return b

-- | identical to the bad version but I do not constrain for each accumulation
goodS :: [S.SBool] -> S.Predicate
goodS prop' = do b <- foldM (helper) S.true prop'
                 S.constrain b
                 return b
  -- | this helper is equivalent to just foldr' (S.&&&)
  where helper x acc = do let b = x S.&&& acc
                          return b

main = do
  -- putStrLn "Running Good:\n"
  -- goodRes <- testS goodS 1000

  putStrLn "Running Bad:\n"
  badRes <- testS badS 1000

  -- just ensuring evaluation
  -- print (length $ show goodRes)
  print (length $ show badRes)

Good test:

Running Good:

15473
     280,648,024 bytes allocated in the heap
      15,749,208 bytes copied during GC
       2,972,112 bytes maximum residency (6 sample(s))
         173,616 bytes maximum slop
              11 MB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0       167 colls,   167 par    0.088s   0.025s     0.0002s    0.0027s
  Gen  1         6 colls,     5 par    0.083s   0.027s     0.0046s    0.0096s

  Parallel GC work balance: 18.75% (serial 0%, perfect 100%)

  TASKS: 10 (1 bound, 9 peak workers (9 total), using -N4)

  SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)

  INIT    time    0.005s  (  0.009s elapsed)
  MUT     time    0.396s  (  0.427s elapsed)
  GC      time    0.151s  (  0.046s elapsed)
  RP      time    0.000s  (  0.000s elapsed)
  PROF    time    0.020s  (  0.006s elapsed)
  EXIT    time    0.001s  (  0.001s elapsed)
  Total   time    0.573s  (  0.489s elapsed)

  Alloc rate    708,712,104 bytes per MUT second

  Productivity  69.3% of total user, 87.5% of total elapsed

gc_alloc_block_sync: 2217
whitehole_spin: 0
gen[0].sync: 0
gen[1].sync: 28
Benchmark auto: FINISH
Completed 2 action(s).

with heap profile:

Bad Test:

Running Bad:

24019
   2,151,236,224 bytes allocated in the heap
   3,536,806,008 bytes copied during GC
     245,950,352 bytes maximum residency (28 sample(s))
       1,026,160 bytes maximum slop
             488 MB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0      1890 colls,  1890 par   61.774s  17.736s     0.0094s    0.0435s
  Gen  1        28 colls,    27 par   15.184s   4.362s     0.1558s    0.3335s

  Parallel GC work balance: 77.98% (serial 0%, perfect 100%)

  TASKS: 10 (1 bound, 9 peak workers (9 total), using -N4)

  SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)

  INIT    time    0.004s  (  0.004s elapsed)
  MUT     time    3.664s  (  2.996s elapsed)
  GC      time   67.154s  ( 19.401s elapsed)
  RP      time    0.000s  (  0.000s elapsed)
  PROF    time    9.804s  (  2.698s elapsed)
  EXIT    time    0.001s  (  0.001s elapsed)
  Total   time   80.626s  ( 25.099s elapsed)

  Alloc rate    587,177,866 bytes per MUT second

  Productivity   4.5% of total user, 11.9% of total elapsed

gc_alloc_block_sync: 1812203
whitehole_spin: 0
gen[0].sync: 13
gen[1].sync: 863036
Benchmark auto: FINISH
Completed 2 action(s).

with heap:

[LeventErkok]

It's actually good to know that 'Symbolic' has the same issue.

Currently we store constraints in an IORef that looks like this:

 rConstraints :: IORef [(Bool, [(String, String)], SV)]

That's one list that you'll keep on overwriting all the time. We really need to replace that with a much better data-structure. It may not help solve the issue you're facing, but replacing it with something like a sequence would definitely help.

I'll see if I can code this up sometime tomorrow; shouldn't be that hard. Hopefully it'll fix the issue, but if not, it's an improvement that won't be a wasted effort. I should have something you can try out sometime tomorrow.

[LeventErkok]

@doyougnu

I just pushed in some changes to improve how constraints are stored internally. I'm not sure if it'll fix the issue, but hopefully, it won't make it worse! Can you build from master and give it a shot? I'm curious if it'll make any difference. We can then investigate further.

[doyougnu]

@doyougnu

I just pushed in some changes to improve how constraints are stored internally. I'm not sure if it'll fix the issue, but hopefully, it won't make it worse! Can you build from master and give it a shot? I'm curious if it'll make any difference. We can then investigate further.

Will do!

[LeventErkok]

Thanks! I very much doubt this'll solve the problem; as the issue is really with constructing those constraints in the first place; which isn't addressed by this change.

But performance is hard to predict, maybe we'll be pleasantly surprised!

[doyougnu]

Unfortunately your suspicions were correct. This had no impact on the issue whatsoever. Note I upgraded to 8.0.5 from 7.13 to build from master:

-- | the test runner, this time f returns a predicate and we have no query
-- | operations whatsoever
testS :: ([S.SBool] -> S.Predicate) -> Int -> IO S.SatResult
testS f n = S.sat $ (S.sBools $! stringList n) >>= f


-- | I fold over the string of SBools here constraining at each accumulation,
-- this seems to blow up the internal cache severely leading to about 95% GC
-- time
badS :: [S.SBool] -> S.Predicate
badS prop' = do b <- foldM (helper) S.sTrue prop'
                S.constrain b
                return b
  -- | combine the current sbool with the accumulated sbool, constrain the
  -- two and then return the accumulated result
  where helper x acc = do let b = x S..&& acc
                          S.constrain b
                          return b

main = do
  putStrLn "Running Bad:\n"
  badRes <- testS badS 1000

  -- just ensuring evaluation
  writeFile "badres" (show badRes)

and running:

Running Bad:

   2,139,086,472 bytes allocated in the heap
   3,984,221,968 bytes copied during GC
     244,111,848 bytes maximum residency (30 sample(s))
         969,408 bytes maximum slop
             484 MB total memory in use (0 MB lost due to fragmentation)

                                     Tot time (elapsed)  Avg pause  Max pause
  Gen  0      1871 colls,  1871 par   66.052s  19.048s     0.0102s    0.0496s
  Gen  1        30 colls,    29 par   18.475s   5.342s     0.1781s    0.3477s

  Parallel GC work balance: 82.21% (serial 0%, perfect 100%)

  TASKS: 10 (1 bound, 9 peak workers (9 total), using -N4)

  SPARKS: 0 (0 converted, 0 overflowed, 0 dud, 0 GC'd, 0 fizzled)

  INIT    time    0.003s  (  0.002s elapsed)
  MUT     time    3.986s  (  3.311s elapsed)
  GC      time   72.316s  ( 20.981s elapsed)
  RP      time    0.000s  (  0.000s elapsed)
  PROF    time   12.210s  (  3.409s elapsed)
  EXIT    time    0.001s  (  0.000s elapsed)
  Total   time   88.516s  ( 27.704s elapsed)

  Alloc rate    536,708,535 bytes per MUT second

  Productivity   4.5% of total user, 12.0% of total elapsed

gc_alloc_block_sync: 1478284
whitehole_spin: 0
gen[0].sync: 13
gen[1].sync: 1315257
Benchmark auto: FINISH

Here are some extra heap profiles:
By Type:

By Data Constructor:

Cost Center but with longer description:

What exactly goes on under the hood when constrain is called? I may have some time to hack on this tonight. This doesn't seem to me to be a space leak but rather a combinatorial explosion of storing combinations of the sBool accumulator. I'm unsure if this is really a problem or bug with sbv as I seem to be using it outside of its design space.

[LeventErkok]

In the end, it may not be something we can fix easily; but I'd like to understand it better. Right now I don't see any smoking guns.

Would it be possible for you to create a github repo and scripts etc? I'd like to clone it so I can run the profiler myself to experiment. I'm hoping that's not a lot of work for you!

[doyougnu]

Sure I can set this up tonight. You prefer cabal over stack? As for scripts I'll make a few to automatically build and execute with the profiling I've shown here. Is that sufficient?

[LeventErkok]

stack is just fine; cabal is OK too; whichever you already have. a "doit" script that spits out these graphs would be fantastic!