Monday, May 4, 2009

Hibernate Wars: The Query Cache Strikes Back

Not so long ago, in a galaxy not very far away... a middle-aged programmer battled to free memory from the dark side of the hibernate query cache.  He was successful, or so it seemed. For the query cache memory problems had risen from the ashes -- stronger and more bloated than ever...


What's With All This Garbage?


We hit a case at work (again), where the java server process came to a grinding halt.  It wasn't dead, it just wasn't making much forward progress either.  A quick look at the heap stats showed we were nearly at capacity and that garbage collection was eating all CPU time, thus not allowing any real work to happen.  Looks like it is time to whip out the ol' memory profiler.  This time I went with Eclipse's Memory Analysis Tool .  It's pretty slick.  I suggest you try it.

Query Cache Waste Redux


Not terribly unexpected, the hibernate caches were the top consumer of heap space.  I have previously explored ways to trim down the memory used by the query cache.  But, we depend heavily on it and size it pretty big, so that's why I wasn't too surprised to see them at the top of the heap report.  Drilling down a little further showed that it was not the contents of the query cache results that were causing the problem.  That was the unexpected part.  Rats, our friend QueryKey is once again a giant waster of memory.



What is it this time?  In A Dirty Little Secret, I showed that you should use object identifiers in your HQL and as parameters so that full objects are not stored as part of the QueryKey.  This was a major win.  I also showed that by providing a decorator query cache implementation, you can reduce all duplicate stored parameters by replacing them with canonical representations.  Using MAT (Memory Analysis Tool), I proved that this was still working as expected.



What I hadn't previously accounted for was the QueryKey.sqlQueryString field.  Since we use a lot of natural id query cache optimizations, we have tens of thousands (and in some cases, well over 100,000) copies of identical queries tucked away in QueryKey as the sqlQueryString (thus, differentiated only by the query parameters).  And since hibernate generated SQL is not exactly terse, we have a nice formula for memory explosion.

Different Tune, Same Dance


We're already using my decorated query cache which eliminates duplicates in the parameters.  So I decide to modify it to also swap out the sqlQueryString for a canonical representation.  One caveat is that sqlQueryString is private and final.  Lo and behold, you can modify private final fields with reflection in Java 1.5!  Granted, you could really do some silly stuff and fake out the JVM and screw up compiler optimizations, but we're only replacing the field with another String that should be functionally equivalent, so hopefully any 'weirdness' is mitigated.  Again, for licensing reasons I won't paste the whole query cache decorator.  Creating the full one (and the factory to instantiate it) is left as an exercise to the reader.



The modified query cache decorator looks like this:

 
private final Map<Object, Object> canonicalObjects = new HashMap<Object, Object>();

 
public boolean put(QueryKey key, Type[] returnTypes,
@SuppressWarnings("unchecked") List result, boolean isNaturalKeyLookup,
SessionImplementor session) throws HibernateException {

// duplicate natural key shortcut for space and time efficiency
if (isNaturalKeyLookup && result.isEmpty()) {
return false;
}

canonicalizeKey(key);

return queryCache.put(key, returnTypes, result, isNaturalKeyLookup,
session);
}

private void canonicalizeKey(QueryKey key) {
try {
synchronized (canonicalObjects) {
canonicalizeParamValues(key);
canonicalizeQueryString(key);
}
} catch (Exception e) {
throw Exceptions.toRuntime(e);
}
}

private void canonicalizeParamValues(QueryKey key)
throws NoSuchFieldException, IllegalAccessException {
final Field valuesField;
valuesField = key.getClass().getDeclaredField("values");
valuesField.setAccessible(true);
final Object[] values = (Object[]) valuesField.get(key);
canonicalizeValues(values);
}

private void canonicalizeQueryString(QueryKey key)
throws NoSuchFieldException, IllegalAccessException {
final Field sqlQueryString;
sqlQueryString = key.getClass().getDeclaredField("sqlQueryString");
sqlQueryString.setAccessible(true);
Object sql = sqlQueryString.get(key);
Object co = ensureCanonicalObject(sql);
if (co != sql) {
sqlQueryString.set(key, co);
}
}

private void canonicalizeValues(Object[] values) {
for (int i = 0; i < values.length; i++) {
Object object = values[i];
Object co = ensureCanonicalObject(object);
values[i] = co;
}
}

// assumes canonicalObjects is locked. TODO: consider a concurrent hash
// map and putIfAbsent().
private Object ensureCanonicalObject(Object object) {
Object co = canonicalObjects.get(object);
if (co == null) {
co = object;
canonicalObjects.put(object, co);
} else if (co != object) {
// System.out.println("using pre-existing canonical object "
// + co);
}
return co;
}



As you can see, we simply change the sqlQueryString via reflection just like we do the param values.

Your Mileage May Vary

How much of a win this is for you depends on your use case.  As I said, since we heavily use the natural id query optimization, we had tons of repeated sql strings.  So, exactly how much memory you 'reclaim' in this fashion totally depends on the variety and uniqueness of the queries you run through the hibernate query cache.

Bonus: L2 Cache Reduction


While we're at it, I noticed a few other things that seemed odd in the L2 cache.  There was more HashMap overhead than there was contents in the cache itself.  I poked around the source a bit and saw that every entry in the L2 cache was being turned into a property->value map before it was stored in your L2 cache provider (and the inverse process occurs on the way out).  This seemed odd to me, as we already have a decomposed CacheEntry object which is an array of the Serializable[] properties from your persistent entity.  Why create another (less efficient) representation as well as introduce unnecessary conversions?  After some google-fu, I realized you can bypass this conversion by setting hibernate.cache.use_structured_entries to false in your hibernate configuration.



Any docs I found on hibernate.cache.use_structured_entries merely seemed to suggest that it stores the properties in a more 'human friendly format.'  And, who wouldn't want that?  And, all examples we built on when first starting with hibernate turned it on, so... so did we.  What they don't mention is what it actually does, and the penalty you pay for doing so -- which is apparently too much HashMap overhead for what should be pretty simple in memory storage.



However, be aware -- this only works for non-clustered, in-memory use of L2 cache.  Apparently, if you cluster your JVMs and need L2 cache clustering, the in/out of the property order cannot be guaranteed between JVMs.  Thus, you have to use structured entities in the cache so they can be re-hydrated properly by property name.

Right-Size Your Caches


We moved to the 1.6 release of EH Cache, so... this may only apply to that version.  But, I noticed that whatever you configure as 'max elements' for ehcache, it uses as the 'min capacity' for the ConcurrentHashMap.  Needless to say, if you size them for your potential largest usage, but then deploy to a smaller-usage environment, you can end up with some waste in the overhead for the hash tables.  It didn't seem terribly significant in my case, but it did show up on the radar.

Wrap-Up


Even in today's multi-gigabyte server (heck, even desktop...) environments, memory is still a precious resource.  Even if you have lots of it available, wouldn't you want to make more use of that memory instead of having it wasted?  Freeing up wasted memory means there is more for the 'transient' objects that come and go quickly.  There's less pressure to garbage collect and try to find available heap.  And, there's more memory available to do smart things with useful caching. In short:



  • Use a query cache decorator to reduce parameter and SQL string duplication.

  • If you are in a single JVM using in memory cache only, use hibernate.cache.use_structured_entries=false in your hibernate configuration.

  • Right-size your caches to reduce hash table overhead.


8 comments:

Alois Reitbauer said...

Good postand nice sample. Possibly you also want to check out my Hibernate posts and tell me what you think.

- Alois

http://blog.dynatrace.com

Steve Ebersole said...

Hey Oldag,

Wanted to drop you a response to this post and the other Hibernate L2 cache one.

First, in regards to the natural id look up cache, there is already a more correct (imho) solution to this : http://opensource.atlassian.com/projects/hibernate/browse/HHH-2896 Feel free to comment there if there are specific points you'd like to see addressed, or discuss it on the hibernate-dev mailing list or #hibernate-dev irc channel on freenode...

WRT the parameter values used in the CacheKey, I have to believe it is just an oversight that the parameter value itself is used and not its "disassembled" state. The key already has access to the Type objects needed to disassemble them. Would you be willing to give that a try and see how it works out? The disassembled state of an entity is its id -> http://fisheye.jboss.org/browse/Hibernate/core/branches/Branch_3_2/src/org/hibernate/type/ManyToOneType.java?r=16097#l148

WRT the query string I actually don't get exactly what you are doing. You swap one string for another string where both have the same content. Not really seeing the point. Reusing the same string instance each time would alleviate the calculation of the hashCode (which String caches on a delayed basis) but you use the initial string in the HashMap lookup, so its hasCode would get calculated as well; so no savings there (and that would just be a time saving anyway). Also, this code is functional similar to String.intern(). So I am confused with this one, could you expand?

Steve Ebersole said...

Of course that should have read "there is already a more correct (imho) solution to this planned..." ;)

Oldag said...

@Steve,

I think HHH-2896 looks pretty interesting. Definitely our biggest use of the query cache is for natural id lookups... and the existing issue comment(s) already point out that the issue should take those into consideration. For sure, something other than the criteria API would be more 'natural' to use for natural id centric loading, imho. If i think of anything to add, I'll be sure to do so.

So, I'm not sure if you are referring to the L2 "CacheKey" or specifically the query cache's "QueryKey." I haven't found any problems with CacheKey, and it appears to already store the Serializable id identifier for the object in L2.

QueryKey is another matter. It does have the Type[] for the positional parameters, and the named parameter map is just TypedValue(s), so the types are available there, too. Manuel posted a comment in my other blog entry (Dirty LIttle Secret) with some sample java code that appears to do nearly what we want (http://www.residencialaguardia.com/temp/QueryKey.java). However, it doesn't use that ManyToOne.disassemble path you reference. And I'd venture a guess your suggestion is more on target with my thinking. I just looked at the Type interface, and 'disassemble()' appears to do the right thing for each type appropriately. Good show.

I'd like to try that out "when I can." But, as it stands, we have a solution in place that works for us, and getting some free time to play around a bit more might not be in the cards in the short term.

As for the swap of the query string -- the goal there was to eliminate string duplication (and thus massively reduce heap use) as much as possible. The more I think about it, the more I come to realize it is the 'natural id query lookup optimization' that is biting us. If it were another query, be it a named HQL query or some static HQL (or sql) in our application -- then it would likely ALREADY be cached or singly-referenced JVM wide, and not be duplicated.

However, since the sql for the criteria query is built up dynamically, each time -- the JVM cannot optimize that as a constant and you have the same sql text over and over and over in the QueryKey(s).

I only did this to reduce memory overhead. I didn't consider, at all, any cpu-centric hashCode and comparison overhead. I purposely only "canonicalize" on the put (write) path, as to not introduce any extra churn or changes on the get (read) path. I didn't want to change anything there, like add a lock or do any QueryKey swapping, because we are way way more read-centric once the cache is populated. So, as long as we stay hashCode/equals compatible with both the modified put query key, and the query key that comes in on a read lookup, then nothing needs to change on the read path. That isn't to say perhaps some other benefit could not be found and utilized in this path, it just wasn't a profiled cpu hotspot for us.

Alas, String.intern() reads like the perfect solution -- on paper. If all those sql strings were intern'd, we could resort to identity based map lookups. And, they would be guaranteed to use minimal storage. But, a bit of 'research' on the subject showed some myth and mystery around String.intern(), and my tech lead architect said "don't use it if you can avoid it." So, basically I ended up writing an intern-like map anyway, but that will use regular heap and not sit there in PermGen. On 2nd thought, I should probably use a weak hash map here so they go away if all the query keys get flushed. But, that is seriously not likely to happen in our case (at least, not for the sql test repetition case.

Steve Ebersole said...

My brain said "QueryKey" and my fingers typed "CacheKey". Never trust the fingers :)

disassemble() is defined on the base Type contract : http://fisheye.jboss.org/browse/Hibernate/core/branches/Branch_3_2/src/org/hibernate/type/Type.java?r=16537#l313 meaning its available for all types. It (and its compliment assemble()) are the methods used to stored property values into the L2 cache and read them back. I was just saying it is probably just an oversight that those parameter values are currently not disassembled.

Oldag said...

then i propose "fix it" per issue HHH-3383;)

Again, Manuel replied in my other post that this might work, except that SessionImplementor is not currently available in the scope where QueryKey is constructed. Straightforward to fix, but probably touches a lot of code. Just a Simple Matter of Programming at that point. And, yay for test suites! (so one can be unafraid of refactoring)

indian said...

Hi,

Was this query key issue fixed in hibernate 3.5.1?

I am using the following maven dependencies:


org.hibernate
hibernate-entitymanager
3.5.1-Final



org.hibernate
hibernate-ehcache
3.5.1-Final


Can they be used in production and live environments or should i wait for another release?

Oldag said...

@ indian --

I've sorta stopped tracking the issue, as what we have is working for us. Perhaps check the hibernate jira issues or pester steve. :)