Wednesday, June 18, 2008

Hibernate: Cache Queries the Natural Id Way

I work on the MySQL Enterprise Tools team, formerly of MySQL and now with Sun Microsystems. The 2.0 version of the Enterprise Monitor is well under way. As part of this, the Java server backend has been refactored to utilize Spring and Hibernate. Honestly, I didn't know either one of those technologies before starting this project. Oh, what a fun road it has been...

A big draw for using an off-the-shelf ORM was so that we didn't have to write our own (kind of bad and slightly wrong -- those darn transactions) caching implementations for the custom one-off ORM that existed previously. A lot of our internal meta-model is very static, so clearly caching would be a HUGE win for performance, right?

Not so fast, turbo. Let me continue...

The headline feature for the 2.0 Monitor is "Query Analysis." Coupled with the MySQL Proxy, the Monitor receives captured query data to/from a MySQL server. Once at the monitor, the data can be aggregated, analyzed, and reported upon. What better test for this feature than to use it on ourselves, to tune ourselves!

And this brings me back to hibernate caching. In the course of monitoring ourselves, I noticed that a certain query was happening WAY more than it should be, just based on my gut feeling. The query in question loaded an object that was generally static -- save for one value that represents the frequency of how often some data should be collected. Its the only mutable value, and once in place, it rarely changes.

Hrm... how to debug. First, we checked the cache settings. Whoops -- WAY too low for both the cache expiry timeout and the max cache elements. Fix that. Still sucks. Some cursory hibernate source and log perusal showed that the cache for these objects was being invalidated at a rapid rate. Yes, the entire cache. Even though the objects are essentially static, the query cache takes the safe route and says any change to a related table invalidates any and all queries that reference that table. This makes sense, from a generic cache standpoint. But I thought to myself -- surely there has to be a way. *I* am smarter than hibernate in this case, and *I* can more rightly determine when the query results should be invalidated. Lucky for me, hibernate allows you to extend the StandardQueryCache "up to date" policy checks. w00t. I implemented one, overrode the timeout policy for the object(s) in question, and re-ran tests. FAILURE. Turns out I am not smarter than hibernate.

However, in the process of implementing the custom query cache policy, I had debugged through some more hibernate code and noticed that "natural id" queries are treated special. Some more google-fu, and quickly I come across Queries by Natural Identifier in the hibernate docs.

Now, the docs just aren't real clear on what optimizations can be made internally by utilizing the Criteria with a natural id restriction. But, as I was just in that section of code, I could correlate it. Here's the meaty bit -- if you make a natural id / key lookup, and hibernate recognizes it as such, it can bypass the table timestamp invalidation and go directly to second-level cache to fetch the object. Hibernate knows, with an immutable and unique natural key, that a table modification will not effect the composition of the object in question (of course, an object modification would, and it would have been evicted from L2 cache).

I cannot overemphasize the utility of this discovery. You see, we were making frequent inserts into the table. But existing objects (rows) where changed almost never. But without the natural key lookup, the inserts invalidated all results in the query cache. There you have why I was seeing way more selects for the same objects than I had anticipated.

Some quick assurances that we mapped the natural id correctly, some quick refactoring of the HQL into Criteria queries with natural id restrictions, and whammo, we're good. Lets run the tests and query analysis again... ruh roh. OH COME ON! (not my exact reaction, but I think you can guess what it was really like).

Confident that the natural id cache lookup optimization was what I really really really wanted, there had to be something else going on. More debuggage ensued. I set a breakpoint near the same area in StandardQueryCache where I first noticed the query cache optimization in the first place. Lo and behold, the hibernate metadata for saying "i am a natural key lookup" was returning false.

I am not amused. I am confident my hibernate mapping is correct, because the unique index was present in the schema. Think. Think. Think. Well, I had recently been on an effort to move from the hbm xml mappings to hibernate annotations mappings. @NaturalId support was, in fact, the very reason I had recently upgraded the annotations jar. On a hunch, I reverted the persistence mapping back to the xml form. Debug, the metadata returns the correct value... test, and YES, finally -- the queries issued are in line with my expectations and the rows present in the database. I. Have. Won.

Being the good open-source citizen, I made a hibernate forum post that detailed my findings, including simplified sample code demonstrating the problem. The good folks on the hibernate forum (after questioning the sillyness of my contrived example) were quick to recognize the problem, and I got a hibernate jira issue opened.

The workaround, obviously, is leave the xml mapping in place until the fix makes it into a hibernate release. Not too bad of a deal, I guess, considering the overall win I now have in my cache hit ratio.

In conclusion -- if it makes sense for your data model, the natural id query cache optimization can be a huge performance win for your app. If you have immutable, or rarely changed objects with a constant natural key lookup -- look into the Criteria natural id restriction. And, use the xml mapping until the bug is fixed.

ps -- there is one other performance note to consider, actually. If using the natural id query, and it returns no rows, this NULL result will not be cached. So, if you have more of these than 'object/row found' results, you will still get tons of these queries that you don't expect. Either stop using the natural key optimization (if 'not found' is more common), or extend your object/schema to include a 'not supported' field. In our case, the lack of a row meant "not supported" and we had a flag "not supported" in case it was supported, but then went away. In those cases where something was frequently "not supported" I simply went ahead and created the object/row and just set the flag to false -- thus ensuring the natural key optimization was not subverted.


151 comments:

Hardy Ferentschik said...

Bug in Hibernate Annotations is fixed now. Haven't traced the code all the way into the caching part, but the meta data is correct now. I am intrigued to hear if this will solve your problem.

Oldag said...

I pulled the latest annotations src from their trunk, built, and ran with it. I'm pleased to say the @NaturalId mapping is fixed, and the query cache optimization works as expected.

You can read more about it back in the hibernate forum thread linked to above.

thanks again, Hibernate team!

Darin Keever said...

So.. if you write a long enough blog, someone fixes something? Guess I need to start blogging.. capriciously.

Unknown said...

Hi,

We are facing a problem with memcache. We are using hibernate-memcached-version1.5-SNAPSHOT and spymemcached-provider-version3.0.2.

The following are the configuration

persistence.xml
---------------

!-- Enable 2nd Level Cache --

property name="hibernate.cache.use_second_level_cache" value="true"
property name="hibernate.cache.use_query_cache" value="true"

!-- MemCache Configuration --

property name="hibernate.cache.region.factory_class" value="com.googlecode.hibernate.memcached.MemcachedRegionFactory"
property name="hibernate.cache.use_minimal_puts" value="true"
property name="hibernate.cache.use_structured_entries" value="true"
property name="hibernate.memcached.servers" value="${hibernate.memcached.server.host.port}"
property name="hibernate.memcached.cacheTimeSeconds" value="1800"

dto
----

@Cacheable(true)
@org.hibernate.annotations.Cache(usage = CacheConcurrencyStrategy.READ_WRITE)

In GenericDao we are setting query hints cacheable to true for loadAll().

We are using loadAll() method to fetch all the records.
Whenever we made a request the loadAll query is executing and also the queries based on id are executing.

when i refer to the log am able to notice that the data is fetching from database and setting in memcache for a request and when we make another request instead of fetching the data from the memcache it is hitting again to the db and again setting to memcache. Am unable to understand that without modifying any data why it is hitting to db?

Please let me know we are missing anything.

Aliya Manasa said...

I have read a few of the articles on your website now, and I really like your style of blogging. I added it to my favourites blog site list and will be checking back soon.
python Course in Pune
python Course institute in Chennai
python Training institute in Bangalore

service care said...

I think this is a great site to post and I have read most of contents and I found it useful for my Career .Thanks for the useful information. Good work.Keep going.
oneplus service center chennai
oneplus service center in chennai
oneplus service centre chennai

Soumitasai said...

Outstanding blog thanks for sharing such wonderful blog with us ,after long time came across such knowlegeble blog. keep sharing such informative blog with us.

Check out : big data training in velachery
big data analytics training and placement
big data training in chennai chennai tamilnadu
big data workshop in chennai

Gowri said...


It’s interesting content and Great work. Definitely, it will be helpful for others. I would like to follow your blog. Keep post

Check out:
best hadoop training in omr
hadoop training in sholinganallur
best institute for big data in chennai
big data hadoop course in chennai with placement

Diya shree said...

Good job and thanks for sharing such a good blog You’re doing a great job. Keep it up !!

PMP Certification Fees | Best PMP Training in Chennai |
pmp certification cost in chennai | PMP Certification Training Institutes in Velachery |
pmp certification courses and books | PMP Certification requirements |
PMP Training Centers in Chennai | PMP Certification Requirements | PMP Interview Questions and Answers

Kayal said...

Nice post! I love this blog and I got more kinds of techniques in this topic. Thanks for your sharing.

Primavera Training in Chennai
Primavera Course in Chennai
Pega Training in Chennai
Unix Training in Chennai
Power BI Training in Chennai
Excel Training in Chennai
Oracle Training in Chennai
Social Media Marketing Courses in Chennai

vinith said...

Its as if you had a great grasp on the subject matter, but you forgot to include your readers. Perhaps you should think about this from more than one angle.
python training in bangalore

Gowri said...

Wow! Really a nice blog. Thank you so much for you effort.

Check out:
reactjs interview questions pdf
reactjs training in chennai
react js tutorial


Gowri said...


It’s interesting content and Great work. Definitely, it will be helpful for others. I would like to follow your blog. Keep post

Check out:
Selenium training courses in chennai
Selenium training center in chennai
Selenium training in chennai quora
Selenium course fees in chennai

Chris Hemsworth said...

The article is so informative. This is more helpful for our
software testing training courses
selenium classes Thanks for sharing

IT Tutorials said...


Get the most advanced Python Course by Professional expert. Just attend a FREE Demo session.
For further details call us @ 9884412301 | 9600112302
Python training in chennai | Python training in velachery

Tyler Worgan said...

This post is good enough to make somebody understand this amazing thing, and I’m sure everyone will appreciate.residential tree services palm beach island

Tyler Worgan said...

Hi, This is nice article you shared great information i have read it thanks for giving such a wonderful Blog for reader.
fence repair orlando

Tyler Worgan said...

Great article and a nice way to promote online. I’m satisfied with the information that you provided
metallic epoxy floor treasure coast

Lee Davis said...
This comment has been removed by the author.
neomacooper said...

I have read your article, it is very informative and helpful for me.I admire the valuable information you offer in your articles. Thanks for posting it..
broward county remodeling contractors

Lee Davis said...

Superbly written article, if only all bloggers offered the same content as you, the internet would be a far better place. vinyl fence jacksonville

haroonullah said...

I have read your article, it is very informative and helpful for me.I admire the valuable information you offer in your articles. Thanks for posting it..

Haroon Ullah

pamelapeterson said...

Thanks for a wonderful share. Your article has proved your hard work and experience you have got in this field. Brilliant .i love it reading.

Visit Website

Tyler Worgan said...

Very nice bro, thanks for sharing this with us. Keep up the good work and Thank you for sharing information
privacy fence tampa

Unknown said...

This post is good enough to make somebody understand this amazing thing, and I’m sure everyone will appreciate.bathroom remodelers fort lauderdale

Tyler Worgan said...

Very nice bro, thanks for sharing this with us. Keep up the good work and Thank you for sharing information
bathtub refinishing and fiberglass expert atlanta

Dana Milton said...

Thanks for a wonderful share. Your article has proved your hard work and experience you have got in this field. Brilliant .i love it reading. mold remediation companies miami

Alvin Steve said...

Great post.
tree stump removal boca raton

Clark said...

Thanks for sharing.
kitchen remodel south florida

Alvin Steve said...

Useful post. emergency ac repair west palm beach

Clark said...

Great post.
solariums west palm beach

Alvin Steve said...

A useful post shared. 24 hour air conditioning repair royal palm beach

Clark said...

Nice post.
kitchen and bath remodeling south beach

Clark said...

Amazing post shared.
concrete overlay contractors port st lucie

Clark said...

Great post.
kitchen and bath remodelers palm beach

Clark said...

Great post.
commercial septic systems san diego

Clark said...

Nice post.
commercial septic systems west palm beach

Training for IT and Software Courses said...

Linking is very useful thing.you have really helped lots of people who visit blog and provide them use full information.devops Training in Bangalore

Training for IT and Software Courses said...

Very useful and information content has been shared out here, Thanks for sharing it.Amazon web services Training in Bangalore

Realtime Experts said...

thank you so much for this nice information Article, Digital marketing is tha good skill in grouth tha career For website creation, promotion and development contact here. For your digital marketing needs just have a look at Click Perfect.dot net training in bangalore


AI said...

"Just saying thanks will not just be sufficient, for the fantastic lucidity in your writing. I will instantly grab your articles to get deeper into the topic. And as the same way ExcelR also helps organisations by providing data science courses based on practical knowledge and theoretical concepts. It offers the best value in training services combined with the support of our creative staff to provide meaningful solution that suits your learning needs.

Business Analytics Courses "

Bangalore Training Academy said...

Your topic is very nice and helpful to us … Thank you for the information you wrote.

Bangalore Training Academy located in Bangalore, is one of the best Workday Training institute with 100% Placement support. Workday Training in Bangalore provided by Workday Certified Experts and real-time Working Professionals with handful years of experience in real time Workday Projects.

Softgen Infotech said...

Very interesting, good job and thanks for sharing such a good blog.

Advance your career as a SharePoint Admin Engineer by doing SharePoint Admin Courses from Softgen Infotech located @BTM Layout Bangalore.

AI said...

"Just saying thanks will not just be sufficient, for the fantastic lucidity in your writing. I will instantly grab your articles to get deeper into the topic. And as the same way ExcelR also helps organisations by providing data science courses based on practical knowledge and theoretical concepts. It offers the best value in training services combined with the support of our creative staff to provide meaningful solution that suits your learning needs.

Business Analytics Courses "

eTechno Soft Solutions said...

Really i appreciate the effort you made to share the knowledge. The topic here i found was really effective...

Learn Blue Prism Course from Experts. Softgen Infotech offers the Best Blue Prism Training in Bangalore .100% Placement Assistance, Live Classroom Sessions, Only Technical Profiles, 24x7 Lab Infrastructure Support.

The Gabay Group said...

פוסט מרענן במיוחד. לגמרי משתף.
קבוצת גבאי נדלן

Softgen Infotech said...

Thank you for sharing such a nice post!

Become an Expert In Python Training in Bangalore ! The most trusted and trending Programming Language. Learn from experienced Trainers and get the knowledge to crack a coding interview, @Softgen Infotech Located in BTM Layout.

Unknown said...

Nice post.

פרגולה מאלומיניום

Cinema furniture said...
This comment has been removed by the author.
Dafmatok Hosting Trays said...

לגמרי פוסט שדורש שיתוף תודה
מגשי אירוח חלבי כשר

Cinema furniture said...

הייתי חייבת לפרגן, תודה על השיתוף.
פינת אוכל

Comfi furniture said...

סגנון כתיבה מרענן, תודה על השיתוף.
שולחן אוכל

Amediciercosmetic said...

הדעות שלי קצת חלוקות בעניין הזה אבל ללא ספק כתבת מעניין מאוד.
טיפול prp לשיער

Unknown said...

valueable post keep posting.
זרוע לטלויזיה

Unknown said...

Your articles really impressed for me,because of all information so nice.
Thai lottery

Planner Productions said...

סופסוף מישהו שתואם לדעותיי בנושא. תודה.
אטרקציות לחתונות

gemcreature said...

Wow!!! It was really an Informational Article which provide me with much Insightful Information.
playboy bunny necklace silver

Bloxi said...
This comment has been removed by the author.
alasksecurity said...

רציתי רק לשאול, אפשר לשתף את הפוסט בבלוג שלי?
מערכת אזעקה לבית

Reputation management said...

מזל שנתקלתי בכתבה הזאת. בדיוק בזמן
ניהול מוניטין באינטרנט

Unknown said...

Very Nice post.

Wound electrical stimulation

Unknown said...

Great post.

123 movies

Planner Productions said...

תמשיכו בפרסום פוסטים מעניינים כמו זה. תודה.
אינדקס פלאנרז

Unknown said...

Thanks for Sharing This Article.
global asset management

Sitelinx said...

אין ספק שהפוסט הזה דורש שיתוף. תודה.
מומחה קידום אתרים בגוגל

Unknown said...

מעולה. תודה על הכתיבה היצירתית.
מיטת מעבר

Keerthi SK said...

i have been following this website blog for the past month. i really found this website was helped me a lot and every thing which was shared here was so informative and useful. again once i appreciate their effort they are making and keep going on.

Digital Marketing Consultant in Chennai

Freelance Digital Marketing Consultant"

Unknown said...

הדעות שלי קצת חלוקות בעניין הזה אבל ללא ספק כתבת מעניין מאוד.
השקעות מניבות

Dogma said...

רציתי רק לשאול, אפשר לשתף את הפוסט בבלוג שלי?
אפיון חווית משתמש

Global Asset Management Udemy said...

Thanks a lot!
global asset management

How To Asset Manage said...

Wow, great post.
global asset management

david said...

פוסט שכיף לקרוא. תודה ששיתפתם
אשתף בבלוג - קולאז תמונות מתנה

akshaya said...

I am glad to me a fan of your stupendous blog.This article is both instructive and informative. All the information’s are furnished to the point. Web Designing Course Training in Chennai | Web Designing Course Training in annanagar | Web Designing Course Training in omr | Web Designing Course Training in porur | Web Designing Course Training in tambaram | Web Designing Course Training in velachery

Clark said...

good.
מצלמות לבית

intercom4u said...

Thanks for such a nice content.
אינטרקום וידאו

Fixadoor said...

Nice one
Garage door repair Toronto

radhika said...

Thanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
AWS training in Chennai

AWS Online Training in Chennai

AWS training in Bangalore

AWS training in Hyderabad

AWS training in Coimbatore

AWS training

Petersons said...

Congrats, determination and hard working never get bad result.


Bastion Balance Seoul

lavanya said...

Thank you for the nice article here. Really nice and keep update to explore more gaming tips and ideas.
I am really enjoyed a lot when reading your well-written posts. It shows like you spend more effort and time to write this blog. I have saved it for my future reference. Keep it up the good work.Java training in Chennai

Java Online training in Chennai

Java Course in Chennai

Best JAVA Training Institutes in Chennai

Java training in Bangalore

Java training in Hyderabad

Java Training in Coimbatore

Java Training

Java Online Training

Jayalakshmi said...

Learned a lot of new things from your post! Good creation and HATS OFF to the creativity of your mind
angular js training in chennai

angular js training in tambaram

full stack training in chennai

full stack training in tambaram

php training in chennai

php training in tambaram

photoshop training in chennai

photoshop training in tambaram

deiva said...

Very interesting, good job and thanks for sharing such a good blog.
web designing training in chennai

web designing training in omr

digital marketing training in chennai

digital marketing training in omr

rpa training in chennai

rpa training in omr

tally training in chennai

tally training in omr

ramesh said...


Very nice post here and thanks for it .I always like and such a super contents of these post.Excellent and very cool idea and great content of different kinds of the valuable information's.


Azure Training in Chennai

Azure Training in Bangalore

Azure Training in Hyderabad

Azure Training in Pune

Azure Training | microsoft azure certification | Azure Online Training Course

Azure Online Training

Petersons said...

Subsequently, after spending many hours on the internet at last We've uncovered an individual that definitely does know what they are discussing many thanks a great deal wonderful post.

Bastion Balance Korea

aravind said...

Good job in presenting the correct content with the clear explanation. The content looks real with valid information. Good Work it’s really nice and meanful. it’s really cool blog. Linking is very useful thing.you have really helped lots of people who visit blog and provide them useful information.
DevOps Training in Chennai

DevOps Online Training in Chennai

DevOps Training in Bangalore

DevOps Training in Hyderabad

DevOps Training in Coimbatore

DevOps Training

DevOps Online Training

jeni said...

Its such as you learn my mind! You appeаr tо grasp ѕo much approximately this, such as you wrote the book in it or something.
java training in chennai

java training in velachery

aws training in chennai

aws training in velachery

python training in chennai

python training in velachery

selenium training in chennai

selenium training in velachery

shiny said...

This post is very informative.

web designing training in chennai

web designing training in annanagar

digital marketing training in chennai

digital marketing training in annanagar

rpa training in chennai

rpa training in annanagar

tally training in chennai

tally training in annanagar

vivekvedha said...

It’s interesting content and Great work. Definitely, it will be helpful for others. I would like to follow your blog.
acte reviews

acte velachery reviews

acte tambaram reviews

acte anna nagar reviews

acte porur reviews

acte omr reviews

acte chennai reviews

acte student reviews

prabhu said...

Excellent and very cool idea and the subject at the top of magnificence and I am happy to this post..Interesting post! Thanks for writing it.
IELTS Coaching in chennai

German Classes in Chennai

GRE Coaching Classes in Chennai

TOEFL Coaching in Chennai

spoken english classes in chennai | Communication training


radhika said...

Good Post! Thank you so much for sharing this pretty post, it was so good to read and useful to improve my knowledge as updated one, keep blogging.



AWS Course in Bangalore

AWS Course in Hyderabad

AWS Course in Coimbatore

AWS Course

AWS Certification Course

AWS Certification Training

AWS Online Training

AWS Training

Garage Door Pros Ontario said...

Well, this post is quite good
garage door repair

vivekvedha said...

Your articles really impressed for me,because of all information so nice.
acte chennai

acte complaints

acte reviews

acte trainer complaints

acte trainer reviews

acte velachery reviews complaints

acte tambaram reviews complaints

acte anna nagar reviews complaints

acte porur reviews complaints

acte omr reviews complaints


a1garagedoors said...

Thanks you for sharing this unique useful information content with us.
garage door repair

James said...

Thanks and keep up the good work
Overhead Door Pros

Petersons said...

Really!!! I am very impressed after reading this blog. thanks for providing deep information.


Kald Gart Garage Doors Calgary

euro cars said...

חשבתי שאתקל בסתם עוד מאמר שטחי. טעיתי.
פיאט דוקטו

euro cars said...

אם כי אני לא מסכים עם כל מה שנכתב, מאמר מעניין
יישור שיניים

un known said...

hanks for Sharing This Article.It is very so much valuable content. I hope these Commenting lists will help to my website
devops online training
best devops online training
top devops online training

James said...

Excellent post.
broken extension spring repair ottawa

James said...

Superbly written article
garage door repair brock

James said...

Keep writing article like this.
garage door track adjustment

Petersons said...

This is really great news. Thank you for sharing it with us!


garage spring replacement pittsburgh

James said...

This article content is really unique and amazing.
Fix it mag

Petersons said...

Thats great. I got the right one information at the right time for the right situation. Thanks for sharing.


etobicoke washing machine repair

James said...

wonderful Blog
24 hr door repair service calgary

Petersons said...

Valuable service for individuals as well as businesses.


garage door repair yaletown

Unknown said...

Great post!

Tree Removal Charlotte, NC

rstrainings said...

I am really happy to say it’s an interesting post to read . I learn new information from your article , you are doing a great job . Keep it up

Devops Training in Hyderabad

Hadoop Training in Hyderabad

Python Training in Hyderabad

Unknown said...

Great info!
Hickory Towing

Micky said...

Very informative post.
UFA888

Petersons said...

Thanks for your refreshingly straight talk on this amazing topic .


24 hour garage doors ottawa

Micky said...

Nice blog
ประวัติดาราในตำนาน

Petersons said...

thanks… I’ve been bookmarking them for a while now and just decided to create a post to provide them to others…


Garage Door Cable Repair

Petersons said...

yes definitely this type of blogs are really help for people great job.


GDR Tech Houston

Python said...

Thanks for sharing the information with us.
Data Science Online Training
Python Online Training
Salesforce Online Training

Petersons said...

מאמר מצוין נהניתי מכל רגע של קריאה


נכסים מניבים

Petersons said...

לא כל כך מסכים עם הכל כאן האמת.

התקנת אינטרקום

Petersons said...

Wow, what a blog! I mean, you just have so much guts to go ahead and tell it like it is. Youre what blogging needs, an open minded superhero who isnt afraid to tell it like it is. This is definitely something people need to be up on. Good luck in the future, man


garage door repair cooksville

Parker said...

Nice idea, i am gonna try the same for my web site. thanks.


garage door repair Edmonton

Selenium Training in Pune said...

That's really impressive and helpful information you have given, very valuable content.
we are also into education and you also can take advantage of Selenium Training in Pune

Python said...

Thanks for this wonderful post.
Data Science Online Training
Python Online Training
Salesforce Online Training

Ganesh said...

Amazing..! I was impressed by your great post and Keep doing well...
Microsoft Dynamics CRM Training in Chennai
Node JS Training in Chennai
Node JS Course in Chennai

Huongkv said...

Aivivu chuyên vé máy bay, tham khảo

vé máy bay đi Mỹ bao nhiêu

giá vé máy bay từ mỹ về việt nam

ve may bay gia re di nha trang

vé đà nẵng phú quốc

giá vé máy bay đi Huế

brentmatt04 said...

Towing Burlington, NC

Anonymous said...

Thank you for posting this. Cleveland, TN Appliance Repair

Anonymous said...

Wonderful article. I'll be waiting for more of your post, keep it up Towing Spartanburg, SC

Anonymous said...

Excellent post. I was checking constantly this blog and I am impressed! Extremely helpful information. Landscaping

Anonymous said...

Thank you for sharing your wonderful story with us. You inspired me a lot. Cleveland, TN Appliance Repair

Anonymous said...

Great information comes from great ideas. Very relevant information. Thank you so much! Towing Jacksonville NC

salome said...

thanks for this informative blog
best-angular-training in chennai |

CloudShine said...

Thanks for sharing great post . Keep it post!
Oracle Fusion Training

vivikhapnoi said...

Nice post brother, I have been surfing online more than 3 hours today, yet I never found any interesting article like yours. It is pretty worth enough for me. In my view, if all web owners and bloggers made good content as you did, the internet will be much more useful than ever before. There is certainly a lot to know about this issue.I love all of the points you’ve made. I am sure this post has touched all the internet viewers.
vé máy bay từ texas về việt nam

vé máy bay khứ hồi từ đức về việt nam

ve may bay tu anh ve viet nam

khi nào có chuyến bay từ úc về việt nam

Vé máy bay Vietnam Airline tu Dai Loan ve Viet Nam

giá vé máy bay từ canada về việt nam

vivikhapnoi said...

The article was up to the point and described the information very effectively.
vietnam airlines quốc tế

Có chuyến bay từ Hàn Quốc về Việt Nam không

Cách săn vé máy bay giá rẻ tu Nhat Ban ve Viet Nam

đặt vé máy bay từ singapore về việt nam

Cách săn vé máy bay giá rẻ tu Dai Loan ve Viet Nam

đặt vé máy bay từ canada về việt nam

Reshma said...


Nice blog! Thanks for sharing this valuable information
RPA Training in Bangalore
RPA Training in Pune

manasa said...

I need to thank you for this great read!! I most certainly partook in every single piece of it. I have you book-set apart to check out new stuff you post…

Machine Learning Training in Hyderabad

Unknown said...


A Boutique Management Consulting Outfit Providing Actionable Insights
& Strategic Advisory to Clients
Services

Unknown said...

C-GUARD has a wide range of scrub suits like the basic scrubs that can be used by all Doctors, medical & para medical staff, medical students, interns etc.Cguard

Peter Johnson said...

I am inspired with your post writing style & how continuously you describe this topic...Best MicroNutrients Company in India

eddielydon said...

I am very much pleased with the contents you have mentioned. I wanted to thank you for this great article. Royal Enfield Jacket

George Mark said...

Statistics students and professor are worried to find the deviation calculator because their work depends on it. Billionaire Boys Club Varsity Jacket

Hi Every One said...

Goodness, this is enticing research. I'm cheerful I seen this as and got to section it. extraordinary occupation upon this content material. I leaned toward it bounty. much thanks to you for the inescapable and specific data. Dll File Fixer Full Version

Besthyiptemplate said...

Hi there, I would like to subscribe for this web
site to take hottest updates, thus where can i do it please
help. BUY HYIPS

Beckman said...

Great blog. Thanks for sharing such a useful content. fence-companies-raleigh

Beckman said...

Great blog. Thanks for sharing. It service west palm beach

Beckman said...

Thanks for the blog. fence company milwaukee

designingcourses said...

https://designingcourses.in/graphic-designing-courses-in-bangalore/

Learn graphic designing courses in bangalore we provide best graphic designing training and 100% placement assistance

seodm said...

Python's syntax is straightforward and quick to learn, which emphasise readability and lowers maintenance costs. This object-oriented, high-level, interpreted programming language has dynamic semantics. click here to check our python training in Bangalore!

seodm said...

If you are considering taking up Data Science training in Bangalore!
then it can help you in shaping a successful career. Today, there isn’t any business sector which is untouched by the data scientist. So, be it the eCommerce sector, IT sector or the Medical sector, each sector is embraced with qualified data scientists who help in processing huge amount of data into valuable information

Huongkv said...

đại lý China Airlines tphcm

đổi giờ vé máy bay Japan Airlines

cách mua thêm hành lý ký gửi Eva Air

Priyanka Singh said...
This comment has been removed by the author.
Jasua said...

Quadrobits, a digital advertising corporation, specializes in tailored strategies that include seo and p.C, handing over advertising fulfillment via understanding and clean objectives. Here at social media marketing

Mishra A said...

Great post! Your insights are spot-on. I appreciate how you break down intricate subjects, making them accessible and understandable.
Best Software Training Institute In Electronic City Bangalore