@Scale Conference 2010!

In today's world of internet scale applications, enormous companies and cloud many applications and application developers are struggling provide the needed scale/ha. They've developed their middleware toolkit of application server, hibernate, caching, and scheduling in order to achieve the business goals and single node performance required of them. But now they need to take those applications and frameworks to the next level.

They need scale out!

That's where the @Scale conference can help. In San Francisco at the same time as Oracle Open World/JavaOne attendees of this one day conference will learn from the leaders in cloud and application scale.

• Jeff Barr of Amazon
• Simon Crosby, CTO of Citrix
• Rich Wolski, founder of Eucalyptus
• Adrian Cole, found of JClouds
• Paul Querna of CloudKick and libcloud fame
• Jos Thosman of Canonical--the Ubuntu guys who also do UEC
• Greg Luck - founder of ehcache
• James House - founder of quartz
• Ari Zilka from Terracotta
• Dan Querna, Founder of Cloudkick

Do not miss this opportunity to work with, listen to and learn from the best in the business. The conference is focused on the scale problems you face and real solutions you can apply. Learn about those solutions from the people who created them!

Don't pull your hair out solving problems that are already solved. Come to @Scale 2010!

Signup now and get the early bird prices! Space is limited so signup before it fills up.

What: Sign up here: @Scale conference

Where: San Francisco, SOMA

When: Friday Sept 24th 8am-4pm

<terracotta clustered="true"/>

One challenge with drastically simplifying Terracotta as a tool for application scale is getting the word out that it's occurred!

The subject of this blog is one of two lines of config and a jar on the classpath that move Ehcache to be a coherent distributed cache. The line of config in the subject says, "Hey, cluster this cache." The other line, which looks something like this:

"<terracottaConfig url="localhost:9510" />"

tells Ehcache where to find the Server Array.

This same story is true for scaling out Http Sessions, Quartz Scheduling and Hibernate Caching. A whole new world of scale is available to the applications we all write. It free's us from complex sharding, awkward CAP trade offs and endless DB tuning. And the cherry on top is it's open source.

No recompile, no code changes, no magic tricks or additional knowledge. I could train my dog to scale an application built on these ubiquitous frameworks.

Check out these blogs to learn more:

Add Scale and HA to Ehcache In Less Than 5 Minutes

Add Scale and HA to Quartz Scheduler In Less Than 5 Minutes

Ehcache 2.1 Beta - Lots of Stuff, Still Backward Compatible

UPDATE: 2.1 went GA March 21st

The Ehcache dev team is pretty excited to be getting 2.1 beta out for the world to try. This release is focused on 3 primary goals:

• Build on our vision of an application scale continuum from one node to the cloud.
• Improve Ehcache performance both unclustered and clustered.
• Improve Ehcache applicability for both unclustered and clustered.

What did we do in the name of these goals? Good question!

Scale Continuum

In this release we are focused on taking features that had been added for the Terracotta Clustered version of Ehcache and extending them back to the unclustered version.

The following features fall into that category:

• The Explicit Locking Module - This module allows you to acquire and release locks manually for given keys. It required some significant rework in the unclustered stores but this now works just as well unclustered as it does clustered supporting fully coherent operations.
• JTA - In 2.0 of Ehcache we added JTA support when clustered via Terracotta. In 2.1 we extended that functionality to unclustered and have begun the process of performance tuning to go along with its XA compliance.

Performance

While performance wasn't a big focus of this release (It is for the next release) we were able to considerably improve Ehcache "put" performance due to the significant work we did on the disk store and locking architecture. We'll also be spending some time tuning JTA but that work did not make it into the beta.

Applicability

Here we did some considerable work.

• JTA for Hibernate Second Level Cache - We added support for using Ehcache JTA in a second level cache both clustered and unclustered.
• UnlockedReadsView - This is a subtle but important feature. For those who are using a coherent cache but have some part of an application that needs to be able to read at high rates without impacting the rest of the cache this view is a huge help.
• NonStopCache - Useful for guaranteeing that your cache can never stop your application. On a per cache basis an application can avoid holdups caused by problems such as a slow disk in an unclustered cache or a network outage in a clustered one.
• New Coherent methods - We've added useful methods like putIfAbsent and replace to simply and easily work with a clustered or unclustered cache in a fully coherent manner. Together with the explicit locking wrapper much is possible.
• We also added a bunch of tests and bug fixes to the web-cache, an extremely useful tool for making performant web applications.

Summing up...

While still in beta we feel like Ehcache 2.1 is another exciting step for our product family. It adds performance, significant features and is still backward compatible. It continues our vision of an application scale continuum from one node to the cloud without burdening the developer with complexity. One set of frameworks, one application, scale out is added at runtime. We are really excited about this vision and are working tirelessly at extending it to all our users' needs.

Please Download Ehcache 2.1 Beta, put it through it's paces and give us lots of feedback!

Application Server Instead Of Web Server

I'm starting to see a small but growing group of sites forgoing web servers and going straight to the Application Server in the Java world. This has some obvious advantages:

• Less infrastructure
• Java doesn't have buffer overflows and is a bit more secure
• You can do much more interesting caching by having the web serving and app serving in the same layer

I was excited to see this blog today and just wanted to point people to it and see what thoughts are out there both pro and con on this trend.

Web Application Architecture - do I really need Apache?

How Much Is Google Really Spending?

This will be a short blog.

Has anyone else noticed that google is using adwords extensively to pimp the nexus one? They must be spending a fortune in lost revenue from other ad sources. Since they don't have to pay for those ads I wonder if even they have an accounting of how much they are spending on this thing. At first I thought the nexus one looked interesting but...

T-Mobile only

No Exchange support

weird app space restrictions

and from what I've read poor touch screen keyboard

On the plus side:

Nice screen

Solid camera with flash

really fast processor

With this kind of marketing muscle it's probably just a matter of time before it gets wide adoption but I suspect it's still a couple years away from being really good.

From Performance to Scale - A Software Story... Part I

When building most applications whether it's JEE, Spring, Grails or plain old Tomcat their are a series of common tools that are used. HttpsSessions is used for managing the state of the web portion, ORM for talking to a database, a cache to hold interimediate application state and a scheduler to make things happen at a certain time, place or interval. Out of the box it is quite easy to get started with these components. However, as an application makes it's way into production it often needs improved performance, often followed by HA and then scale-out.

This blog is about a journey. It's about taking an application from performance all the way to scale-out without rewrites and using the components that you already have.

In the beginning it's just the Application

The good news is that whether you're a JBoss person, a Spring person, a Grails person or a Tomcat person your really using the same key components. Almost everyone in these environments are using Hibernate as an ORM, the Http Session Spec, Ehcache for caching/performance and Quartz for scheduling. So while their is no official standard, the global consensus is pretty clear. If it's me I would start from Grails (which is built on Spring, Ehcache, Quartz and Hibernate) but doesn't require the 2 weeks of picking, deeply understanding and piecing together components that is required with most other approaches.

Round 1, Performance

So you've written your single node application using the usual components. Turns out, it's slow. Now what? Well the first thing you need to do is understand why it's slow at a high level. I usually check the following obvious things:

1) CPU bound - Check machines CPU stats on all nodes (including DB), now day you also need to figure out if one of your CPU's is maxed out which could mean the app needs more parallelism

2) GC Bound - Use verbose GC or your favorite tool

3) Database bottleneck - A number of tools and tricks exist to see if your database is the bottleneck. My favorite ways are:

- Keep track of and monitor query times and or look at hibernate stats

- Take thread dumps. If all threads are blocked on DB calls something is wrong

- Check resource Utilization on the Database machine

4) I also like to isolate operations and see which ones are slow or create slowness. This can tell you a lot about where to focus.

Ok, so now you know why things are slow and your in the same boat as about 90 percent of the world. The Database is the number one bottleneck.

Tuning the ORM

Many start by generating the ORM schema using tools. This usually ends up being a completely Normalized database that requires expensive queries. At this stage of development it's almost certain that this is where your performance problems will be. Aside from the usual mistakes like missing JDBC connection pool or bad JDBC Connection pool (Use C3PO and make sure your seeing parallelism. Many of the others are either useless or broken). The next thing to do is to try hibernate 2nd level caching. If your application is read heavy this can have an amazing impact on performance. Entity caching is generally the way to start. The most common gotcha is it getting invalidated by custom SQL. It is a little known fact that if you execute custom sql through hibernate it clears the cache making it useless. If you are going to do Query caching you MUST READ THIS BLOG.

Now you've got hibernate 2nd level caching turned on and you've tuned it. The database isn't the bottleneck anymore but your seeing a lot of GC and thread dumps show contention on Hibernate itself. Especially when displaying medium size to large tables of data on the screen.

Plus, it turns out that updating the database on every intermediate operation that occurs is pounding the databases locks, cpu and disk. It's also made my schema really unwieldy What do I do now?

Stepping up to Caching

If after 2nd level caching you still have performance problems it is usually time to look at application level caching. This can be extremely helpful for performance. If your seeing:

a) A lot of garbage creation around calculating results

b) cpu usage or contention getting information

c) Latency from IO in retrieving information from disk, a web service or a DB

d) Pounding your database with fine grained updates/information with information that you only needed for a short time.

You'll likely want to start caching. Some things to cache include:

1) heavily used reference data from the database - States, zipcodes, product ID's etc

2) Intermediate state - If a series of operations occur but all you care about is the end result. Cache until the end result is reached then put in the DB.

3) heavily reused calculated values - "Like totally searching for Britney Spears."

4) Temporary values that are only needed for a set amount of time - user sessions or data that is needed at a certain time of day.

Hopefully between the Hibernate caching and tuning and the Application data caching you've now got much of your performance issues under control.

Your application probably does some scheduling and uses http sessions but these rarely create performance problems in one node.

Part II - Now that it's performaning what about availability?

Part III - One node isn't enough anymore I need scale-out

Clustered Quartz Scheduler In 5 Minutes

Need a fast clustered/persistent Quartz Job Scheduler? Quartz Scheduler, the ubiquitous job scheduler built into Spring, JBoss and Grails can be configured to provide those features in under 5 minutes in this brief tutorial.

A Brief Digression Into The Why

Why do I need a persistent scaled out job scheduler? The main use cases for a clustered/persistent job scheduler are:

• HA - You need to be able to restart your application without losing scheduled jobs
• Scale Out - Your application now needs more than one node to handle the load it receives and you want your scheduled jobs to distribute across nodes.
• You are using a database to persist and or scale your scheduler and you are seeing DB load/locking issues and or you find it two difficult to setup or maintain.

Steps:

1) Download Terracotta 3.2 (Which includes Quartz Scheduler) http://www.terracotta.org/dl/oss-download-catalog

2) Put the following jars in your class path (all included in the quartz-1.7.0 directory of the Terracotta kit from above):

quartz-1.7.0.jar - regular quartz jar
quartz-terracotta-1.0.0.jar - Terracotta clustered store

3) Whip up some scheduler code:

 public class QuartzSample {

public void startJobs() throws Exception {
  Properties props = new Properties();
  props.load(QuartzSample.class.getClassLoader().getResourceAsStream("org/quartz/quartz.properties"));


  // **** Begin Required TC props
  props.setProperty(StdSchedulerFactory.PROP_JOB_STORE_CLASS,"org.terracotta.quartz.TerracottaJobStore");
  props.setProperty("org.quartz.jobStore.tcConfigUrl", "localhost:9510");
  // *** End Required Terrocotta Stuff


  StdSchedulerFactory factory = new StdSchedulerFactory(props);
  Scheduler scheduler = factory.getScheduler();
  scheduler.start();
  if (scheduler.getJobDetail("myJob", "myGroup") == null) {
    System.out.println("Scheduling Job!");
    JobDetail jobDetail = new JobDetail("myJob", "myGroup", DumbJob.class);
    Trigger trigger = TriggerUtils.makeSecondlyTrigger(5);
    trigger.setName("myTrigger");
    scheduler.scheduleJob(jobDetail, trigger);
  } else {
    System.out.println("Job Already Scheduled!");
  }
}

public static class DumbJob implements Job {

  @Override
  public void execute(JobExecutionContext arg0) throws JobExecutionException {
    System.out.println("Works baby");
  }

}

public static void main(String[] args) throws Exception {
  new QuartzSample().startJobs();
}

** NOTE: Notice the two lines of properties that set things up for clustering with Terracotta in the sample. That's the only difference from single node unclustered Quartz.

4) Start the Terracotta server by running start

./start-tc-server.sh

in the bin directory of the Terracotta kit

5) Run the sample code above and watch it run the job every 5 seconds. Then kill the sample app and restart it. The app will tell you that the job is already scheduled and the job will continue.

Conclusion

Two lines of configuration and a server takes you from ubiquitous job scheduler built into Spring, JBoss and Grails to scale out and persistence.

Have fun!