DSO Guy

New Home...

2011-09-25T20:16:00.001-07:00

Hey readers. Just a heads up that as of today I'm doing my posting over at http://scaleaholic.blogspot.com. Check it!

What Is Terracotta?

2011-08-01T08:09:00.001-07:00

One of the biggest challenges in software is telling people what your software is in a way that helps them make decisions about it. This challenge can often be as difficult as designing and building the software itself. The Terracotta team recently hooked up with the gang from Epipheo studios to create a 2 minute video to do just that. We spent a lot of time thinking about and refining things in order to get a clear succinct message. I think it came out quite well.

Check it out:

Easy Java Performance Tuning: Manage Objects By Size On The Java Runtime Heap

2011-07-11T18:13:00.000-07:00

The Problem?

For those of you who use caching with things like Hibernate, Spring or anything else, you know what a pain performance tuning can be. The tools you have available for tuning, resource management and avoiding OOME's boil down to counts and age controls. These are actually not resource management controls at all. They are data freshness controls and should be treated as such.

How do you even begin to figure out how many entries to allow in each Hibernate cache when you have a hundred of them? What if things change and objects get bigger, or smaller. What if you change your heap size and/or usage patterns?

You've Gotta Try This...

With Ehcache 2.5 Beta1 you can just specify a percentage of heap or a heap size in bytes that your graphs of Java objects are allowed to use. This is accomplished by passing a simple size description into a Cache or CacheManager. All objects held onto by the cache will borrow their space from the cache's or manager's specified pool. Entries will get evicted from the cache as space runs low without any intervention from the developer. This can be done at the Cache Tier level whether it's on heap, disk or BigMemory. It's another way to reduce tuning, improve performance while avoiding OOME's. Learn more here.

If you do things in config or in code it's just a one line change:

A few More Details...

It works on any 1.5 or 1.6 JVM (Tested on Oracle, JRocket and IBM). Doesn't require any object serialization for the on heap management.

Learn More:

http://ehcache.org/documentation/configuration.html#Memory_Based_Cache_Sizing_Ehcache_2.5_and_higher

http://www.terracotta.org/beta

http://dsoguy.blogspot.com/2011/04/biggest-problems-in-local-caching.html

Try it out and give us lots of feedback on the Terracotta Forums:

http://forums.terracotta.org

Exciting Times... Terracotta and Software A.G.

2011-05-22T23:37:00.000-07:00

Exciting times over here at Terracotta.

Read all about it:

http://www.softwareag.com/corporate/company/terracotta.asp

http://terracotta.org/terracotta-joins-software-ag

http://blog.terracottatech.com/2011/05/terracotta_joins_software_ag.html

This is great news for Ehcache, Quartz, and Terracotta customers and communities!

Local Caching++

2011-04-20T08:22:00.000-07:00

Ok, so you've built your application in Java. You've used all the usual tools. Tomcat, Spring, Ehcache, Quartz, etc. Or maybe you went the JRuby, Grails or Scala route. You test your new application or hand it off to run in production and it's too slow. This is just a single node application at this point. It services 20-100 users. It's churning and burning the database, creating and recreating the same Web Pages, Users and other relevant data. You want to start caching locally to solve your latency and throughput problems. Then, upon a more detailed look at your application, you get scared.

You find your application has:

40 DB tables in Hibernate that can be cached
A web cache
A user session cache

Then it hits you. Caching is easy but cache tuning is hard.

What Makes Cache Tuning Hard?

In conversations with 100's of cache users there are actually a small handful of difficult to work through challenges applying caching to an application:

Hibernate/Lots of caches - When using Hibernate you often end up with as many as 100 tables in your DB. How do you balance a fixed amount of resources(Heap/BigMemory) across 100 caches?
Indirect knobs/Bytes vs Count/TTL - In local Java caching the control points are almost always measured in number of entries and time to live. But wait a minute! When I start the JVM I don't say how many objects the heap can hold and for how long. I say how many bytes of memory the heap can use?
Who Tunes and When? - At some companies the desire is to have the "Application Administrator" do the tuning. At others it's the "Developer." They have different understandings of the application. The developer can tune by knowledge of the application. The app admin can only tune based on what's happening when the application is running.

These are the challenges we are working to solve in the next version of Ehcache. While it's early days on the dev side we would love feedback on our approaches. You can get a sense of how it's going to work from the doc on ehcache.org.

What We Are Building

Greg, me and the dev team spent a bunch of time pondering the above problems over the last few years. We felt that with two key improvements to how people tune we could address most of the above (and a few more items hit the rest):

Tune from the top - Define max resource usage for the whole cache manager and then optionally define it for the individual caches underneath it as needed. So if you have a hundred caches you can start with, "Give these 100 caches N amounts of Heap/OffHeap." Then monitor and see if any specific caches need special attention.
Tune the constrained resource, Bytes - TTL is a cache freshness concern not a resource management concern. Max entry count does not directly map to available heap resources. So we are adding "bytes" based tuning. This eliminates the mistake prone process of trying to control resources by TTL/TTI/count and hope you get it right. Instead you say, "I want to allow caching to use 30 percent of heap." We take it from there.

Wrapping Up

With those two key improvements a developer or admin is now directly turning the knob (Size of cache in bytes) that maps to the resources available in the JVM and doing it at a global level or a local level as needed to avoid hard to tune individual cache constraints.

This will work with all of our JVM level cache tiers (onHeap, BigMemory, Disk).

When you put those features together with other items coming in the next major release like entry and cache pinning and a snapshotting bootstrapper for warming we feel like this will be a very powerful release.

Please check out the new docs and give us feedback by commenting on this blog or posting the the Ehcache forums.

Help us make Ehcache as easy to use and powerful as we possibly can.

http://ehcache.org/documentation/configuration.html#Memory_Based_Cache_Sizing_Ehcache_2.5_and_higher

Where To Buy Your Apple Gear?

2011-04-16T08:20:00.001-07:00

Most people who know me think I'm a bit of an Apple products junkie. I can't deny it. I'm a big fan of Apple products (Typing this blog on my 11.6 inch Macbook Air) and the way they package and reuse great ideas across devices. Being that I'm such a fan(boy) of the Apple ecosystem I'm regularly asked for advice from wood be purchasers when they want to pickup an Apple product. I'm writing this blog in order to provide that product purchase advice for my friends and family as well as for my readers who can't ask me verbally.

"Speed of Acquisition" vs "Help During Purchase" vs "Total Cost"

The decision making process for where to buy Apple products usually comes down to three major questions. How do you rate "Speed of Acquisition" vs "Help During Purchasing" vs "Total Cost?" (See diagram for the big picture on this issue). Once you know the answer to the above question based on your keen sense of self awareness you can then weight the various purchase options appropriately. For best results, Be Honest With Yourself!

NOTE: Many people think after sale help is somehow related to the purchase decision. IT IS NOT! The Apple store Genius bar will always help you no matter where you bought the device.

What's In The Cost?

When buying an Apple product there are 4 components to the cost of the device:

The price tag of the device itself
State sales tax
Shipping and Handling
What you can get thrown into the deal

Take into consideration all four points when looking at the price. You can often save over a hundred dollars in sales tax just by ordering from an out of state, online retailer! In addition, many of these online retailers ship for free. These online retailers usually have the best discounts aka "price tag" on devices as well. Check out the AppleInsider.com price guide. It will tell you where to get the best deal on any given Apple device and is kept very up to date. It's where I look and where you should too. Another thing to consider is that on some items MacMall and MacConnection are actually flexible and may throw in things like printers and bags if you call.

The Need For Speed?

Some people are impatient, some people need a computer right away, and others just like the experience of being first. These are all speed questions. In the speed world there are two categories of Apple products "Hot and Constrained" and "Generally Available."

Hot and Constrained

If we are talking about "Hot and Constrained" usually the fastest way to get a product is buying it from Apple. Either through their online store or going to an Apple Retail location. They stock themselves first. You may have to wait on lines but it's often your best and sometimes your only bet for acquiring new, hot, Apple devices.

General Availability

For "general availability" devices "fastest" is broken down again into two categories. Fastest online (Generally Amazon) and fastest brick and mortar (Generally Apple's stores). Amazon is generally the best place to buy anything online as far as fulfillment (How fast they get it to you, amount in stock, ease of using store, return policies) and Apple products are no exception. Amazon's prices aren't the absolute bottom but they are pretty darn good.

When buying from a retail store, Apple's is second to none. They get you in and out fast, have all the information you need, it's easy to find what you want and they have ton's of stock of everything Apple. The only downside is the sales tax and no discounts to speak of.

Help During Purchasing

This one doesn't apply to me as when it comes to Apple products I tend to know what I want. It's a hobby of mine to monitor Apple's products and product direction. I use product and business learning from Apple for inspiration in my job building Performance and Scale software (Probably a good blog topic as well. "How I Apply My Apple Learnings To My Business"). For the rest of the world, aka "normal people," who spend precious free time doing things like traveling, dating, hanging out with friends, help might be required. For those people you should probably lean towards either the Retail Apple Stores and or the online Apple Product specialists (MacMall and MacConnection) where they know everything about the devices and can help you make good decisions. For those who are like me, stick to "Total Cost" and "Speed" in your decision making process.

Conclusions

When looking to buy an Apple device one must evaluate a number of options. You should evaluate those options against a keen sense of self awareness? Can you wait, are you cost conscious, do you need help? By answering those questions and comparing the results against the "Apple Product Purchasing Guide" diagram below, the AppleInsider price guide and the above criteria you should be able to make the best possible decision for you.

Please let me know if you have any feedback about this blog. How can it be more helpful? Where did I get it wrong?

Please Strengthen My Weak LinkedIn Links...

2011-04-13T13:26:00.001-07:00

State Of LinkedIn

I've had a LinkedIn account for many years. I find it an excellent tool for keeping tabs on ex-coworkers, recruiting new ones, and watching job trends. The problem I have is that those links/connections in LinkedIn can be weak. It doesn't take those connections to the next level by helping me manage the relationships. It doesn't help me manage and monitor the strengthen of my connections.

What Does Manage and Monitor Relationships Mean?

At a macro level I can keep track of my first level connections and second level connections. I can traverse these to help me find people to hire, ask for advice and even look for jobs myself. It even does a ton of stuff to help me find new connections by making suggestions, forming groups etc. But at a micro level it does nothing to help me improve and make value judgements about those relationships. I.e. I have a relationship with Joe the CEO of SomeCompany.com. By looking at that link I can't tell if that's a strong relationship or a weak one (Monitoring). I also can't set goals of improving my relationships and keep track of those goals (Managing). It just gives me a tree/graph of the relationships as if they were all the same. Further, when I'm traversing through my relationships and into my connections relationships I can't tell how strong they are connected to their connections.

What I Want...

What I really want is for LinkedIn to help me manage and improve my relationships as well. It would all be much more powerful if I could apply a rating to each relationship, monitor the relationship's progress and assign goals for each relationship. I could do so on a 1-10 scale.

So instead of a graph like this:

I could have a graph like this where the thickness of the lines indicated the quality of the relationship where it is right now:

That would be a great start. Then I would like it to keep a history of the relationship as well. It could show me how the relationship is progressing over time. Say it started at a 10 but I haven't talked to Joe in 3 years and now I rate it a 4. It could represent it with little arrows or color showing deterioration.

What about goals for a relationship? I could set a goal of 8 and do a query on all relationships that are not at the level I wish they were. It could remind me to make a quick contact with the person in question. It could auto deteriorate a relationship over time if I don't contact people.

Summing Up...

LinkedIn is great. I'm more connected than ever. But... I want to monitor and manage my links to improve my connectedness to certain people and I'm hoping linked in can help me.

"What" "When" and "Where" ... Quartz Scheduler 2.0 Goes GA

2011-03-24T17:02:00.000-07:00

What is Quartz?

Quartz is The Lightweight, Open Source, Enterprise-class Job Scheduler for Java. For years Quartz has put the "When" in Java applications via it's full featured scheduling capabilities. As a result, from Spring to JBoss, Quartz is embedded in just about every major Java product. Quartz is extremely robust and full featured providing things like HA, Transaction Support and all the precise guarantees one needs to assure reliable Job execution.

Terracotta has done a number of incremental improvements since taking over stewardship of Quartz. We've been evolving Quartz one step at a time while collecting what features the Quartz community were really interested in.

Quartz 2.0 is the realization of those user requested features!

"What" "When" and "Where"

Let's step back a bit and review where Quartz fits in the Terracotta world view. As most people know Terracotta is focused on adding Snap-in performance to JVM based applications through Scale-out (Terracotta Server Array), Scale-up (BigMemory) and Speed-up (Ehcache). We do this while maintaining our goals of simplicity, predictability, and density (can your datastore hold 2 billion entries per node in-memory?) throughout our product line. We view this data layer scaling layer as solving the "What" or the data problem of an application. But when going from 1 to many nodes that's just the first of two major scale points.

Up until now, Quartz Scheduler has been solely focused on the "When" part of code execution. Making sure "code execution" happens exactly "When" it's supposed. It has been pretty much ubiquitous in that space with literally 100s of thousands of users. While it Scaled-out with JDBC and Terracotta, it gave precious little control over where jobs execute.

With Quartz Scheduler 2.0 we've now added that "Where." This is about where code executes. Instead of randomly choosing a node for job execution in a scaled-out architecture you can now create constraints that help make the decision based on "Where" it should be executed. You can do this based on:

Resource Availability - CPU available, Memory Available, custom constraints
Ehcache's data locality - Bring the work to where the data is
Static allocation - Just decide where it goes

This gives Terracotta the ability to Snap-in and handle both the major scale-out points.

What's new in Quartz 2.0

The simple answer is A LOT!

Easy to use Fluent API - Quartz 2.0 has a new, easy to use fluent interface that hides the complexity of building out the description of your jobs behind a simple description of what you want to happen and when. I wrote a short blog about this when it was in beta.
Quartz "Where" - Constraint based system for controlling where jobs execute based on things like CPU and Memory usage, OS, and Ehcache data locality
Quartz Manager - A flash based GUI console for managing and monitoring your scheduler in production.
Batching - Helps improve a schedulers throughput by allowing one to make trade-offs between perfect time execution and benefiting from batching.
Ton's of bug fixes and features - Lots of long requested features. Check out the link for the list.

Quartz 2.0 Is Now GA

Quartz 2.0 makes a big leap in usability, power, visibility and management and it's NOW GA. We did this while maintaining Quartz's reliability and predictability. We focused on making sure existing users would have an easy time moving forward so check out the migration guide if you need help. It really is the next generation of Quartz. Give it a try and give us feedback as we are constantly working to make things better!

More Reading

Quick 5:41 Intro To Ehcache Search (Now GA)

2011-02-14T21:02:00.000-08:00

Ehcache 2.4 went GA today. Still lightweight (under 1MB) and still backward compatible to 1.x so no reason not to give it a try. Below are the highlights and a short video on getting started with Ehcache Search:

Search - Brand new search API. Allows one to get beyond the key based lookup of objects (Check out this sample)
Local Transactions - Fast optimistic concurrency without the need for a TransactionManager (Check out this sample)
Bigger BigMemory (ee) - 2 Billion entries, 1.3 million TPS, Extreme predictability for meeting SLA's
Bigger Disk Store (ee) - Swap your Ehcache to disk. Grow to hundreds of gigs with no on heap footprint

In the past I've enjoyed short videos that teach me something. This is my first try at doing one to benefit others. I've created this short 5 minute 41 second video to get people started on using Ehcache Search...

Here's the Slides...

Ehcache At 2 Billion...

2011-01-26T14:19:00.000-08:00

What's Up With Ehcache 2.4

Ehcache is the de facto caching standard for Java that everyone uses (500,000+ production deployments; the majority of enterprise Java applications). Ehcache 2.4 is coming out soon and includes some capabilities that will make it even easier to use, more powerful, while still maintaining it's light weight.

The highlights include:

Search - Quickly find entries based on the criteria of your choosing. String matching, dates, ranges, sums, averages etc.
Fast local transactions - Improved performance of JTA and added a new non-jta transaction api for user level control
Even more capacity and performance

What I've been Testing

I've written before about BigMemory for Enterprise Ehcache and how it solves the problem of long, unpredictable GC pauses in Java. The first release of BigMemory was… well, big. In Enterprise Ehcache 2.4, BigMemory has gotten even bigger.

Using the Enterprise Ehcache Big Memory Pounder I was able to show that Enterprise Ehcache 2.4 now easily handles:

Entry Count: > 2 billion entries (I reached 2 billion on the hardware I had; with bigger hardware, I could probably have gone much higher).
Throughput: 1.3 million operations per second (symmetric read and write; CPU bound)
SLA/Predictability: No GC pauses and a predictable 38-42 ops/thread/millisecond throughout the test
Data Size: 1-350 GB in-memory cache (again, I was limited by the hardware I had; with more RAM, I could probably have gone much higher)
Flexible Efficient Entry Sizes: The cache can now dynamically handle very large (10-100 MB) and very small entries (just a few bytes) together more efficiently with no tuning (This test used small entries in order to fit as many entries as possible into the memory I had. I also ran tests with fewer entries in order to validate wide ranging sizes)
Tuning: All tests were done with NO TUNING. Right out of the box.

Here's the hardware and software stack I used for my testing:

Cisco UCS C250 Server
Dual Intel x5670 2.93 Ghz CPU
384 GB RAM ( 8 GB x 48)
Redhat 5.4 Enterprise Edition
Sun JDK 1.6_22

For this test, all of the data was in memory.

A Bit About Ehcache BigMemory

BigMemory is 100% pure Java and in process with a Java application. No magic or special JVMs (works on IBM and JRocket as well). The cache data is safely hidden away from Java GC and the pauses that occur with large heaps by instead storing data in a BigMemory off-heap store.

Embedding

BigMemory got it's start as a component in the Terracotta Server Array and as a result it is particularly useful for embedding. It's performance characteristics and no tuning approach improves "The Out Of The Box Experience" and saves money on support by removing tuning required by users and problems caused by GC pauses.

You may be thinking...

"I don't have 2 billion entries in my caches?"

That's ok. Ehcache is a lightweight core library (under 1MB) for caching that's ubiquitous and easy to use. When it's needed, Ehcache lets you scale up and out to billions of entries and terabytes of data. It does so at a manageable server density without changing code/architecture and without a bunch of tuning and learning. This protects not only your knowledge investment but your code investment.

More about BigMemory for Enterprise Ehcache:

http://terracotta.org/bigmemory

BigMemory Whitepaper

BigMemory Documentation

More about the 2010 Ehcache user survey:

Ehcache User Survey Whitepaper

Ehcache To The Rescue (Comic Strip)

2010-12-15T09:05:00.000-08:00

Working on a product that has as much adoption as Ehcache is extremely rewarding. Every day millions of people's applications run faster with this tiny little library. It's one of those, "Of course I use it, doesn't everyone " kinds of products and what Greg Luck has accomplished with it is really amazing. One of the challenges we face as we improve and extend it's functionality is getting the story out to those million or so users. Yes, it's still an extremely fast extremely light weight library that pretty much everyone just uses. But it also future proofs your app with snap-in scale-up, scale-out and HA each with just 2 lines of configuration.

In the coming release we are adding Search, Local Transactions and an even better BigMemory (We've been regularly testing with 2 billion entries in memory with linear, predictable, fast, performance but that's another blog) and we want people to know about this stuff, try it, and give feedback so we can constantly make it better.

Anyway, after watching an inspiring talk from Edward Tufte on visual representation I've been experimenting with ideas on how to teach people about Ehcache in quick, fun ways. In the process I wrote my very first comic strip. Check it out...

Ehcache To The Rescue Comic Strip...

Quartz Scheduler 2.0 Beta 1 Welcomes New Fluent API and "Where"

2010-11-29T15:26:00.001-08:00

Quartz Scheduler, the most widely used Java scheduler is getting some major improvements for 2.0. I'm going to talk a little about what to expect but you can also read the release notes here.

Goals

We had two major goals when planning for Quartz 2.0 began.

Simplify/modernize the Quartz API.
Improve the Quartz experience when leveraging a cluster

Quart 2.0 API

In order to improve the usability and readability of Quartz 2.0 over passed versions we spent a lot of time evaluating the parts and how a user interacts with them. We found that too much knowledge of those parts needed to be understood at construction time. As a result we moved to a "Fluent API/DSL" approach. The best way to get a feel for the kind of improvement this gives is to compare the samples from 1.8.4 to the same ones translated in 2.0 Beta 1.

In the simple of case of example 1 from the Quartz Kit you can see the basic philosophy change:

Improvements include:

The date/time related methods have been moved off of the Trigger and Job classes into a Date building class called "DateBuilder"
We've removed the need to know details about which Job and Trigger classes you need and instead infer them through the building methods you call.
The construction now reads more like a sentence. new job withIdentity "job1", "group1". new trigger withIdentity "trigger1", "group1" start at runTime

This is pretty subtle in a simple case like example 1 but gets more obvious as the cases get more complex. Let's look at example 2.

In this case notice how the constructors are growing with no real indication to what each parameter means.

Now lets look at an example where you have to choose a specific Trigger type.

This example shows how in Quartz 1.8.4 you would have to know to select a different Trigger type but in 2.0 it's just abstracted away in the scheduling.

It's worth going through the samples yourself and making any suggestions you may have. Still plenty of time to get your API suggestions in via the Quartz Forum!

Quartz "Where" And Other Improvements

Clustered Quartz has gotten two major improvements in 2.0. Improved performance as node count increases and Quartz "Where"

I wrote a blog about this a while back but now you can play with it. Alex Snaps has written an excellent blog with code samples on the topic. Check it out to really dig in.

Ehcache 2.4 Beta 1 Welcomes Search, Local Transactions and more...

2010-11-29T10:44:00.000-08:00

Ehcache 2.4 Beta has been released. We've been spending the last few years working hard to make Ehcache the best possible solution for users with caching needs. 2.4 beta 1 is another big step forward. The beta includes some significant new features and improvements that can be read about here. This blog is a few notes on getting started with those features and a few comments on why they are important.

Ehcache Search

This is one of the most requested features for Ehcache. I can't count how many times that I have had a map or cache and just wanted to quickly and easily search for an entry. Sure, one can write an iterator and use Java's matchers but that's

A bit of annoying coding
Only practical for unclustered caches

Ehcache 2.4 Beta 1 has a new search interface in core Ehcache that works throughout our scale continuum. We used the "Fluent Interface/DSL" style that we've been moving towards over the last couple of years in both Ehcache and Quartz Scheduler. This style of interface makes an API easy to use and read like english.

To learn more about it check out this self contained sample app and the documenation. Give us feedback early and often on the Ehcache Forums as the API is still under development and can still be improved.

Local Transactions

Back in Ehcache 2.0 we added JTA support. This was helpful but we found that for some use-cases people wanted a few things.

Transactions without a JTA transaction manager
More speed

We improved the speed across the board of our XA compliant JTA stuff and we also added the concept of local transactions. Local transactions provide optimistic concurrency and atomicity across caches. They are like XA transactions but aren't completely tolerant of external resource crashes outside of Ehcache (i.e. cache + databases don't have all the guarantees of XA in the node failure cases. Does provide the guarantees across Ehcaches), don't require a transaction coordinator (though can be used with them) and tend to be less resource intensive (read faster).

I've whipped up a little self contained sample here.

Improved Enterprise Disk Store

In 2.3 we added an Enterprise version of unclustered Ehcache core. The version added support for very large caches through the seamless BigMemory add-on (1 line of config). We also, somewhat more quietly, added the beginnings of a new Enterprise disk store that no longer stored keys in memory and can handle larger restartable caches.

2.4 beta 1 takes this to another level. The disk store is now way more efficient, better leverages SSD drives and can efficiently grow to very large sizes without any additional heap usage or fragmenting. One can exercise the improvements using the Ehcache Pounder.

Other Improvements

While not everything for the final release was done in time for Beta 1 some significant improvements are in.

NonStopCache now built in. Rather than have to add a jar and configure a wrapper to get the non-stop characteristics in clustered land this is now built into the product core and be turned on via configuration
Search now works clustered - The new search API is backed by the Terracotta tier. This is still early and we have a lot of performance and HA work to do here. That said, it is testable and usable so give it a try.
Explicit locking module is now in the core kit
Rejoin now works in non-stop (You can disconnect from a cluster and reconnect to that cluster without restarting)

For those interested in scheduling we have also released Quartz Scheduler 2.0 beta 1. More stuff will be coming in a month or so when we do beta 2 for all products.

Direct Buffer Access Is Slow, Really?

2010-11-15T09:46:00.000-08:00

Lots of claims in the blog sphere around direct memory buffers being slow. We have been working with them a lot and hadn't seen that slowness but I'm more of a try it and see kinda guy so I did just that.

I cooked up a quick single threaded test to compare offheap direct memory buffers to on heap byte buffers. What I found is that at least on my notebook direct is between 2-5 percent of onheap. Even on my notebook I was writing and reading 2G/second in small chunks to random parts of the byte buffer (Every operation on the test does one write and one read).

The nice thing about them is they occupy no heap so the data stored in them is hidden away from the JVM GC. Of course whether that 2 to 5 percent matters depends on how much data your app is trying to crank through them so as always you need to look at your use-case's, latency, through-put and SLA goals and code accordingly.

While more testing can and should(and has) be done, here is a place for people to start.

Here's the code and my results from my 1.6 ghz notebook. I only spent a few minutes on it so suggestions to improve it are welcome:

Type: ONHEAP Took: 8978 to write and read: 10737418368

Type: DIRECT Took: 9223 to write and read: 10737418368

Type: ONHEAP Took: 8827 to write and read: 10737418368

Type: DIRECT Took: 9283 to write and read: 10737418368

Type: ONHEAP Took: 8813 to write and read: 10737418368

Type: DIRECT Took: 9604 to write and read: 10737418368

https://gist.github.com/700690

A Couple Minutes With Ehcache Search...

2010-11-05T12:30:00.000-07:00

Welcome To Ehcache Search

Now that we are releasing our, GC Busting, Ehache BigMemory product we are starting the process of gathering feedback on the new Ehcache Search API. While Ehcache Big Memory solves your Latency, through-put and SLA issues, Ehcache Search gives you the ability to handle a users data retrieval needs in a simple and efficient way.

What is it?

With todays data explosion and hardware memory size explosion, in-memory and caching have become an important part of everyone's software. Now, getting at the data in that cache in a flexible, simple and fast way has really become an important area of focus. Things like finding entries that match criteria, performing aggregations and calculations (eg average, sum, max, min) need to be easy and efficient.

That's where the new Ehcache Search API comes in. It will provide fast efficient searches and aggregations across the Ehcache Snap in, Speed up, Scale out continuum (Single node performance through scaled out architecture).

Please try out the API (It's checked into Ehache core's trunk) by grabbing the this self contained, buildable sample and open source Ehcache nightly jar I've put up on GitHub.

https://github.com/sharrissf/Ehcache-Search-Sample/downloads

It's simple to use and 100 percent open source. We are very excited about feedback and plenty of time still exists for making changes and improvements. Please ask lots of questions and give lots of feedback.

To try it you can just unpack the kit:

tar -xzvf ehcache-search-sample-0.0.1-SNAPSHOT-distribution.tar.gz

and run it:

sh run.sh

The source code is included in the kit and can be edited in this directory:

src/main/java/org/sharrissf/sample/

rebuilt if you have Maven.

mvn clean install

And rerun using the run.sh command.

You can even package it up yourself by using

mvn clean assembly:assembly

Looking forward to hearing from the community. It will be interesting to see how projects like Grails, Hibernate, Liferay and Cold Fusion leverage cache search. Those uses will help set direction for years to come.

Strange Talk On Ehcache BigMemory

2010-10-08T21:10:00.000-07:00

I'll be giving a talk at Strange Loop On "The Life And Times Of BigMemory." I'll put a link to the slides and the recording of the talk here when it's done. It's an excellent conference and I'm really looking forward to speaking there.

A Couple Minutes With Ehcache BigMemory Pounder...

2010-10-06T17:26:00.000-07:00

Introduction

UPDATE: This product was released (No longer in beta)

Ehcache with BigMemory is now out in beta. One of the challenges when confronted with a new technology is exercising and understanding it's characteristics. Sure, you can read the docs, Google for some blogs, integrate it with your application and maybe write some samples, but that's a lot of work. Plus those approaches may not give you a clear picture of the various characteristics of the software. In order to make things a bit easier I've released a configurable pounder application for Standalone Ehcache with BigMemory.

Getting Installed

Here are the steps to get started:

Get the Ehcache with BigMemory Beta and a license key to use it.
Get the Standalone Ehcache Pounder distribution
Unpack the Ehcache with BigMemory distribution

tar -xzvf ehcache-core-ee-2.3-distribution.tar.gz

Copy the Standalone Ehcache Pounder kit into the ehcache kit and unpack it

cd ehcache-core-ee-2.3

cp ../ehcache-pounder-0.0.5-SNAPSHOT-distribution.tar.gz .

tar -xzvf ehcache-pounder-0.0.5-SNAPSHOT-distribution.tar.gz

Copy your license file and your ehcache core jar into the pounder kit

cd ehcache-pounder-0.0.5-SNAPSHOT

cp ../lib/ehcache-core-ee-2.3.jar .

and copy terracotta-license.key to the ehcache-pounder-0.0.5-SNAPSHOT as well.

Running the Pounder

Now you are ready to go. First take a look at the start script in one of the template directories (i.e. templates/1G-BigMemory):

sh run-pounder.sh

Out of the box it looks like this:

java -verbose:gc -Xms200m -Xmx200m -XX:+UseCompressedOops -XX:MaxDirectMemorySize=64G -cp ".:./jyaml-1.3.jar:./ehcache-pounder-0.0.5-SNAPSHOT.jar:./ehcache-core-ee-2.3.jar:slf4j-api-1.5.11.jar:slf4j-jdk14-1.5.11.jar" org.sharrissf.ehcache.tools.EhcachePounder

This should work fine for most people. It uses a small heap of 200 meg (you may need to grow this for really big caches). It is fine to have a MaxDirectMemorySize that is larger than your memory size on your machine. Just don't have a maxOffHeap size in your config.yml that is greater than your available physical memory on your machine. Also, make sure you leave room for your OS and the JVM. For example. If you have 8G of physical memory you might do a 400m java heap, a 6G offheap and leave the rest for the OS to use.

The script defaults to verbose gc because it's useful to see those stats and compressed oops BECAUSE YOU SHOULD BE USING A 64 BIT JVM and this setting makes it much more efficient (closer to 32 bit object pointers when possible using much less heap).

I tend to run this script like this:

./run-pounder.sh | tee out.txt

That way all output of the test goes to both the screen and to the file out.txt

Configuring the Pounder

You configure the pounder using the config.yml file. You can learn about all the options in the README. Here is the sample config included in the kit:

storeType: OFFHEAP

threadCount: 33

entryCount: 1000000

offHeapSize: "1G"

maxOnHeapCount: 5000

batchCount: 50000

maxValueSize: 800

minValueSize: 200

hotSetPercentage: 99

rounds: 40

updatePercentage: 10

diskStorePath: /export1/dev

NOTE: The most important setting here is the offHeapSize. If you set this to a number greater than the amount of memory you have on your machine you will not be happy.

The offHeapSize + the heap size + the amount of memory your OS needs together must be less than the amount of physical available memory on the machine.

Output

When running the output will look like the following:

size:492120 time: 7434 Max batch time millis: warmup value size:796 READ: 0 WRITE: 15151 Hotset: 99

size = total size of cache

time = total time to execute batch

max batch time = either warm up for the load phase or the longest time it took to execute a batch (note that in a multi-threaded test on a cpu bound machine the batch times are going to be impacted.)

value size = size of the value in this batch

READ = number of reads in the batch

WRITE = number of writes in the batch

hotset = percentage of the time that reads are done from the on heap cache

At the end of each round you'll see something like this:

Took: 10899 final size was 995240 TPS: 91751 MAX GET TIME: 20

Took = total time the round took

final size = the total size of the cache at the end (this can be impacted by eviction and be less than what you loaded)

TPS = total TPS during the run

MAX GET TIME = The maximum amount of time it took to get an entry

An interesting design choice of this pounder is that the threads you define do BOTH reading and writing. So the writers can starve out the readers in a hotset test. I could have gone he other way and then the readers could starve out the writers giving overly generous TPS.

What's Next

The biggest gap right now is that I didn't implement a way to specify a rate. It always goes full throttle. I'll try to add that when I get some time if people think it's useful.

Please give me lots of feedback so we can improve both the pounder and Ehcache

Learn more at:

The source for the pounder

http://terracotta.org/bigm emory

My other blog on BigMemory
Check the Ehcache BigMemory docs

Server Array BigMemory FAQ

A Little Bit About BigMemory for Ehcache and Terracotta ...

2010-09-15T00:18:00.000-07:00

Big Memory?

In talking to our users it is clear that applications are getting more and more data hungry. According to IDC, data requirements are growing at an annual rate of 60 percent. This trend is driven further by cloud computing platforms, company consolidation and huge application platforms like Facebook. There is good news though. Server class machines purchased this year have a minimum of 8 Gig of RAM and likely have 32 Gig. Cisco is now selling mainstream UCS boxes with over 380 Gig of RAM (which I have tried and is amazing). On EC2 you can borrow 68.4 Gig machines for 2 dollars an hour (I have also tried this and it is also pretty amazing). Memory has gotten big and extremely cheap compared to things like developer time and user satisfaction.

Unfortunately a problem exists as well. For Java/JVM applications it is becoming an ever increasing challenge to use all that data and memory. At the same time that the data / memory explosion is occurring the amount of heap a Java process can effectively use has stayed largely unchanged. This is due to the ever increasing Garbage Collection pauses that occur as a Java heap gets large. We see this issue at our customers but we also see here at Terracotta tuning our products and the products we use like third party app servers, bug tracking systems CMS's and the like. How many times have you heard "run lots of JVM's" or "don't grow the heap" from your vendor's and/or devs?

So we set out to first identify the problem as it exists today, both in the wild and in-house. We then created a solution, first for us (an internal customer) and then for all of the millions of nodes of Ehcache out there (all of you)

3 Big Problems Seen by Java Applications

My Application is too slow

My application can't keep up with my users. I've got 10's of gigs of data in my database but it's over loaded and or too slow to service my needs. Either due to the complicated nature of my queriers or the volume of those queries. I want my data closer to the application so of course I start caching. Caching helps, but I want to cache more. My machine has 16 gigs of RAM but if I grow my heap that large, I get too many Java GC pauses.

My Application's latencies aren't predictable

On average my Java application is plenty fast but I see pauses that are unacceptable to my users. I can't meet my SLA's due to the size of my heap combined with Java GC pauses.

My software/deployment is too complicated

I've solved the Java GC problem. I run with many JVM's with heap sizes of 1-2 gigs. I partition my data and or loadbalance to get the performance and availability I need but my setup is complicated to manage because I need so many JVM's and I need to make sure the right data is in the right places. I fill up all 64 Gig's of RAM on my machine but it's too hard and fragile.

The other problem

Like many vendors, in the past we told our users to keep the heaps down under 6 gig. This forced our customers to not completely leverage the memory and or cpu on the machines they purchased and or stack JVM's on a machine. The prior is expensive and inefficient and the latter fragile and complex.

Here is a quick picture of what people do with their Java Applications today:

Base Case - Small heap JVM on a big machine because GC pauses are a problem
Big heap - That has long GC's that are complicated to manage
Stacked small JVM heaps - This in combination with various sharding, load balancing and clustering techniques is often used. This is complicated to manage and if all the nodes GC at the same time this can lead to availability problems.

What kind of solution would help?

Here's what we believe are the requirements for a stand-alone caching solution that attacks the above problems.

Hold a large dataset in memory without impacting GC (10s-100s of Gig) - The more data that is cached the less you have to go to your external data source and or disk the faster the app goes
Be Fast - needs to meet the SLA
Stay Fast - Don't fragment, don't slowdown as the data is changed over time
Concurrent - Scales with cpu and threads. No lock contention
Predictable - can't have pauses if I want to make my SLA
Needs to be 100 percent Java, work on your JVM on your OS
Restartable - A big cache like this needs to be restartable because it takes too long to build
Should just Snap-in and work - not a lot of complexity

What have we built?

First we built a core piece of technology, BigMemory, an off-heap, direct memory buffer store, with a highly optimized memory manager that meets and or exceeds requirements 1-6 above. This piece of technology is currently being applied in two ways:

1) Terracotta Server Array - We sold it to our built-in customer, the Terracotta Server Team, who can now create individual nodes of our L2 caches that can hold a hundred million entries, leverage 10's of gigs of memory, pause free and with linear TPS. This leverages entire machines (even big ones) with a single JVM for higher availability, a simpler deployment model, 8x improved density and rock steady latencies.

2) Ehcache - We've added BigMemory and a new disk store to Enterprise Ehcache to create a new tiered store adding in requirements 7-8 from above (snap-in simplicity and restart-ability). The Ehcache world at large can benefit from this store just as much as the Terracotta products do.

Check out the diagram below.

Typically, using either of the BigMemory backed products, you shrink your heap and grow your cache. By doing so SLA's are easier to meet because GC pauses pretty much go away and you are able to keep a huge chunk of data in memory.

Summing up

Memory is cheap and growing. Data is important and growing just as fast. Java's GC pauses are preventing applications from keeping up with your hardware. So do what every other layer of your software and hardware stack does: cache. But in Java, the large heaps needed to hold your cache can hurt performance due to GC pauses. So use a tiered cache with BigMemory that leverages your whole machine and keeps your data as close to where it is needed as possible. That's what Terracotta is doing for it's products. Do so simply, i.e. snap it in to Ehcache and have large caches without the pauses caused by GC. As a result create a simpler architecture with improved performance/density and better SLA's.

Learn more at http://terracotta.org/bigmemory
Check the Ehcache BigMemory docs

Server Array BigMemory FAQ

Application Scale and Quartz "Where"

2010-07-28T07:29:00.000-07:00

I don't think it is to controversial to say that Quartz Scheduler is by far the dominant Open-Source Java job scheduler. It provides fast, flexible, and extremely reliable job execution and it is embedded in just about everything out there. In the last 6 months or so we have added a new Terracotta backed version of Quartz, lots of bug fixes and a GUI for monitoring quartz when clustered with Terracotta. We have a lot more coming. The team is hard at work on what's next. My favorite feature on the Quartz 2.0 list, the one they are working on right now, is Job Location Control or what is code named "Where".

Some of the trends we are seeing in our user base include the leveraging of EC2, larger apps/user bases, demanding HA requirements and hardware farms. As a result, scaled-out architectures are becoming common place in many IT environments. Scheduling work in these large multi-node environments is becoming a part of many software developers lives. Currently, Quartz supports scale-out but with little control over how the work is distributed. While this is a good start one quickly runs into problems like assigning jobs to machines that have the processing power to the work on them at the time the job is fired. Or execute the job where the data is local. Or just perform certain jobs on certain classes of machines do to their location or by purpose.

Well that is "where" we are headed. In the next few months the Quartz guys working with the Terracotta Guys and the Ehcache guys are developing a solution to the above set of problems. We'll be giving the same flexible and reliable scaled out scheduler but adding a new level of control. We are adding the "Where"

Stay tuned...

Hiring at Terracotta - Performance/Testing Engineer and or Lead

2010-07-22T08:40:00.000-07:00

Working at Terracotta is just about the best job someone could want (my humble opinion). Fast paced, super smart people, lots of interesting problems and widely adopted products wrapped up in a nice little package known as a fast growing startup. So send your resume now!

These positions can be either in San Francisco or Noida India.

About Terracotta:

Terracotta is the a fast growing company behind the most widely used software for application scalability, availability and performance. Our software is deployed in more than 250,000 enterprise installations, including the majority of the Fortune 2000.

Snap-in Scale and Performance

Terracotta's software products provide snap-in performance and scale for enterprise applications. With a simple change in two lines of configuration, Terracotta customers can run enterprise applications 10x faster and scale them―from one node, to 1000s, even to the cloud―without re-writing code or compromising performance or reliability.

A Leader in Distributed Caching

In “The Forrester Wave™: Elastic Caching Platforms, Q2 2010,” Forrester named Terracotta a Leader in this emerging market and ranked us strongest in strategy among eight elastic caching platforms.

http://www.terracotta.org

Where we are:

Our main headquarters are in San Francisco, CA in the SOMA area. We have an office in Noida, India and we have super star developers all over the world.

Description for LEAD QA Engineer

At Terracotta quality and stability in our product are our primary

objectives. Join our highly motivated, fast paced, agile, quality driven development team

where you will have many great opportunities to make an impact on

product capabilities and success in your role as QA Lead.

As a valued member of our tech lead team you will:

* Develop, maintain, and enhance both unit testing and performance

testing frameworks

* Design test strategies, develop test tools and implement test cases to

ensure highest quality deliverables for maintenace and new feature releases

* Improve the overall productivity of all of your co-workers by

identifying tools and processes to increase overall efficiency

* Create and maintain functional, performance, stress and endurance tests

* Diagnose and debug issues in a production environment

* Work closely with Engineering to understand the Product Architecture

and work on identifying, designing or enhancing existing test frameworks

to support backend test development

* Mentor and manage QA Engineers in a distributed team

Qualifications

* Proven track record as a lead in development and/or QA

* Motivated to improve existing processes, test strategies

* Strong knowledge of Java or other related programming languages

* Strong Knowledge in at least one scripting language such as Perl or Ruby

* Experience in creating back end test frameworks

* Ability to work independently to triage issues and prioritize tasks

* Strong understanding of QA Process

* Strong communication skills (verbal and written)

* Experience with code coverage and test tool development

* Experience with UNIX

* Experience with distributed caches, high availability products, and/or NoSQL solutions

* Experience with common java frameworks and containers such as Spring, Jetty, Hibernate, Ehcache, Quartz

* Ability to focus on multiple projects while in differing SDLC phases

If that's you and you meet most of the below criteria send us your resume careers@terracottatech.com.

Hiring at Terracotta...

2010-07-20T18:48:00.000-07:00

About Terracotta:

Snap-in Scale and Performance

A Leader in Distributed Caching

http://www.terracotta.org

Where we are:

Our main headquarters are in San Francisco, CA in the SOMA area. We have super star developers all over the world.

Who we are looking for:

Have you written or worked on a distributed system, messaging system or NoSQL solution? Do you love cool, hard problems? Are you good at design API's that work well in todays Java Applications? Can you work both in a group and on your own?

Qualifications:

Works hard and solves hard problems
Strong OO/Framework design sense
Understand the Java Landscape in a deep way (i.e. Spring, J2EE, Ehcache, Quartz, Rest, SOAP, NoSQL)
Excited about performance, caching and scale-out
Love to code and write tests
Works well in a team and individually
Believes that the only way to know if something works is to test it in a repeatable way

Nice to have:

Experience with open-source
Live in/near San Francisco
Tech Lead experience

Responsibilities:

Design and build the next generation of scale-out, performance and HA software.
Contribute to and extend Terracotta, Ehcache and/or Quartz Scheduler, some of the most popular and widely used frameworks in Java

If that's you and you meet most of the below criteria send us your resume careers@terracottatech.com.

A Couple Minutes With Some Toolkit Samples

2010-06-24T14:42:00.000-07:00

Last night we released Beta 2 of the Terracotta 3.3 release. It has a number of improvements and updates to the Terracotta Toolkit including improved naming and factoring and additional clustered classes like AtomicLong and List. You can grab that or to get the absolute latest you can grab a nightly (my preference because the beta doesn't work with maven)

I pushed a couple of toolkit samples to GitHub (which by the way is awesome!). I didn't do hardly any cleanup or comments so just post questions to the blog if you need help.

http://github.com/sharrissf/Terracotta-Tools-and-Samples/

If your a Git user already I don't need to tell you how to check it out. If your not just hit "download source" on the upper right of the web page and don't worry about Git.

Four examples are included:

PlayingWithMapOfLocksExpress - A quick sample that show's how to create a clustered Map of Locks.

PlayingWithToolkitBarrier - Example of using a Cyclic barrier to coordinate between processes

PlayingWithToolkitClusterInfo - Example showing how to register a listener for cluster events.

PlayingWithToolkitQueue - Little sample on using a queue between two nodes

PlayingWithToolkitClusterCounter - Basic sample that show's using a clustered atomic long

You can run these from the command line by:

Downloading and unpacking the nightly build or beta from Terracotta
Staring the Terracotta server by calling the ./bin/start-tc-server.sh
Running 2 instances of the compiled versions of any of the above

java -cp target:INSTALL_DIR/common/terracotta-toolkit-1.0-runtime-ee-1.0.0-SNAPSHOT.jar PlayingWithToolkitClusterInfo

You can also run them using the Terracotta Maven Plugin:

mvn tc:run

To switch between the samples edit the pom.xml

A Couple Minutes With The Terracotta 3.3 Beta

2010-06-09T12:49:00.000-07:00

With Terracotta platform version 3.3 our goal of creating an accessible application performance and scale-out solution is taking another big step forward. It's important to us that developers can find success using the ubiquitous Ehcache for performance, Hibernate for an ORM, Http Web Sessions for user state and Quartz for scheduling in a single node application. Then, without much thought or effort, add a couple lines of config to achieve scale-out and HA serving needs all the way through enterprise apps and into the cloud.

We have focused on a couple of high level areas in this release to take our solution to the next level.

Simple Scale - Reduce the need for tuning and tweaking with an improved next gen datastore. It will allow the everyday user to achieve the kinds of scale needed for massive applications both in data size and number of nodes.

Improved Visibility - We have added panels for Quartz and Sessions to our developer console giving full visibility to the full suite of performance and scale-out products. We have also added a more product focused organization of information in the tool.

Simple HA - A new panel that makes it easier to monitor interesting events that occur in a cluster. Pre-built templates for various configurations. Simplified way of migrating nodes, better defaults.

Modularity - We have exposed some of our most powerful pieces and parts as a versioned standard API that can be simply coded against to get things like, locking, queuing, maps and cluster topology. We use this API to build all four of our core products (Ehcache, Quartz, Hibernate 2nd level cache, Web Sessions).

While not everything made into the beta, enough is there to get a taste of where we are going. I encourage people to download it and try it as soon as possible. GA is just around the corner. Learn more on the release notes page.

Steve Jobs Stanford Commencement

2010-05-18T13:12:00.000-07:00

Was reading a couple stories on the web this morning and I ran into two somewhat interesting lists over at Time Magazine. I kinda mocked the first one upon reading the title "Top 10 College Dropouts." I laughed and said to my Coworker and friend Orion, "Must be a slow news day!" Followed by spending the next 10 or 15 minutes dissecting the article. How many were either from or lived in California? how many dropped out of Harvard? Who the heck is Buckminster Fuller? "About 10 minutes in Orion rightly pointed out, "Article can't be too bad since you've spent the last 10 minutes talking about it." Of course he was right.

Followed that up by clicking a link taking me to the next list called "Top 10 Commencement Speeches." I know what your thinking now, "How much free time does this guy have?" I was enjoying a little candy after a hard working morning ;-). Anyway, I digress. I noticed that one of the names was on both lists. You guessed it, Steve Jobs, someone that anyone who has ever built anything of any kind should at least respect.

I set about to watch the commencement speech he gave in 2005. He basically tells 3 stories from his life experience. Much like Randy Pausch's last lecture their is a lot to be learned here. The first two stories are awesome and inspirational and many of us can relate to and learn from them. The third is also good/inspirational but a bit dark and kind of scary and depressing.

Anyway, since in this speech Steve talks so much about time, I will lead in with the opinion that listening to this commencement is time well spent. Check it out!

A Couple Minutes With Non-Stop Ehcache

2010-05-07T13:57:00.001-07:00

Sometimes you just want to have an SLA (Service Level Agreement) when dealing with a component like a cache. You want to know that no matter what goes wrong, disk failures, deadlocks, network down, database locked up (for write-through) your threads won't be blocked for longer than your SLA. In the security world this is analogous to Layered Security where one is protected at multiple layers from breaches. In this case you are protected at multiple layers from hangs.

Non-Stop Ehcache

In comes NonStopCache. This is a decorator for Ehcache that allows one to specify an SLA. When using this decorator no operation is allowed to take longer than the SLA provided. All operations are isolated from the cache via a thread pool providing complete protection. The NonStopCache has multiple ways of configuring it. One can setup whether to serve stale data, perform noops or throw exceptions when SLA's are being violated covering most of the cases one needs.

Getting Started

Here's what I did to get started using clustered Ehcache and NonStopCache:

Download Ehcache 2.1
Download the NonStopCache 1.0
Start the Terracotta server

ehcache-2.1.0-beta/terracotta/bin/start-tc-server.sh

run the below program (source code below)

java -cp .:ehcache-core-2.1.0-beta.jar:slf4j-api-1.5.8.jar:slf4j-jdk14-1.5.8.jar:ehcache-terracotta-2.1.0-beta.jar:ehcache-nonstopcache-1.0.0-beta.jar MyFirstNonStopEhcacheSample

NOTE: Kill the Terracotta server when the printed output instructs you too.

Let's review the output:

Regular cache. No Decorator
The size of the cache is: 0
After put the size is: 1
Here are the keys:
Key:0
Done with cache.

Sleeping, Stop your server
Disconnected NonStop with noop cache.
The size of the cache is: 0
After put the size is: 0
Here are the keys:
Done with cache.

Disconnected NonStop with local reads cache.
The size of the cache is: 1
After put the size is: 1
Here are the keys:
Key:0
Done with cache.

Disconnected NonStop with exception cache.
Exception in thread "main" net.sf.ehcache.constructs.nonstop.NonStopCacheException: getKeys timed out
at net.sf.ehcache.constructs.nonstop.behavior.ExceptionOnTimeoutBehavior.getKeys(ExceptionOnTimeoutBehavior.java:114)
at net.sf.ehcache.constructs.nonstop.behavior.ClusterOfflineBehavior.getKeys(ClusterOfflineBehavior.java:120)
at net.sf.ehcache.constructs.nonstop.NonStopCache.getKeys(NonStopCache.java:264)
at MyFirstNonStopEhcacheSample.addToCacheAndPrint(MyFirstNonStopEhcacheSample.java:45)
at MyFirstNonStopEhcacheSample.(MyFirstNonStopEhcacheSample.java:40)
at MyFirstNonStopEhcacheSample.main(MyFirstNonStopEhcacheSample.java:60)

What Just Happened?

The cache is first loaded into an ordinary undecorated cache. This is performed before the server kill and proceeds without incident. The next round of operations on the cache were performed with the server down.

Since that cache is configured as noop all operations are ignored and the size of the cache appears as zero. This is great for when you just want to not bother with the caching if it isn't available.

Next round is local reads. This allows you to use the values that are available locally while the cache is unavailable. This is especially nice for read only or mostly read only caches that fit in memory and are always available.

After that we go to the exception version. If you have a cache of important coherent data but you don't want threads blocked this setting is for you. It will blow cache operations out of the thread so your container can just keep going by either showing an error page or getting the data from elsewhere.

Some interesting points here:

These decorators are all being used on the same cache. This way you can make the behavior specific to the user of the cache. It gives tremendous flexibility.
You'll notice this little sample flies through despite the timeout being set to 13 seconds. This is because it's in fail fast mode. In this configurable mode if the cache knows it can't communicate it will return the failure case immediately. If that's not what you want you can instead set it up to not fail fast and wait the full timeout no matter what.
I did this work in config but the same setup can be done in code

The Code

MyFirstNonStopEhcacheSample.java

And the Config file ehcachenonstop.xml:

A Couple Minutes With Terracotta Toolkit Nightly

2010-05-01T08:44:00.000-07:00

A couple days ago I wrote a short blog about our goal of creating a Terracotta Toolkit for others to easily build simple to use scaled out frameworks, tools and in some cases apps. This same exact toolkit is used to build Ehcache, Hibernate, Http Sessions and Quartz with Scale and HA just by adding 2 lines of config. I was pretty shocked and excited about what a tremendous reaction this blog and the toolkit received. Usually when I write a blog most of it's traffic comes from awesome sites like DZone but this blog was different. I actually got considerably more direct traffic from things like e-mails and twitter than anything conventional. That kind of viral excitement shows a much deeper level of curiosity and is tremendously motivating for the team here. Anyway, I digress.

Me being the curious type I decided to do something a little dangerous yesterday. I asked Geert, one of the leads on the Terracotta toolkit project, if we were in a state where I could play around with it a bit and get a sense of how hard it is to get started. We discussed the usual caveats, "We haven't agreed on some of the naming and factoring stuff yet?" which I clearly understood. Knowing that fact and that I'd be working off a nightly(this link downloads it) I set out to follow his simple instructions and see just how easy it was. Here is what I did.

Geert gave me a simple cyclic barrier sample to work off of. This is a sample I've whipped up my self a few times over the years. Many of our earliest users leveraged Terracotta strictly as a distributed lock manager and that's what this simple sample does as well. You start the app with a barrier name and a node count and it just hangs until the node count is reached. Seems like as good of a sample as any to highlight how to get started with the toolkit.

Downloaded the nightly and unpacked

tar -xzvf terracotta-trunk-nightly-rev15534.tar.gz

Grabbed a quick sample app

import org.terracotta.api.ClusteringToolkit;

import org.terracotta.api.TerracottaClient;

import org.terracotta.coordination.Barrier;

public class PlayingWithExpressBarrier {

public static void main(String[] args) {

final String barrierName = args[0];

final int numberOfParties = Integer.parseInt(args[1]);

//Start the Terracotta client

ClusteringToolkit clustering = new TerracottaClient(

"localhost:9510").getToolkit();

//Get an instance of a barrier by name

Barrier barrier = clustering.getBarrier(barrierName,

numberOfParties);

try {

System.out.println("Waiting ...");

int index = barrier.await();

System.out.println("... finished " + index);

} catch (Exception e) {

e.printStackTrace();

}

I worked in eclipse so at this point all I had to do is add the toolkit jar to the classpath to get it to compile

terracotta-trunk-nightly-rev15534/common/terracotta-toolkit-1.0-runtime-1.0.0-SNAPSHOT.jar

Now kickoff the Terracotta server

terracotta-trunk-nightly-rev15534/bin/start-tc-server.sh

And run the sample 3 time

java -cp .:terracotta-toolkit-1.0-runtime-1.0.0-SNAPSHOT.jar barrierName 3

The first two nodes should hang waiting for the 3rd node. Once the 3rd node is started all three should run to completion. This is just a little taste of what is coming.

The toolkit will have highly concurrent maps, locks, counters, queues, evictors and more so stay tuned. All of the toolkit classes will have tests in our TCK. The toolkit will run on any 1.5/1.6 JVM and requires no boot-jars, agents or container specific code. Given versions of the TCK will allow people to just drop in that toolkit jar at runtime for any app developed using that version of the API provided by the version of the TCK. I'll talk a bit more about how versioning the toolkit API will work in a future blog but it will work like any other spec. You can use any implementation that implements the spec. Enjoy, much more to come.

Countdown To The Terracotta Toolkit Beta

2010-04-29T09:23:00.000-07:00

Countdown To The Terracotta Toolkit Beta

Over the last year, Terracotta has launched four great products backed by the snap-in HA and scalability of the Terracotta Server Array: Enterprise Ehcache, BigMemory for Enterprise Ehcache, Web Sessions and Quartz Scheduler. Building these products has left us feeling a bit greedy. In order to free ourselves from this guilt we've created a toolkit that has allowed us to rapidly build these easy-to-use, high-scale/HA products.

The Revolution Begins

So why should we be the only ones who have all the fun. Lots of smart people out there can build clustered caches, frameworks, and tools? So we are creating the Terracotta toolkit and adjacent Terracotta Compatibility Kit. We will be releasing a set of standard parts like highly concurrent distributed maps, clustered evictors, queues, locks, counters, cluster events etc that can be used by anyone so that they might realize their vision of what the developers of the world should use. This kit was built with a few core goals in mind:

Ease of use is paramount - For both the developers that leverage the Terracotta toolkit and the people who use the stuff built using the Terracotta toolkit
Stable API matters - We are building a compatibility kit and will maintain a strict and clear versioning scheme so that framework developers can rely on and clearly know what versions of Terracotta can work with the API version used in the application. Your users can just drop in any version of the terracotta-toolkit.jar that implements the version of the API you coded against.
Parts is Parts - Get all the useful parts we use to build our products packaged and out for others to use.
Scale Continuum - The parts should work both clustered and unclustered continuing our vision of a scale continuum.

The beta of this toolkit is a few short weeks away and we can't wait to see what great stuff people will build on it. Come show us how a cache should be built, come show us what tools we haven't even thought of.

With this toolkit pretty much any framework developer will be able to build out their HA, distributed and scale out dreams.

Dave Klein's Scale Grails Webinar

2010-04-26T18:32:00.000-07:00

As someone who spent years in the Smalltalk world I have a special place in my heart for powerful/flexible frameworks and languages that strive for simplicity for the developer. Grails is one of the most powerful and flexible frameworks around and modern representation of those Smalltak values. Some would argue that it's only major missing features are scale-out and HA. The great news on that front is that Grails is built on Quartz, Ehcache, Http Sessions and Hibernate all of which can be scaled with Terracotta in a couple lines of config each. By scaling these four frameworks you can completely scale Grails.

Check out this webinar from the guy who wrote the book on Grails Dave Klein and and learn about it for yourself:

@Scale Conference 2010!

2010-04-22T15:59:00.000-07:00

In today's world of internet scale applications, enormous companies and cloud many applications and application developers are struggling provide the needed scale/ha. They've developed their middleware toolkit of application server, hibernate, caching, and scheduling in order to achieve the business goals and single node performance required of them. But now they need to take those applications and frameworks to the next level.

They need scale out!

That's where the @Scale conference can help. In San Francisco at the same time as Oracle Open World/JavaOne attendees of this one day conference will learn from the leaders in cloud and application scale.

Jeff Barr of Amazon
Simon Crosby, CTO of Citrix
Rich Wolski, founder of Eucalyptus
Adrian Cole, found of JClouds
Paul Querna of CloudKick and libcloud fame
Jos Thosman of Canonical--the Ubuntu guys who also do UEC
Greg Luck - founder of ehcache
James House - founder of quartz
Ari Zilka from Terracotta
Dan Querna, Founder of Cloudkick

Do not miss this opportunity to work with, listen to and learn from the best in the business. The conference is focused on the scale problems you face and real solutions you can apply. Learn about those solutions from the people who created them!

Don't pull your hair out solving problems that are already solved. Come to @Scale 2010!

Signup now and get the early bird prices! Space is limited so signup before it fills up.

What: Sign up here: @Scale conference

Where: San Francisco, SOMA

When: Friday Sept 24th 8am-4pm

2010-04-22T09:37:00.000-07:00

One challenge with drastically simplifying Terracotta as a tool for application scale is getting the word out that it's occurred!

The subject of this blog is one of two lines of config and a jar on the classpath that move Ehcache to be a coherent distributed cache. The line of config in the subject says, "Hey, cluster this cache." The other line, which looks something like this:

"<terracottaConfig url="localhost:9510" />"

tells Ehcache where to find the Server Array.

This same story is true for scaling out Http Sessions, Quartz Scheduling and Hibernate Caching. A whole new world of scale is available to the applications we all write. It free's us from complex sharding, awkward CAP trade offs and endless DB tuning. And the cherry on top is it's open source.

No recompile, no code changes, no magic tricks or additional knowledge. I could train my dog to scale an application built on these ubiquitous frameworks.

Check out these blogs to learn more:

Add Scale and HA to Ehcache In Less Than 5 Minutes

Add Scale and HA to Quartz Scheduler In Less Than 5 Minutes

Ehcache 2.1 Beta - Lots of Stuff, Still Backward Compatible

2010-04-20T21:10:00.000-07:00

UPDATE: 2.1 went GA March 21st

The Ehcache dev team is pretty excited to be getting 2.1 beta out for the world to try. This release is focused on 3 primary goals:

Build on our vision of an application scale continuum from one node to the cloud.
Improve Ehcache performance both unclustered and clustered.
Improve Ehcache applicability for both unclustered and clustered.

What did we do in the name of these goals? Good question!

Scale Continuum

In this release we are focused on taking features that had been added for the Terracotta Clustered version of Ehcache and extending them back to the unclustered version.

The following features fall into that category:

The Explicit Locking Module - This module allows you to acquire and release locks manually for given keys. It required some significant rework in the unclustered stores but this now works just as well unclustered as it does clustered supporting fully coherent operations.
JTA - In 2.0 of Ehcache we added JTA support when clustered via Terracotta. In 2.1 we extended that functionality to unclustered and have begun the process of performance tuning to go along with its XA compliance.

Performance

While performance wasn't a big focus of this release (It is for the next release) we were able to considerably improve Ehcache "put" performance due to the significant work we did on the disk store and locking architecture. We'll also be spending some time tuning JTA but that work did not make it into the beta.

Applicability

Here we did some considerable work.

JTA for Hibernate Second Level Cache - We added support for using Ehcache JTA in a second level cache both clustered and unclustered.
UnlockedReadsView - This is a subtle but important feature. For those who are using a coherent cache but have some part of an application that needs to be able to read at high rates without impacting the rest of the cache this view is a huge help.
NonStopCache - Useful for guaranteeing that your cache can never stop your application. On a per cache basis an application can avoid holdups caused by problems such as a slow disk in an unclustered cache or a network outage in a clustered one.
New Coherent methods - We've added useful methods like putIfAbsent and replace to simply and easily work with a clustered or unclustered cache in a fully coherent manner. Together with the explicit locking wrapper much is possible.
We also added a bunch of tests and bug fixes to the web-cache, an extremely useful tool for making performant web applications.

Summing up...

While still in beta we feel like Ehcache 2.1 is another exciting step for our product family. It adds performance, significant features and is still backward compatible. It continues our vision of an application scale continuum from one node to the cloud without burdening the developer with complexity. One set of frameworks, one application, scale out is added at runtime. We are really excited about this vision and are working tirelessly at extending it to all our users' needs.

Please Download Ehcache 2.1 Beta, put it through it's paces and give us lots of feedback!

Application Server Instead Of Web Server

2010-04-19T21:11:00.000-07:00

I'm starting to see a small but growing group of sites forgoing web servers and going straight to the Application Server in the Java world. This has some obvious advantages:

Less infrastructure
Java doesn't have buffer overflows and is a bit more secure
You can do much more interesting caching by having the web serving and app serving in the same layer

I was excited to see this blog today and just wanted to point people to it and see what thoughts are out there both pro and con on this trend.

Web Application Architecture - do I really need Apache?

How Much Is Google Really Spending?

2010-02-03T23:10:00.001-08:00

This will be a short blog.

Has anyone else noticed that google is using adwords extensively to pimp the nexus one? They must be spending a fortune in lost revenue from other ad sources. Since they don't have to pay for those ads I wonder if even they have an accounting of how much they are spending on this thing. At first I thought the nexus one looked interesting but...

T-Mobile only

No Exchange support

weird app space restrictions

and from what I've read poor touch screen keyboard

On the plus side:

Nice screen

Solid camera with flash

really fast processor

With this kind of marketing muscle it's probably just a matter of time before it gets wide adoption but I suspect it's still a couple years away from being really good.

From Performance to Scale - A Software Story... Part I

2009-12-30T22:47:00.000-08:00

When building most applications whether it's JEE, Spring, Grails or plain old Tomcat their are a series of common tools that are used. HttpsSessions is used for managing the state of the web portion, ORM for talking to a database, a cache to hold interimediate application state and a scheduler to make things happen at a certain time, place or interval. Out of the box it is quite easy to get started with these components. However, as an application makes it's way into production it often needs improved performance, often followed by HA and then scale-out.

This blog is about a journey. It's about taking an application from performance all the way to scale-out without rewrites and using the components that you already have.

In the beginning it's just the Application

The good news is that whether you're a JBoss person, a Spring person, a Grails person or a Tomcat person your really using the same key components. Almost everyone in these environments are using Hibernate as an ORM, the Http Session Spec, Ehcache for caching/performance and Quartz for scheduling. So while their is no official standard, the global consensus is pretty clear. If it's me I would start from Grails (which is built on Spring, Ehcache, Quartz and Hibernate) but doesn't require the 2 weeks of picking, deeply understanding and piecing together components that is required with most other approaches.

Round 1, Performance

So you've written your single node application using the usual components. Turns out, it's slow. Now what? Well the first thing you need to do is understand why it's slow at a high level. I usually check the following obvious things:

1) CPU bound - Check machines CPU stats on all nodes (including DB), now day you also need to figure out if one of your CPU's is maxed out which could mean the app needs more parallelism

2) GC Bound - Use verbose GC or your favorite tool

3) Database bottleneck - A number of tools and tricks exist to see if your database is the bottleneck. My favorite ways are:

- Keep track of and monitor query times and or look at hibernate stats

- Take thread dumps. If all threads are blocked on DB calls something is wrong

- Check resource Utilization on the Database machine

4) I also like to isolate operations and see which ones are slow or create slowness. This can tell you a lot about where to focus.

Ok, so now you know why things are slow and your in the same boat as about 90 percent of the world. The Database is the number one bottleneck.

Tuning the ORM

Many start by generating the ORM schema using tools. This usually ends up being a completely Normalized database that requires expensive queries. At this stage of development it's almost certain that this is where your performance problems will be. Aside from the usual mistakes like missing JDBC connection pool or bad JDBC Connection pool (Use C3PO and make sure your seeing parallelism. Many of the others are either useless or broken). The next thing to do is to try hibernate 2nd level caching. If your application is read heavy this can have an amazing impact on performance. Entity caching is generally the way to start. The most common gotcha is it getting invalidated by custom SQL. It is a little known fact that if you execute custom sql through hibernate it clears the cache making it useless. If you are going to do Query caching you MUST READ THIS BLOG.

Now you've got hibernate 2nd level caching turned on and you've tuned it. The database isn't the bottleneck anymore but your seeing a lot of GC and thread dumps show contention on Hibernate itself. Especially when displaying medium size to large tables of data on the screen.

Plus, it turns out that updating the database on every intermediate operation that occurs is pounding the databases locks, cpu and disk. It's also made my schema really unwieldy What do I do now?

Stepping up to Caching

If after 2nd level caching you still have performance problems it is usually time to look at application level caching. This can be extremely helpful for performance. If your seeing:

a) A lot of garbage creation around calculating results

b) cpu usage or contention getting information

c) Latency from IO in retrieving information from disk, a web service or a DB

d) Pounding your database with fine grained updates/information with information that you only needed for a short time.

You'll likely want to start caching. Some things to cache include:

1) heavily used reference data from the database - States, zipcodes, product ID's etc

2) Intermediate state - If a series of operations occur but all you care about is the end result. Cache until the end result is reached then put in the DB.

3) heavily reused calculated values - "Like totally searching for Britney Spears."

4) Temporary values that are only needed for a set amount of time - user sessions or data that is needed at a certain time of day.

Hopefully between the Hibernate caching and tuning and the Application data caching you've now got much of your performance issues under control.

Your application probably does some scheduling and uses http sessions but these rarely create performance problems in one node.

Part II - Now that it's performaning what about availability?

Part III - One node isn't enough anymore I need scale-out

Clustered Quartz Scheduler In 5 Minutes

2009-12-16T15:58:00.001-08:00

Need a fast clustered/persistent Quartz Job Scheduler? Quartz Scheduler, the ubiquitous job scheduler built into Spring, JBoss and Grails can be configured to provide those features in under 5 minutes in this brief tutorial.

A Brief Digression Into The Why

Why do I need a persistent scaled out job scheduler? The main use cases for a clustered/persistent job scheduler are:

HA - You need to be able to restart your application without losing scheduled jobs
Scale Out - Your application now needs more than one node to handle the load it receives and you want your scheduled jobs to distribute across nodes.
You are using a database to persist and or scale your scheduler and you are seeing DB load/locking issues and or you find it two difficult to setup or maintain.

Steps:

1) Download Terracotta 3.2 (Which includes Quartz Scheduler) http://www.terracotta.org/dl/oss-download-catalog

2) Put the following jars in your class path (all included in the quartz-1.7.0 directory of the Terracotta kit from above):

quartz-1.7.0.jar - regular quartz jar
quartz-terracotta-1.0.0.jar - Terracotta clustered store

3) Whip up some scheduler code:

 public class QuartzSample {

public void startJobs() throws Exception {
  Properties props = new Properties();
  props.load(QuartzSample.class.getClassLoader().getResourceAsStream("org/quartz/quartz.properties"));


  // **** Begin Required TC props
  props.setProperty(StdSchedulerFactory.PROP_JOB_STORE_CLASS,"org.terracotta.quartz.TerracottaJobStore");
  props.setProperty("org.quartz.jobStore.tcConfigUrl", "localhost:9510");
  // *** End Required Terrocotta Stuff


  StdSchedulerFactory factory = new StdSchedulerFactory(props);
  Scheduler scheduler = factory.getScheduler();
  scheduler.start();
  if (scheduler.getJobDetail("myJob", "myGroup") == null) {
    System.out.println("Scheduling Job!");
    JobDetail jobDetail = new JobDetail("myJob", "myGroup", DumbJob.class);
    Trigger trigger = TriggerUtils.makeSecondlyTrigger(5);
    trigger.setName("myTrigger");
    scheduler.scheduleJob(jobDetail, trigger);
  } else {
    System.out.println("Job Already Scheduled!");
  }
}

public static class DumbJob implements Job {

  @Override
  public void execute(JobExecutionContext arg0) throws JobExecutionException {
    System.out.println("Works baby");
  }

}

public static void main(String[] args) throws Exception {
  new QuartzSample().startJobs();
}

** NOTE: Notice the two lines of properties that set things up for clustering with Terracotta in the sample. That's the only difference from single node unclustered Quartz.

4) Start the Terracotta server by running start

./start-tc-server.sh

in the bin directory of the Terracotta kit

5) Run the sample code above and watch it run the job every 5 seconds. Then kill the sample app and restart it. The app will tell you that the job is already scheduled and the job will continue.

Conclusion

Two lines of configuration and a server takes you from ubiquitous job scheduler built into Spring, JBoss and Grails to scale out and persistence.

Have fun!

Welcome James House and The Quartz Community

2009-11-06T13:39:00.001-08:00

I'm excited to be welcoming James House and Quartz to the Terracotta and Ehcache Fold. The Terracotta dev and field teams have long believed that scheduling and coordination are hugely important parts of applications from single node to scaled out architecture. In Java that leads you to one place. Quartz! We believe James and Quartz are an excellent fit with our community and suite of products that are useful from single node to the cloud.

Quartz Scheduler is both a best of bread and a ubiquitous product. We are getting right to work on contributing. The first step along the path is the Mavenization and Hudsonifaction of the Quartz project which has already been completed! We also are ready with a beta version of Quartz Terracotta Express edition. It's an HA/Durability/scale-out version of Quartz that requires no DB and is so simple it can be leveraged in minutes by existing Quartz users. We look forward to working with James to create the most usable and useful enterprise class open source scheduler available and the enterprise class support product to go with it. We have lots of great feature ideas and we look forward to the journey ahead.

What To Do If Your iTunes Gets iHacked...

2009-10-26T18:35:00.000-07:00

Please make sure you watch your iTunes receipts/charges. In the last week or so someone somehow started using my iTunes account to purchase music illegally. Yep, I got iHACKED! Fortunately I only got 2 bills worth around 75 bucks but if I hadn't been paying attention it could have been much worse. So make sure you read those statements!

If this happens to you immediately change your password and remove your credit card information from the iTunes site/app. You do this by:

Clicking on the store menu while in iTunes and then selecting the "View my account" menu item. From there click "Edit" on the main screen where it says "Payment Information" and select credit card type "None".
Next change your password. In the same "View My Account" screen click "Edit Account Info." That will take you to a screen where you can change your password.

Ok, now that you've protected yourself you need to get your money back. Unlike Amazon or Zappos contacting support and reporting the fraud DOES NOT lead to a refund. They will direct you to your credit card company to refute the charge. I did this and it appears to be successful (I found it disappointing that Apple didn't just take care of this for me).

As a side note, when shopping online make sure you have a credit card company that takes responsibility for stuff. Amex has always been great and I'm sure other good companies exist as well.

UPDATE:

Looks like I'm not alone here. This blog reads like I could have written it.

5 Hints You're Using A Map When You Should Be Using a Cache?

2009-10-26T16:29:00.000-07:00

When developing software in Java one almost always ends up with a few maps that contain keyed data. Whether it's username -> conversational state, state code -> full state name, or cached data from a database. At what point do you move from a basic Map (or one of it's more fancy variants like LinkedHashMap subclasses or ConcurrentHashMap) to an open source, light weight cache like Ehcache?

Here are 5 quick things to look for:

5) You've built your own Map loader framework for bootstrapping and/or reads triggering loading

4) You need to be able to visualize and/or control your Map via JMX or a console. For example you want to watch the hit rate of your Map or track the size of your Map.

3) You're hacking in "overflow to disk" functionality and or persistence for your Map in order to handle memory pressure and/or restartability.

2) You're hacking in special behavior to cluster Maps when you scale out. This includes things like writing your own invalidation and/or replication solution over jms or RMI

1) You find yourself implementing your own eviction strategies.

Avoid The Slippery Slope:

It's a slippery slope. First you add in one feature, then another, and next thing you know you've reinvented the cache wheel. No point in doing that. There are great caches out there that are apache licensed, light weight and have the above features you need and more.

To learn more about Ehcache check it out at ehcache.org »

Excellent Blog On Code Smells...

2009-10-23T15:51:00.000-07:00

This is a really good/short blog that highlights some code smells that everyone should look out for. It's not a complete diary or anything but when I read it I felt like it could have been written by me.

8 Signs Your Code Sucks...

He also has a follow on which I agree with.

Comments are a sign of bad code and I am not sorry for saying it

Improved Web Browsing By Controlling Flash

2009-10-23T14:09:00.000-07:00

I have a few pet annoyances when surfing the web.

* I find it disruptive to have audio play when I hit a web page (though I usually keep my sound off)

* I don't like when video plays automatically when I hit a web page.
* I don't like when my computer heats up and the battery drains when I'm not doing anything just because
I left a web page/tab open.

* Some pages that look rather slim take a disproportionately long time to load (there are lots of reasons for this but flash seems to be one of them)

Turns out that I was able to mostly solve those problems by using one of the many flash control
plugins. I surf on Safari for the most part so I went with ClickToFlash. FireFox has FlashBlock which I haven't tried.

The way it works is it shows you frames where flash usually should be. If you want to see what is there then just click on the box and it loads the real flash. It has many other nice features around content but the important one is the one I described.

I have to say, when I installed this thing I was absolutely amazed by how many things that looked like regular adds and images were actually flash. You will be astounded. Gotta wonder what these companies are doing with flash when they are showing a static image? I didn't take any official benchmarks but after installing I noticed an increase in battery life and decrease in heat on my computer. This experience makes me actually believe (didn't really at first) Apple's battery/cpu excuse for not supporting flash on the iPhone.

If your like me and only want flash when you want flash. Try it out.

Distributed Coherent EhCache In less than 5 Minutes...

2009-10-02T16:09:00.000-07:00

Need a fast clustered/persistent cache? Ehcache, the ubiquitous cache built into Spring, JBoss and Grails can be configured to provide those features in under 5 minutes using this brief tutorial.

A Brief Digression Into The Why

Why do I need a persistent scaled out cache? The main use cases for a clustered/persistent cache are:

I'm using hibernate and it's pounding the database or it's too slow. Use a coherent second level cache to deflect load off the database, reduce latency without getting stale data.
Have a bunch of intermediate data that doesn't belong in the database and/or is expensive to store in the database that I want to keep in memory. Problem is if a node goes down or if someone asks for the data from another node the data is lost
I'm already caching but I have to load data over and over again into every node even though hot data for one node is hot for all (Known as the 1/n effect). If the data is cached for one node it is cached for all.

Steps:

1) Download the latest Ehcache www.ehcache.org

2) Put the following jars in your class path (all included in the ehcache kit):

ehcache-core.jar - Core Ehcache

ehcache-terracotta.jar - Terracotta clustering

slf4j-api-1.5.8.jar - Logging API Used by Ehcache

slf4j-jdk14-1.5.8.jar - Implementation of the Logging API

3) whip up some cache code:

 package org.sharrissf.samples;

import net.sf.ehcache.Cache;
import net.sf.ehcache.CacheManager;
import net.sf.ehcache.Element;

public class MyFirstEhcacheSample {
 CacheManager cacheManager = new CacheManager("src/ehcache.xml");

 public MyFirstEhcacheSample() {
      Cache cache = cacheManager.getCache("testCache");
      int cacheSize = cache.getKeys().size();
      cache.put(new Element("" + cacheSize, cacheSize));
      for (Object key : cache.getKeys()) {
           System.out.println("Key:" + key);
      }
 }

 public static void main(String[] args) throws Exception {
      new MyFirstEhcacheSample();
 }
}

4) Whip up some quick config

 <?xml version="1.0" encoding="UTF-8"?>

<ehcache xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
 xsi:noNamespaceSchemaLocation="ehcache.xsd">

 <terracottaConfig url="localhost:9510" />

 <defaultCache />

 <cache name="testCache" eternal="true">
                <terracotta clustered="true"/>
 </cache>

</ehcache>

5) Download Terracotta

6) Start the terracotta server in the bin directory with the start ./start-tc-server.sh

Now just run that Java snippet a few times and see your cache grow.

Welcome EHCache Community

2009-08-18T07:59:00.000-07:00

I'm excited to be welcoming Greg Luck and the EHCache community to the Terracotta family. EHCache is an extremely useful/usable product and nearly ubiquitous in the caching space. Greg has spent years solving the important real world problems associated with building highly performant applications. The Terracotta Dev team is very much looking forward to helping accelerate EHCache's development as well as provide the best possible integration with the Terracotta product family.

EHCache will remain under the Apache 2 license and we have created the beginnings of a new website at www.ehcache.org. Greg will continue to drive EHCache's vision and direction, as well as being highly involved in it's development. He will also be instrumental in helping Terracotta to define and build out our caching strategy as a whole. His vision, as well as the EHCache community's help are essential in allowing us to together take these products to the next level.

We see a great future of product offerings for your desktop app, on your servers and in your cloud solving the scale/performance problems of today, tomorrow and beyond.

Distributed Data Structures: ConcurrentDistributedMap

2009-08-12T09:56:00.000-07:00

Concurrent Distributed Data Structures?

Many challenges exist when developing a high scale multi-node application. Our goal at Terracotta is to take on those challenges in ways that remove them from the plate of those architecting and developing applications and place them squarely on our shoulders.

In order to accomplish such a lofty goal we first had to create some core pieces of infrastructure on which many higher order abstractions could be built. One such "piece" is our ConcurrentDistributedMap. This data structure is a fundemental piece of our Distributed Cache, our Hibernate product and our Web Sessions product and is also available for use in custom solutions for those using Terracotta as a platform.

Challenges and Tradeoffs

Developing a data structure that is Distributed as well as Concurrent and Coherent has very different trade-offs from developing for a single JVM. If one took a standard concurrent data structure like ConcurrentHashMap and just clustered it "as is" one would likely run into performance and memory efficiency issues. Even a really cool concurrent data structure like Cliff Click's Non Blocking Hash Map would not do well if one used the algorithms without thought in a coherent cluster.

The challenge is that the trade-offs change when you add the latency of a network and data locality in the middle of the game. In normal concurrent data structures you care about:

- How long you hold locks

- How much is locked while you hold it.

- CPU usage

- Memory Usage and Object creation

In the clustered case you add the following:

Lock locality - Is the lock you need already held on the local machine or do you need to go get it over the network. If you need to go get it how long does that take. While a little of the question of "How long does it take to get the lock" exists on a multi-cpu single machine it's not nearly to the same degree.

Data locality - Is the data I need already local or do I need to go get it. If I need to get it how long does that take

Data change rate - How much clustered data am I changing and how long does it take to send it around? Also, do I send it around?

Data size - In a clustered world one often uses data structures that don't fit entirely in a single node. One has to take pains to control the size and amount of the data in each JVM for efficiency.

There are other implementation specific/point in time issues like number of locks and their cost but those can mostly be optimized away at the platform level.

Single JVM ConcurrentHashMap

ConcurrentHashMap adds concurrency by collecting groups of entries into segments. Those segments are grouped together both from a lock perspective, they share a lock, and from a physical space perspective, all entries in a segment are generally in one collection. In a single JVM the only risk of sharing a lock between the entries is that one can contend on the in-memory speed look-ups. This is a very effective way to handle large numbers of threads making highly contended gets and puts to the map. If one runs into contention with this kind of data structures one can just up the number of segments in the Map.

Concurrent Map In A Clustered World

In a clustered world problems occur with a data structure like this. First, getting a lock or an object can be either in-memory speed or take many times in-memory speed depending on whether it has recently been accessed locally. In some cases this is no problem and in some cases it's pretty bad. It's also a space issue. If a segment is brought in as a whole and it's entries are in that segment strictly because of it's hashCode then the natural partitioning of the app's usage won't help save space by only loading the entries needed locally. Instead it will load the needed objects and anything else in it's segments. This elimenates the benefits of any natural or forced locality that occurs in a multi-node application.

Use-Case Analysis

In order to highlight some of the pro's and con's of CHM (ConcurrentHashMap) I'm going to vet it against a few use-cases.

Use-case 1 - An 8 node app sharing a clustered ConcurrentHashMap

All the data in the map is read only and it's used in all nodes evenly and the data fits entirely in a single JVM's heap.

GOOD NEWS! you will be fine with a regular clustered ConcurrentHashMap. Lets look at why.

1) All data will be loaded everywhere so unnecessary faulting (the act of pulling a data item into a node) won't be happening

2) All locks will be read locks and will be local everywhere so your latency will be nice and low (Due to greedy locks)

3) Won't have contention on the segments because reads are pretty much concurrent

Use-case 2 - The same as use-case 1 but now the map data is bigger than memory and you have a sticky load balancer.

Some good and some bad:

1) Since data is batched into segments by hash code and your load balancer hashes on something completely different than your map hashes on you will end up loading data into each node that is not needed. This is a result of the ConcurrentHashMap segmenting strategy.

2) Locks will still be fine because it's all read and read locks are very concurrent so segment contention won't be an issue.

So the memory manager may be doing unnecessary work and whether you will be in trouble depends on how big the ConcurrentHashMap is

Use-case 3 - Same as use-case 2 with the exception that now we are doing 50 percent writes. Something similar to caching conversations.

1) Still have the above problem of loading unneeded batches

2) But now, due to the writes, you are also maintaining the state of the objects that have unnecessarily poor locality in all the nodes where they don't belong.

3) Now you have a locking problem. While writing an entry to a segment you are blocking people in other nodes from reading or writing to that segment adding some serious latency. Plus the locks are getting pulled around to different nodes because even though your load balancer provides locality it is on a different dimension that of the internals of the map and is therefore not helpful.

Reviewing the problems highlighted by use case 3:

- Lock hopping leading to slow lock retrieval

- Lock contention due to grouping of multiple unrelated entries with locks.

- Faulting and Memory wasting due to unfortunate segmenting of data

- Broadcasting of changes or invalidations to nodes that shouldn't care

What did we do?

We built a specialty highly concurrent map tuned for distribution and the above challenges call ConcurrentDistributedMap.

Locking:

Instead of breaking things down into segments for locking we lock on the individual keys in the map. This gives the same correctness guarantees while giving the maximum concurrency. This drastically reduces lock hopping and contention and provides in-memory lock speeds most of the time.

Segmenting:

The segments go away completely. Key Value pairs are managed on an individual basis so no unnecessary faulting occurs.

Broadcasting and invalidation:

The above, plus an efficient memory manager means that values are only faulted into nodes where they are used. Since those values aren't in all nodes anymore invalidation and or broadcasting of changes for those entries is no longer needed.

This data structure takes excellent advantage of any natural partitioning that may occur at the application level.

Summary

Building a fast, coherent, concurrent, distributed data-structure requires thinking about an extended set of concerns. However, if one pays attention to the issues it is possible to create a highly useful solution. To learn more check out the ConcurrentDistributedMap described above.

Additional Reading:

To learn more about the ConcurrentDistributedMap you can check out these links:

http://forge.terracotta.org/snapshots/projects/tim-concurrent-collections-root/

http://forge.terracotta.org/snapshots/projects/tim-concurrent-collections-root/apidocs/index.html

For more information on Terracotta's distributed data structures one can always look here:

http://www.terracotta.org/web/display/docs/Clustered+Data+Structures+Guide

Micro Review for Safari 4 Beta...

2009-02-24T18:20:00.001-08:00

I don't know what they are doing as far as tricks but HOLY COW THIS THING IS FAST!. Love the new top sites pane (like chrome), Don't care about the new tabs thing but some may. I thought Safari 3 was really fast. This puts it to shame from a perceived speed point of view. Coverflow for bookmarks is a nice touch and the smart guessing feature in the field where you enter the URL seems to be much better.

Download it now from Apple.

Maven Is Cooler than you think...

2009-02-06T16:44:00.001-08:00

I'm sure I'm not the only one who has heard people curse Maven. But Maven is cooler than you think. Back in the day when I wanted to start a project I always had to get a whole bunch of gunk setup before I even wrote a line of code. Especially when trying a new framework or tool. Today I was whipping up a new project for a simple micro-benchmark on some Terracotta stuff and it reminded me why Maven really can be quite awesome. It took me 10 minutes and about 7 steps. The next time around I won't need to do the installs and then it's 4 steps.

These were the steps I took to get started:

1) Install Maven

2) Used the Pojo Archetype to create the build and test environment for my project.

- Creates a Mavenized directory structure ready for build, test, run etc. Hooks up to Terracotta maven plugin as well.

- make sure you replace the group id and project id in the command line.

updated - with the latest eclipse plugin this is unnecessary

X 3) In my new project directory type: "mvn eclispe:m2eclipse"

- This takes your Maven project and readies it for eclipse

4) Install the Maven Eclipse Plugin (I already had eclipse installed)

- Makes dealing with Maven from eclipse much easier

5) Install Terracotta Eclipse Plugin

- Makes dealing with Terracotta from eclipse much easier

6) File-> Import-> Maven projects and import your project into eclipse

- Loads up the project directory created from the archetype into Eclipse

7) Select the project and hit Terracotta->Add Terracotta Nature

What you end up with here is a complete project setup ready to be built and tested from both Eclipse and the command line using Maven.

Literally took me about 10 minutes to get started. Notice what you didn't have to do.

1) Didn't have to build a pom.xml or other kind of build file

2) Didn't have to download or install Terracotta or any of it's pieces

3) Didn't have to think about your directory structures, where you want to put your tests, how you want to run those tests

4) Didn't have to figure out how to do all this stuff in Eclipse or the commandline

Sure, Maven can be challenging at times, but in cases like this, when the vendors have things setup for you, it can be a huge time saver.

update:

Looks like we've reduced the number of steps to 6 the first time and 3 after that. If we take the guy's idea about auto-applying the Terracotta Nature in archetype we could reduce it to 5 and 2.

Web 2.0 Scaled Out Reference App...

2008-11-10T06:32:00.000-08:00

It has struck me that not many good, scaled out, start to finish, Open Source Java reference applications exist for developers to learn from. Something that can demonstrate a modern and diverse stack of software handling large amounts of users.

Well check out the Examinator Web Reference App. It's goal is to demonstrate and document everything from "Build and test" to "Deployment" for a high scale, realistic, web application. It strives to be well written and as simple as possible while still being a good demonstration of best practices. It also strives to document all the relivant pieces so others can learn from it. This is quite a challenge and I'm sure dissagreement will exist on some of the choices. I think it's an excellent start and the right set of goals.

Here is a taste of the stack and tools:

Spring Webflow
Spring MVC
Sitemesh
Spring
Spring Security
Freemarker
Terracotta
JPA
Hibernate
Apache DBCP
MySQL
Apache mod_proxy
Tomcat
Jetty
Maven
Eclipse (+WTP)
Cargo
JUnit
HttpUnit
Selenium
Crosscheck

Click here to learn more about the choices and why they were made.

It's pretty good now and I'm pretty sure it's going to get better and better over time so check in early and often. I hope others can learn as much from reading about it and hacking on it as we did writing it.

JVM Wish List

2008-09-24T15:35:00.000-07:00

I'm sitting at the JVM Summit at Sun for the next few days. It is an interesting group of about 80 people mostly represented by language implementers who target the JVM. Then their are these two guys (me and tim) who work on clustering of the JVM. Seems that our wish list for improvements to the JVM is a bit different than what others here are looking for. I asked one of the speakers about the proxify stuff below and he shot me down like I was Dick Cheney's pal on a quail hunt. Anyway, what the heck, here's my list.

Proxify - In Terracotta we dynamically swap objects in and out of the JVM and back to a server in order to create a Virtual Heap. We do this in a way that maintains Object Identity. We rely on a bevy of tricks to do this but it would be much more efficient if we could point to an object (or more than one object) and say proxify (or if a become: call existed we could use that as well, see Smalltalk for more info). All references to the original object would now be pointing to a light weight proxy. If someone touches the thing we would then inflate it back to being a full fledged object.

Array Instrumentation - Currently we instrument arrays in java by instrumenting the classes that reference the array. This is a bit messy and expensive and forces us to instrument classes that we might not otherwise need to. It also forces us to do more magic than I like to associate a shadow object with the array. Would be great if we could make this go away.

Native Method Replacement - When we want to muck with things like System.arraycopy we can't just wrap and replace because native methods get unhappy with that approach. So we have to replace at the caller. Would be nice if we didn't have to do that as much like arrays this is messy and forces us to instrument extra classes.

JVM support for meta-level monitoring of objects and code - This is not a new idea but it would be simpler and more efficient if we could ask the jvm to callback on us when things like field changes, lock acquires, field accesses occur on objects we care about. Would also be cool if we could associate meta-data with an instance (shadow objects) and ask for stats on a live instance (for our memory manager to make really good decisions on when to proxify). We do all these things now but they are complicated and more expensive then they need to be because we do them at the JRE level.

Good Hot swapping - If we can't have any of the above, good Hot swapping would also make things easier. We could decide what to instrument on the fly to simplify config a little bit and it would enable some optimizations that are very difficult to do now.

Solve the Int Size Problem - Currently arrays and all collections in Java are sized by an integer. As JVM's go more and more 64bit and people start creating collections larger than a couple billion in size it sure would be nice if they didn't bump up against this limitation.

That's it, my quick and dirty brainstormed wish list. Maybe some JVM fairy God Mother will some day grant me three wishes and bring some of these to reality.

Who's Serious About Search...

2008-05-08T09:34:00.000-07:00

I was chatting with someone from Yahoo last night and I mentioned that while I use Yahoo as my home page and e-mail I almost always search on Google. I never gave it much thought but a reason must exist for this to be true. I remember a LONG time ago Google had better search results and that's why I used it. I suspect that little difference exists now. So, why?

My theory is that it has little to do with the search itself and everything to do with two things:

1) Accessibility - By far the biggest reason I use Google is because it is just flat out the easiest search to get to. It is in my tool bar on both safari and firefox on my desk and in my phone. Yes I know that the others are in there too but... Does anyone actually change from the default?

Question:
If Yahoo and others are serious about search how can they hand this slot that is SO important to Google?

A second part of accessibility is more subtle. Go to www.google.com and www.yahoo.com. On Google's site search is what you see. It is the only thing on the page aside from some fine print. This screams "I care about search". On Yahoo's site the page is so busy you barely notice search.

2) Marketing - Google managed to get people to refer to searching as Googling! Need I say more. Does anyone even know of another name for a Band-Aid?

A couple of side points. I like yahoo's home page (old not new) a lot. I actually like yahoo's e-mail better than Googles. It is entirely possible that they have intentionally taken a more balanced approach to the web and not focused on being number one in search. I have no idea. But... if someone wants to take on Google in a serious way, be RIGHT THERE when I want you. On my IPhone, on my desktop, in my browser. I don't "think" about search. It's a tool and I mostly grab the one near my hand that fits the general problem I'm trying to solve.

An Optimization For Garbage Collectors...

2008-05-06T13:50:00.000-07:00

For the last few days I have been thinking a lot about GC as Terracotta moves towards our first major rewrite of that subsystem. Lots of relatively large changes have been bouncing around in my head as I read papers, blogs and talk to people. Maybe I'll blog about those later but one pretty simple one occurred to me. I have a theory that most Shared Objects are actually only directly referenced by one parent object (I haven't run stats on this yet so I might be full of it). I started from wondering whether we could take advantage of this to improve the efficiency of GC. Here is what I came up with:

We can keep a Set of Object ID's for objects that only have one direct reference to it. We have an implementation of a compressed Set of ID's so this can be quite space efficient.

If an Object gets a second reference to it then it is removed from that Set
If that one reference is removed and in the Terracotta world that object is not reachable from a client or in a non-terracotta world it is not reachable from the stack then the object is garbage.
If an Object has no references but is reachable from the stack or is still on a client then add it to the no-refs Set so that when those two things are no longer true they can be marked as garbage or if the object is re-referenced it can be accounted for properly.
You can also recurse through the objects that the new garbage object referenced doing the same check.

One might be wondering, "Does steve think he just invented reference counting?" Nope, I don't and I haven't decided if this idea is any better than just having a first phase of garbage collection based solely on reference counting. I'm just theorizing that it might be. I don't even know that I invented this shortcut. Most real world GC's are hybrids of multiple approaches that best fit the set of restrictions and limitations faced in the environment. This one seems like it could improve things significantly in real world apps and potentially drastically reduce the required frequency of full GC's in our world without adding too much overhead.

Anyway, blast away :-)

More Advice To A Young Developer...

2008-04-24T09:55:00.000-07:00

I read Alex Miller's blog post on "Advice To A Young Developer" and it got me thinking. What general advice would I give to young developers. I came up with a few things. Hopefully they will help somebody.

1) Become your best by NOT being the best - This may seem obvious but the best way to be your best is to put your self in positions where your surrounded by people who are better than you on at least one dimension. This leads to point number 2.

2) Listen more than you talk - Use a bit of insecurity and a bit of wanting to be the best to drive you to suck the knowledge and experience from the people around you. Even the people you may view as less than you on some dimension or another will still teach you things if you listen. While your talking you are NOT listening.

3) Be Stupid, Stupid! - Sometimes being smart is a smart person's worst enemy. Don't ever use your intelligence as a crutch. Do research, talk to people, listen to people, read code, read books. In short allow yourself to evolve or risk not knowing how bad you are at things that you could have been great at. Watch out if you find yourself doing things from first principles all the time.

4) Slow Down - Of course a developer should work ones ass off. Almost all of the good ones do. But allow yourself time to think. Step back and look at what you are doing. In software it is very easy to rat hole on the wrong thing so take brakes, think, and get back to it. No more than 50 percent of your time should be spent typing (I actually think for an experienced dev it should be more like 25 percent).

5) Be A Tester - This is really for all Software Engineers, but... Be a tester, learn to write great automated tests that exercise at the unit, component, and system level. NOTE: This doesn't mean write tons and tons of tests. It means learn to write good ones.

6) Read your code - Software is complicated. Go back and read your code. You'll find all kinds of interesting things

7) Refactor - Refactor your code for readability, testability and maintainability. A myth exists that doing something in a messy poorly organized way is somehow quicker than doing it in a clean way. This is flat out untrue. I've actually timed myself. It takes longer to do things poorly. You are not saving any time by leaving bad code around. If it's a state machine write it as a clean well factored state machine. If you see a ton of nested if's extract methods, use null objects, what ever is needed.

8) Don't Guess - I first started with don't pre-optimize but I figured I would go one step further. Don't Guess! Write tests to show a perf problem before fixing them, don't guess about what features will be needed in the future, don't guess about whether your change fixes a bug. PROVE IT!

9) Trust Me - If you find yourself trying to win an argument by saying "trust me, I've done this or that before", or "I just know" then STOP. If you can't explain your point then you probably don't really have one so either figure out what the point really is or just admit your wrong.

10) Wishful thinking - Any decent list always has 10 items in it right? Anyway program by wishful thinking. Keeps you focused.

Reading back on this list it is completely all over the place. I have like 50 more ideas but this blog is already kind of preachy and pedantic so I'll stop :-). Hope someone can find some benefit from it.

Anyone else have ideas?

The New "My Yahoo" Homepage

2008-01-30T13:13:00.001-08:00

I have used my.yahoo.com as my home page for a long time. Here are my comments on the new version they are pushing lately.

- I'm not thrilled with the giant square advertisement in the upper left and most important space on the page. You can move it to the right side with the layout control but it still isn't my favorite. It's giant, square and flashy and for me a bit annoying as compared to the old layout.

- The left side blocks are too wide. I like to glance at that stuff but it doesn't have a lot of text and tends to not need the extra width. It is about 50 percent wider than it was in the old version.

- I like the idea of the personal assistant thing. It has mail, stocks and some less useful stuff in a very useful spot. My problem with it is that it grows on hover. Things that take major actions on hover kind of bug me. Mostly they just make me do the action by accident and add no real value.

- I'm not sure if it's a bigger font or more white space but it feels like I have to do a lot more scrolling to see my rss feeds.

Fear not, it's not all bad, the personalization stuff while lacking enough control is visually pleasing and trivial to use.

DZone Suggestion, More Comments

2008-01-28T13:08:00.000-08:00

I like DZone a lot. I read it regularly and post to it when practical. I like the rising links, I like the popular links, I even like the voting system. However, one thing that would make DZone just a touch better is to encourage people to comment on blogs when they vote. I personally learn from both the positive and negative feedback and anything that can be done to encourage that feedback is welcome. So, my suggestion is to pop up an optional "Add comment" field with proceed and cancel options after any vote on DZone. Anyone is free to not comment at that point but if someone has something on the tip of their tongue they might be more likely to say it. Our experience at Terracotta in building our .org website tells us that even subtle changes can have a broad impact on what people do. I would be interested to see this A/B tested and see if it increases comment volume.

Why I love and hate statics in Java

2008-01-24T22:16:00.000-08:00

I was chatting with some fellow geeks earlier this evening and it occurred to me that I've said to people that they should almost ALWAYS use static and also told people they should almost NEVER use static. Am I schizophrenic, a hypocrite, or just dumb. Maybe all three but it has nothing to do with this blog. I'm talking about two different language usages of the static reserved word.

USAGE 1, where the love is:
Inner classes. I hate non-static inner classes. IMHO non-static inner-class is unnecessary syntactic sugar that leads to hard to read code and subtle bugs. For those who don't know, non-static inner classes maintain a hidden instance variable holding a parent instance. It then uses specially generated methods to give access to the parent's private fields and auto-magically calls methods on the parent if no local method of the used name exists. I've seen this lead to memory leaks (people passing around instances of inner classes and not realizing that they are keeping around parents), all kinds of confusing issues with methods of the same name in inner and outer classes and variable problems of the like. On the occasions I use inner classes I almost always go with the static kind.

USAGE 2, no love here:
Static variables. With the exception of constants I have a strong dislike of static variables. Why you ask? When used to create various versions of singletons it leads to messy hidden code dependencies . It also makes it hard to do mock object stuff for testing, creates hidden initialization stuff and makes it difficult to create multiple environments in a single JVM. Just darn inflexible for no gain. I would go into details but this has been covered quite nicely here

Anyway, in summary, STATIC inner classes good, STATIC variables bad.
Goodnight ...

Saved by a Time Machine...

2007-12-19T20:00:00.000-08:00

In the last 4 months I have had the good fortune of twice having my hard drive crash on me on my Macbook Pro. The first time I lost of bunch of stuff but was lucky to be able to recover some data off of various other machines. This last time I was saved by Time Machine. I'll be taking my computer to the apple store tonight for another hard drive replacement but I already have access to all my files. I just connected my backup drive to my wife's mac and clicked the time machine button. Selected use previous configuration, and I was able to recover some of the directories and files I needed to her machine. It was very simple and effective. So for those of you who have had lousy backup experiences and have given up. I recommend giving it another try with time machine. It bailed me out this time.

First experiment with JDK 1.6 on OS X

2007-11-28T00:26:00.000-08:00

Ran into this article on JDK 1.6 on OS X from slashdot. I figured I would give it a whirl with Terracotta as an experiment. Turns out I just needed to hack the Terracotta OS version check in the ./common/src/com/tc/util/VendorVmSignature.java class because this jvm calls the OS Darwin instead of osx and then things just worked. Was kind of odd to see the JTable demo come up as an X window but things worked like a charm.

Using Web Clipping to Monitor DZone

2007-10-28T09:38:00.000-07:00

Now that I've installed Leopard I figured I would try out a few of it's new features. At first I didn't really get this new Web clip button on Safari. Turns out, it's really kind of useful. It allows one to choose a piece of a web page and auto magically turn it into a widget.

I like DZone's little rising links section on the top left of the home page so I thought I would try it out on that. It's pretty much so easy that even the least technical person should be able to handle this. Here are the steps:

Go to the web page that has a part that you like to watch. Some examples might be my.yahoo.com or dzone.com (the one I chose)
Click on the button in the top bar of your web browser that looks like a pair of scissors and a dotted rectangle. Should be between the refresh button and the add link button.
Move your mouse around and see what clips get highlighted. When one you want to use as a widget is highlighted just click on it.
Towards the upper right click the "add" button if you like what you have selected and you are done.
Now, when you go to your widgets it will be there and refreshed every time.

Enjoy...

No Java, Bye Bye Macbook Pro's...

2007-10-28T09:19:00.000-07:00

I have liked Mac's for as long as they have had OS X, didn't like them before that. Over 50 percent our Development/Field Eng team, have a mac. I've been accused of being an Apple fan boy on a few occasions. Unfortunately Apple is backing us into a corner. We develop Java infrastructure software and if Apple isn't going to take JVM on OS X seriously we will have to move off the platform.

http://www.cio.com/article/149200

http://javablog.co.uk/2007/10/26/apple-os-x-leopard-doesnt-have-java-6/

This would be very unfortunate for me, since I like my Macbook Pro, my team, because they like theirs and for Apple itself because we will stop spending thousands of dollars a year on Apple hardware. Hopefully they resolve this issue as I will be unable to purchase more Macs if this situation doesn't get resolved soon.

Goodbye Randy Pausch

2007-09-21T19:36:00.000-07:00

Don't need to be a geek to learn from this one. One of the best CS professors I ever had is going too pass in the next month or so at the way to young age of 46. He has inoperable pancreatic cancer. For those who had the luck to have met or learned from him and for those who have not, this is his must see last lecture.

The Lecture
WSJ piece on him

Also check out some of his projects:

Alice
A bunch of his papers

My best to his family and friends and may his legacy live on. I only wish I could take one more class with you at the helm.

3.5 Rules in Distributed Algorithm Design

2007-09-10T22:07:00.001-07:00

In highly concurrent distributed computing their are all kinds of algorithms one can learn. One can read books like Concurrent and Distributed Programming in Java and Distributed Systems: Principles and Paradigms. One can learn how to write scalable servers by reading things like these papers on SEDA. But when creating a distributed algorithm their are 3 simple Goals/Rules you need to follow. While these rules were developed with the Terracotta approach in mind they are mostly applicable to any distributed computing approach:

Use algorithms that allow for maximum concurrency

Seems obvious but design your algorithm to allow maximum concurrency. In a single JVM concurrency is important. In a distributed environment where lock acquire and release is almost certainly more expensive it is that much more important
Use read locks where possible. If you can have multiple readers look at something, duh, don't stop them.
Try strategies like striping, lock on fine grained objects etc (this is some of what concurrent hash maps do).

Minimize chatter

Many really cool concurrent algorithms are less good for distributed computing because of the amount of chatter (data that needs to be communicated between nodes).
Algorithms that require too much cross node book keeping are a problem. Networks are slow and relatively thin pipes. Chattiness plays into that weakness and can also eat cpu.

Take advantage of locality of reference

Use algorithms that partition well

Use algorithms that can mostly or entirely act on local data.

Scale-out architectures don't have all the data everywhere. They rely on various levels of caching both near and far for optimal performance. When hitting data try to hit data in the same node where possible.

Rule 3.5 exists as well. Remember not to guess at the performance of something. Test it, time it and automate as much as possible of those tests and timings.

This has been a public service announcement.

E-mail Sucks!

2007-07-29T23:03:00.000-07:00

E-mail is absolutely an awful way to communicate with people. Orion, who works with me at Terracotta, and I joke about this all the time. More lost information, mistakes, fights and miscommunication can be attributed to e-mail than almost any other cause I can think of. Here are a few of the problems with email communication that I’ve noticed over the years:

1. It is almost impossible to properly convey emotion in e-mail (so don't try).
2. Most people only skim e-mails because they get so damn many of them
3. E-mail is slow and problematic for conversations
4. E-mail is permanent.

Solution:

Here are some rules that I try (and fail sometimes) to follow.

1. Never try to express anything but the simplest of emotions via e-mail. You can't see the persons face, or even make judgments based on communication pauses or tone. So don't send emotion in e-mail.
2. Don't send e-mail if you are angry. (Alex miller pointed out that it is often helpful to write the angry e-mail but not send it. It's a good way to vent). Always make your e-mail twice as positive and/or emotionless as you think you need to. As they say on dragnet: Just the facts ma’am.
3. Don't assume that your e-mail was read. How many times have you heard, "But I sent an e-mail." Doesn't mean squat. Check in with people on chat, drop by, or call and make sure that they got/understood your e-mail. If you didn't get an ack (geek for response), you should assume that they didn't get it. Let me say it again. Don't ever defend yourself with "I sent an e-mail"
4. If you want to have a conversation, use a synchronous communication mechanism like chat, irc, talk in person or on the phone. E-mail is strictly good for relaying facts (and not even very good for that because they get lost in the ton of other e-mails you have. Try a wiki for facts).
5. If you send a dumb e-mail be assured that the receiving person has passed it around to a few friends to get a laugh or opinion. It is a permanent record of your stupidity (kind of like this blog is for me). So don't say anything in e-mail that you don't want to see on the cover of the NY times.

I must admit that I learned some of the above the hard way. I assume most people have received an e-mail that made their blood boil. You’re just dying to write up a response outlining what a jerk someone is. Well don't do it!

The thing to keep in mind is that there are a million ways to get yourself into trouble with email, but you almost never get yourself into trouble by NOT sending one.

A Job building Terracotta

2007-07-21T08:09:00.000-07:00

Opportunistic would be the best way to describe our hiring style at Terracotta. We always have our eyes open for the super sharp so I figured I would post about what we generally look for and see if anyone out there shows up to make more magic happen for us. I talked a bit about what I think people should look for when hiring in my blog on teams.

If you are:

Passionate - Do you actually love having complex problems to solve. Can you not sleep at night thinking about how best to design, factor, improve software. Are you always striving for improvement and learning.

Knowledgeable - I'm not going to list a bunch of frameworks here. The knowledge we mostly look for is about knowing how to solve tough problems. The one thing that is a must is you must know multi-threaded programming. Having experience building some kind of infrastructure software is a big plus. Whether it's building an App server, a database or jms system isn't so important but knowing how to build things that need to be scalable is a big plus (just to be clear, not using those things, building them). For some of our stuff knowing classloaders cold and byte code manipulation is a big plus as well. Most of all you must love and be good at writing software.

Respect/Teamwork - This is complex stuff and we are not a big company so you need to be able to talk to and work with others. No room for people who are pedantic and or self serving. It has to be about the product and the software for you not the title. This is true no matter what role you are looking at filling.

Judgement - Each person has to have excellent judgment. Must be able to focus on what's important. If you don't know something, that's not a problem but communicate, ask questions don't rat hole and don't hide problems.

Intelligence - Not going to lie. You must be smart. You should be a person who can analyze and design complex algorithms. Debug and solve complex problems. We will ask you to answer algorithm questions and write some code in the interview.

I know that's a pretty tough list but if you think what we do is cool and fit most of that please shoot your resume to careers at terracottatech dot com.

For those who don't already know about Terracotta here is a brief overview:

We are a well funded startup based in San Francisco but with developers all over the world. Our product is open source network attached memory for Java and is probably the most interesting and diverse product one could work on stretching from distributed computing, to byte code manipulation.

What were those results again?

2007-07-19T14:19:00.000-07:00

This is a small follow up to my blogs about anti-patterns. When trying to debug a complex logic or performance problem one of the most important things one can do is take notes. I had a conversation with someone the other day where the person said, "I'm in a rush, I don't have time to take notes about my runs." To this I replied, you don't have time not too.

So often we are in such a rush to solve a problem that we cut the wrong corners. When performance tuning, and/or tracking something down that requires multiple runs or configurations of your software always always always take notes on each run. They don't have to be super formal but you should write down all the details you can think of. Some examples include, what were my settings, what did the cpu usage/machine stats look like. What problems did I run into. Always keep a date/time stamp on the tests. This will prevent the inevitable rerunning of tests because you forgot the results, or mixing up what you have tried and not tried. It only takes one mistake to use up more time than tons of note taking would require.

What do I do when my iPhone gets hot?

2007-07-13T13:11:00.000-07:00

This will be a quick one. Since the gang at Terracotta has about about 5 iPhone users, I've had a few people ask me what to do when an iphone starts to get hot or the battery seems to be draining to fast. This is usually an indication of a rogue app. Same thing sometimes happens on my notebook. On my notebook I usually run the Activity Monitor and then kill the process that is using a bunch of cpu. On the iphone you just have to make an educated guess which app is the problem. It's usually the web browser or Mail for me.

Just hold the home key (The only button on the front of the phone) for 10 seconds with the app you want to kill open and it will get restarted. That will do the trick. This should lead to longer battery life and a cooler phone.

Good luck

UPDATED 4/29/2010:

I've noticed a number of people still hit this blog. I'm pretty sure that iphones now use that home button for screen capturing and the above advice won't help anymore.

So here is what to do now:

From the 3g on the thing to do is restart your iPhone by:

Hold down the power button (top of the phone) until the slider comes up asking you if your sure you want to shutdown (This can take about 10 seconds).
Just slide it to say yes and wait while it shuts down.
When it's done shutting down push the power button again to start up again.

Everything should work better now and the heat should dissipate. Good luck!

More Lies - Distributed Performance Testing Anti-patterns Part 3 of 4

2007-07-13T12:36:00.000-07:00

In part 3 out of 4 of this blog, much like parts 1 and 2 I will hit on anti-patterns that allow your performance testing of clustered and or distributed software to lie to you. I'll be following up part 3 of this blog, the last 4 anti-patterns, with a blog about a simple distributed testing framework I have begun. Hopefully enough of you will be interested, try it and maybe even contribute to it.

Anti-pattern 7: In-memory vs. Distributed Performance Comparison

Description

Writing a test that compares the speed of adding objects to a local, in-memory data structure vs. adding objects to a clustered data structure.

Problem

To avoid suspense, I'll tell you the results of that test without running it. Adding things to a local, in-memory data structure stakes virtually no time at all. In-memory object changes happen so fast, they are hard to even measure. However, when you are making changes to a distributed data structure, no matter what, those state changes have to be shipped off to another location. This takes instructions to be executed to make this happen on top of the ones used for the original task. This isn't just slower, it is way slower. The comparison between in-memory object changes and distributed object changes is useless.

Solution

Figure out how much data you are going to be clustering and what the usage patterns of that data becoming clustered will be. Then simulate and time that. Once again, focus on total throughput with acceptable latency.

Anti-pattern 8: Ignore Real-world Cross-node Patterns

Description

Reading and writing the same data in every node.

Problem

Generally speaking, whether reading or writing, it is more expensive to access the same data concurrently across all nodes. Depending on the underlying clustering infrastructure, this can be more or less of a problem. If you are using an “everything everywhere” strategy, the performance hit of random access across all the data on all the nodes is less, but the “everything everywhere” sharing strategy generally does not scale well. Most other strategies perform better when data access is consistently read and or written from the same node

Solution

Write your performance tests in a way that allows you to set a percentage for locality of reference. Is an object accessed on the same node 80%, 90%, or 99% of the time? You should usually have some cross-node chatter, but usually not too much—although you should be as realistic to the problem you are trying to solve as possible.

Anti-pattern 9: Ignore Usage Patterns

Description

The performance test either just creates objects or just reads objects

Problem

In the real world, an application does a certain amount of reading, writing, and updating of shared objects. And those reads, writes, and updates are of certain sizes.

Solution

If your app likely changes only a few fields in a large object graph, then that is what your performance test should do. If your app is 90% read from multiple threads and 10% write from multiple threads than that is what your test should do. Make your test be true to what you need when it comes to data and usage.

Anti-pattern 10: Log Yourself to Death

Description

Last, but far from least, doing extra stuff like writing data out to a log chews up CPU. Logging too much in any performance test can render the test results meaningless.
This anti-pattern generally covers any extra CPU usage on a load-generating client that affects the performance test. In general, if one or more of your nodes is CPU bound in a cluster performance test, you likely have not maxed-out the performance of your cluster. Let me say that again, if you are resource constrained on any node, including your load generating nodes (but not including your server if one exists) then you are probably not maxing out what your cluster as a whole can handle. Investigate further.

Problem

If the individual load-generating nodes—or even the clustered nodes—are resource constrained, it is likely to create a false bottleneck in your test. You are trying to figure out the throughput of the cluster and your cluster nodes are likely busy doing other things like logging.

Solution

First, always have machine monitoring on all nodes in a performance test. Any time one of the nodes or load generators becomes resource constrained make sure you test with an additional node and see if it adds to the scale. If a node is unexpectedly resource constrained, then take a series of thread dumps (java only) and figure out where all the time is going.

Alright, that is the end of my anti-pattern list for now. I could probably come up with a few more but I'll save them for another day. The moral of this section of the blog is to be curious and skeptical with your testing results. Don't just ask what the numbers are. Find out why and you will end up a much happier person.

Distributed Performance Testing Anti-patterns Part 2 of 4

2007-07-02T12:51:00.000-07:00

In Part 1 of this 4 part blog I hit upon 3 Anti-Patterns that can make one's performance testing a poor representation of reality. Here I'm covering 3 more and will be following up with the last 4 in a few days. After that I'm going to talk about a simple distributed performance testing framework I'm going to give away to try and help people be more successful with this stuff.

Anti-pattern 4: Fake Data Fake Performance

Description:

Using data in a distributed performance test that looks nothing like your real data.

Problem:

Distributed computing solutions use all kinds of strategies to move data between nodes under the covers. Just representing a size of data to be shared ignores those strategies and in many cases misrepresents the performance of a real system with real data under real load, both positively and negatively. You may be testing specially optimized flattening tricks that make the system look faster than it is; likewise, you may be testing a particular case that doesn’t perform well, but that isn’t representative of the true performance of the system with real data.

Solution:

Make sure you test with object graphs that vary in size, type, and depth in similar ways to the data you plan to use in your application. Don't assume Maps of Strings will behave anything like the way real object data will behave.

Anti-pattern 5: Incoherent Cluster

Description:

Some clustering products are coherent, some are not, and some have both modes. Don't ignore whether you are testing the performance using the mode you really need for your application.

Problem:

While it is quite possible to have a coherent cluster that has the same throughput as an incoherent cluster, it is certainly harder to do. Coherently clustered software frameworks require the provider to do some fancy locking, batching, windowing, and coherent lazy-loading tricks that aren't for the faint of heart (in the internals of the clustering engine, that is, not for the application developer). You can't assume that performance between a coherent and incoherent clustering approach will be the same.

Solution:

Make sure that if what you need is coherently clustered data that you are actually testing that way. Also, if it’s coherence you’re after, it’s a good idea to verify the end-state of a performance test to make sure the system actually is coherent. Sort of post test verify phase.

Anti-pattern 6: The World by a Thread

Description:

Distributed tests that only use one thread per node.

Problem:

For most clustered software, the name of the game is throughput with acceptable latency. Pretty much all distributed computing software does batching and windowing to improve throughput in a multi-threaded environment. Maxing out a single thread will usually not even approach the max throughput of the JVM or the system as a whole in the same way that a single node will not.

Solution:

Make sure your test uses multiple threads for generating load in each JVM. Check to see if you are cpu bound on any node. If you are not cpu bound you might have a concurrency issue or just need to add more threads.

Conclusion:

I have 4 more anti-patterns that I'm going to publish next week. Keeping an eye on the full 10 will help greatly reduce mistakes in clustering and distributed computing. Once again I'll then be following up with a framework to help develop and run useful tests.

Why Your Distributed Performance Tests Are Lying to You: Anti-Patterns of Distributed Application Testing and Tuning - Part 1

2007-06-27T13:32:00.000-07:00

Clustering and distributing Java applications has never been easier than it is today (see Terracotta). As a result, writing good distributed performance tests and tuning those applications is increasingly important. Performance tuning and testing of distributed and/or clustered applications is an important skill and many who do it can use a little help. Over my next few blogs I'm going to cover a series of anti-patterns in this area. I'll be following it up with a simple open distributed testing framework that I hope can help people out (hint, hint, the testing framework itself is distributed to best test distributed apps).

Here are the first 3 anti-patterns...

Anti-pattern 1: Single-Node “Distributed” Testing

Description

Running your “distributed” performance test inside a single JVM.

Problem

Depending on the framework, this can tell you either: 1) nothing, because the clustering framework recognizes it has no partners so optimizes itself out or 2) very little—it might give one an idea of maximum theoretical read/write speed for that framework.

Solution

When trying to evaluate the performance of any kind of clustering or distributed computing software, always use an absolute minimum of 2 nodes (Preferably more).

Anti-pattern 2: Single-Computer “Distributed” Testing

Description

Putting all (or just too many) of the resources for a performance test on one machine.

Problem

This has two problems. First, distributed applications running on the same machine have different latency and networking characteristics than distributed applications on different machines. This can hide various classes of problems around pipeline stalls, batching, and windowing issues.

The second problem is a variation on another anti-pattern I will discuss later around resource contention. By running multiple JVMs on one machine you are now contending for CPU, disk, network, and potentially affecting context switch rate, etc.

Solution

The only real way to test a distributed application is to run it in a truly distributed way: on multiple machines. If you must have multiple nodes/JVMs on one machine, make sure you are running one of the many resource-monitoring tools and make sure you aren't resource constrained (I use iostat/vmstat for simple tests).

Anti-pattern 3: Multi-Node, Load Only One

Description

Testing with multiple nodes but only sending load/work to one of those nodes while leaving the others just hanging out doing little or nothing

Problem

Depending on the distributed computing architecture chosen, the nodes that are not receiving load may be actually doing a lot of work. If that's the case, only loading one of the nodes is giving a false sense of performance. Also, in some cases, data is lazily loaded into nodes so only putting load on one node could be putting you in the same boat as the single-node tester where no actual clustering is happening.

Solution

When testing clustering software, make sure you are throwing load at all nodes.

Be sure to check back soon as the next few anti-patterns will cover the data aspects of distributed performance testing...

Latency v Throughput

2007-06-14T13:35:00.001-07:00

Which is the faster way to get your cargo across the United States. A plane or a train? Some might think the answer is obvious. A plane travels 500 mph (or so) and a train does maybe 80 mph. Therefore the plane is faster. Or is it? The question is really a matter of latency vs. throughput.

Imagine you have to move a bunch of coal across the country and deliver it to a coal processor. Now say that on the west coast, the receiver of the coal can process 100 units of coal an hour. You have 1 train that can haul 10,000 units of coal and takes 48 hours to get to its destination. You have 1 plane that can deliver 100 units of coal in 12 hours.

If the most important thing was to have the coal soon, then the plane is faster (lower latency). But, if the most important thing is to have the coal-processing pipeline filled on the west coast over time then train is faster (higher throughput). Every 96 hours they get 10k units of coal with the train (remember there’s only one train and, just like the plane, it must make the return trip to the east coast). That works out to about 100 units an hour which is just what you need. With the plane, every 96 hours you get 800 pounds of coal. Not nearly fast enough.

The above discussion may seem obvious but I have this conversation all the time when talking about Software: what is fast and what is slow. I've had people tell me it's impossible to do 10 thousand transactions per second in Terracotta when persistent because the disk seek time is 10 millis. Well they would be right if you serialize things. But in infrastructure software, the game is throughput with acceptable latency and it turns out 10 thousand transactions per second isn't all that hard. With parallelism, batching, and windowing, the disk isn't even usually the bottleneck.

Anyway, just wanted to get the throughput v latency thing off my chest.

Now that's fast...

2007-06-12T19:00:00.000-07:00

Alright, I promise I'll get back to blogging about Java and Terracotta stuff next time but... I've been reading a lot of negative press about Apple's Safari 3 beta and while some it is fair I haven't seen a lot of talk about the good stuff about it. So before people flame me let me start with:

Yes, I know it has security holes and those need to be fixed
Yes, I know it has some bugs (like it doesn't work with Zimbra for me)

But...

It still has all the features from Safari 2:

reset browser so when your surfing on someone else's computer you can clean up everything you've logged into like e-mail.
Really good tabbed browsing
Private browsing (for when you want to pause the caching and recording of what your browsing)
plus RSS, popup blocking and the other usual suspects.

Good new stuff in 3.0:

This thing is blindingly fast? I haven't taken actual timings but just from eyeballing it this thing is super fast. It is much faster than what was already fast Safari 2.0. And much much faster than Firefox. I don't have the time to do real benchmarks on this but I would love to see some.

Much improved inner search. How many times have I hit command F, typed some text, seen the window move but not be able to find where the highlighted word is. Safari does a really nice animated bubble highlight that is impossible to miss. Kudos to the Apple guys for simple subtle improvements.

Plus I think it is supposed to be more standards compliant and it has this resize textbox feature which I haven't tried yet. update: Works great but worth noting that it only works for multi-line test fields not the single line variety.

update2: Someone pointed out that Safari3 also now has WYSIWYG editor support. Should have mentioned it since I used that when writing the blog :-)

Anyway, don't want to sound like a fanboy boy, and I might be alone, but I actually like Safari 3.0 and think windows users should give it a go to.

Why Ning is the thing...

2007-06-07T19:43:00.000-07:00

Ning, a website that allows one to create their own social networking website (a meta site for those geeks reading this). It is my pick for the sleeper coolest thing I've played with lately. I peeked at this a year or so ago because I wanted to run a social network for my wife's sports league. Back then I thought it needed more bake time to be useful (putting it nicely). Well, while not perfect, it is quite nicely baked now. After playing with it for less than an hour my brain was teaming with cool ideas on how I can take advantage of this site.

First thing I did was create a social network for my extended family. Everyone posts pictures and vids and it's so simple an adult could do it. Next I got my wife to create that social network for her sports league. She used the business version so that she could use adsense with it and maybe make a few bucks :-). Next on my list is to create one for my kids pre-school. This thing is flat out addictive. It even has an api and gives pretty low level access to things like css. One can pay a little extra and have a custom domain name and remove all references to Ning.

Support is good do. I sent a suggestion that I wanted a calendar for my family on ning and they got back to me in under 24 hours that one was on the way. Hey maybe someday they will cluster with Terracotta ;-).

Anyway, when I played with this way back when I was a little sceptical but now I'm sold. Kudo's for the team at Ning for coming up with a cool idea and executing on it well.

update: One person pointed out to me that they didn't feel this entry was particularly developer focused and that it is more of a review of a website (not their is anything wrong with that). I see that point but I believe I am not reviewing a website but am reviewing a development tool to a certain extent. Ning is a meta-website. Anyway, dig in and tell me if I'm crazy.

Performance Architecture

2007-05-19T11:32:00.000-07:00

I was reading this article from another guy named Steve Harris (not me). I think he makes some interesting points but I thought I would follow up. Optimizing architecture to me mostly means designing for flexibility and testability. It also means having a way to test ones app for performance and scale. I've seen a number of people write apps, put them in prod without ever performance testing them.

Write your app so that you can overlay the performance architecture after the fact. Have a way to EASILY validate your performance and you'll be ok. Optimizing code early (which he doesn't mention) is actually quite evil and almost always leads to badness.

Teams, starts with the people

2007-05-19T01:30:00.000-07:00

What are the needed traits across a company to have the greatest chance of success? How much of each trait does each individual have to have? These are questions that everyone should ask themselves on both sides of the interview process and in ones everyday life.

My list is as follows (not in any order):

Passion/Pride/Work ethic
Knowledge/experience
Respect/Teamwork
Judgement
Intelligence

The list itself can vary though probably not as widely as how much one values each item on that list. The most important thing is to have the list and to use it as a framework for how you choose jobs, hire, value and motivate existing people, and at times part ways.

Since words tend to have very different meanings to different people despite the existence of dictionaries and wiki's I'm going to cover a bit what my list means to me.

Passion/Pride/Work Ethic

First let me start with what this doesn't mean. Passion and pride does not have anything to do with outward energy. One does not have to be "Rah Rah" to have passion and pride. Many of the people who I feel have had the most passion have actually been super quiet. Passion is about commitment to the task at hand and the greater goal. It's about going the extra mile to get things done well, on time, and sometimes in ways that nobody else thought of. It's about finding the things that need to get done that nobody noticed and being the person to do them while not dropping the things they are supposed to be doing :-).

Knowledge

Knowledge is what you "know" coming into a problem. Intelligence is what you can learn/how well one solves problems. In many cases knowledge is the least important of the five. For given positions one should always give some thought to how much specific knowledge is actually needed. I constantly see job postings with a long list of knowledge items and almost no indication of the other 4 items importance. IMHO If you can find people who excel in the other four areas, knowledge of a given specific task can be lower (though not zero. Working with people who have zero knowledge of their domain can be very expensive. It's less of an issue for large companies with time and money).

Respect/teamwork

I focus on respect more than one might imagine. I worked with a guy who said he spends 75 percent of his time in an interview trying to figure out if the person will work well in the team. I think that's a good idea. It's not about whether the person is social or fun. It's about whether they present their ideas with an understanding that those around may have better ones. Do they listen to feedback, even from people they do not expect to have good ideas. Do they listen in general or do they just wait for their turn to talk. This is important stuff. Don't underestimate it.

Judgement

This is another really important one. Their is a nice coverage of how important this is in the book "Producing Open Source Software.(A free version of this book exists)" Whether someone is your boss, your co-worker or your employee, you need to be able to trust their judgement more than anything else. If the people around you are smart but have bad judgement they will create excellent solutions to the wrong problems. If people are not smart and have bad judgement they will create poor solutions to the wrong problems. On the other side of the coin, if they are smart and have good judgement they will create great solutions to the right problems. Most importantly, if they are not smart (and on any given day all of us are not smart) but have excellent judgement they will still make good decisions about what to do, and most usefully, what not to do. While I find it really really hard to interview for, Judgement is huge!

Intelligence

Having some Intelligence is important. More important for some jobs than for others. When hiring, know how much intelligence (balanced against the others) is needed for the position/type of work. For some jobs, judgement and hard work with some intelligence is what is needed. Have questions in mind that will indicate whether a person is smart enough. Since intelligence is the easiest of the bunch to test for Don't Forget to Do it! To many people mistake liking someone, "talking the talk" or even looks for intelligence. Do the work, have your intelligence questions in mind when you interview someone.

The above are some high order bits. When I again get some time and motivation I want to cover team building from a balance perspective. Recognizing and filling ones personal and team gaps.

Ok, it's 2:30 am and I'm starting to question my judgement for being up so late. What's my point here? Think about what you want in employees coworkers and bosses. Sometimes even assign numbers to the core values you look for so you can compare people.

I read that we make up our mind about people we meet in seconds. Before they even speak in most cases and that those first impressions are VERY hard to overcome. So have the tools ready to make wise informed decisions and fight past human nature.

The int size problem...

2007-04-05T10:13:00.000-07:00

I'm just curious. Of the people out there using 64bit jvms, has anyone run into the int collection size problem? One of the things I've been quitely worried about for awhile now is that with both 64 bit jvm's and the virtual heap stuff provided by Terracotta DSO it is possible for an ArrayList or a HashMap to grow beyond what an integer can hold. Sure 2 gig of objects is a ton to be in a collection but if one has a 128g heap it isn't crazy anymore.
It's easy to make the collections able to handle the larger sizes but so much code is written to the current collection interfaces it might be a problem. Have the smart people at Sun started tackling this already? Maybe some JSR?

Motorboard 2000x First days...

2007-02-08T19:55:00.000-08:00

Overview:

While most of my posts tend to be about software I recently got a new electric scooter for commuting to work (shameless plug for the coolest new software product in years!) so I thought I would blog about it. After doing a bunch of research it came down to a choice between the goped 750 sport and what I ended up choosing the motorboard 2000x. The goped seemed a bit more powerful/rugged and the motorboard looked much much better and was less than half the weight. Since I plan to use this thing to commute around a mile each way to work on flat ground and I wanted to be able to easily dump it in the trunk of my car I went the cooler looking lighter route (also is cheaper 399 v around 599 but that wasn't a strong factor as neither were as expensive as all the parking tickets I was getting).

First Impressions:

I've had this device for two days so I will update this as I learn more.

It looks even cooler in person than it does on the website. At first glance you wouldn't even think it is electric powered. So I'm happy about that.
No setup at all really. Took it out of the box and had it together with no tools in about 1 minute.
The battery stuff is a bit of a pain in the butt. It actually matters whether you plug the charger into the wall first or into the device first. I messed this up because, unlike the rest of the world, I didn't carefully read what was written on the charger.
I can't ride it to and from work yet because the ground is wet from rain. That is a bit annoying but I knew about this limitation before I bought it so I can't complain.
I got it all charged and tooled around our office with it. It's fast and smooth. Really fun to ride.
When people in the 200 pound range from my office tried it they would often get the dreaded blinking red light. I think this could be a combo of them being close to the weight limit and riding on low carpet putting strain on the device.
Did I mention all the rules around charging this thing are annoying?
I live in a building so I took it into the garage and rode around for about 1/2 an hour. It was fun and the thing was smooth and worked great. It is nice and fast. It didn't make it up the hill between the first level and second level of our garage but that hill is a bit steep so I can't fault it to much.
Another thing to note is that both before and after my purchase I sent support questions via e-mail and they always got back to me within 24 hours. In our world of automated phone systems and outsourced support I was pretty pleased with this.

Conclusions:

So far I am happy with my purchase. I haven't seen anything else I would have rather had. If these guys can get the electronics to be a bit less sensitive this thing would be the greatest thing since sliced bread. I'll update when the rain in SF stops...

Programming by Wishful Thinking

2007-01-27T17:15:00.000-08:00

I'm currently reading an excellent book called Everyday Scripting. I'm really enjoying it as a nice primer on whipping things up with Ruby. When I hit chapter 7 the author began to discuss a programming style that I have used for a while now. I had no idea this simple technique had a name. Many of the things I've picked up throughout my career I've learned from books, articles, reading code, and still others I've devised from first principals. This particular practice I picked up while pair programming with this super smart guy Peter Seibel. Apparently it's called "Programming by Wishful thinking."

The idea is that when you have some code to write you first do so as if methods existed already to perform the high level operations. Say you want to write some code that tells you how many songs and albums you have by a given artist. Programming languages don't have methods do to this baked in but you pretend they do. An extreme example of the practice would be:

I came up with this example in about thirty seconds so I hope it isn't to opaque.

You may then go on to write the above methods in the same way:

and so on and so on. Anyway, don't know how many people out there use this style but I've found it really helpful when I need to keep focused on the task/flow at hand without getting to distracted by the details of an issue. Maybe it will help someone else too.

Wisdom and Terracotta... Good mix

2006-12-19T14:30:00.000-08:00

This guy wrote about two topics that I like. The book wisdom of crowds and Terracotta DSO:

http://blogs.techrepublic.com.com/opensource/?p=31

Pairing can make you stupid...

2006-09-26T11:16:00.000-07:00

Pretty much everyone has heard of or tried pair programming. I personally have found it to be quite valuable. One very interesting thing I was reminded of when watching some people on my team do some pairing is how dumb pair programming makes the typer. Conversely, it also makes the watcher (don't know what it's really called) considerably smarter.

I can vividly remember pairing with someone much smarter than me on a transaction logging system a few years back and noticing this effect. When he was typing, I was always coming up with the cool ideas. Of course it's not always an intelligence thing. Around the office I have noticed multiple pairs demonstrate this. The typer becomes dump. The non-typers make excellent judgments and decisions.

I think the obvious explanation for this is just how much energy we put into things like typing, picking variable names, formatting, etc... I've often joked that we should hire secretaries with a little programming knowledge for each programmer.

Has anyone else out there noticed this? Is their anything to be gained from this observation? Or am I completely off base.

Performance Tuning with Thread Dumps

2006-09-26T03:12:00.000-07:00

As a follow on to my earlier tuning blog I whipped up a quick sample to make the Performance tuning with Thread dumps point clearer (see my previous blog). Take a look at the following code and make a guess at what you think the slowest part of the code is and why.



package softwareandstuff.performance;

import java.util.ArrayList;
import java.util.List;
import java.util.Random;

public class SomethingToRun {

 public void run() {
   List stuff = new ArrayList();
   Random r = new Random();
   long total = 0;
   long count = 0;
   while (true) {
     long start = System.currentTimeMillis();
     String s = " " + r.nextInt() + " " + r.nextInt();

     for (int i = 0; i < i =" 0;" 1000 ="="">

On my slow computer it took around 43 seconds per 1000.

Now use jps to find out what process id the little app has and do a "kill -3 " 10 to 20 times fast in succession.

Note: jps requires jdk1.5 to be installed. Figure out how to use task manager in windows or ps on unix based platforms if you are not on jdk1.5


steve:~ steve$  jps

7595 Jps
7594 SomethingToRun
7440 startup.jar

steve:~ steve$ kill -3 7594

Note: See the following link for more on how to do thread dumps and what they are A little on thread dumps

Should see something like the following:



Signal Dispatcher" daemon prio=9 tid=0x005098d0 nid=0x1820a00 waiting on condition [0x00000000..0x00000000]

"Finalizer" daemon prio=8 tid=0x00509030 nid=0x180fc00 in Object.wait() [0xf0a04000..0xf0a04ab0]
at java.lang.Object.wait(Native Method)
- waiting on <0x266101e0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:116)
- locked <0x266101e0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:132)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

"Reference Handler" daemon prio=10 tid=0x00508c40 nid=0x180ec00 in Object.wait() [0xf0983000..0xf0983ab0]
at java.lang.Object.wait(Native Method)
- waiting on <0x26610260> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:474)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
- locked <0x26610260> (a java.lang.ref.Reference$Lock)

"main" prio=5 tid=0x005013b0 nid=0x1804600 runnable [0xf07ff000..0xf08002c0]
at softwareandstuff.performance.SomethingToRun.run(SomethingToRun.java:25)
at softwareandstuff.performance.SomethingToRun.main(SomethingToRun.java:36)

"VM Thread" prio=9 tid=0x00508460 nid=0x1803c00 runnable

Note: ignore any thread that is in Object.wait() for now. In fact, searching on runnable is usually your best bet for skipping to the things that are interesting from a performance perspective.

My guess is what you will see is that a decent percentage of the time is spent in the remove call. hmmm, wonder why that is. Now go look at the implementation of remove on an ArrayList and it will all be clear. Now try using a linked list instead. For me that dropped it down to 6 seconds per 1000.

busy today but more to come...

Performance Tuning Tips for Java - No tools required

2006-09-23T18:15:00.000-07:00

Over the years I've developed a simple methodology for doing performance tuning. To tell the truth, I didn't even know it was a methodology until I realized that whenever people said "The software is slow!" I always responded with the same actions. In this article I'm going to talk about what I now fondly call, "Performance tuning by Steve" (Can you tell I've seen one to many infomercials in my day?)

First things first:

Someone comes to you as a developer and says, "The software is too slow." They usually have between 2 and 5 guesses as to why it's slow and maybe so do you. What do you do? I'll tell you what you NEVER do. You NEVER take action based on a guess of what the problem is. The only thing worse than an outsiders guesses as to why your software is slow are your own guesses. Do the investigation. I can't tell you how many times I hear people insist that the reason some piece of software is slow is because of networking, disk or database even though the machine is cpu bound. Before I touch a line of code I always ask these two basic questions.

1) Are you CPU bound

(if it is a multi process test I ask if they are cpu bound on any of the boxes involved in the test). If it is a multi cpu machine I ask if any one processor is pegged.

Answer I usually get is "I don't know." Ahh, I say. Can you run your test again with vmstat (translates to iostat on the mac and that Perfomance thing in the task bar on Windows). Don't forget to tee the output to a file.

2) Are you GC bound?

Answer I usually get, "I don't know." I usually follow with. " Can you run your test again with all processes having some variant on the -verbose:gc flag turned on. And don't forget to tee it to a file. If they are GC bound then one needs to do GC/Garbage creation tuning which I will save for another day. It is a big topic on its own.

If they come back and say, "Yes I'm cpu bound! and no I'm not spending tons of time in GC." My mouth waters a bit. I scratch my chin gently and say, "Great! can you take a series of thread dumps while your app is running." At this point I get a funny look. Why? "I don't think it's deadlocked" They usually say. Well this is an extremely effective way to see where an application is spending it's time. In my experience this is more effective than even a good profiler. If you run a load test that is cpu bound and take a series of thread dumps you will find out where your code is spending time. For a little more on thread dumps check this out. A little on thread dumps

I usually perform analysis on the thread dumps, fix the thing that is the worst offender than try the test again. Rinse dry repeat...

Now you are not CPU bound anymore. What next? I'll tell you what still isn't next. DON'T GUESS! you will be wrong. I promise!
The trick here is to find the bottleneck. It is best to think about a non cpu bound performance problem as thread starvation. Not a network problem and not a disk problem (at least at first and for this their are exceptions).

If you are using SEDA style development (think work queues) then your lucky. The next thing you have to do is just find out what queue is backing up. Really, any statemachine style programming can use this technique.
- Which ever queue is backing up is where your performance problem is.
- You may need to add more worker threads to the end of the Queue. Especially if the threads are doing blocking tasks of any kind.
- You may need to do a better job of separating out CPU tasks from blocking tasks.
* If you have a lot of disk writing to do than the thread that writes to disk should do almost 0 cpu work otherwise you will get pipeline stalls going to disk. Big mistake.
- You may need to more effectively synchronize something if your threads are all blocked waiting for the same resource.

If you are not doing SEDA style development or if you are but you have found the queue that is backing up but don't know why yet then here is the next fun step.
Warning, it is time consuming but very effective.
Wrap large swaths of your slow code like so:


long start = System.currentTimeMillis();
.
.
-your code-
.
.
long t = System.currentTimeMillis() - start;

count++;
total += t;
if(count % 1000 == 0){   //<-- obviously the number you divide by here is dependent on how often the code is called
 System.out.println("T1 Average: " + (t/count)+" count:"+count); 
}

Do this in multiple parts of your code and then basically keep moving the timing in tighter and tighter until you have found the part or parts of your code that are taking the longest. This takes some practice. You'll probably get fooled by thread starvation a few times. Maybe get fooled by some blocking calls, ie. network or disk. Keep your mind clear. Try and draw on a white board the hand offs between threads.

One last thing. TAKE NOTES, lots of NOTES. Save the output from every run and make a little readme that reminds you what each experiment was and even any thoughts you had during said experiment.

Summary:

1) Figure out if you are CPU bound
2) Eliminate GC as a concern (tuning GC is a whole other article)
3) If you are cpu bound use thread dumps to figure out where time is being spent (ignore things in wait, only runnables)
4) If you are not then do the following:
- Still take thread dumps to understand where threads are blocked and where they are runnable
- Bottleneck analysis, find out where things are backing up moving through your state machine
- use timings over many calls of each code path and then move those timings in on the areas that are slow until you figure out where the problem is.
5) Take lots of detailed notes and save all artifacts of each run.

Good luck