The Blog-Health-o-Meter™ reads Fresher than ever.

Crunchy numbers

The average container ship can carry about 4,500 containers. This blog was viewed about 15,000 times in 2010. If each view were a shipping container, your blog would have filled about 3 fully loaded ships.

In 2010, there were 3 new posts, growing the total archive of this blog to 14 posts.

The busiest day of the year was March 10th with 108 views. The most popular post that day was Hibernate, Spring and multiple datasources (Part 1).

Where did they come from?

The top referring sites in 2010 were stackoverflow.com, en.wordpress.com, google.co.in, swik.net, and blog.smart-java.nl.

Some visitors came searching, mostly for hibernate multiple data sources, spring multiple datasource, spring hibernate multiple data sources, spring hibernate multiple datasources, and opensessioninviewfilter.

Attractions in 2010

These are the posts and pages that got the most views in 2010.

Hibernate, Spring and multiple datasources (Part 1) March 2009
9 comments

Hibernate, Spring and multiple datasources (Part 2) March 2009
4 comments

Hibernate, Spring and multiple datasources (Part 3) March 2009
6 comments

Hibernate inheritance ClassCastException (Part 1) February 2009
1 comment

Hibernate inheritance ClassCastException (Part 2) February 2009

http://sysin.files.wordpress.com/2011/05/introducetion-to-ddd.pptx

Checkout http://dynamicquerygen.sourceforge.net/ for more details

Spring comes with an inbuilt support for the some of the popular frameworks in the ORM world including Hibernate, JDO, TopLink, Ibatis and JPA. Spring provides a template for each of these frameworks and also a base Data Access Object which your DAO layer can extend. The Hibernate template can be used either directly in the application layer or in conjunction with the DAO. This template acts like a wrapper over the Session and abstracts all the session handling code from us. So how about transaction management? Spring provides different flavors of transaction management, you can use TransactionTemplate and define the transaction scope or use the HibernateTransactionManager to define the scope or annotations or AOP to define the scope. This means that the underlying orm is completely abstracted from the application layer and you can at any point switch to some other framework.

I don’t wanna repeat how to use these features with Hibernate (trying to follow DRY ) as there are hell lot of articles in the net. There’s nothing much in this post as the aim of this post is to just kick start a discussion on the much talked about Domain-Driven-Design (DDD). I’m in the process of coming up with an example for DDD using Hibernate and Spring in the middle tier. Keep reading my posts for updates. Thanks !!

One typical bug (user induced and not hibernate) that a beginner would face is with composite key associations. Say that you have a couple of tables linked by a foreign key, and the foreign key is the primary key of the second table. Now, while defining your many-to-one or one-to-many associations in the mapping file, you gotta be careful with the key ordering. The keys should be in the same order as that of what is defined in the composite-id of the second table’s mapping file. What hibernate does is, applies the values to the keys to the prepared statement in the order the keys are defined in the associations as long as the type is same.

Take a look at the below association mapping,

<class name="Entity1">
	<id name="id1"/>
	<property name="property1" column="property1" type="long"/>
	<property name="property2" column="property2" type="long"/>
	....
	....
	<many-to-one name="entity2" class="Entity2">
		<column name="property1"/>
		<column name="property2"/>
	</many-to-one>
</class>

<class name="Entity2">
	<composite-id name="id1">
		<key-property name="property1" column="property1" type="long"/>
		<key-property name="property2" column="property2" type="long"/>
	</composite-id>
</class>

This would work perfectly fine as the key ordering in the many-to-one is exactly same as the second entity. But when you change the order say, property2 as the first and property1 as second in the association, Hibernate subsitutes the value of property2 to the property1 and vice versa. The outcome is not an exception but rather an empty result set and its very very tough to debug this issue. Believe me I spent nearly a couple of days without getting any clue on whats happenning and had to debug line by line of Hibernate code to check what it’s doing internally.

• You can reduce the number of connection pools
• The number of session factories can be limited as the pools have been reduced.

The first thing we have to do is to make the DataSourceIdentifier return the catalog,

public abstract class DataSourceIdentifier {

	protected String dataSourceId;
	protected String dataSourceCatalog;

	public String getDataSourceId() {
		if(dataSourceId == null) {
			resolve();
		}
		return dataSourceId;
	}

	public String getDataSourceCatalog() {
		if(dataSourceCatalog == null) {
			resolve();
		}
		return dataSourceCatalog;
	}

	// Add the partitioning logic here
	public abstract void resolve();
}

Then we have create one session factory (assuming that all partitions reside in the same db instance otherwise create one per db instance). We have to create a custom session factory bean to make sure the correct catalog is set before the connection is used by the session.

public class CustomLocalSessionFactoryBean extends LocalSessionFactoryBean {

	@Override
	protected void postProcessConfiguration(Configuration config) throws HibernateException {
		String connectionProvider = config.getProperty(Environment.CONNECTION_PROVIDER);
		if (connectionProvider != null) {
			if (connectionProvider.equals(TransactionAwareDataSourceConnectionProvider.class.getName())) {
				connectionProvider = CustomTransactionAwareDataSourceConnectionProvider.class.getName();
			} else if (connectionProvider.equals(LocalJtaDataSourceConnectionProvider.class.getName())) {
				connectionProvider = CustomLocalJtaDataSourceConnectionProvider.class.getName();
			} else if (connectionProvider.equals(LocalDataSourceConnectionProvider.class.getName())) {
				connectionProvider = CustomLocalDataSourceConnectionProvider.class.getName();
			}
			// Set Spring-provided DataSource as Hibernate ConnectionProvider.
			config.setProperty(Environment.CONNECTION_PROVIDER, connectionProvider);
		}
		super.postProcessConfiguration(config);
	}

	@Override
	protected void afterSessionFactoryCreation() throws Exception {
		SessionFactoryImplementor implementor = (SessionFactoryImplementor) getSessionFactory();
		ConnectionProvider provider = implementor.getConnectionProvider();
		if(provider instanceof CustomConnectionProvider) {
			((CustomConnectionProvider)provider).setFactory(implementor);
		}
		super.afterSessionFactoryCreation();
	}

	public interface CustomConnectionProvider {

		public void setFactory(SessionFactory factory);
	}

	public static class CustomLocalDataSourceConnectionProvider extends LocalDataSourceConnectionProvider implements CustomConnectionProvider {

		private SessionFactoryImplementor factory;

		@Override
		public void setFactory(SessionFactory factory) {
			this.factory = (SessionFactoryImplementor) factory;
		}

		@Override
		public Connection getConnection() throws SQLException {
			Connection connection = super.getConnection();
			DataSourceIdentifier dataSourceId = ContextHolder.getDataSourceId(this.factory);
			if(dataSourceId != null && dataSourceId.getDataSourceCatalog() != null) {
				connection.setCatalog(dataSourceId.getDataSourceCatalog());
			}
			connection.setAutoCommit(false);
			return connection;
		}

	}

	// Repeat the same for CustomTransactionAwareDataSourceConnectionProvider and CustomLocalJtaDataSourceConnectionProvider as well

}

The session factory configuration will look like this,

<bean id="db0" class="CustomLocalSessionFactoryBean">
	<property name="hibernateProperties" ref="hibernateProperties" />
	<property name="mappingResources" ref="dbPartitionMappings"/>
	<property name="dataSource" ref="db0DataSource" />
</bean>

And that is it. We are ready to test this out. It has been working pretty good in my laptop I would love to get back from you guys on optimizing the design or the issues that you find with this logic.

The idea here is to use a datasource identifier to decide which partition to use.

public abstract class DataSourceIdentifier {

	protected String dataSourceId;

	public String getDataSourceId() {
		if(dataSourceId == null) {
			resolve();
		}
		return dataSourceId;
	}

	// Add the partitioning logic here
	public abstract void resolve();
}

I just borrowed the idea of ThreadLocal from other posts to identify the current working datasource. But unlike the other solution (routing-data-source) referred in the last post, this would allow us to work with multiple datasources simultaneously.

public final class ContextHolder {

	private ContextHolder holder;
	private ThreadLocal context;

	// Private, to make it singleton
	private ContextHolder() {
		context =  = new ThreadLocal();
	}

	public ContextHolder getInstance() {
		if(holder == null) {
			holder = new ContextHolder();
		}
		return holder;
	}

	public DataSourceIdentifier getLastUsedDataSourceIdentifier() {
		DataSourceIdentifier dataSourceId = (DataSourceIdentifier)context.get();
		return dataSourceId;
	}

	public void use(DataSourceIdetifier dataSourceId) {
		context.set(dataSourceId);
	}
}

Now, we gotta implement our custom Spring HibernateTemplate class to make things work.

public class CustomHibernateTemplate extends HibernateTemplate {

	public void setContext(DataSourceIdentifier dataSourceId) {
		ContextHolder.getInstance().use(dataSourceId);
	}

	public  T load(Class entityClass, Serializable id, DataSourceIdentifier dataSourceId) throws DataAccessException {
		setContext(dataSourceId);
		return super.load(entityClass, id);
	}

	// Override the super class methods and throw an exception as we need DatasourceIndetifier to work with a datasource.
	public  T load(Class entityClass, Serializable id) throws DataAccessException {
		throw new IllegalArgumentException();
	}

	// Likewise you can do for all the super class methods

	public SessionFactory getSessionFactory() {
		DataSourceIdentifier dataSourceIdentifier = ContextHolder.getInstance().getLastUsedDataSourceIdentifier();
		if(dataSourceIdentifier == null) {
			// TODO get the default datasource (partition)
			dataSourceIdentifier = getDefaultDataSourceIdentifier();
		}
		return getSessionFactory(dataSourceIdentifier);
	}

	protected SessionFactory getSessionFactory(DataSourceIdentifier dataSourceIdentifier) {
		// TODO get from application context or do an dependency injection or whatever you like/know of
	}
}

Thats it, we are almost done. We gotta use this template in the DAO class.

public class HibernateDAO extends HibernateDaoSupport {

	public void delete(T entity, DataSourceIdentifier dataSourceId) throws DAException {
		try {
			getTemplate().delete(entity, dataSourceId);
		} catch (DataAccessException e) {
			throw new DAException(e);
		}
	}

	public void refresh(T entity, DataSourceIdentifier dataSourceId) throws DAException {
		try {
			getTemplate().refresh(entity, dataSourceId);
		} catch (DataAccessException e) {
			throw new DAException(e);
		}
	}

	@Override
	protected HibernateTemplate createHibernateTemplate(SessionFactory factory) {
		return new CustomHibernateTemplate(sessionFactory);
	}

	public CustomHibernateTemplate getTemplate() {
		return (CustomHibernateTemplate) getHibernateTemplate();
	}
}

Make sure all your subclasses of HibernateDAO calls getTemplate() instead of getHibernateTemplate() as the later will return an HibernateTemplate and you would have to cast it to CustomHibernateTemplate. So better to use getTemplate() in the DAO.

If you go with container managed transactions, you can sit back and relax, nothing to worry about. But for those poor guys like me who don’t use JTA, we need to customize the transaction manager as well.

public class CustomHibernateTransactionManager extends HibernateTransactionManager {

	public void setContext(DataSourceIdentifier dataSourceId) {
		ContextHolder.getInstance().use(dataSourceId);
	}

	public void getTransaction(TransactionDefinition definition, DataSourceIdentifier dataSourceId) {
		setContext(dataSourceId);
		super.getTransaction(definition);
	}

	// Unfortunately, we can't override the super class method getTransaction(definition) as its final ;-(.
	// But there's a workaround for that as well around. I'm not going to confuse you explaining it here.

	public SessionFactory getSessionFactory() {
		DataSourceIdentifier dataSourceIdentifier = ContextHolder.getInstance().getLastUsedDataSourceIdentifier();
		if(dataSourceIdentifier == null) {
			// TODO get the default datasource (partition)
			dataSourceIdentifier = getDefaultDataSourceIdentifier();
		}
		return getSessionFactory(dataSourceIdentifier);
	}

	protected SessionFactory getSessionFactory(DataSourceIdentifier dataSourceIdentifier) {
		// TODO get from application context or do an dependency injection or whatever you like/know of
	}
}

Yep.. We are done with all the fixes (Hacks ???). Even though we save the dataSourceId in the threadLocal, we should still be able to work with more than one partitions within the same thread provided you use Spring’s HibernateTemplate in your code.

What if you want to use the same connection to connect to all the partitions?. Meaning, all the partitions are there in the same database instance and you want switch the partitions by changing the catalog. I shall discuss this in my next post.

That’s a pretty good solution but comes with its own limitation,

• Works good if the application receives request for a single customer. But what if I have process a request for multiple customers, say I have a cron job that fires a request to the servlet to do something on these customer accounts. The thread local will be constantly overwritten that could lead bizarre effects.
• The above holds good for transactions as well. If you are using JTA, you will be safe but if you let your application manage the transaction, this solution is not for you.

If you closely take a look at the example in that post, you will notice that the database being partitioned based on the customer (horizontal partitioning). So why not go for Hibernate Shards in this case? I will leave that to you to decide on that. Hibernate Shards is a powerful library when you want to pull out collective data from multiple partitions. But in this case, the partitioning is done based on the customer. So we will be interested only on one partition at a time.

If you are not interested in Shards, then you might be interested in the below solutions. Multiple datasources doesn’t mean that database is partitioned, you can still have multiple databases one per an independent module. So the solution would differ for both of them.

Multiple datasources – no partitioning
If you are using JTA, the solution is straight forward. You have to create session factories for all the datasources and attach them to your DAO’s (Yep, this solution is for those who use Hibernate in conjunction with Spring). And a JTA transaction manager to manage the transactions. That’s it, you need not bother about the datasource as any table will have in no more than one data source; remember you have not partitioned your database. Since you go with the JTA, its the container that manages the transaction. So the transaction is going to be across all the connections created within that transactional scope.

But when you let the application handle the transactions, you might have to create one transaction manager per session factory. When you create the transaction, choose the transaction manager that you want to use based on the session factory (you can get them from the DAO, no thread local is required here). I don’t have any example to demonstrate this and I feel that my solution descriptive enough to understand. If you feel that you need to an example, please do ping me.

I shall discuss the multiple data sources with partitioning in my next post..

• Selecting a shard for an object at runtime
• Support for other hibernate features like Criteria, query etc…
• Clean implementation in the application and DAO layer

Although the shards documentation has listed out of a lots of limitations like, lack of support for HQL and other hibernate features, I wasn’t concerned much as anyways I’m not using HQL in my application. But all my database partitions are there in the same MySQL instance. So we thought of using the same connection for all the partitions and decide which partition to switch to at runtime by setting the catalog over the connection.

connection.setCatalog(partitionName);

This piece of code will execute.

use partitionName;

using the MySQL connection and there by switch the catalog to the new partition. The idea here is to minimize the total number of connections used by all the partitions thereby allocate more connections to other applications using the same database.

But unfortunately I don’t find such a feature in shards. Shards always insists in creating one session factory per partition (and group them together using ShardedSessionFactory). Moreover there’s no active development going on in this project lately and what I have heard from the developer is this project has been shelved stalled.

So I thought of coming up with my own version of shards (though not a very good design like shards but certainly satisfies my needs). I will try to post that design sometime later in another post.