Redefining the Integration Test

Recently I had been discussing with another developer the distinction between "integration test" and "unit test", and he echoed the same wisdom heard elsewhere. An integration test is any time a test case requires outside resources (i.e. external database, other servers, etc) in order to execute. This is the same definition proffered by a No Fluff Just Stuff conference I had attended a while ago. While I agree that this has been a good rule of thumb, I feel as if it may be time to reexamine this position.

The problem these days its so easy to spin-up resources from within one's test case that I'm not sure that's a fair unit/integration litmus test. Presently, I count libraries like Unitils, DbUnit, and Hypersonic as tools in my testing arsenal. The latter is an in-memory database, and by leveraging it within my test case I eliminate the need to have, for example, MySQL running during test execution.

In many of my tests, I've swapped out use of MySQL with Hypersonic, allowing my test case to be truly autonomous. Using the classic test definition, I've made my test case go from integration to unit test. Merely by changing the spring context file to point to an actual running database, Presto! -- its an integration test again. There's something amiss with the test definition, I feel, when traversing between the two types of test is so easily made.

I'm compelled then to reflect on what unit testing means to me. One of my core beliefs is that unit testing is about isolation -- truly testing only that code which is executed within a class. Take your typical DAO method (excuse the formatting):


public Widget getWidgetByLabel(String label) {

List results = hibernateTemplate.find("select w from Widget w where w.label=?", label);
if(results.isEmpty()) {
throw new WidgetNotFoundException("No widget exists with the label:" + label);
}
return (Widget) results.get(0); // ignoring too many results for now.

}

When I put on my unit-testing peering specks, I see very little logic to test here. This is a happy conclusion -- having too much logic within one's DAO is not a good thing. My unit test for this method would merely insure:

1. The proper arguments are passed to the hibernateTemplate.find() method.
2. An empty List will trigger an exception.
3. A non-empty List will make the method return the first object in the list.

However, now that I can easily create an instance of Hypersonic, create the database schema, and initiate a Hibernate session factory, my test can easily verify things like proper Hibernate mapping of my Widget object, and that the HSQL syntax is correct. All this without the necessity of external resources. But these sorts of verifications are about the collaboration -- or integration of other objects and tools.

Therefore, I think I'm revising my personal litmus test. Only when a test case is truly testing in isolation will I consider it a unit test. If a test case involves logic in other classes and libraries, I shall regard these as integration tests, even if it requires no outside resources.

Bamboo Disappointment

Being a big fan of Atlassian's Confluence and Jira, it was with much anticipation that installed Bamboo, the continuous integration (CI) engine they've released. Perhaps these high expectations led to my ultimate disappointment with Bamboo, but truly the features I've come to expect in a commercial CI product are nowhere to be found.

No concept of inherited project structure.
If you have 20 modules you would like to build, defining behavior and properties for each is a daunting task. QuickBuild and AntHillPro both allow for a hierarchal organization of modules, so that a child may inherit properties (like environmental variables, build targets, etc) of its parent. In Bamboo, when creating a "Plan", I can clone an existing module, but that's it. Should I have the need to change a property for all plans, I'll be forced to configure each through the web GUI. A tedious process -- even with Cruisecontrol I could search & replace config.xml in a text editor to make wholesale changes.

Alien nomenclature
In Bamboo, you have top-level "projects", and beneath them you have "plans", which represent the modules being built. I've never used the word "plan" before when describing a module's build, and frankly the limited options offered by Bamboo to govern build behavior makes it a dubious word choice.

No passing of properties
It is sometimes desirable to direct a CI engine to pass arbitrary properties to one's build process, and vice versa. I don't mean "static" environmental variables, rather I refer to dynamic properties like "version number". No such functionality is present in Bamboo. Luntbuild and Quickbuild both allow for this using OGNL expressions.

No concept of build promotion
Most commercial CI products have evolved to include "Application Lifecycle Management", so you may describe how a build can go from being development-status to gold. Implicit in this is a workflow allowing QA to promote and release builds. None of this is even hinted at in Bamboo -- it does not even tag your build.

Summary
Features that Bamboo stresses seem misplaced. You'll see much mention of its heralded "Build Telemetry", on the product's site. Through this feature, you can generate nice graphs and statistics about one's build. But honestly, I've regarded charts and graphs in CI engines as a feature present because they're easy to add. While its fun to see how often one's build has been broken, I've never seen a lasting need for such statistics.

These are merely the shortcomings I've experienced, but I'm sure there are others -- like remote builds (where the CI Engine is deployed to multiple servers) for which I've no immediate need, but will no doubt be experienced by others. Its hard to see what advantages Bamboo offers beyond a free product like Luntbuild besides a pleasing user interface. Indeed, Luntbuild's features like OGNL expressions and Eclipse plugin makes it, in my opinion, more feature-rich than Bamboo.

My favorite build server remains QuickBuild. It was with tremendous ease that I was able to control and manage many modules at once. To be fair, though, its been awhile since I've looked at AntHill Pro or TeamCity from JetBrains, both products that looked promising when I reviewed early versions.