Keeping Your Wits in the Candy Store

Build or buy -- sadly, those are only two among a myriad of decisions when you're looking at software purchases that'll be used to run your core business.

• By Mike Gunderloy
• 04/20/2006

Lately I’ve been spending a good deal of my time helping a relatively small consulting shop get its business off the ground. As part of this effort, we’ve been looking at a variety of different types of infrastructure software, from profilers to source code management packages to time-tracking software to document management systems. Of course, as developers, our first impulse in nearly every case is, “Hey, we could write that ourselves!,” but that’s seldom the smartest answer. Yes, you or I could spend all of our time developing the software that we need to run our own businesses -- but then, where would we find billable hours to sell to our customers?

In nearly every case, the smart choice when faced with a build or buy decision in this sort of situation is to buy. But even having made that decision, you’re still faced with a dizzying array of choices. There are literally hundreds of commercial and open-sourc, issue-tracking applications on the market. Given this plethora of options, how do you pick just one in a reasonable amount of time? In this column, I’ll share some of the ways we’ve come up with to make systematic software buying decisions.

Features First
Even as a small company, we brought a good deal of experience to the table: collectively, this team of under a dozen people had over a century of work in the field under their belts. This meant that for every category of tool (say, bug-tracking systems) people had used something in the past. Often, they’d used several different things in each category. And everyone had their opinions about which features worked, which ones were in the “must-have” category, and which bells and whistles sounded nice but turned out to just be wastes of time and money.

So, we started with an information-gathering process that concentrated on those "must have" features. This was done in part over lunchtime bull sessions, in part via e-mail discussions, and ultimately on a private wiki. For the most part, we conducted this part of the exercise in a spirit of brainstorming, with everyone being invited to contribute their ideas of what features would make the perfect tool and no one being allowed to say that someone else’s input was stupid. The exception was the company’s owner, Mr. Big, who reserved the right to pronounce some features absolutely essential and others absolutely not required. But since the funding for the purchases was largely coming out of his pocket, this seemed a reasonable part of the process.

The end result of these discussions was a wish list for each tool: a list of anywhere from ten to fifty features that our ideal tool would embody, preferably at a reasonable cost. To this point, no one had actually looked at any software (though most of us had our favorites in mind already).

Surveying the Landscape
While the group feature-gathering was going on, we also had one person working on gathering potential products to buy in each category -- for example, I hunted down time-tracking software while another employee was looking for issue-tracking systems. As a group we set some very broad parameters (we knew that we wanted to have all of our systems use SQL Server for data storage and run on ASP.NET for their user interface), and then we simply tried to find every single package that fit our respective categories within those parameters. While gathering candidate products, we didn’t worry about cost or features; the goal at this stage was just to find everything that was even remotely likely to fit the bill. We ended up with anywhere from five to 30 products to look at in most of our categories.

Build the Matrix
With a feature list in one hand and a list of products in the other, we could then each sit down to build a product matrix for our chosen category of software. This was a simple Excel spreadsheet, listing the products across the top and the features down the side, and with “yes,” “no,” or occasionally some commentary in each cell to show which features were supported by which products. We also added some information to these spreadsheets for each product:

• Product name
• URL
• License type (commercial, GPL, or whatever)
• Price
• How recently updated
• Overall comments

Building these spreadsheets was actually fairly tough. Getting started required reading Web sites, but then in many cases we had to download trial copies and read help files or even install the products to figure out what was actually supported, or swap e-mail with sales people (always a dangerous practice!). We also ended up modifying the feature lists a bit during this process. If one product heavily promoted a particular feature as essential, and our category expert liked the feature, he’d have to go back and look to see which other products also had the same feature.

Making Choices
You might think that after all this work (and it took weeks to get the feature matrices put together for the eight or so product categories we looked at) that making the actual decisions would have been easy: Just pick the least expensive products that covered all the "must have" features. If only life were that simple!

For starters, in almost no case was there any product that hit our entire list of "must have" features. In some cases, the discrepancy was so wide that we made notes of future business opportunities. For the most part, we went back to the bargaining table and talked about what we’d be willing to live without, and which pieces we could tack on ourselves with an appropriate use of batch files, scripting languages, spit, and baling wire. Products with well-defined extensibility APIs tended to score well with our crowd of developers, because we knew we could bend them to our wills (in our yet-to-be-discovered copious spare time).

Another factor that vastly complicated things is that the boundaries between products turned out to be very fuzzy. There were issue trackers that also tracked people’s time, customer portals that could also manage our internal documents, source code control software that would plug into the issue tracking, and so on. We had to make a whole series of decisions about whether to go for all-in-one products that might not do anything well, or single-use products that were best of breed but might not integrate as well. Ultimately, we chose the latter strategy for the most part - again, confident in our ability to hack up integration as necessary. For a company that wasn’t loaded with developers, I’d probably think harder about the integrated solutions.

Cost and licensing proved to be as important as features in some cases; there were categories where we literally had prices ranging from “free” to $25,000 for supposedly the same capabilities. While we were by-and-large disappointed by the open-source solutions we tried, we also didn’t buy a lot of “enterprise” software directed at people with large corporate budgets. There are quite a few full-featured packages for the midrange these days.

And finally, when the group as a whole had come to its best consensus, Mr. Big broke ties. Whether you let the one big boss do this or defer to subject matter experts in each category, I recommend you have some plan for cutting off debate and making a choice. Otherwise you’ll end up spending $5,000 in staff time to buy a $2,000 tool, which doesn’t make a lot of sense.

Parting Advice
Looking back after a couple of months, this process worked very well for us. But it wasn’t perfect. Evaluating software based on sales literature, Web sites, and single-user test drives isn’t a perfect substitute for actually using it. Despite our best efforts, we did end up with one package that didn’t scale to our level of usage, proving to be a complete dog when trying to use it with an Internet-connected database (despite the salesperson’s blithe assurances to the contrary). Fortunately, we had a good negotiator on our side and we ended up with a 30-day, money-back guarantee that we exercised before our 30 days were up. If you’re investing a substantial chunk of money and betting your business on a tool, I suggest you negotiate hard for similar terms. It’s good insurance, and reputable vendors will be happy to agree.