Software Evaluation, Selection, and Procurement

A 1995 Standish Group research study based on over 8000 software implementation projects found that nearly one third of the projects were canceled before completion. And over 50% of the projects cost nearly twice their original estimates. The study further concluded that only 16% of all software projects are completed on time and on budget (only 9% for large companies)¹.

With these grim statistics firmly in hand, a plethora of books, white papers, and articles have been written on implementation success factors and software implementation methodologies.

While these publications offer sound and sage advice, the problems remain and the project failure statistics have not improved.

One area receiving little scrutiny, however, is the software evaluation, selection, and procurement processes employed by most organizations. This article reviews common problems associated with software evaluations, and introduces several ideas for improvement.

The Capital Investment Paradigm:

Financially, GAAP requires companies to account for most software license and implementation costs as capital investments. This makes perfect fiscal sense, as the software lifecycle is long and depreciating the investment over multiple years is sound financial practice.

Unfortunately, the software procurement process does not follow a similar, long-term capital investment approach. Instead most companies handle software purchases like departmental operating expenditures. This flawed paradigm is the root of many implementation failures and the cost overruns cited above.

To illustrate the capital investment approach, let's use the example of production machinery for a manufacturing firm. When considering an investment in new machinery, multiple departmental teams with varying requirements participate in the decision making process. Typical activities might include:

Team	Activity
Manufacturing Management	Build business case for investment in new machinery Investigate alternatives to new machinery purchase (i.e. outsourcing, continued use of existing machinery, etc.) Investigate the service and warranty support provided by the candidate vendor(s) Investigate the financial viability of the candidate machinery supplier(s)
Engineering	Test the machine to confirm it will produce the necessary product within tolerance
Factory Operations	Test the machine to make sure it fits within the allocated floor space, is easy to operate, and will improve productivity as expected
Quality Assurance	Test actual runs of product through the new machine to confirm the error rate is within requirements
Customers	Manufacturing process changes may need to be reviewed by external customers to ensure they conform with agreed to specifications
Suppliers	Raw materials will need to be confirmed on the new equipment and possibly alternative suppliers must be evaluated
Procurement	Negotiate final contract terms
Finance	Arrange for acquisition of needed capital

Other forms of capital investment (such as facilities or real estate) generate similar analysis and due diligence activities from a variety of internal and external teams. This model recognizes the fact that a capital investment must serve the organization well for multiple years. Spending time and effort up front to ensure a smooth transition to the new machinery (facility, etc.) is simply good business practice.

Software Evaluation Challenges:

Most companies do not treat software procurement in a similar manner. A series of common flaws can be found in most selection processes. The following table outlines the most common software evaluation and selection flaws, and the associated risk.

Flaw	Leads to...	Project Risk
Insufficient time devoted to evaluation and selection	Ill informed decisions �	Poor software "fit" against business requirements Improper expectations on implementation cost, time, resources, etc. Erosion of project support Increased user resistance
No business case or Poorly defined business case	Inaccurate projection of: Costs Benefits Alternatives Risks Assumptions etc.	Cost overruns due to poor planning Lack of business benefits Lack of "Ownership" of each individual benefit Erosion of project support
Affected departments not part of selection team	One-sided evaluation of software fit Unidentified software gaps Poor software "fit"	Cost overruns due to unplanned work to resolve new software gaps Erosion of project support Increased user resistance
Incomplete or inaccurate set of business / technical requirements	Unidentified software gaps Inability to sufficiently "test" the system	Cost overruns due to: Unplanned work to identify new requirements at a later date Unplanned work to incorporate additional functionality Unplanned work to perform additional testing Erosion of project support Increased user resistance
Insufficient testing of candidate solutions	Unidentified software gaps Unidentified integration issues Poor software fit	Cost overruns due to: Unplanned work to resolve new software gaps Unplanned work to build interfaces Erosion of project support Increased user resistance
Insufficient verification with external customers/suppliers	Customer/supplier discontent with new system	Lost sales Lowered customer service Supplier cost increases
Insufficient due diligence of software vendor (i.e. references, viability, future development, support, etc.)	�	Poor support Lack of future functionality for upgrade, enhancement Increased total cost of ownership

The software evaluation and selection process builds the foundation for the entire implementation project. A foundation that includes one or more of these flaws will inevitably crack under the pressure associated with a mission-critical software implementation. A structured and disciplined software evaluation and selection approach will mitigate this risk and establish a solid foundation from which to move forward. The following section outlines our recommended guidelines for such an approach.

This concludes Part One of a two-part article. Part Two makes Recommendations for Improvement.