Program Testing Methodology Part Two: Running Tests and Getting Approval

Tuesday, July 7, 2009
Before any system can be completely implemented on a production computer, analysts and programmers working on the system must be able to unequivocally state that the programs work exactly as they were designed to work and that any errors or mistakes found in data processed by the system will be handled properly. Programming testing methodology should accommodate any scheduling slippage that may result from the test results. Once testing and data parameters have been determined testing can begin. After link or string testing; system testing; and backup and restart testing have been conducted with confidence, approval from management and users should be sought.
Link or String Testing:

After the individual programs have been tested using data designed by the programmer, the programs within the system which depend upon one another must be tested together. This is called link or string testing. For example, in a system with an edit and an update program, the output of the edit program must be properly formatted and contain the correct data to input to the update program. The programmer who wrote the edit program may not be the same programmer who wrote the update program. Therefore, there may be some discrepancy between the format in which the data is created in the edit program and the format in which the data is expected in the update program. The series of programs must be tested together.

Another reason for testing the series of programs is to ensure that the job stream is correct. In computer systems there is the operating system, and the features and functions of the operating system must be invoked through the use of the job stream. The job stream consists of those job control statements which are necessary to invoke the proper programs to be processed, to define the files to be created and processed, and to designate the devices which are to be used for the files. If files are defined incorrectly or programs are called improperly, the system will not function in the specified manner; therefore, the testing of the job stream is as important as the testing of the programs within the system.

The data which is used in string or link testing should be developed by the systems analyst. This data should include data which will test certain conditions which may not have occurred in program testing. For example, three programs may process a certain field and the data in this field is accumulated in the three programs. Although each program may be able to process a maximum size for the field, the accumulation in the three programs may cause the field to be too small in the last program. When the programs were tested individually, each programmer allowed the proper size field but when the actual processing takes place, the field size may have been improperly specified by the analyst. It must always be noted that not only is each program being tested by the string testing, but the system itself is being tested, and all of the specifications developed by the analyst, are also being tested. Therefore, data must be developed which not only tests the programs but also tests the specifications for the system which were designed by the analyst.

It is very important for the analyst to define what is to be tested during any given test run. It is inefficient and error-prone to attempt to test everything on the first test run. Therefore, the analyst should specifically designate portions of the system to be tested on each "pass" through the related programs within the system. It may be necessary to design 10�20 "passes" through the system in order to adequately test each operation, each file definition and size, and all the other variances which can be found in data and in the processing.

The testing procedure should run from the normal to the abnormal. That is, the first tests of the related programs within the system will process test data which represents those cases which occur most often within the system and present no abnormal situations. For example, in a payroll system, the first data tested may be for employees that work a standard forty hour week with no overtime and with standard deductions. When this data is tested properly, the analyst should then mix this data with data which includes overtime and test the overtime routines. The analyst should then switch to test data which has some invalid times in the transaction input to be sure that this data is processed correctly.

System Testing:

After the link or string testing is completed, the entire system must be run through a new series of tests. In these "systems tests", the entire system should be tested beginning with the preparation of on-line transactions running all of the processing programs within the system, and generating output from the tests. The main objectives of the systems test are as follows:

  1. To perform a final test of the programs against the original programming specifications.

  2. To ensure that the computer operations staff has adequate documentation and instructions to operate the system properly on the computer and properly handle the incoming and outgoing data from the system.

  3. To ensure that the user departments are able to properly enter data into the system and to properly disburse and use the output information.

  4. To ensure that the "flow" of the system works properly, that is, the channels for the delivery of information from the user or other departments are established, the input data moves smoothly from the point where it is received through its preparation for use on the computer, and the output data is properly handled to allow for its distribution to the user departments.

As can be seen from the objectives of the system testing, more than just the testing of computer programs is involved, and, additionally, more personnel and departments are involved than just the Information Systems Department. Because of this, there are a number of elements which must be considered by the systems analyst, including the following:

  1. A determination must be made concerning what constitutes a satisfactory performance of the system. That is, at what point can the system be declared ready to go into production.

  2. The methods of evaluation must be determined, that is, how is the performance of the system to be measured.

  3. The different "cases" to be tested must be determined by the analyst and the data to test these cases must be designed and prepared.

  4. The tests must be scheduled, both in the Information Systems Department and in conjunction with the user departments which may take part in the tests.

  5. All efforts in terms of data preparation, scheduling, and evaluation must be coordinated so that the testing takes place on schedule and both the data processing department and the user departments can make accurate determinations concerning the readiness.

During systems testing, the test data should be designed by the analyst exclusively to satisfy the objectives of the systems testing. After all of the testing has been satisfactorily completed, however, it may prove useful to run several complete systems tests using live data, that is, data which has been previously processed through the currently existing system.

System testing is one of the most important steps in ensuring that a reliable system is being placed in production yet is often one of the most neglected areas in testing a system. The primary reason it is neglected is that it is the last step before system implementation. Therefore, if any schedule slippage has occurred, it is likely that the system is behind schedule when systems testing is to take place and, to try to meet the schedule, systems testing is quickly performed, if at all. This can be a mistake because it means the complete system has not been adequately tested and, therefore, any errors which are in the system will not be found until it is actually in production. Thus, it is imperative that complete systems testing be performed, even if it means the system will not be implemented exactly on schedule.

Backup and Restart Testing:

Backup and restart testing is performed to ensure that the files which are kept as backup are adequate and that the programs which are used to both copy the master files and other files which are saved and the programs used to reload the files work properly. Even though these programs are not part of the system in terms of daily, weekly, or monthly use, it is important that they work properly because without them, recovery will not be possible when recovery is necessary.

Restart testing must be performed to ensure that the restart points within the system are carefully and accurately defined. If they are not, then it will not be possible to restart the system with all files in tact. Although in a batch system there are not a great deal of restart problems, this problem can become more difficult when the programs processing the data are very long running programs and checkpoints are used, or when an on-line application is used and there is not an accurate record of transactions received or updates processed. If a file is destroyed, then restarting is difficult and it is in these areas that testing and organizing must take place if a system is to be restarted properly.

No matter what the backup and restart procedures are for the system, and no matter how simple or difficult they may be, they must be tested. Therefore, in addition to testing the entire system as a whole, the analyst must be sure that all backup and restart procedures are adequate. If they are not, and if the analyst does not fulfill his responsibility in this area, there could at some future time be an adverse result that impacts business operations because of a failure within the system. It is important that these backup and restart procedures be reliable before the system is implemented.

Management and User Approval:

After the final systems testing is completed, the analyst should obtain management and user approval. The purpose of management and user approval is to show them what they are getting out of the system and to ensure that the functionality of the system is working properly. In addition, the procedures for processing the data through the system should be exhibited so that all concerned are aware of what is necessary in order to efficiently operate the system.

There is no way that changes in files or on-line screens, processing routines, or report formats can be accommodated at this time without substantially delaying the project. These changes should have been made when the presentation was given after the system design phase. The only changes which should be made at this time are corrections of errors which are discovered when the users and management review the output of the system. If there are errors in programming, then the errors will have to be corrected and the system re-tested. Again, however, it is imperative that no changes, other than corrections, be accepted by the analyst after the system has been extensively tested and before it has been implemented.

Changes which must be made to the programs in order to add additional features or restructure files, reports, etc. can be made to the system after it is in production and when there is time to properly make changes.

It is important to have the user's and management's approval of the system which is shown to them. As with previous meetings and review sessions with management and the users, it is quite helpful to the analyst to have approvals on everything which is going to be produced from the system. In this way, if there is a problem at a later date with some component or module in the system, the analyst can point out that whatever caused the problem was approved at the conclusion of the testing of the system. There is some protection for the analyst and it many times causes the user to very closely examine the system.

Program testing and debugging is one of the most critical aspects of implementing a computer system because without programs which properly work, the system will never process information and produce that output for which it was designed.

HIPAA-Watch for Security� Speeds Up Compliance Part Two: Phase III and IV, and Product and User Recommendations

The HIPPA-Watch for Security� tool was developed by RiskWatch a company founded in Maryland (US) in 1993. The tool is designed to aid companies through US risk analysis to eventual US regulatory compliance. Its risk analysis engine is embedded in the product and consists of four phases. Phase I assists users with establishing compliance case boundaries, and phase II values are defined, audit questions are created, and respondents are determined in order to formulate boundaries. Phase III and IV pertain to evaluation and reporting.
Phase III and IV: Evaluation and Reporting:

Phase III launches the risk analysis engine and performs the evaluation. Clearly preparing for the evaluation is a lot more time consuming than running the evaluation engine. Before you actually run the evaluation however, HIPAA-Watch allows you to review the links created between Asset Categories with Loss Categories. If you need to change the default recommendations for the links between Asset Categories and Loss Categories, it is simple to make the change. You simply uncheck the assets that are not prone to the type of loss indicated. For example, supplies and consumables are likely not prone to data disclosure and therefore should not be linked. Figure 4 illustrates how Assets are linked to Losses.

Figure 4. Linking Assets with Losses

In Phase III, you decide which calculations you want to compute based on the relationships of the threats, assets, vulnerabilities, and seriousness of potential incidents.

Phase IV generates a final report that has a variety of options that can be included. The options include

  • an executive summary
  • recommendations for resolving vulnerabilities
  • a full asset report
  • a summary by asset report
  • a full threat report
  • a summary by threat report
  • a full vulnerability report
  • a vulnerability distribution report
  • a full safeguard report
  • a cost benefit report
  • a safeguard threat report
  • an audit trail question report
  • an audit trail respondent report

The reports generate color pie charts and bar charts and can be saved in either rich text format or Microsoft Word format. While the reports are verbose in their recommendations, most organizations will want to apply some edits to customize them further.

Suggestions for Product Improvement:

Relevant Technologies would like to see the aesthetics of the user interface improved in HIPAA-Watch for Security�. The engineering of the tool is so sophisticated, that this product deserves a user interface with cutting edge aesthetics and a vanguard look. While the existing graphic design and reporting engine is adequate, it could evolve into a market sensation if the developers enlisted the help of a top-notch design artist. Relevant Technologies believes that software is art, and when a product excels, we expect the look and feel of it to excel also. The look and feel of HIPAA-Watch for Security� is basic and for that reason, using it may not elicit as many "oos and ahs" as it might otherwise receive given its capabilities.

Relevant Technologies would prefer to see the survey questions worded in the form of a true interrogative sentence instead of a statement with a question mark at the end. For example, instead of "Access to system log data is restricted to approved personnel?", we would prefer the question to be worded, "Is access to system log data restricted to approved personnel?" However, it's fair to say that the survey questions that exist are certainly on topic and apropos to a HIPAA audit.

Since LAFE values vary according to geographic location, Relevant Technologies would like to see this feature automated so that when you put in your organization's zip code, the LAFE values are automatically adjusted. For example, if your organization is in Omaha, Nebraska (US), you would have a much higher likelihood of tornados that if your organization is in Portland, Maine (US). Today HIPAA Watch for Security� allows you to manually adjust these values, however, this presumes that you know what the adjustment should be and it may take you some time to look it up and find out.

Recommendations for Users:

HIPAA-Watch for Security� works as advertised and has all the appropriate features that experts in risk analysis expect to see. It's ability to make appropriate calculations from which quantitative risk-based decisions can be made is first-rate. The automated reports that it generates will be useful for chief financial officers, chief information officers, and chief security and privacy officers. Since HIPAA-Watch for Security� has the ability to accommodate multiple respondents that can login to the system from different locations, it can be particularly useful for large, disparate organizations. By using HIPAA-Watch for Security�, it is possible to understand which safeguards will give you the greatest return on investment, ranking them from highest to lowest. If you are ready to tackle a HIPAA compliance risk analysis, and don't know where to start, using HIPAA-Watch for Security� will likely speed up your ability to comply with the CFRs.

Aside from helping your organization comply with the Final Security Rule, HIPAA-Watch can help your organization make better business decisions by making recommendations on how cost effective it is to apply particular safeguards. To take advantage of the sophisticated business decision recommendations, users of HIPAA-Watch may want to educate themselves on basic quantitative risk analysis equations including how to calculate annualized loss expectancy (ALE), single loss expectancy (SLE), annualized rate of occurrence (ARO), and exposure factor (EF). The information HIPAA-Watch generates can also be used to populate a Disaster Recovery planning exercise.

An auxiliary addition to HIPAA-Watch is a bonus CD that includes a data collection kit, that has forms, PowerPoint presentations, and various shortcuts and tips that will make the analyst's job easier. A complete risk analysis project plan is included both in Microsoft Project and in Excel formats for reference purposes.

Consultancies that specialize in assisting healthcare organizations on the road to HIPAA compliance may want to consider using HIPAA-Watch for Security� as a tool for standardizing their service offering. Since the audit questions can be refined and added to, it is possible to build up comprehensive question libraries that can be used with different types of covered entities. The different types of covered entities that can take advantage of HIPAA-Watch for Security� include

  • health care providers
  • health care plans
  • health care clearinghouses

Health care providers include hospitals, doctors, clinics, pharmacists, and mental health care specialists. Health care plans include insurance companies, health maintenance organizations (HMOs), medicare plans, Medicaid Plans, veteran's health care Programs, and Indian health service programs. Health care clearinghouses include organizations that process or facilitate billing or transmittal of electronic health information data for other covered entities such as community or local health information systems.

Conducting a risk analysis manually is not an intuitive process and use of HIPAA-Watch for Security� will be a definite timesaver for any organization that wants to conduct a true risk analysis. A two-day training class is available every month at RiskWatch's headquarters in Annapolis (US).

A feature that Relevant Technologies found to be particularly notable was the ability to actually see the HIPAA Final Security Rule, which is expressed as a control standard. This feature enables organizations to actually understand why they need to pay attention to a particular security policy and whether or not it is considered a required or addressable CFR. While Required CFRs are mandatory, addressable CFRs are optional.

Relevant Technologies spent a considerable amounnt of time researching possible market competitors and was not able to find any other HIPAA security compliance products that appeared competitive with HIPAA-Watch for Security�. However, since the market for HIPAA compliance products is still young, Relevant Technologies expects new competing products to emerge within the coming year.

US federal agencies will like that the safeguards list includes the deliverables that are typically required to pass a FISMA-based security certification and accreditation audit. Federal agencies that already have a Certification and Accreditation (C&A) package can apply these C&A reports to their HIPAA risk analysis and reuse much of the pre-existing information.

Automated Enterprise: Many High-ROI Opportunities

Major systems management vendors are presenting a new vision of the future data center, and success-minded CIOs should begin constructing a roadmap to the automated data center, with moderate steps that ensure return on investment (ROI).

Automated data centers self-configure, self-heal, self-optimize, and self-protect. The underlying solutions combine intelligent management software and resilient hardware to deliver better asset utilization, make data center operations less expensive, increase flexibility to meet changing business demands, and proactively provide more resilience.

Companies that haven't yet adopted the automated data center face a number of potential challenges. For example, changes to a large financial services organization's Microsoft's Active Directory accidentally brought down Internet access for the company's trading desk. (Active Directory, an essential component of the Windows 2000 architecture, allows organizations to centrally manage and share information on network resources and users while acting as the central authority for network security.) As users logged in, they weren't able to get on-line, and could not access vital information and mission-critical applications.

After eight hours, the problem was traced to an accidental change made by the Active Directory administrator, and the proper settings were restored. The trading desk was adversely affected and this minor issue caused an impact that was estimated in the millions of dollars, in both productivity and lost business.

Whether for availability, security, or general IT operations, the automated enterprise delivers significant productivity and business benefits. Depending on the business, even slight improvements may generate exponential returns—and provide tangible business value that extends far beyond IT. Organizations looking to become more automated should follow established strategies and best practice features, including

  • User and Resource Provisioning—add, move, and modify resources or configurations to enable or enhance performance of mission-critical applications, customers, partners, or employees on a priority and demand basis.

  • Infrastructure Availability—ensure consistent and readily available access to key business resources by managing availability, loss prevention, and recovery.

  • Security Management—establish identities and manage security of key business resources.

Following these best practices has significant financial benefits, as they'll expedite the automation processes in four critical areas: IT operations and administration; virtualization and provisioning; security; and availability.

IT Operations and Administration:

Operations and administration typically consumes 30 to 40 percent of IT spending—an average of $4,400 (USD) per employee in an average US enterprise. Of this, 65 percent is dedicated to ongoing maintenance and asset management, 25 percent for migrations and upgrades, and only 10 percent for innovation. To improve the value of IT, it's important to decrease the ongoing maintenance and asset management through task avoidance and productivity enhancements, while increasing the resources toward innovation.

Organizations should target the three biggest areas of gain:

  1. Reduce the number of administrator tasks
  2. Reduce each task's steps and cycle-time
  3. Reduce the skills required

Data centers deploying consolidation and self-optimization handle their workload with 30 to 40 percent fewer assets; this saves 20 percent on overall administration. Self-healing and other best practices yield another 5 to 10 percent in labor savings. This can amount to an annual savings of $1,320 (USD) per employee for a typical enterprise.

More than 85 percent of companies experienced security breaches within the last 12 months, and more than 60 percent of companies acknowledged financial losses as a result.

When a security incident occurs, IT organizations scramble to meet the challenge. Even if harm is prevented, many tangible and intangible costs are incurred

  • Repair and Mitigation—the time and cost of finding the problem, repairing damage, recovering data, and ensuring that the vulnerability is addressed to prevent future harm.

  • Downtime—lost productivity, revenue, and profit while the systems or applications are unavailable.

  • Competitive Impact—loss of customers and market share due to system unavailability or customer dissatisfaction.

These costs can be broken down by type of cyber security incident, including internal and external attacks. It's important to understand the typical total cost of each type of successful attack to fully understand the financial benefits of mitigating them.

Security Threats and Estimated Impacts
Typical Impact per Incident (USD)
Virus $24,000
Denial of service $122,000
Physical theft or destruction $15,000
Data destruction $350,000
Theft of proprietary information $4.5 million
Illegal system access—outsider $225,000
Unauthorized insider access $60,000
Installation or use of unauthorized software or hardware $250,000
Insider abuse of net access or e-mail $360,000
Financial fraud $4.4 million

Estimated security impacts per incident for various internal and external security issues—Source: Alinean

The automated data center's self-protecting features proactively reduce vulnerabilities, automatically distribute patches, and reconfigure systems as needed, reducing security risks, and saving companies 20 percent per year on security management and business impact costs.

Virtualization and Provisioning:

Automated data centers' virtualization and provisioning features are estimated to save companies 30 to 40 percent on hardware and software, by avoiding establishing the systems for peek load. The automated data center automatically allocates assets where needed, supporting changing business priorities and meeting routine and peak performance requirements.

Net savings can easily top $1,000 (USD) per year, per employee, based on a typical enterprise, which spends $1,633 (USD) per year, per employee for data center hardware and software, and an additional $1,496 (USD) per year, per employee on purchased software.

High Availability:

How long downtime lasts is crucial. A workgroup losing just a few minutes can easily make up the time, but hours of downtime can mean invalid transactions or a permanent loss of clients.

Estimated Outage Cost per Minute
Business Impact (USD)
Supply chain management $11,000
Electronic commerce $10,000
Customer service center $3,700
ATM/POS/EFT $3,500
Financial management $1,500
Human capital management $1,000
Messaging $1,000
Infrastructure $700

Estimated downtime impact per minute for various business applications—Source: Alinean

Downtime for a typical computing infrastructure is estimated at $42,000 (USD) per hour. At this rate, a 1 percent improvement in availability can lead to millions in reduced risk and productivity losses.

Unplanned Downtime (Mission Critical)
Typical Uptime
Hours Down per Year
Cost per Unplanned Downtime Hour (USD)
Downtime Risk (USD)
Worse than average 98.000% 174.72 $42,000 $7,338,240
Average 99.000% 87.36 $42,000 $3,669,120
Better than average 99.500% 43.68 $42,000 $1,834,560
Good 99.900% 8.736 $ 42,000 $366,912
Best in class 99.999% .09 $42,000 $3,780

The automated data center promises to be more resilient to downtime issues, helping companies achieve best-in-class or "good" availability—typically a 50 percent reduction in downtime. For most organizations, this can mean saving millions of dollars annually.

The technology will continue to advance throughout the next three years and IT management will have to augment its own skills and processes to profit from the promised benefits.

All companies are different, but for those needing a resilient high performance infrastructure for business process improvement and e-business mission critical applications, the automated enterprise will deliver a solid ROI.


When Provider's Value Is Not In Synch With Customer's Value

It may seem obvious to all of us, but the fundamental process by which technology and equipment firms engage with clients have somewhat opposing processes and needs. Modern service best practices attempt to harmonize these goals.

On October 20th, I attended the annual Stanford/Wharton Service Supply Chain Forum. At the outset, Morris Cohen of Wharton set the stage for our cross industry and cross competitor sharing of ideas.

Said Professor Cohen, "Customers' main goal is to extract value from the goods and services they buy—and in most cases, extend that value (use) as long as possible."

Various panel members reinforced this basic concept.

The longer that cycle continues, the more likely the supplier will have to institute and manage a so-called service business to support them. A trillion dollars later—give or take a few billions—the B2 Bomber is still in use, the Under Secretary of Logistics of the DoD reinforced. Less exciting, but also challenging to service, a tool firm like Hasbro has equipment in the manufacturing facilities built in the forties and fifties that still must give service (no one has invented replacements for these).

There are a few key issues that service providers need to understand if they intend to design and deliver products and services that create and maintain that value:

  • Focusing on performance is the key metric—in the customer's eyes. Gone is the concept of response time and fill rates, but rather availability or uptimes. It's not about fixing broken things. But more important, it's about having a separate line item for inventory or part that a customer pays for. Customers expect and will pay for the performance, and leave the rest to the service provider. We have several profound examples of firms attempting to create these business models—Lockheed and Boeing working with the DoD on performance based logistics (PBL).

  • The second key concept we learned was the need for total life cycle integration for end-to-end knowledge sharing. Service data or returns data holds many keys to product innovation design, yet many firms' supply chains are highly outsourced, or pay little mind to returns, and therefore never mine this knowledge.

  • Eco system innovation, the point of performance—using and gaining value from goods and services—is the result of a complex web of partnerships. With long life value at stake, the decision to buy products is beyond just feature or function, but buying into architecture—a continual stream of long-term innovation.

  • Outsourced models of service frequently put someone else in charge of the dissatisfier zone—using the product for a lifetime. These models, we heard, are designed mostly with sales reach, supply chain efficiencies and cost in mind .. .hmmm ... how about delighting the customer? When we heard the word customer used, most frequently it related to the OEM. This is still a huge issue in service models.

Study after study has shown that customers will pay for extra service, yet the basic model for service puts all customers on the same platform. One of the few things the customers got right—special 800 numbers for Platinum customers. High tech and a very bad rap with customers now with their outsource services (referenced in other articles in this month's PV). Those who buy a $3400 laptop with a service contract should get higher-grade service than those with the $599 product. Yet, all wait in the queue to reach unskilled service call centers where employees read through manuals and notes. Shame! Shame!

The issue, as well as for the brand firm, is that this induces a POOR RELATIONSHIP with the customer, destroying loyalty and future sales.

Jim Molson says Selectron is thinking through these models, and gave some insights on the insourcing or outsourcing of services. Warranty dollars are shrinking, due to the lower costs of the price of hardware. (But one would assume there is still decent margin here, since the reliability has gone up.)

Forces in Outsourcing Versus Insourcing
Theoretic Central Control Access to new markets or geographies
Customer access to developers and engineers Asset recovery or after market reselling
Hardware prices are dropping—OEMs want to capture all revenue opportunities Multi-vendor knowledge and leverage

Aligning goals between partners seems to work over time since the hard definition of value—cash—is understood. Yet the dissatisfied zone needs a lot of work. One interesting discussion occurred about chemical services versus chemical sales in automotive. The ultimate goal is not to buy and store chemicals, but rather to have just enough to manufacture the car. Chemical service provides and charges for that outcome and managing the supply chain, the disposal, recycle etc., and only charges based on each manufactured car. Clearly, a lot of data drives this process—end to end.

Integrating the chain is crucial to aligning the goals of customers and provider. Left unsaid is who will be responsible for integrating the channel. What also appears to be true is the lack of dialogue and innovation at that last mile, the dissatisfied zone.

Want to make more money? Be understanding and deliver high-end care for the customer. Part of the analytic tool kit can help model this—again, that end-to-end data may have some of the keys!

Customer Relationship Management Strategies Part Two: Creating Your Strategy

Customer relationship management is a total business strategy; therefore, proper planning is crucial. After you have communicated and established support for your initiative, outlined the steps your company needs to take for implementation, and marked your parameters for success and failure, you can then create your implementation strategy
Implementation Issues and Strategies:

What Robert Burns said about the best-laid plans are true: they do often go awry. However, preparation is everything because it will help you to accommodate change. The road map to a successful CRM implementation is filled with pot holes, but if you know the road signs, you will be able to plan ahead. Large and small organizations alike have stumbled through the same dangers and get caught in the same traps. Recognizing these issues in advance will help guide you through or around these problems.

Overcoming Lack of Requirements

Ask any number of IT professionals what the number one reason software development projects fail and many will say "lack of requirements", more specifically, lack of good requirements. Not understanding user needs can result in inadequate functionality. This is usually a result of insufficient user participation and involvement. Though the system is being developed to serve the user, most business people are too busy with their job demands to spend the time needed to clearly vocalize and document what they want the system to do.

Gathering the Right People

Also falling in this category is inconsistent requirements, which can result when many users are involved in the development of a CRM system. One division's needs may be in direct conflict with the requirements of another division, and diverse needs may compete for priority. One way to solve this problem is to develop some common goals and to focus on the customer. After all, you are implementing a customer management system to take care of the needs of customers. Ironically, often the customer is silent in CRM. A 2003 study by Bob Thompson, published for CRMGru.com, stated that CRM strategies focusing on the customer are more likely to succeed than those that don't.

Getting the right people to participate can be a challenge. The mix should include representatives from the software vendor, key users, and customers.

Software vendor associates: Must be knowledgeable not just in the technical aspect of the product but with best practices and industry standards. Avoid using sales people to help you develop requirements. Instead, ask for someone with functional knowledge to help guide you through the implementation process.

Key business users: Should participate in brainstorming meetings and joint application development sessions (JADS) to create effective use cases, and to write scenario-based test scripts. This type of test script is based on a user's daily workload and not a some vague functional description. Involve your users early on. Start with current business process flows such as an outbound sales call, or a prospect qualification process. Knowledge gaps can be filled with team members who have different knowledge and background.

Users: Look for someone who can see the big picture and has intimate knowledge with the day-to-day workings of the company. Users are required to contribute time and effort to the project beyond attending a few meetings. They must be able to speak for others in the company and understand the needs of the organization. Most important of all, they must share a customer centric vision.

Customer participants: Involve a customer with whom you have a very strong relationship with. This is not necessarily your biggest customer. As a matter of fact, make sure it is not your largest client so that any negative impact will not affect your bottom line materially. Customer participants must not let their special agendas override the overall goals of the organization. When involving your customers, be prudent and realistic. Avoid selecting customers who are negative about the company or have a bias against technology. Just as importantly, do not project pessimism. Be cautious not to present your company in a negative context.

Documenting the Review Processes

To harness the expertise of the team, take a "before" snapshot of your organization. Understand the needs of the organization by looking at the current business environment. How are customers being handling today? What policies are in place? How does information flow? Map out current process flows, documenting procedures, policies, key players, and organizational hierarchy. Gather documents on policies on controls and interview people involved in these processes. If policies are not documented, draw a process diagram or document what users do and have your users check for accuracy. Many users find it insightful to see on paper what they actually do—they might even realize that steps are missing or they are doing redundant work. In one organization, users discovered that they make twelve copies of a form and verify its information eight times—and they still had a 74 percent error rate!

Next, review these flows to see where CRM can help facilitate your processes. Highlight processes that can be replaced with a CRM and note where a CRM system can increase efficiency, improve quality, or automate manual processes. One company enlarged their process diagram, taped it to the wall and stuck sticky notes with the words CRM and a numbering convention. They then organized a use case referencing those CRM sticky notes.

Capturing requirements in an organization that does not have existing, global processes or has departmentally isolated processes is challenging because employees improvise throughout the business cycle. No centralized repository of data exists. No policies are in place and procedures are not documented. Customer relationships are managed via business lunches, golf games, and friendly phone calls. Contacts and phone numbers are typically handwritten in leather-bound notebooks and some sophisticated traders keep their own Rolodex; however, when these employees leave, their fancy notebooks go with them.

In one energy-trading arm of a former Fortune 100 company, traders scribbled down deals on scrap pieces of papers or even on cocktail napkins that were then handed in to the back-office for fulfillment.

Users find it hard to fathom a process they've never had but know they need. Get users to think outside the box, challenge their comfort level, and document anecdotal processes. White papers and cases on similar companies are also a good way to glean information. If you have an implementation partner, whether professional services from your software vendor or a third party consulting firm, inquire about requirements they used in other projects. They will usually remove all the proprietary information first but key implementation information will still be there. Some companies also may have generic requirements that you can tailor to your organization.

Combating Scope Creep:

A second reason for CRM implementation failure is scope creep. Changing business conditions can lead to changing requirements. However, undefined requirements and lack of prioritization can cause expanded scope and increased effort. Document your requirements in detail. Write a good-use case to illustrate what the system is expected to accomplish and what the primary actors (direct users) are expected to do.

Be sure to get sign offs on everything! Start with agreements between developers and users, and also involve your sponsors who are the real stakeholders. Business sponsors are typically in management. They are the people who have the vision to support your efforts, acquire funding for your project, and secure resources when you need them.

Break down your deliverables into manageable chunks and prioritize your deliverables. Believe it or not, this does more to prevent scope creep than other techniques. If you deliver the quick wins first, it helps your users to reap some early benefits and consequently you will gain more buy-in.

Perform "sound checks" from time to time to ensure continued agreement and focus. Reinforce the scope of your project and clarify requirements as needed. If the business environment changes, you might have to be flexible and provide solutions to incorporate these changes or alter the requirements to meet these new challenges. When changes occur, document and get sign off on your scope changes.

Compensating for Lack of Skilled Resources:

Mid-market companies may not always have the luxury of a full-fledged IT department and are often so overburdened with the cost of new technology and software licenses that they hesitate to invest in technical labor. They might hire a few people to work on the implementation and retain them for support after go-live or consultants are brought in to supplement the team.

However, mid-market companies must take the bold move to increase their technical staff to avoid being at the mercy of their software vendors and third-party consultants. An experienced program manager and senior developer are two key personnel you need to hire.

Your program manager should have strong experience in software development, with least three to five implementations, and ideally has some experience with your industry. To implement a CRM system, a good project manager, should be able to handle all the traditional tasks of project management such as budget control and project plan updates, keep good meeting minutes, and document issues. Risk assessment, project planning, resources management, problem solving and negotiation skills should all be part of the package. Your program manager should also be able to effectively communicate to upper management, users, and of course, to the team,.

Project managers in mid-sized companies cannot afford not to leave technical management to others or trust individuals to manage themselves. Your project manager must have enough technical experience to do their job, but not necessarily write hundreds of lines of code. If your project manager does not have the technical skills to solve technical issues, you should make sure other people in your team can. A strong technical developer would complement a less technical project manager.

Next, acquire a senior developer with over five years experience in development and knowledge of query tools such as Access, SQL or PLSQL, programming best practices, database experience, and be willing to learn new technology from different people. A good candidate should have development skills in the programming language, which your software is built on such as Java, JavaScript, C++, Visual Basic, or PERL. Additionally, experience with application methodology is more important than most people realize. Your developer must understand the software development cycle and practice good, controlled, organized methodology.

In addition to a program manager and a senior developer, a qualified business analyst, database administrator, and technical account manager should also compromise your team. A good business analyst can translate the business needs to software developers. A good applications database administrator (DBA) is a must because CRM applications rely heavily on the underlying database, no matter what database platform you use, Oracle, MS SQL Server, or IBM DB2. A seasoned applications DBA will be able to anticipate problems, resolve difficult issues and optimize performance.

Also invest in a technical expert from your vendor if one is not provided for you. One leading vendor instituted a very effective professional services program where a technical account manager (TAM) is assigned to every implementation. Minimum consulting hours are often negotiated at time of licensing and additional hours can be purchased as needed. These consultants are not sales oriented and are not involved in any licensing negotiations. Instead, they are usually from technical backgrounds, and have a broad knowledge of all CRM modules. For what they don't know, they can usually find someone who does. They also serve as the liaison between the customer and technical support, helping to escalate technical issues and bug reporting.

Because of the limited resources of mid-market companies, one way to avoid increasing head count and overhead is to supplement the technical team with independent contractors. Though consultants are often pricey, their cost can be capitalized and depreciated along with the project costs. Good consultants can offer diverse experience and are usually very well trained. They can bring proven methodology, rigorous training programs, certified technical skills, and multiple implementation experience. An additional advantage is that as a third-party unconnected to the software vendor, they can provide impartial feedback and input. Ensuring that your consultants are not also resellers of any software can help and. importantly, make sure you look for one with experience in your industry and has ample references when seeking consulting services.

Developing Infrastructure

In addition to possibly lacking an in-house technical team, mid-market companies may not possess a robust enough infrastructure to support CRM initiatives. There are two solutions to this dilemma. If an organization does not have an infrastructure to support its other growth initiatives, such as employee count, expanded sales territory, emerging market penetrations, increased technology, cutting edge products, then an injection of cash to overhaul the current infrastructure is called for. This organization needs the infrastructure to support the growing needs of a growing employee base, provide the technology to venture into new markets with new sales channels such as the Internet or telemarketing, and attract skilled IT programmers to develop cutting edge products and services.

A second solution is to use a hosting company. Many hosting companies have sprouted over the past decade, and they provide low cost storage and hosting solutions to companies who do not want to invest in perpetually changing technology. They provide the necessary hardware and network solutions to support CRM initiatives as well as the technical resources to help manage the environment. This solution also works for companies who want to try new technologies, but are not convinced that the ROI would justify the effort and capital investment in such highly depreciating assets. Another advantage of third-party storage and hosting is that a full time monitor does not have to be hired to watch servers or troubleshoot issues.

When selecting third-party storage or hosting solutions, ensure that they have redundancy systems, escalation processes, and disaster recovery plans. Request references and talk to their employees. Your consultants or technical staff often has experience working with these companies. Ask them for a recommendation. The IT world is close-knit. It seems everyone knows everyone. Ask around to see if anyone has heard anything negative about the hosting company you are contemplating engaging. Ask to see their financial statements and do some research on the Internet. In these days of consolidation and mergers, you want to look for a stable company.

Data Quality

According to one Gartner Group report, Successful CRM Hinges on Data Quality (April 2002) the number one killer of CRM initiatives is poor data quality. Many implementations focus so much on the big picture, that they overlook the most important component of their new system. The biggest asset your company is your customer data. It is the heart of every CRM system. Without it, you have no CRM system. So why not take good care of it?

Start from the beginning and evaluate your current data and how you manage and maintain it. How do you collect your data today? Is it a reliable source? Can you depend on this source in the future? What does your company do with the data? Do you mine it to provide different information? Understanding how your data fits in with your overall CRM strategy is key not just to the implementation, but to ongoing CRM initiatives as well. Identify the issues you have with your data today. What data issues plague your staff? Is it sufficient data? Is it timely? Is it accurate? If not, now is the time to cleanse it. Remove redundant data. Standardize your nomenclature. Verify your data. Are your addresses up-to-date? Spend the time to make sure your data does not present roadblocks to growth or sabotage your CRM initiative later down the road.

Converting bad data is never a good practice unless the system you are converting to has the tools to help with the data cleansing process. Some CRM applications allow you to merge customer records to remove duplicate data. Third party data quality tools that integrate with your CRM system can help standardize your data. For example, all instances of "street" is abbreviated to "St." or vice versa. Data validation tools help to verify your data. Is the street address your customer provided you a valid address? Are the zip codes correct? If you do business-to-business commerce (B2B), you might want to verify your data with the Dun and Bradstreet database or other B2B data source. Once your data is sufficiently "clean", develop a process to maintain the quality. Customer data is used throughout the organization so it is important to develop a customer data quality management (DQM) strategy that works for and involves everyone. Many companies acknowledge the value of their data and guard it as a critical assets by appointing a position or a department to monitor to control data quality.

An idea that has gathered popularity in the last few years is the concept of "data stewards." As old-fashion as this term sounds, it implies a position of control, careful attention, and responsibility. Data stewards are charged with not just monitoring customer data, but they are also responsible for establishing standards and procedures for maintaining data quality throughout the organization. These stewards must incorporate all departments and divisions in their total DQM strategy, recognizing that each have special needs, limitations, and challenges. Systems must also be considered. If an application has certain data requirements, such as data length, formats, or fixed elements, policiess for collecting, cleansing, or maintaining such data must account for these requirements. For example, if your mass marketing application requires a nine-digit zip code, you might want to validate your addresses with a United State Postal Service database. Stewards should also institute standardization of data. Published data standards help to get everyone in the company on the same page. Data entry uses these standards, and end-users should understand the different abbreviations, formats, and data elements. CRM initiatives must not forget this most crucial asset, and must carefully tend to and cultivate the data

Change Management

Most organizations resist change. As homeostatic animals, human instinct dictates that we remain the same regardless of our changing environment. However, in today's business environment, change is the only constant. While most employees recognize this, recognizing is not realizing. Changes must be positive, well communicated, gradual, and well planned. Employees like to be part of the solution. If benefits are publicized and presented in a tangible way, employees will recognize the value of a CRM system and will see themselves as enablers, adding to the value. Ultimately, the company's success is tied to individual job security, self-worth, job satisfaction, and resume building. Changes must also be gradual. Sudden changes are stressful and require more energy to adapt to. People must have time to change their habits, learn new technology, and understand the reasons for the change. This is not to say that drag out your implementation until your employees are 100 percent comfortable—you will never go live that way. Decide for your organization what the happy medium is.

Strategy for Change

Communicate your CRM initiative early, at the inception if possible. By the time you are in the testing phase, everyone in your company should be mentally prepared for this new "thing." The transition phase must be well planned. The key elements of a well-executed change management strategy are:

  • Motivating promotion
  • Thorough testing
  • Sustainable training
  • Long-term usage campaigns

Motivating Promotion: Your employees cannot touch, see, or feel the change that is about to happen. Immerse them in your CRM initiative. Print posters to publicize the company's goals. Tell them what benefits they will experience personally, less work, better quality, less frustration, smoother processes. Hold town hall meetings to air issues and concerns. Employees want to be heard. They want to be reasoned with. They want to do what is best for the organization. Involve key users in your transition plans, anoint "super users" to spread the message and extend the knowledge. Use role playing to help users understand the system in the context of their everyday work environment.

Thorough Testing: Develop good test scripts that incorporate real life scenarios. System testers cannot replace user acceptance testing (UAT). Utilize users in your testing phase. Your system will only succeed if it could withstand the scrutiny of the users. Don't be afraid that users will discover bugs and faults in the system. Use this as an opportunity to improve it. Develop a realistic timeline for testing. Establish a bug tracking/error reporting tool and follow up with your users. You could not pay someone to do a better job of testing than your users.

Sustainable Training: Provide hands-on training to familiarize users with the system. Lecture-style training classes do not work as well as getting the users in front of the computer. Have users plug-in real data, and experience the results. Show them how their work fits into the whole picture. Supply easy-to-understand training manuals to help maintain knowledge. Train the whole organization. Schedule training classes so you don't interfere with daily work. Start training before go live starting with key users and maintain a training schedule a few days or weeks after go-live. Keep a log of who was trained.

Often, management are left out of the training phase because of their demanding schedule, their ego, or because they may not be direct users of the system. The myth that leaders of the company do not need to know how to use the system should be dispersed. In truth, leaders should attend training to gain knowledge of the technology they advocated. Secondly, they need to feel their employees' gains and pains. Thirdly, CRM initiatives involve everyone in the company, from the lowest to the highest. Executives are the drivers of the process; they must learn to understand and use that the system that enables the company to meet its stated business goals. Finally, employee morale is greatly affected by what leaders do. People will follow if their leaders lead by example.

Long-term Usage Campaigns: CRM is an ongoing process. It does not stop when the software has been implemented and the last person is trained. How do you keep your employees motivated? How do you increase productivity? What metrics will you use for success? How do you increase usage? How do you wean your employees from one system to another? What are the consequences of not using the system? All these questions must be addressed to ensure long-term usage of your CRM system.

Some companies use phased releases to ease the pain and keep the excitement increasing from one phase to another. You can also use promotional products to motivate usage. Customized mouse pads with exciting graphics or motivational phrases help to help to encourage users. Bright, user-friendly overlays prompt users of keyboard shortcuts. Small, laminated quick reference guides help users to remember key procedures. Weekly or monthly themes can highlight certain CRM goals. For example, your company wants to understand why customers call into your technical support line, if you would want to increase usage of the call log feature of your CRM application. Have a contest on which team documented the most calls, not just answer the phone. Make a large chart and post it in your call center. Use system logs to report how many times users log in each day, each week, each month and report by department, by division, or by title. Also get feedback from users to find out why the system is or is not being used.

You don't have to make using the CRM application a condition of employment as some companies do. However, non-compliance could affect performance. It could be a sign of inflexibility, lack of skills or knowledge, or disagreement with the company's goals and mission. Employers often present CRM as just another software to be trained on, another technology to get used to but, it is a shift in how your organization thinks, operates, and performs. Pay attention to the human factor of change. Plan for it, but be flexible enough to modify your plans as needed.


What's Really Driving Business Intelligence?

Finding the Deeper Trend in Business Intelligence:

If you follow the logic of the major analysts covering the business intelligence (BI) market, the market drivers for business intelligence software are based on fairly simple environmental factors. The most commonly cited market drivers are the following:

  1. Increasing Regulation—New laws in both the US and Europe are requiring companies to make their external reporting more transparent, forcing business to develop better systems for storing and retrieving the most current and detailed information on operations.

  2. Information Overload—Having invested heavily in CRM, ERP, and SCM systems, many businesses are awash in data, but short on actionable intelligence. Being able to aggregate, mine, and analyze data in order to prepare for and respond to business and market events is the next step in making IT investments pay off.

  3. Demand for Accountability and Metrics—A slow economic recovery has forced many businesses to continue trimming budgets, while requiring greater accountability for every area of spending. Business intelligence, and its associated data-mining, analytics and scorecards, provides the tools necessary to track performance metrics tied directly to strategic corporate goals.

  4. Need to Improve Competitive Responsiveness—With markets exposed to increasing competition, customer demand and pricing pressure, businesses need to reduce cycles by accelerating processes that support aggressive competitive strategies. BI initiatives provide real-time information that can help businesses eliminate process delays and streamline management to improve decision-making and market response.

There's nothing wrong with these descriptions of existing market conditions. Each of them is a correct and compelling reason for businesses to support BI initiatives. However, they don't tell the whole story. In fact none of these market drivers, taken individually or taken as a whole, are enough to explain the level of investment being made in business intelligence software.

Think about it. Businesses have found ways to skirt or delay the impact of increasing regulations for decades. Why would they suddenly respond so rapidly to new regulations today? While information overload is acute, many businesses took a soaking in IS investments over the past few years. What smart CEO would throw good money after bad to try and rescue a previous investment? Metrics and accountability are certainly in high demand while budgets are tight, but how many businesses would invest millions of dollars just to be confident their million-dollar investments are sound? That kind of long-term thinking doesn't move markets in our quarterly-driven world. And finally, yes, businesses are being compelled to be more efficient and effective in order to compete, but that battle has been shaping up for decades among TQM, Six-Sigma, lean production methodologies—does anyone really believe the end of the rainbow is only a dashboard away?

While, each of these market drivers is accurate, they're only symptoms of a much deeper drive—a drive that is shaped by a concern far greater than the threat of regulation, information overload, accountability or competitive response. It's a drive that reflects the deepest fears of a CFO. It's a drive that shapes the search for CEOs that can move the businesses that move markets. It's a drive that cuts straight to the bottom line of the corporation, because it's about the single, all-important factor that defines the success of every business today.

It's all about how the value of a business is measured.

Measuring the Value of a Business:

To understand how the value of a business is measured, you need to know what investors look at when they assess a company's worth. Investors consider three things above all else: profitability, growth, and risk. A business needs positive cash flow, clear potential for increasing it, and some hedge against the uncertainty of achieving growth in the face of volatility.

Investors, typical of most people involved in finance, like their money easily counted. They like the kind of value that shows up on a balance sheet—concrete worth that lends itself to crisp calculations showing money in the bank. It's a mentality that relies on relentless consistency, structure, and predictability. The fundamental concepts of valuation, concepts like net present value, are conceived as formulas that everyone, everywhere, in every situation writes the same way.

So when people started talking about "changing fundamentals" during the dot-com boom, a lot of financial people just snickered. Sure, a lot of companies were being sold at values ridiculously inflated over the value shown on the balance sheet, but all that meant was an impending correction—and boy did that correction come. The boom turned into a bust, and many of the highest-flying companies went broke. Markets change, fundamentals don't.

But a few quarters into the recovery, some financial analysts found an interesting trend in the post-bubble numbers. Those market-to-book ratios that had been so inflated during the bubble—the difference between what a business is worth on its balance sheet and what the market would be willing to pay for it—did indeed correct, but not as much as many expected. Since 1982, the businesses making up the S&P 500 had been growing in market value over and above what they appeared to be worth. After the bust, the trend didn't reverse, it merely corrected to the curve that had been steadily increasing for two decades. According to Jonathan Knowles of Brand Finance, a consultancy that specializes in the valuation of businesses, the tangible assets that used to account for 75 percent of a company's stock market value in the 1980s now only accounts for 22 percent of market value.

What on earth does this have to do with BI, you ask? Let's cut to the chase.

Turning Intangibles into Value:

We've just seen that the way markets value a business is growing increasingly disconnected from the hard assets of the balance sheet. No one is certain why business valuations keep climbing above assets, but there is a good hypothesis. The theory is that businesses are realizing an increasing level of value from intangibles—things that help a company create value, but that don't show up on the balance sheet. Identifying those intangibles so they can be modeled, measured and optimized is a powerful market driver that is sending managers scrambling into every corner of business operations.

There are lots of intangibles that have the potential to create value, including intellectual property, business processes, specialized training, skilled employees, customer intimacy, corporate culture, brand equity, and too many others to mention. The question for businesses and investors is how to get your arms around intangibles. How do you identify the intangibles that contribute to the creation of value? How do you measure those intangibles to understand the nature of the value they create? How do you improve the value of those intangibles to measurably grow the bottom line?

If, as the trend line of market-to-book ratios shows, the value of intangibles is on a growth curve, these are important questions indeed. The need to identify, quantify and optimize the underlying mechanisms that are driving an increasing portion of a business's value is far more compelling than concern over things like information overload or increasing regulation. Sure, BI is evolving to help you get more intelligence out of your data, faster, to support better decision-making with real-time responsiveness while keeping your business in compliance with new regulations. But don't "miss the forest for the trees". Beyond the drumbeat of short-term pressures, there are more sustainable advantages to be gained from a solid understand of BI tools. Perhaps the most important advantage is the most intangible: the knowledge of what adds value to your company's bottom line.

BI Technology and Beyond:Business intelligence tools provide tremendous utility in running a more responsive and streamlined business. But they also provide the power to learn what dials can be turned to tune your performance in generating market value. Remember taking chemistry in school? BI tools are like a lab full of chemistry equipment: it's all good for measuring and reporting information from events, but if you don't have a hypothesis to test, you can be incredibly efficient in measuring events and still learn nothing about how things really work.

Along with the growing trend in BI initiatives, many businesses are developing parallel programs for modeling and testing their intangible assets, particularly business processes. Some of the more popular programs, like the balanced scorecard, facilitate the identification of value-creating intangibles directly linked to corporate strategy. Like a scientific experiment, it's all about developing a hypothesis about how things work, and using the tools to prove assumptions right or wrong. The ultimate goal is to develop a model that can help you not only measure important events, but to predict results and repeat specific outcomes on demand.

The first place to start is by identifying the intangibles that add value to your business, and how they function. How does your company's corporate strategy shape the business processes of your department? What business processes in your department create real value for your company? How do those processes function, and how can they be best measured for performance? How can they be optimized to provide greater value?

If you're not familiar with concepts like the balanced scorecard, it's a good introduction to the concept of value-creating intangibles. A recent book released by the creators of the balanced scorecard, called Strategy Maps, gives a decent overview of the process of reducing corporate strategy to supporting business processes and their underlying intangibles. It's heady stuff, but if you want to understand what's really driving the BI trend, you need to move beyond information into real knowledge.