Software Decisions and Opportunity Costs

As usual, the DIA annual meeting this year was a great success. The thing I enjoy the most is a walk down ALL the aisles of the increasingly dramatic trade show floor, where vendors, consultants, and service providers of all shapes and sizes get to display their wares and offerings. "Reading" the show floor can create a very vivid snapshot of the issues facing the industry in any given year.

Paul Bleicher

The grand illusion

There seem to be dozens of different vendors offering any particular technology, many with an expensive and well-designed booth with carefully worded descriptions of their systems. If somebody is interested in piloting a particular technology and goes around to all the vendors at the show for screen demos, it will become increasingly difficult to distinguish one from another. They all seem to have a great Web demo, full of functionality, and all the necessary 24/7/365 services that one could want.

I wonder how any nontechnical, nonexpert can possibly distinguish between the different technologies and vendors. I can imagine that many decide that there isn't really a difference between the offerings. "Why not," they might conclude, "just pick one and try it on an unimportant project. Even if it doesn't work so well, what have you lost?" they might say.

To anyone who has taken economics, the answer is obvious: opportunity cost. From the Wikipedia (today, at least): Opportunity cost is...the cost of something in terms of an opportunity forgone (and the benefits which could be received from that opportunity), or the most valuable forgone alternative.

An allegorical tale

With opportunity cost firmly in mind, let me relate to you a brief personal story of the last few days. As the title to this section suggests, consider the parallels between this story and a clinical trial pilot of some technology.

The story starts, innocently enough, with a little pet project I have been putting off. I was asked to design a timeline of the history of my company with photos, links, and descriptions of key milestones. It is, in my mind, an ideal task for everyone to participate in. But somehow, I knew that sending an email out to the company was unlikely to unleash an onslaught of descriptions in my inbox. In addition, I figured, few would be formatted properly, thereby increasing my work.

As I thought about it, I came upon what seemed to be the perfect solution. I would use an open-source timeline program (http://simile.mit.edu/timeline/) that takes XML and/or JSON files (typical export files from a database application) and automatically creates a very functional timeline. To get the data into the database, I would create a Web page with a single form that would allow people to enter the properly formatted information onto a Web page. Now the task would be easy—send people a request to help with the history along with a link to the form on a Web page. If I did it right, they would see the data appear on the timeline as they entered the Web page.

The best rule about technology that I know is: Use the software that best fits the task at hand. For a simple Web page form, I wouldn't need to build a relational database with a sophisticated application on top simply to collect five or six fields. In fact, I realized that this would be a great opportunity to try out one of the really cool new Web-based programming tools that don't require any programming experience.

These programs are part of a new breed of Web 2.0 Software as a Service (SaaS) offering. SaaS applications are provided entirely through a Web browser and many use technology that gives them the feel and functionality of a client application. Many of these are so functional they are being discussed as threats to the hegemony of Microsoft Office, including Google Spreadsheets and Google Docs.

In fact, Google isn't completely dominating this concept. One company that has an impressive array of SaaS programs is Zoho (www.zoho.com). Zoho offers free or very inexpensive applications, including a word processor, spreadsheet, presentation software, Web meetings, email programs and more. These programs are highly functional and offer more than many people would ever need in these areas.

Zoho's newest offering was exactly what I (thought I) needed for my project. Called Zoho Creator, it is an application that allows nonprogrammers to create database applications for simple business use involving the entry and viewing of data. Without worrying about database design, the Zoho user simply drags the particular type of data entry fields onto a form (dropdown boxes, text boxes, checkboxes, etc.) and voila: Automatic database entry screens and reports are created.

With similar drag and drop validation elements, it is very easy to do range checks and logic checks on and between fields. In literally minutes it is possible to have a working data entry form with underlying database. These forms can be very easily inserted into Web pages, creating the look and feel of a professional Web form. In fact, some companies and organizations are using Zoho forms for simple surveys, course signups, and other simple tasks.

I toyed with using Zoho's newest competitor, Coghead (http://www.coghead.com), but decided to stick with Zoho because I found it to be simpler to use and faster to implement.

And here my troubles began.

From hours to days

Within the first 30 minutes of using Zoho, I had built the form I wanted and tested it on a Web page. I was actually amazed at how easily and quickly the application came together. Despite this, I knew I had a few hours of tweaking and developing range checks to make sure the application would collect clean data. I began to enter some scripts, and the minutes turned to hours as they always do when programming. I added, tested, and modified in iterative cycles. Some of the features that made the Zoho application easy to use became annoying, as they took more steps and longer than a real Web development tool. Despite these minor annoyances, the tool was fast and easy to use.

At some point during the evening I entered a snippet of script that prevented my application from working and issued an error message. Not knowing which validation script item it was that caused this, I spent an hour or two taking pieces of script out of the application, each time restarting and entering data. Over time, the application became really slow and funny things began happening. I couldn't enter data. I couldn't delete data. I couldn't even delete the application itself. Finally, in exasperation, I tried to delete my entire account and start over. After going through this same cycle three times, the entire Zoho Creator "execution engine" choked on two lines of my script. Apparently, nobody, including me, could log in for a period of hours. Through the whole process, I had several calls to the help desk with friendly responses and rapid problem solving.

Before I knew it I had spent about 12 hours over two separate days developing a simple Web page that I could have done in about three hours with a variety of well-tested Web development tools.

Of course, going in I understood that all Web 2.0 applications are in "beta" for months to years (take a look at the Gmail logo!) and Zoho Creator 2.0 was only one month old. I reasoned that all software had bugs and recently launched beta software would have a bunch, but somehow I thought I would be the one to avoid all the obstacles.

The next morning the entire development team from Zoho Creator called me from India. It seems that a bug in their code caused my little program to bring their application server to a grinding halt, preventing me and anyone from logging in. They were incredibly helpful and very responsive, fixing the problem on the spot. Zoho Creator, they explained, was a work in progress, still young and prone to these sorts of problems. Although it looks great and does exactly what I want it to, it isn't the robust timesaver I hoped it would be today. By my next project, I am sure it will be robust and reliable.

When I related this story to our CTO, he told me that this was the very reason that he spends a great deal of time researching technologies and tools—to make sure that the ones our developers use are stable and don't create these same sort of problems, which can cause delays and endless frustration.

The moral of the story?

When I decided on a new, flashy Web 2.0 development tool for this simple project, I thought I was matching functionality with task. However, I wasn't taking into account the opportunity cost, which was significant. I had spent all of my free time for two days in frustration and wasted a time window when I could have completed my project and several others. Although I learned a lot about a new tool, I did little else during these two days.

In choosing Zoho Creator, I had the same problem that the DIA attendee has at the booth demos—I was considering the superficial features and not the quality, robustness, and scalability of the tool. While an evening or two of frustration isn't a large problem for a one Web page project, it can be greatly magnified when implementing clinical trial technology, even for a single pilot.

The moral of the story is: When choosing clinical trial software of any sort for your important projects, respect your opportunity costs and do your due diligence. Seek out and interview those with experience (preferably recent experience) in using the application. Find out how many companies are using it, on how many clinical trials, and how quickly they were able to scale up. Don't get fooled by the allure of feature-rich, "easy-to-use" software that has great promise but has had only limited use. It may turn out to be a major source of frustration, or much worse. Go with the features you need and the tools that work, and work reliably.

Paul Bleicher MD, PhD, is the founder and chairman of Phase Forward, 880 Winter Street,Waltham, MA 02451, (888) 703-1122, paul.bleicher@phaseforward.com, www.phaseforward.com. He is member of the Applied Clinical Trials Editorial Advisory Board.