How to find the best content management system (or any other tool)

Every day I see questions posted on various sites asking for the "best content management system" or the best XML authoring tool or the best... In order to find the best, you have to define what that means. How else are you going to judge your success at finding it?

I'll say up front that the conclusions drawn in this article should not be a surprise to anyone. In fact, I bet you can guess the biggest one: Gather requirements before you start. Whenever I give this presentation, audience reactions generally fall into one of two categories—“Uh-oh” or “Well, duh!”

Whichever camp you fall into, you’re still probably like the rest of us who take one or more simple steps to uncover the justification that will permit the purchase the tool that we already have our hearts set on:

• We go looking for reviews online.
• We read just enough articles to confirm our choice.
• We ask for a demo.
• We play with a trial version.
• We compare pricing.

We think that after all of that, the last step—comparison shopping on price—is due diligence. It's not.

We all know we should do real requirements gathering, but we don’t want to do that much work and, what’s worse, in today’s social-media connected world, we don’t think we need to. We’ve checked the review online, right?. Even the “Well, duh!” crowd falls into this trap. Almost no one really gathers requirements.

In this article, I hope I can convince you to gather requirements before you buy anything that is more than a simple retail product, whether it's your "best content management system" or anything else. I’ll tell you why you need to do this. I’ll give you strategies to make the exercise easier. I’ll tell you what to watch out for. I’ll tell you who you can trust (and how far you can trust what they’re telling you).

Any product that requires input, approval, buy-in, or consensus from more than one person requires requirements gathering.

I’ve been on both sides of the fence. I’ve been a customer and I’ve been a vendor. I’ve bought and sold software in Silicon Valley, a place that is notorious to software sales people for being well known as the hardest place to sell software. I’ll share what I’ve learned with you because I want you to be wise, savvy, and powerful purchasers. I want you to know how to serve yourselves and your companies well, today and in the future.

Here we go.

What kinds of purchases need requirements gathering?

Any product that requires input, approval, buy-in, or consensus from more than one person.

For example, my husband, Paul, and I share a car. We regularly talk about buying a second one. I am part of a virtual team. Paul works in an office approximately 7 miles away from home. I can walk to most amenities and have many public transportation options. He bikes to work several days a week. We work together to make sure each of us gets the when we need it. Neither one of us can buy a car without the other’s agreement.

In a corporate organization, if you want to buy a new authoring tool, or "the best" content management system, or conference tuition, or even new break room coffee maker, you need authorization and approval. Someone else has to agree to your purchase so the vendor gets a check or you get reimbursed.

Financial power is not the only factor that determines who is part of the decision making process.

Back to the car example. For me, buying a car means having someone else perform the maintenance. I am not interested in learning to be a mechanic. I have other things that I’d rather spend my time one. For me, an acceptable second car is one whose maintenance is both easy and accessible—the car must be robust, not requiring constant maintenance; qualified mechanics must be easy to find and relatively near by. For Paul, he doesn’t mind doing some things himself. In fact, an acceptable second car is one that he can take on the trail. Repairs on the road are fun! It’s perfectly OK if the specialized mechanic is over an hour away. If I agree to the second car, I’m agreeing to have my workload added to especially if I’m the one who has to drive it to its maintenance appointment because Paul has a conflict.

In a corporate environment, it’s no different. Who’s time or skills do you need to make your tool function. What about the IT people who must purchase, install, and maintain that server for that content management system you want? Will you be adding to their workload or their responsibilities? If the answer is yes, you need their buy-in, their agreement. They are a part of your decision whether you like it or not.

If you think, “well that’s their job, right?” Try looking at it from a different angle. What is necessary for you and your purchase to be successful?

Does your purchase require:

• Extensive implementation effort?
• Consultants or Sales Engineer assistance?
• New skill development (for you, someone else, or several people)?
• Negotiation with, and support from, other entities inside or outside your nucleus?
• A sizeable budget?
• Resources to be directed to your project—and away from someone else’s, even if for a limited time?

If you answer yes to any of these questions, you’re dealing with a complex product and you need to have your ducks in a row before you go asking to buy something.

If all we have done is the superficial due diligence I described earlier, we are limiting ourselves and doing our companies a great disservice. We have tried to treat a complex situation as if it’s a simple retail purchase. We have put ourselves at risk of “confirmation bias” (our own or someone else’s). We have failed to take into account essential requirements that are part of our unique and individual situation. We are almost guaranteeing failure and unnecessary expense.

Several years ago, I saw Ann Rockley speak at LavaCon. She was talking about how her company was experiencing a revitalized interest from customers that had spoken with them, but not hired them, years before. They were in a pickle. They had decided to do it themselves. They had rushed tools decisions, plowed full steam ahead into XML publishing. Now they were coming back to her and when she looked into the situation, she’d discovered they had the “same mess, different tools.” If we all know that requirements gathering is the key to success, why don’t we do it more often?

I think that it’s because we have applied simple selection processes—processes that work fine for commodity products—to the purchase of complex products. We do this is because we believe, somehow, that there is a single, best product out there. A view that is reinforced by sensationalist product marketing.

This brings us to myth #1.

Myth #1: There is a best content management system (or "best" anything)

Not every tool is right for every company, every industry, or every situation. There can be a lot of reasons that a fit isn’t right. For example, a long time ago I worked for Juniper Networks, which was started by ex-Cisco employees. Despite the fact that people used to bounce back the two companies had a history of doing the opposite of the other. What Juniper did, Cisco would not do, and vice versa. I heard this same story from several software sales professionals who had tried to sell their product into both companies. They would frequently get one, but not the other. Getting one was often a black mark in the eyes of the other.

Corporate culture can color what best means. What works for HP won’t necessarily work for their OEM, despite the close working relationship. What works for Schneider Electric doesn’t necessarily work for a startup with only one or two writers. Big companies have resources that small companies don’t. The economics differ greatly. Just because you’re in the same industry doesn’t guarantee a fit.

We put ourselves at risk of “confirmation bias” (our own or someone else’s). We have failed to take into account essential requirements that are part of our unique and individual situation. We are almost guaranteeing failure and unnecessary expense.

You can’t take someone else’s results and assume they are right for you. Unfortunately, it happens all too often. You have to know your reality. No one else can do your work for you. And yet, every week I see questions like these on LinkedIn:

• “What is the best content management structure to manage content?
• “What software do you use for content management and version control”
• “Results are in”...”Here’s the list of top tools”

When you see questions like these and the answers that follow, do you ask yourself:

• Did the reviewer get paid (in one way or another) to write that review?
• Did the reviewer leave some tools out because there was no compensation provided from the vendor?
• Why did they equate version control with content management?
• “Top tools” according to what criteria? Is it spelled out?
• “Top tools” for which situation? Industry? For what purpose?

There are several well known consulting firms that only work or recommend with products that they have compensatory relationships with. They will never recommend or even discuss any product for which they don’t already have a compensation plan in place and they rarely, if ever, declare those relationships publicly. Compensatory relationships are not the same as partnerships. Partnerships may or may not have compensation involved. Be wary of the assumption of neutrality in any “Top Tools” list.

Assumptions are all over these kinds of questions. It’s the assumptions that will get you into trouble when applying someone else’s question and answer to your situation. Just look at the assumptions embedded deeply in a question like this one:

“How do I go about converting FrameMaker to XML? Are there scripts available...can you give me some pointers?”

There are entire companies who make a living doing data conversion. If there are entire companies like that, you can probably bet that it’s going to take more than a script to do the job.

And what about this one:

“Recommendations for open-source help authoring system”

This question has so many assumptions built into it, it's hard to know where to start deconstructing it.

Try this:

In 2010, Sarah O’Keefe gave a presentation at LavaCon. She provided some statistics about the cost of DITA Open Toolkit projects. The DITA-OT is a free, open source project. Scriptorium is a well-known DITA-OT implementer. She reported that Scriptorium found the average DITA-OT project to cost over $100,000 ^[1].

And that brings us directly to myth #2.

Myth #2: Product cost is total cost

Technical writing departments struggle for budgets. We are often an afterthought, a cost center. It’s not surprising that we try to do a lot under the radar.

We look to open tools because the cost associated with them is seemingly invisible to management:

• No capital expenditures.
• No approvals for expense reports.
• No discussions with executives.

We feel that we can “spend our own time” making things work. That somehow, it’s zero cost when the tool is free.

Unfortunately, time and effort are not zero cost. I don’t know where you get free labor or copious amounts of spare time, but i’d sure like to!

And, if you think it’s a sunk cost, because you’re a salaried employee, you’re still missing something. If you’re working a technical writer, you already have a full-time job (and it probably isn’t to develop software or tools). Given that we know that the cost of the time and effort to implement the DITA-OT is over $100,000, we’re looking at the the cost of for one full-time technical writer here in Silicon Valley. Unless you’re working 80 hours a week—40 at the job they hired you for and 40 more for the tools you’re building—you’re actually an underperformer. In addition, they’re probably not paying you double for all those extra hours. That “free” tool, cost you $100,000 that you gave away free to your employer. I’m sure they appreciate getting all that work for free. A year down the road, will you?

Don’t forget to take into account the cost of long-term maintenance for those tools you’re building. Developing software means that you’re responsible for fixing bugs, upgrades, implementing changes when the next DITA version is approved and released, just like any other software company. You are effectively growing a software company inside your company. If your company isn’t already a software company, this is probably outside their core business. Even if they are a software company, unless their core business is building DITA tools, you’re still probably outside your company’s core business model.

Even if you think you’re only going to try it out—get a feel for how it all works—you’ve got to be careful about how you position it, so you don’t end up somewhere you don’t want to be.

Which brings us to myth #3.

Myth #3: Pilot will tell

Too often, a “pilot” is an excuse for hands-on, free play. Pilots are rarely done correctly because:

• No goals are set.
• No specific tests are done.
• No criteria for viability or acceptability are defined in advance.

If you’re going to do a pilot, you must understand exactly what you’re testing and what the goals are. How else will you evaluate a successful conclusion?

A real pilot requires full implementation. In reality, pilots are typically “phase one” implementation. And once you’ve spent all that money (time and effort for you and others), getting it to the point where you can run your evaluation, are you really going to toss that product to the curb? Practically speaking, the answer is almost invariably no. (At least until you can’t take it any more. Remember the Ann Rockley story.)

Think about remodeling your kitchen. Do you ask three to five contractors to come in, one after the other, to remodel your kitchen? Do you tell them you want them to come in, set everything up, let you try it out for a while, then have the next one come in and repeat the process three to five times? Do you get them to do all of this for free for you in the hope that you will choose them? Even if you could get them to agree, given the average kitchen remodeling time, you will spend a year without a kitchen and have very little to show for it.

You’re not going to do this with your kitchen and you’re certainly not going to do it with your work environment because you can’t have all that disruption and still meet your deliverables and corporate goals. It’s not a test of the tools and when it is, it’s disastrous.

Rebekka Anderson from UC Davis wrote a paper ^[2] about following a team who did a “pilot” that was hands-on, free play, and she evaluated the criteria for successful implementation. Her research showed that the tool would not guarantee you success or failure. It’s the change management of the implementation, the process, and the people. Tools only make up about 10% of a project’s success. All of the failure has to do with process and people.

Pilots are rarely done correctly. No goals are set. No specific tests are done. No criteria for viability or acceptability are defined in advance. If you’re going to do a pilot, understand exactly what you’re testing and what the goals are.

If you’re a tactile learner and really need the hands-on free play, go take a training class from the vendor. Even with travel costs, the cost of taking a training class is a lot less than doing a full implementation just to get your hands on the product. It’s like a field trip. You’ll be able to talk to the trainer, who will assume you’ve already purchased, and to other customers. You will learn far more from the class than from a pilot.

Which speaks directly to myth #4.

Myth #4: Reviews will reveal

If 90% of success or failure is in the change management of people and process, reviews that focus solely on tools misrepresent the problems the project experience. For any complex product, there are a lot of moving pieces, a lot of contributors, to success or failure.

A good reviewer will give you a score card. They will list out all the project phases... They will distinguish the tool from the implementation. They will address training of staff. They will talk about the order of implementation. They will be able to tie every success or failure to a specific requirement.

Be careful of reviews where misunderstood requirements or bad implementation is reported as a tools failure. Does the review identify which part of the project really led to success or failure?

A friend of mine at a software company in Seattle had her project hamstrung because the final purchase order left off the services component that they knew they would need to get the content management system up and running. They were delayed six months while the management team debated spending more money than originally allotted. The rest of the writing team didn’t see any of that. The writers saw only that instead of the new product they had been promised they got a delayed implementation, lots of workarounds, and bad user experience that had nothing to do with the product or the software company that made it.

At Juniper, I would regularly hear from writers that “Arbortext wasn’t working.” Every time I went to their desk to find out what was wrong, it turned out they hadn’t mapped the drive on their Windows machine and so couldn’t access the files. Arbortext wasn’t down at all, but because it was front and center in this new process, this new experience, it got the brunt of dissatisfaction and poor reviews in the internal discussions.

Be careful. A good reviewer will give you a score card. They will list out all the project phases, the number of people involved in each phase and the role of each person. They will distinguish the tool from the implementation. They will address training of staff. They will talk about the order of implementation. They will be able to tie every success or failure to a specific requirement.

There's one last myth I want to address. And it's one that I know you will find hard to believe.

Myth #5: Sales is the enemy.

We all have an uneasy relationship with sales professionals, that comes out of bad experience. However, buying a complex product isn’t like buying a used car and you are not alone in your evaluation, purchase, and implementation.

If you understand the economics of how sales works, your sales rep can be a partner for you.

Sales professionals for complex products are just as concerned as you are that their product is a good fit for you. If you want to see this for yourself, spend some time listening in to your sales team when they get together.

No one wants to waste time or money. Sales professionals are acutely aware of the need to be economical with their time and attention. They don’t want to pursue leads that are a bad fit. A bad customer can kill a company and if the company fails, no one gets paid. They are responsible for making sure that they bring in the money that pays the salaries of everyone in their company, not just themselves. Most of all, they want to be able to qualify a customer as much as a customer wants to qualify a product. And they want to do it as quickly as possible.

Consequently, you don’t need to play coy. It is their job to be an expert in their product. They know what factors are critical for success and failure. They know the problem space their product addresses. They know what long-term resources are necessary for success. They know what interdependencies need to be taken into account. When they’re asking you all those questions that feel invasive, this is their way of sorting through the list of questions that would disqualify you as a customer.

Ultimately, you are both trying to answer the same question:

What is the irrefutable reason that you will buy a specific product from a specific vendor within a specified time frame?

• Irrefutable: Your reason is, by definition, undeniable and everyone agrees that your purchase is essential. No other project has priority. You must have the resources necessary. Everyone agrees.
• A specific product: You have identified that this specific product is the right one. You cannot purchase if you’re still comparing. You can’t have an irrefutable reason if you don’t have a specific product picked out.
• A specific vendor: You know exactly who you will buy from.
• A specified time frame: Schedules are cleared. Resources are allocated. You know when the purchase will happen and when you will start implementing.

The sales reps from the various products you investigate will help you answer this question because it’s the same question they have for you. They need to know absolute—buy or not. From them or not. When will the money will come in? When can they move on to the next sale?

Sales is a tough job. I highly recommend spending some time with your sales team, if they’ll let you. It’s an eye opening experience that will change your relationship with sales forever.

If you treat a sales rep like a partner and not like an enemy who is out to cheat you, you’ll be surprised at the results. About a week after I gave this presentation to the San Francisco Bay STC Chapter, I got the following in email:

However, if you still don’t trust your sales rep, there are at least three other possible resources at your disposal.

Another option: Go to IT for help

IT buys, implements, migrates, and maintains complex products every day. Your project will require their support as well. They may need to buy servers or install the software. They will need to do patches and ongoing maintenance. They will be delighted to help you talk to sales reps. However, you need to remember that they are focused on the part that intersects with their jobs, which is as it should be. They will not understand your workflow, your needs. They will expect that you know those things and that you have applied all the same due diligence and requirements gathering that they do. They will not do it for you. They can help, but you still need to do your own requirements gathering.

One option: Enlist a consultant

They evaluate and implement complex products for a living. Even if you only have them for a couple of hours to review your requirements or to listen in on a phone call or demo, their wider experience can help you. They will hear things you don’t. They will understand jargon and differentiators. They have more reach and more experience with a wider variety of complex products. Just be sure that you understand where their biases lie and who their partners are.

Get the strategy right, not the technology

John Mancini from AIIM wrote a white paper ^[3] about “the importance of spending more time on the planning end of the project before hopping into vendor roulette.” He also said that if you don’t have expertise with the product you’re buying or the class of products to “hire an independent consultant...to help you with the strategy roadmap and definition.” If you get the strategy right, everything else will work out. He confirmed Rebekka Anderson’s conclusions, “Focus on getting the strategy right, not the technology” if you want to succeed.

Gather your requirements because you don’t want to be the one responsible for the “same mess, different tools.” You want to succeed. I want you to succeed. Everyone wants you to succeed. Now that you have the tools, do the work. It’s worth it.

Need some examples?

Anyone who's heard me give this presentation has heard the many personal anecdotes I give during it to illustrate the points. While funny, the stories I tell during the presentation are real-life examples that have encouraged many people to seek me out for advice. Inevitably, as I'm sure you can tell by now, I ask them many questions about why they whatever it is and what purpose it is to serve for them.

Recently, I decided to write up one of my personal stories and the method by which I defined my requirements before making a particular recommendation for, of all things, a computer headset for conference calls. The one I used finally broke down and I needed a replacement. I could have just looked at a site like LifeHacker and picked a pair, but I knew my requirements were not for gaming or music listening. Rather, I needed something that served me best in a business situation.

So, after doing my research, I wrote up my recommendations and, better still, the requirements I had going in. That way, if you are looking for something similar, you can decide for yourself. You can determine which requirements we share, which you don't care about, and can make a determination that best suits your needs (not mine).

If you want to see an example, head on over and read more about my journey to find The Best USB Headsets for Audio and Web Conferencing.

If you want another, you can see exactly what I keep in my travel kit for frequent presenters on the road. It's all written up in my article Technology that solves the top technical problems for presenters (My Presenter's Tech Travel Kit).

View the Slides

If you want to see the slide presentation based on this article, you can find them attached to the video recording of the session I gave at the TC Dojo: How Do I Pick the Right Tool For Me?

This article was published originally in STC Intercom in September 2016

About Liz Fraley

Liz Fraley, CEO of Single-Sourcing Solutions, is well known for her advocacy of defining requirements. She has founded two companies, sits on the boards of three non-profits, and is constantly coming up with new ways to share knowledge in the technical communications and content industries. She has worked in high-tech and government sectors, at companies of all different sizes (from startups to huge enterprises). She advocates approaches that directly improve organizational efficiency, productivity, and interoperability. If you ask her, she’ll say she’s happiest when those around her are successful. She's the author of “Arbortext 101: Best Practices for Configuring, Authoring, Styling, and Publishing with Arbortext,” available on Amazon. She has several more planned.

References and Links

1. “The state of structure in technical communication.” 2009. http://www.scriptorium.com/2009/04/structured-authoring-in-technical-communication/ (Since this presentation was given, it has been moved behind a pay wall.)
2. "Component Content Management Technology Diffusion and Adoption Challenges." Proceedings of the 2010 Society for Technical Communication Annual Conference, May 2-5, 2010. Dallas, TX, May 2010. 93-102.
3. “23 Things I Wish I Knew When I First Implemented a Content Management (ECM) Project”. 2015. http://info.aiim.org/digital-landfill/23-things-i-wish-i-knew-when-i-first-implemented-a-content-management-ecm-project
4. "The Best USB Headsets for Audio and Web Conferencing." 2016. http://headset.travelkit.ninja
5. "Technology that solves the top technical problems for presenters (My Presenter's Tech Travel Kit)." 2016. http://presenter.travelkit.ninja
6. This article appeared in STC Intercomm in September 2016. Login required.