10 Criteria to Evaluate When Choosing a New Technology

10 Criteria to Evaluate When Choosing a New Technology
Organizations often choose technologies for short-sighted reasons. Here’s a better way with 3 questions to ask and 10 criteria to evaluate.
10 Criteria to Evaluate When Choosing a New Technology
When I look back at our last nine years and more than 500 social sector technology projects (and the systems we’ve replaced), it strikes me that organizations often choose technologies for silly and short-sighted reasons. These range from “my cousin works at Oracle” to “everyone here uses Excel, so of course we’re going with Sharepoint / Dynamics / PowerBI” to “the maps look really cool” to “their hotline is open 24/7” to “we can’t afford any license fees” to “we only use open-source technology / pro-bono vendors”.
In the last year, we’ve honed in on a common-denominator model of three questions organizations should ask and ten criteria they should evaluate when choosing a technology and have shared it with dozens of clients and partners. The model seems to fit big and small organizations, cloud and mobile technology, and a range of system types (CRM, ERP, MEL, portfolio management). The feedback has been so positive that we’ve decided to publish the model here so others can use it (and help us improve it).
10 Criteria to Evaluate When Choosing a New Technology

Question 1: To what extent will this technology meet our needs today?

While often overweighted in nonprofits’ technology selection, this question is critical because it shapes how well the tool or platform will deliver business value in the short-term and, by extension, end-users’ early impressions of the technology. To answer this question, break it down into three criteria:

1. Features: Does the technology provide out-of-the-box features that meet your requirements? If not, can the gaps be filled through configuration?

List out all the requirements you can currently imagine addressing with the new technology, grouping ones that are related and categorizing into buckets like Automation, Analytics, Mobile, etc. Then identify each requirement as ‘must-have’, ‘should-have’, or ‘could-have’, making sure it’s described specifically enough (e.g. “Automatically send email reminders when reports are overdue” or “Display data in a point map”). Then get vendors to honestly score their technologies as follows: 3 points if the feature is provided out-of-the-box, 2 points if the feature can be configured without code, 1 point if the feature can be delivered through custom development, 0 points if the requirement cannot be met. You don’t have to have a hundred-million-dollar annual budget or be running a 6-month competitive RFP to do this exercise.
Tip: Gaps on ‘should-have’ features shouldn’t be a deal-breaker, especially if features can be configured without code. Some great technologies may underperform on this criterion but provide exceptional flexibility and/or value for money.
2. User-friendliness: How intuitive and easy-to-understand is the technology’s user-interface and user-experience?

You might have strong opinions yourself, but end-users are best positioned to evaluate this criterion. It’s important to look at user-friendliness through each lens of the technology. Often we see user-friendliness evaluated only from the perspective of end-users doing data collection or capture – that’s a crucial lens, but the technology’s friendliness for data management, data analysis, data visualization may be equally important.
3. Security: Can you trust that your data are safe on the technology?

This criterion is not simply about ticking a box that the technology provider has a white paper about GDPR or ISO 27001 – when it comes to data protection, GDPR compliance has as much to do with your practices as with the technology you use. Rather, think about security through two lenses: internal and external.

For internal security, evaluate how well you’ll be able to control users’ access and permissions with the technology, ensuring the right staff are able to see and do the right things. Look for technologies that allow you control access and permissions at object/table-level, at feature-level, and at field-level. You’ll also want an audit trail that helps you track who made which changes and when (both for data and metadata). For external security, you want to ensure your system will be safe and that only authenticated users will have access. You might want features like: single-sign-on, two-factor-authentication, and the ability to insist on password requirements. And assuming you’re storing data on the cloud, you’ll certainly want a technology provider with a strong reputation for how it manages its servers both physically and digitally.
10 Criteria to Evaluate When Choosing a New Technology

Question 2: To what extent will the technology meet our future needs?

This question is almost always more important than the first question, yet often receives less attention in the selection process. If you ask anyone at Vera Solutions why we build on the Salesforce Platform, nearly everyone’s answers will revolve around the platform’s sustainability and never-ending improvement. Again, break this down into four criteria.

4. Flexibility: How easily can the solution evolve as your organization and your requirements evolve? Try to picture 5 years from now – is the technology propelling new ideas and ways of working or is it struggling to keep up as the organization changes and matures?

To assess a technology’s flexibility, you’ll need to understand, for instance:
  • How configurable are its automation features? Do business rules need to be hard-coded or can they be managed through modular, adaptable automation tools.
  • How configurable are its analytics features? Do reports and dashboards require customization or can they be adapted in a drag-and-drop way by end-users?
  • How easy or hard is it to adapt or extend the underlying data model? Over time, you will certainly need to capture new data points, modify picklist values, inactivate certain fields, introduce entire new data tables – do you need to rely on developers to make these sorts of commonplace changes or will you be able to make them in-house?
  • Can your organization extend the technology to handle major new requirements and new entire use cases that may come up in the next 5 years?
Tip: In achieving long-term ROI and success with the new technology, this is the most important but often the most underrated criteria. Remember that a technology’s adaptability needs to come hand-in-hand with your organization’s capacity to adapt it. This is enhanced by a strong ecosystem (criterion #7).
5. Interoperability: How well does the technology ‘speak’ to other technologies your organization uses? How easily can data flow from this tool to other tools or vice-versa? How nicely does it play with other tools staff use and will continue to day-to-day (e.g. for email, documents, analytics, etc)? 

If you think about a solution you are still thrilled with five years from now and work backwards, one of the most important characteristics to optimize for is interoperability, or how easily the tool can integrate with other tools. On an interoperability scale from 1-10, a ‘1’ would be a tool on a custom, proprietary stack with no documented API and few examples of successful integration to date, while a ‘10’ would be a technology with an extensively documented set of APIs, a wide range of plug-and-play integrations, and myriad published examples of successful integrations including with tools your organization uses. Investing in a middleware can help speed up integration efforts and reduce the cost of maintaining integrations over time.
Tip: Don’t mistake ‘Open-source’ with ‘highly interoperable’ or ‘easy to integrate with’. Just ask the team at OpenFn how hard it was to build the first ODK-to-Salesforce integration. In our experience, open-source tools tend to score lower in interoperability than paid SaaS or PaaS solutions, though you’ll certainly find some great open-source tools and some terrible SaaS tools.
6. Innovation: How much investment and effort is going into improving the technology and its adoption? How many releases does the technology have per year and how helpful are the new releases?

The best SaaS companies invest heavily in R&D, ensuring their products stay ahead of the game when it comes to features, user-friendliness, security, flexibility, and interoperability. In answering the question of meeting future needs, don’t just consider what a tool or technology offers today, extrapolate from its innovation track-record to anticipate what it will offer in the future. To evaluate innovation, you’ll want to know how many releases per year to expect, what the current high-level roadmap looks like, how the roadmap is prioritized, and what kind of influence you’ll have (as a customer) on the roadmap.
Tip: Ask current customers about their experience with the technology and its most recent three or four releases. Do they feel like the tool/technology is getting significantly stronger, improving incrementally, or staying fairly static? Do they feel that the technology provider cares about and listens to organizations like yours in prioritizing its roadmap?
7. Ecosystem: How strong and connected is the community of users and partners around the technology? How widely available is information that will help you troubleshoot or improve your implementation of the technology?

Perhaps the most underrated criterion to evaluate is the ecosystem of customers, partners, and knowledge surrounding a technology. On one side of the spectrum, you may have a brand new tool with shiny features but only a few customers, a few developers who can adapt it, and a few online resources. On the other side of the spectrum, you have platform solutions, like Salesforce, with a huge and well-connected online community providing help and training, hundreds of partners globally, 150,000+ customers, and thousands of hours of free, gamified online learning modules on Trailhead.
10 Criteria to Evaluate When Choosing a New Technology
Tip: Try scoring each technology you’re considering 1-10 on each of these criteria using a decision matrix like the one above. Feel free to adjust the weights as you see fit for your organization, then multiply the weight times each score and add up to an overall weighted score for each tool across the above seven criteria. This will help provide an objective, balanced lens through which to identify the best-fit technology.
10 Criteria to Evaluate When Choosing a New Technology

Question 3: To what extent does the technology work within your budget?

Depending how spending works in your organization, it may be tempting to look at this question purely from the perspective of “how do we minimize ongoing license fees?” or “how do we get a solution up and running on a budget of X?” Making a sound, strategic decision hinges on understanding the technology’s total cost of ownership (TCO). TCO breaks down into three components: setup costs, license costs, and maintenance costs. (Note: If you want a much more detailed guide on working out TCO, check out Brightwork’s methodology.) Some of these will be tangible, direct costs; some will be indirect like staff time, staff motivation, or staff turnover. For marketing and fundraising systems, you will want to look not only at the cost, but also at the expected return-on-investment; for MEL and program technology, the return-on-investment tends to be far less tangible and harder to calculate.

8. Setup Costs: How much will it cost us–directly and indirectly–to design, configure, and rollout this tool?

Firstly, work out what upfront investment will be needed to get the tool up and running. A familiar tool that requires little or no configuration (e.g. Google Docs or Box.com) will require much less investment than a tool that requires months of complex setup (e.g. a CRM or M&E system). Setup costs not only include consulting services required to tailor and train, but also internal staff time required to help design, test, and adopt the tool. Setup costs will vary from provider to provider and will depend on whether you’re using a fixed-price or time-and-materials model. Unless your organization has developed firm, mature requirements or you need a very simple, straight-forward solution, a capped and carefully managed time-and-materials contract with a partner you trust is usually the most reliable model for ensuring a win-win project.
Tip: Sadly, things will not always go according to plan. It is a sobering fact that — according to the Standish Group’s CHAOS Report on 50,000+ technology projects — fewer than 1 in 3 technology projects succeed on-time and on-budget with a satisfactory result. The report cites executive sponsorship, user involvement, and emotional maturity as key factors influencing success. Whatever technology you select, ensure your leadership is bought in, that end-users are consulted, and that you’ve planned and budgeted for the unexpected.
9. License Costs: What do we need to pay each year to license this technology? 

While many great tools offer a ‘freemium’ model, most technology that is worth using is licensed. While many in the nonprofit sector have been taught to believe that open-source is always better, the costs of relying on open-source tools often outweigh the benefits. The advantages of licensing software are similar to the advantages of renting an apartment instead of crashing on your friends’ couch. When it comes to Salesforce, most nonprofits are eligible for 10 free licenses, but remember that Salesforce is only ‘free’ like a free puppy is free.
Tip: Make sure you understand exactly what you are and aren’t paying for. Often, software providers will bake-in features and benefits you may not need into their initial quote. Do your homework to make sure you procure licenses that fit your needs (for instance, if you’re building a project management or M&E system, consider Salesforce Platform licenses instead of Salesforce Enterprise or Unlimited licenses which cost more because they provide access to the Sales Cloud and Service Cloud).
10. Maintenance Costs: What will it take and what will it cost to support this technology?

The final and perhaps most important component of answering the budget question revolves around the cost of maintaining and supporting the technology once it has been launched. “How is maintenance the most important cost component?” you ask.

Jeff Hanby answers that question well, breaking out examples of the corrective, adaptive, preventive, and perfective maintenance needs that come up and citing five articles that suggest maintenance costs tend to make up 40-80% of TCO.

The more customized your solution is, the higher your maintenance costs will be. Direct costs here may include:
  • Staff time from System Administrators (anywhere from 20% to 300% full-time-equivalent, depending on the system’s size and complexity) spent managing and adapting the system
  • Support services from the technology provider or implementation partner to troubleshoot, debug, or make minor adaptations (often provided on a retainer basis or as part of a license fee)
  • Additional services required to adapt the technology to meet your evolving requirements (mitigated by choosing a technology with excellent flexibility and a strong ecosystem)
  • Buying and maintaining servers or any other hardware required to use the technology
  • Time or services needed when retiring the software (exporting data, keeping the system running while a replacement is getting rolled out)
Indirect costs, meanwhile, may include:
  • The cost of staff frustration, burnout, or attrition resulting from using the unfriendly, inflexible, non-interoperable technology you chose (though it may not show up on the budget, there is a very real cost to your M&E Officers ripping their hair out)
  • The inefficiencies and time wasted by relying on manual data manipulation and having easy answers to common questions at your fingertips
  • The cost of uninformed decisions made because your decision-makers don’t have the data or information they need when they need it
Tip: Consider whether you want support from the technology provider at 1st, 2nd, and/or 3rd level. If you have a Service Desk or IT Support team internally, you will likely get the fastest resolution time if they handle 1st level support. Make sure you’ve got, at a minimum, adequate support at 3rd-level to resolve any issues you can’t handle internally.

In Summary

Whether you’re after a new CRM, ERP, MIS, or ICT4D solution, choosing a technology can be intimidating and nuanced. We hope this framework will help your organization arrive at a balanced, carefully considered choice that meets your current needs, future needs, and budget.
Need help thinking through a technology decision?
Comments are closed.