IDEA ASSESSMENT AND SCREENING

During the Idea Generation and Capture activity, hopefully a lot of ideas have been generated. Idea Assessment and Screening is closely related to the idea generation activity. During the Idea Assessment and Screening activity, organizations must decide which ideas are ready to be pursued further during the Conceptualization activities. Since the ultimate goal of innovation within an organization is to provide a sustainable profit, the organization must carefully weigh the costs vs the benefits of developing an innovation. Important to note is that in the lists that follow, some approaches might be overkill for the type of organization that wants to implement them, for the size of the evaluated projects and for this early phase of the innovation process. Some methods will also be relevant in later phases of the innovation process – “Concept Assessment and Screening” and “Portfolio Management”.

Don Hofstrand makes the following recommendations (Hofstrand, 2009):

• Form a project committee: An internal project committee bringing together individuals with an expert knowledge of the “business” can do the first screening of ideas;
• Formulate general business idea(s) or concept(s): The business idea must be formalized in a document and potential merits must be described: Customers must be willing to pay for the product or service;
• Identify alternative business models or scenarios for the idea(s): According to Hofstrand a business model describes ‘how the business will function in producing the product or service and providing it to the customer’. A business scenario has another meaning: ‘a logical assemblage of the essential business elements starting with raw materials procurement and ending with the sale of the final product, and all the stages in between’.
• Investigate idea/concept and alternative business scenarios: In this step an initial investigation can be done whether the idea(s), the business models and the business scenarios are viable. Don Hofstrand indicates that this phase can easily be done listening to some (external) knowledgeable individuals;
• Formal investigation: Some aspects of the ideas, business models and business scenarios can be investigated and evaluated by external consultants to get an advice on the feasibility of the proposal;
• Refine scenarios: At the end of the assessment subtheme phase, the project committee will eliminate ideas, models and scenarios from the original list. Only a shortlist will remain for further investigation.

 

A list of methods for idea assessment and screening is provided in (Bradac, 2014):

• ABC analysis: in this approach, ideas are listed in a table according to their importance (what “importance” means has to be defined beforehand). A items are very important, B items are somewhat important and C items are only marginally important. Only the A items get through the filter to the next subtheme.
• AHP-based approach: (link to AHP-based Approach paragraph on template page)In the Analytical Hierarchy Process an idea is evaluated based on a hierarchical set of criteria. The approach is very detailed and mathematical, making it potentially more objective, but also more difficult to use than other, simpler techniques.
• Anonymous voting: (link to Anonymous voting paragraph on template page)this group technique is particularly useful for groups where there is a lot of internal pressure or anxiety. The disadvantage of this technique is that it is very subjective, since it is based solely on the opinion of the participants. In this technique, all the facilitator has to do is prepare a table for each participant, where the participant can add the idea to the correct rank. The facilitator decides on the list of ideas to be included.
• A+T+A+R model: (link to ATAR model paragraph on template page)this pretest model, based on the “diffusion of innovation theory” allows a business to estimate the percentage of the population that might become a regular customer. ATAR is an acronym that stands for Awareness Trial Availability Repeat. Once one knows the chance for each of these stages of familiarity with a product, one can calculate the actual chance of a random person becoming a regular customer. A person first has to become aware of the existence of the product (e.g. through an advertisement campaign). Next, that percentage of the population that is aware, needs to actually try the product. The percentage that has tried the product, needs to have general and repeatable access to the product (availability through shops). Finally, that percentage that is aware, tried the product and has good access to the product, needs to buy the product regularly. Estimating the percentages for these various levels leads to a general estimation of market year. It is advised to perform this exercise for the first ten years of a product launch.
• Check lists for business idea evaluation:  various simple checklists have been produced over the years allowing managers to assess ideas. Good examples are the Princeton Creative Research checklist for evaluating ideas for a business or a product, and the 21-Point Invention Evaluation Checklist (Entrepreneur.com).
• Consensus mapping : the consensus mapping technique is a qualitative (and complex) approach that helps teams structure ideas into organized and interrelated clusters. Small teams of five to nine participants oversee and guide the effort. Two steps are key in this effort. The first is the team making a unified classification scheme to identify key parts of the problem that is to be solved with the ideas, thus obtaining a classification scheme. The second comprises creating a ‘Strawman map’, used to display the relations and time dependencies among the ideas investigated (i.e. drawing up sub-activities necessary to implement the ideas and the sequences between them).
• Cost-benefit analysis: the cost-benefit analysis is a widely used approach to evaluate ideas. The concept is relatively simple: the costs are weighed against the benefits, and the outcome is used to prioritize ideas. Whereas the costs and benefits are often calculated using financial data, it may include other kinds of costs and benefits as well (e.g. social benefit of hiring people). The analysis is generally carried out in three steps: First all cost elements are identified and valued, after which the same is done for the benefit elements. Finally, the costs are compared to the benefits.
• Decision trees: A decision tree uses a tree-like graph of decisions and accompanying consequences to support the decision-making process of selecting ideas. A decision tree consists of three types of nodes. The first type are decision nodes, which represent decisions that need to be taken (e.g. to pursuit or not to pursuit an idea), the second type are chance nodes, representing uncertain outcomes (e.g. 40% chance of selling 1000 items), and the third type are the end nodes (e.g. the final profit to be expected).
• Delphi method: This approach attempts to reach a consensus over which idea is the best to pursuit. The Delphi method relies heavily on the opinion of individual experts. Experts answer questionnaires, after which a facilitator summarizes the responses and the experts are given the opportunity to revise their answers from the previous round based on these (anonymized) responses. This process stops after a pre-defined number of rounds or after a certain consensus has been reached.
• Grid matrix analysis:  Several different names exist for the ‘grid matrix’ technique, a.o. decision matrix, evaluation matrix, weighted criteria matrix, etc. Simply put, ideas are evaluated based on a list of criteria, which is visually translated in a matrix. Each criterion may be given a weight, after which the final scoring of an idea is calculated as the sum of the scores of each criterion. The score of each criterion can be calculated as the rating given multiplied with its weight.
• FMEA-Failure Modes and Effects Analysis:  The key purpose of this method is to identify all possible failures in case it is decided to pursue the idea reviewed, so that actions can be taken to eliminate or reduce the risks for these failures to actually happen. This method is mainly used to evaluate ideas in the design and production field.
• Force field analysis: The force field analysis technique is easy to use, but is also based on subjective assumptions. Group members identify and list two types of forces: forces in favor of the idea, and forces against the idea. This list is usually the result of a brainstorming session. The facilitator will group the forces, and will also attempt to visually show the impact and strength of each of the forces described. This diagram should then be used to verify the possibility of neutralizing the negative forces related to a particular idea.
• Idea advocate: This technique is appropriate in case a pre-selection of ideas has already been made, and only a few ideas are still in the running. Each idea is presented by an advocate, e.g. the person who came up with the idea or the person who will need to implement it. Then, a jury group eliminates ideas until only one is left, possibly after a certain amount of elimination rounds. It has to be ensured that all advocates are equal in status and power within the organization.
• Impact analysis: Impact analysis is a subjective technique in which participants brainstorm about the changes that will result from the implementation of an idea. It allows for the visual forecasting (e.g. in a matrix) of all possible effects before they arise.
• Kano model: The Kano model is a technique to analyze customer’s satisfaction, by classifying customer preferences into five categories: must-be qualities (which are taken for granted, and cause dissatisfaction when not fulfilled), one-dimensional qualities (which result in satisfaction when fulfilled and dissatisfaction when not fulfilled), attractive qualities (which provide satisfaction when fulfilled, but do not cause dissatisfaction when not fulfilled), indifferent qualities (neither good or bad), and reverse qualities (causing dissatisfaction for some customers, and satisfaction for others). The latter two are often omitted in a Kano diagram. It is a rewarding technique when used in the product development stage, to decide which features to include in the product/service and which not to include. It is less effective to be used for idea selection.
• Kepner Tregoe matrix: This approach is a detailed and complex method which is applicable in many areas, not just for idea selection. In the analysis, alternatives are weighted by a) assigning numerical weights to a series of values, b) giving a rating to each of the values for every alternative, and c) computing the numerical scores for every alternative.
• NAF–Novelty Attractiveness Feasibility: In NAF, every idea is given a score between 1 and 10 for three criteria: Novelty, Attractiveness, and Feasibility. All three scores are then summarized and the ideas are ranked. This technique is simple to use, and is thus often used in the process of idea selection.
• Nominal group technique: This approach is easy and very useful to assign priorities to ideas, and can therefore be used early in the idea selection process. The method consists of 5 main steps. In the first step, the facilitator explains the problem or describes the situation, after which the participants are given ten minutes to write down ideas. In the second step, every participant reads out all ideas which are then written on a flip chart by the facilitator. In the third step the participants may engage in discussions. These discussions have as a sole purpose the clarification of ideas, rather than lapsing in a yes or no debate. In the fourth step an anonymous vote is carried out, and the last step consists of further discussion and voting (steps 3 and 4), until a clear prioritization has been reached.
• Paired comparison analysis: In this approach, a range of ideas are scored by determining what the preferred idea would be if one would have to choose between one idea or the other. This is usually executed in a visual diagram, as it may become complex in case many ideas are to be compared. The idea with most points (i.e. which was preferred most often) comes out as a winner.
• Pareto analysis: This technique is often referred to as the 80/20 rule. The underlying assumption is that 20% of all possible causes determine 80% of all problems. Thus, the purpose of Pareto analysis is to identify the most important causes, without spending too much time on finding the more detailed causes. It is very similar to ABC analysis, as only the most important causes are taken into account. The result of the Pareto analysis will consist of a list of ideas which address these most important causes.
• PMI analysis: In this technique, all positive effects resulting from implementing the idea are classified in a Plus column, all negative effects in a Minus column, and all interesting effects in an Interesting column. Plus items have positive scores, minus items have negative scores, but interesting items can have either positive or negative scores. In the end, all scores are summarized, and the idea with the highest score is considered as the ‘best’ idea.
• Prioritization: Prioritization is a general technique to order ideas based on their value and viability. Many of the other techniques explained in this list can be used to prioritize ideas.
• Repeatable questions diagrams: This approach is a simple technique to gain an in-depth understanding of ideas presented, as well as to distinguish between viable and less viable ideas. It is generally believed to be based on the behavior of children, asking “why?” over and over to their parents. As many questions as possible are written down, all following one of the two basic structures: why/why or how/how. Thus, two lists of serial questions are defined, one always starting with “why”, the other always starting with “how”. Such diagrams can be used to identify problems that there might be with the implementation of the idea, to identify possible causes of problems, to identify new problems, etc.
• Sticking dots: This is a popular and simple technique to quickly assign priorities to ideas. The general method is the following. The list of ideas is clearly itemized on a flip chart. Secondly, all participants are divided into groups, and each group receives a different colored set of dots (e.g. group B has yellow dots). Then, each group is given a number of dots (e.g. 15), after which the members of the group will discuss to which ideas they will assign the dots. One member of the group then sticks their dots next to their preferred ideas on the flip chart, and finally a short-list of best ideas is created using the dots and a final group discussion.
• SWOT analysis: SWOT stands for Strengths, Weaknesses, Opportunities, and Threats. It is a widely adopted method to evaluate products, industries, organizations, and may also be used to evaluate ideas. Its goal is to identify internal and external factors that are favorable and unfavorable to monetize an idea. The general idea is to assess the strengths of the idea, its weaknesses, as well as investigating if other opportunities or threats exist in the landscape which may cause an idea to be more powerful than another idea.
• TRIZ: TRIZ stands for “Teoriya Resheniya Izobretatelskikh Zadatch”, which is Russian for “The theory of inventive problem solving (TIPS)”. It is a complex problem-solving, analysis and forecasting methodology. The method is generally deemed too complex to use for idea selection, however it is a widely known method which is why it has been listed here. Three primary findings support the TRIZ methodology. The first is that problems and solutions are often repeated across industries. Secondly, patterns of technical evolution are often repeated across industries. The last and primary finding is that creative innovations often use scientific effects outside the field where they were developed.
• Value Analysis: The value analysis technique has its roots at General Electric Co., a company that was confronted with a shortage of skilled labour, materials, and parts during World War II. GE then looked for acceptable substitutes, and it was noticed that these substitutions often reduced costs, improved the product, or both. The value analysis is therefore used to improve the value of a product (or decrease costs), by gaining a thorough understanding of the different steps that are required to come to the end-product, and trying to optimize each of these steps. Value analysis in the context of idea selection can thus be used to generate/evaluate ideas by questioning the improvements they bring to potential customers.
• Vroom-Yetton-Jago Decision Model: This model is not an idea selection technique, but rather a model which helps to decide on the leadership technique that is used during the idea implementation phase. Therefore, it will only be described briefly here. The model identifies five different leadership styles: Autocratic I, Autocratic II, Consultative I, Consultative II, and Group. The decision model is shaped as a decision tree, in which eight yes or no questions must be answered when moving across the tree from left to right, finally reaching one of the above mentioned leadership styles.

 

In addition to the above list of methods for idea assessment and screening, the tools and techniques identified in the Portfolio Management subtheme, also offers relevance to assessing alignment and fit with organisational strategy. After having executed one or more methods to assess and filter ideas, a selection of those ideas will proceed to the conceptualization phase.

 

AVAILABLE TEMPLATES

• ABC analysis: 
  ABCAnalysis
• AHP-based approach: 
  AHP
• Anonymous voting: 
  AnonymousVoting
• A+T+A+R model: 
  ATAR
• Check lists for business idea evaluation:
  ChecklistForBusinessIdeaEvaluation

• Consensus mapping : 

  ConsensusMapping

• Cost-benefit analysis : 
  CostBenefitAnalysis

• Decision trees : 

  DecisionDocument

• Delphi method: 
  Delphi
• Grid matrix analysis: 
  GridMatrixAnalysis
• FMEA-Failure Modes and Effects Analysis: 
  FMEA
• Force field analysis: 
  ForceFieldAnalysis

• Idea advocate : 

  IdeaAdvocate

• Impact analysis : 
  ImpactAnalysis

• Kano model : 

  KanoModel

• Kepner Tregoe matrix: 
  KepnerTregoeMatrix

• NAF–Novelty Attractiveness Feasibility 

  NAF

• Nominal group technique : 
  NominalGroupTechnique

• Paired comparison analysis : 

  PairedComparison

• Pareto analysis : 
  ParetoAnalysis

• PMI analysis : 

  PMIAnalysis

• Prioritization ; 
  Prioritization

• Repeatable questions diagrams ; 

  RepeatableQuestionsDiagram

• Sticking dots : 
  StickingDots
• SWOT analysis: 
  SWOT

• TRIZ :

  TRIZ

• Value Analysis : 
  ValueAnalysis

• Vroom-Yetton-Jago Decision Model : 

  Vroom-Yetton-JagoDecisionModel

• Action Priority Matrix : 
  ActionPriorityMatrix

• Idea Ranking : 

  IdeaRanking

• Urgent Important Matrix : tbf

 

BIBLIOGRAPHY
Bradac, B. (2014). Idea evaluation methods and techniques. Institute for Entrepeneurship and Small Business Management.
Entrepreneur.com. (sd). 21-Point Invention Evaluation Checklist. Opgehaald van Entrepreneur.com: http://www.entrepreneur.com/encyclopedia/checklists/article81922.html
Hofstrand, D. (2009, October). C5-02: Idea assessment and business development process.