Help Contents

Overview

Quick Start Guide

Portfolio Lists

Portfolios

Attributes

Prioritization

Optimization

Bubble Charts

Bar Charts

Ranking Charts

Efficient Frontier Charts

Sensitivity Charts

Menus

How Optsee® SMART Scoring Works

Here's whats covered:

  • Definition of SMART Score curves
  • Overview of creating attributes and building a portfolio
  • How to use 2D and 3D bubble charts
  • How Optsee® Simulation-Prioritization works

SMART Scores

The built-in SMART (Simple Multi-Attribute Ranking Technique) project prioritization system was developed by researchers from MIT, Harvard, and the University of Southern California. This system allows you prioritize your projects in a way that integrates your key project data into a single value score that is clear, understandable and rigorously defensible.

In the SMART prioritization system, each project attribute (such as ROIs, risks, costs, etc.) is transformed onto a curve representing a “0” to “100” scale where “0” is the worst value (for example, lowest ROI or highest risk) and “100” is the best value (for example, highest ROI or lowest risk). These individual attribute scores are then weighted by the analyst and then summed to yield the overall SMART value score for each project in the portfolio.

For example, when you purchase a car you compare each car's "package" of individual attributes (the combination of price, appearance, reliability, gas mileage, etc.) with those of other cars. You make your selection based on which car has the highest overall value to you. The relative importance of individual attributes is usually different for each consumer. For some consumers, horsepower is more important than gas mileage; for others, the opposite is true. Decision-makers select projects that have the "package" of attributes that provides them with the highest overall value or value.

Optsee prioritizes your projects according using the SMART Score to represent the overall value of each project, based on its attribute values, and the relative importance that you assign to each individual attribute. Optsee uses a logical step-by-step process to build a portfolio model that can be used to objectively rank projects based on your, or your team's, values and then graphically displays the results in charts to provide a clear picture of the results.

In addition, Optsee calculates and displays sensitivity charts that you can use to quickly understand the impact of changes in the portfolio or in the projects themselves. The Optsee Prioritizer™ can run Monte Carlo simulations in which the projects are evaluated in up to 100,000 portfolios to provide you with highly insightful statistical rankings of all your projects.

Finally, the Optsee Optimizer™ can easily optimize your portfolios against up to 100 independent constraints, including project dependency constraints.

SMART Score Curves

Optsee models your value by generating SMART Score curves for each attribute in the portfolio. A SMART Score Curve assigns an attribute's worst outcome to have 0 units of value and the best outcome to have 100 units. You can select from linear, s-type (logistical), step (incremental), categorical, or custom curve types; see the Attribute form for more information about creating attributes.

Example:  Figure 1 illustrates an attribute curve showing SMART Score versus Profit. Since the SMART Score curve is a straight line (a "neutral value" curve), projects that have the lowest profit potential gain 0 points, projects with the highest profit potential gain 100 points, and projects in the middle (the mid-point where the two blue arrows meet) gain 50 points.

Figure 1: Straight-line (Neutral) Value Curve

If each project had only attribute associated with it, decision-making would be easy - you'd simply pick the project with the highest SMART Score value (highest profit or lowest cost, for example). However, projects usually have more than attribute that impacts the decision. Furthermore, decisions involving projects with multiple attributes become even more complex because the relative importance of one attribute to another can change based on strategy and circumstance.

This is why using SMART Scores to prioritize projects is so valuable. Using SMART Score integrates both multiple attributes as well as strategy into project selection.

A Simple Exampe of SMART Scoring: Selecting a restaurant

For the purposes of illustration, we're going to create a simple portfolio for choosing a restaurant. Each restaurant has a common set of attributes to use for comparison, such as location, cost, atmosphere, quality of food, etc. The resteraunt we choose depends on which restaurant has the highest SMART Score based on our current needs. If we are choosing a restaurant for a nice long dinner, then atmosphere and quality of food would be more important than if we wanted a quick lunch before an important meeting, where travel time may be given higher value weight. The attributes for the restaurants would not change; however, the importance of the attributes could change relative to meeting our value requirements.

Similarly, in business decisions the relative importance of different attributes will be different for different decision-makers, business strategies, and circumstances. Budget planners prioritize attributes differently based on the strategic goals and needs of the company or division. New product designers prioritize attributes differently based on the market segment whose value appeal they are trying to maximize. Financial portfolio planners prioritize attributes differently based on long- or short-term investment strategies

You prioritize attributes in Optsee by assigning them with numerical weights. Attributes with higher weights are more important than attributes with lower weights. Optsee uses your assigned weights with your SMART Score curves to evaluate and rank your projects.

Building a Simple Portfolio Model

Now, let's look at how this all comes together by using a simple restaurant example, as discussed above. The desired decision outcome is to select the best restaurant for a nice dinner with an old friend. We'll choose from five restaurants and eva;uate them using four attributes. Of course, you wouldn't use Optsee to make this decision, but this is a useful example to understand how Optsee works and how it can be applied to your business decisions.

Figure 2 shows an Optsee Attribute list form with four attributes chosen to determine which restaurant to go to for the desired outcome.

Figure 2: A Weighted Attributes for Selecting a Restaurant

On the Attribute list form, the attributes are automatically ranked in the first column (Rank) based on the assigned weights in the second column (Weight). In this model, the Atmosphere attribute has the largest weight and highest rank, whereas the Travel Time attribute has the smallest weight and lowest rank. The magnitude of the weights indicates that Atmosphere is twice as important as Cost (1,000÷500=2) and more than three times as important as Travel Time (1,000÷300=3.33). Thus, the relative importance between the different attributes can be calculated by dividing the larger attribute weight by the smaller attribute weight.

The "Atmosphere" and "Food Quality" attributes are assigned to "Categorical" attributes in the the Attribute form. This allows you to assign a SMART Score value to individual categories, and then assign those categories to your projects using a pop-up menu in the the Project form.

Figure 3 shows the categories and corresponding values assigned to "Atmosphere" and "Food Quality" in the the Attribute form.

Figure 3: Categories and Corresponding Values

Figure 4 shows how a category value is assigned in a the Project form using a pop-up menu. This assigns a value of 80 to that attribute for that project.

Figure 4: Selecting the "Very Good" Category in the Project form

Approximate cost and travel time attributes are both quantified in real units (dollars and minutes, respectively). The SMART Score Curves for both of these attributes are Neutral, which indicates a straight-line linear relationship between the attribute value and SMART Score value. (See the Attribute form for more information on "Diminishing return" and "Increasing return" SMART Score curves.)

Let's assume that we've narrowed our restaurant selection to choices. The following figure reveals how these restaurants are displayed in an Optsee Portfolio form. (Figure 5):

Figure 5: A Portfolio of Five Restaurants

Each restaurant was automatically ranked according to its Overall SMART Score as it was added to the portfolio. The blue header indicates that the portfolio table has been sorted according to the Overall SMART Score, thus the restaurants are listed in order of diminishing scores. In this portfolio and portfolio, Japanese Lotus is the most attractive restaurant and the Blue Crab Grill is the least attractive.

How the Overall SMART Score is Calculated

The following is a general explanation of how the SMART Score for each individual project is calculated:

1) The individual unweighted SMART Score for each project attribute value is calculated according to the SMART Score Curve or Category assignements. These are the unweighted SMART Score values.

2) The individual SMART Score values are adjusted in proportion to their relative weight or importance. These are the individual weighted SMART Score values.

3) The weighted individual SMART Score values are summed together to yield the combined value or "Overall value."

For example, click here to see a numerical example that explains in detail how the overall SMART Score of the Japanese Lotus restaurant in this example was calculated.

Clicking the blue [Calc.] button at the top of the Portfolio form populates the attribute columns with the individual weighted SMART Score values (Figure 6). Clicking the [Actual] button toggles the column values back to the actual entered attribute values (Figure 5).

Figure 6: Displaying Calculated Weighted Attribute SMART Scores

Visualize Your Data Using Bubble Charts

In addition to using the Portfolio form to analyze your models, you can also use Optsee's Bubble Charts.

Bubble charts are a type of X-Y (scatter) chart that are used to plot three sets of data simultaneously. Each data point has X and Y values that correspond to the location of the bubble center, and the size (area or radius) of the bubble represents the third value. In Optsee, the user can use both the bubble pattern and color to distinguish one project representation from another. For example, the bubble chart in Figure 7 displays the overall SMART Score on the Y-axis, Atmosphere on the X-axis, and Cost is represented by the bubble size. In this view, you can easily see why Japanese Lotus is the most attractive restaurant, since has the highest SMART Score and is less expensive than Le Bon Chateu and Gardenville Inn.

Figure 7: Atmosphere and Cost Bubble Chart

Rotating 3D Bubble Charts allow you to see 4 dimensions of data: X-axis, Y-axis, Z-axis, and buuble size simultaneously. In Figure 8, you can view the restaurants in relationship to all four attributes. Buttons at the bottom of the chart allow you to rotate it for different views.

Figure 8: Atmosphere, Cost, Travel Time, and Food Quality 3D Bubble Chart

Bubble charts can provide important insights into the distribution of your projects within the portfolio parameters, which are not readily apparent from merely studying the portfolio, particularly when you have several dozen or more projects. Bubble charts are extremely useful for understanding and communicating all aspects of the decision. The Optsee bubble charting tool is fully integrated within the application. It is easy to use and has a variety of features for viewing, comparing, annotating, and exporting your charts.

You can also perform advanced sensitivity testing on your portfolios as discussed in Creating and Using Sensitivity Charts. Click here for an example of using sensitivity testing on this Restaurant Portfolio.

However, the Optsee Prioritizer™ removes the need to do advanced sensitivity testing.

Get a Statistical Analyses With the Optsee Prioritizer™

Now that we've seen how to use Optsee to study a single portfolio where the user sets fixed attribute weights, let's look at one of Optsee's most powerful features: the ability to create thousands of models in a single click, and study the resulting project rankings based on a statistical analysis of these models. Optsee generates up to 100,000 models using a Monte Carlo simulation and then displays the results in specialized charts and tables.

The Monte Carlo simulation methodology was named for Monte Carlo, Monaco; a city that is famous for its casinos and games of chance such as roulette wheels, dice, cards, and slot machines. Games of chance exhibit random behavior within the context of the game equipment and rules. For example, a shuffled deck of cards will contain 52 cards, but the card order is random. An Optsee Monte Carlo simulation involves creating thousands of random portfolios within a set of defined parameters, testing your portfolio in each model, and then calculating the average rank, standard deviation, and cumulative percentage ranking for each project in your portfolio.

In the Restaurant Portfolio, the four attributes had fixed rank and weight values (Figure 8). Weight sensitivity testing allows you to vary a single attribute weight while holding the other attributes constant. A Monte Carlo simulation allows you to vary all of the attribute weights at once in either a fixed sequential or random rank order. A fixed sequential rank order generates random weights that maintain the same rank order as the portfolio (for example, in the Restaurant portfolio Atmosphere would always have the highest rank, Food Quality the next highest, Approximate Cost the third highest, and Travel Time the lowest). A random rank order randomly varies both the weights and the rank order. This allows you to explore the entire portfolio space.

Figure 8: Weight and Rank

Let's look at creating a Monte Carlo Simulation prioritization. Figure 9 shows the Prioritizer form used to set up a simulation test.

Figure 9: Simulation-Prioritization Set-up Form

The [Attribute Order] radio button allows you to generate the random portfolios in either fixed sequential order (attributes are always in the same rank order as the parent portfolio) or in random order (attribute rank order is also randomly varied). The Number of Tests drop-down menu allows you to set the number of models that you want to generate in the prioritizer (from 1000 to 100,000). The Weight Range drop-down menu allows you to set the range between the highest and lowest weight (from 1000 to 10,000). A setting of 1,000 means that the largest possible weight is 1,000 and the smallest is 1.

Running the Prioritization

During the prioritization, a portfolio is tested in each of the random portfolios, and the projects are ranked sequentially based on the Overall SMART Score values. After running the prioritization, the Prioritizer Summary List Form is populated in the (Figure 6). This form displays the results in list form including the average rank, standard deviation, highest rank, lowest rank, and cumulative percentage rank. The cumulative percentage rank is discussed in more detail below.

Figure 10: Summary List Form

You can easily view the data by creating a Project Ranking chart.(Figure 11) This chart lets you view the Simulation-Prioritization data in many different views. In Figure 11, the chart displays the mean, standard deviation, maximum and minimum, and the bubble area is proportional to the cost.

Figure 11: Simulation Ranking Chart

Clicking the [Cum. %] button at the top of the Summary List Form, you can display the Cumulative Percentage bar chart (Figure 12). This chart displays the normalized cumulative percentage of the number of portfolios that a project was ranked at a particular rank or higher. This is a good indicator of the strength of a project's ranking. Projects that were ranked higher more often have longer cumulative percentage bars.

Note here that Japanese Lotus and Le Bon Chateau Inn are clearly the strongest restaurant candidates.

Figure 12: Cumulative Percentage Chart

The results of this prioritization show that the top two projects, Japanese Lotus and Le Bon Chateau Inn, respectively, would both be good restaurants based on the Simulation-Prioritization results. The Japanese Lotus has a slight edge, but statistically the two restaurants are about the same in the analysis.

As mentioned earlier, you probably wouldn't use Optsee to determine where to go to dinner; however, the purpose of this example is to illustrate how Optsee works so that you can understand how to apply it to complex and important business decisions.