Regardless of what you want to evaluate, the process for developing an evaluation framework is the same. Ideally, the whole team develops the evaluation framework together to create a shared understanding of the goals of a project or an organization. How long it takes depends on the complexity of the project and the loftiness of the goals.
1. Identify your goal.
You can’t measure success if you don’t know what you’re trying to achieve. This is the most important step. Try framing the goal in terms of desired change, and include the direction of the intended change (increase productivity, reduce poverty…). Once goals are set, activities can be planned that seek to achieve the goals. Without goals, activities are just make-work projects.
2. Determine performance indicators.
What evidence can be collected that will demonstrate success? What questions do you have that need to be answered about whether or not your daily activities & products / services are contributing to the goal? Two types of indicators are used: Efficiency indicators measure value for money spent; effectiveness indicators measure the desired impact. Effectiveness indicators can be developed for the output of a project (for example, a training course) and for long term outcomes (for example, increased productivity of those trained).
There is an inherent conflict between the most relevant indicators and the difficulty to collect the information, for this reason you may need to use less than ideal indicators. However, the closer you are able to link the logic of the indicators to the goal, the more useful the data will be to help you make tough decisions around whether or not your goals are being achieved.
3. Set performance targets.
Sometimes it’s not realistic to expect 100% success rates. There are factors beyond your control that will hinder acceptance of ideas or change in behaviour. For example, the goal of my blog is to reach other public servants interested in web performance measurement, however, I would consider it successful if it reached even 1 person in 20 different Departments.
4. Collect data to measure against the performance targets.
In my opinion, this is the hardest part and often interferes with step 2. Frequently, people look at the information they have instead of what they need. Out of the two most commonly used web analytics products, Google Analytics (GA) outperforms WebTrends in this area by helping us determine relevance of the data, for example, by reporting on “visitor loyalty” rather than “returning visitors”.
Share the information in a digestible and understandable way with colleagues and managers. Frame the data as the answers to the questions about success you developed in step 2. For example, “Are we reaching our target audience?” “what ratio of our target audience are we reaching?” or “Are people able to find our web site through search engines?” Rather than providing lists of raw data like visitor demographics, visits, views or referrers.