2. What is the purpose and scope of the evaluation?

• Why is the evaluation being done?
• How will the information be used?
• What evaluation form(s) and approach(es) might be most suitable for this study?

Why is the evaluation being done?

Once the project to be evaluated is clearly defined including its intended design, implementation, outputs and outcomes, it is essential to identify the purpose for which the evaluation is being carried out. The purpose is the primary reason for doing the evaluation. For example, the evaluation might be to provide information to project designers on how to improve their design, or intended to assess the extent to which the project achieved student learning outcomes. Each evaluation should have a primary purpose around which it can be designed and planned, although it may have several other purposes. It is a common problem in evaluation studies that they are expected to be all things to all people, whereas the reality is they have limited resources (time, funds, expertise) and thus can only focus on a limited range of purposes. Evaluation studies which are too much of a shotgun approach are unlikely to adequately address the needs of any stakeholders.

How will the information be used?

In defining the purpose of the study, it is helpful to identify how the information collected and reported by the study will actually be used and by whom. This is likely to narrow down the purpose of the study. For example, if it is decided that the project requires information about how well a project is being implemented so different groups can learn from each other (perhaps in several schools or institutions), it is essential that information is collected about implementation and disseminated to these groups in time to be of use in modifying the project implementation. This type of evaluation information is termed formative, whereas information collected to make judgements about the outputs, outcomes or impact of the project is termed summative.¹

Role of the evaluator

The purpose of the evaluation will also help clarify the role that the evaluator will play.  Is it to be a critical friend to the project team, involved in project meetings, asking searching questions to challenge thinking, monitoring the progress of the project, recording the processes of the project including decision-making and management, and providing regular formative feedback to the project team?  Or is the role to be an independent collector and analyser of existing and new data, reporting on summative questions of project performance and outcomes?  Or as is likely, is it to be some combination of these two?  If so, what is the balance?  How independent should the evaluator be in this particular context?   Answers to these questions should be made explicit in an evaluation plan and will help clarify expectations and resolve problems that may arise.

What evaluation forms and approaches might be most suitable for this study?

Owen (2006) outlines the following 5 main forms of evaluation studies² which serve as a very useful framework for identifying the purpose of the study and setting the boundaries for what the study will focus on.

The circumstances of each evaluation differ and it is important to ensure that the methodology fits the type of project and the outcomes to be measured. In designing an evaluation framework, it is necessary to bear in mind that there are a number of forms of evaluation. Evaluations can be proactive in order to scope the environment in which the project is to take place. Still in the early stages of a proposal, another form of evaluation is to clarify the objectives and ensure that the outcomes and the objectives are logically connected. Once a project is operative, it may be necessary to modify the design and an interactive form of evaluation is used to obtain data from the participants to establish if the design of the project is working well or needs to be changed. To ensure that the project meets its objectives it is necessary to monitor the progress of the project being evaluated. Finally, and most commonly, the impact of the project may have to be measured to ascertain if the objectives have been achieved and whether any modifications are recommended for the future. Most evaluations will focus on more than one of these forms.

The eventual form of the evaluation activity is determined by the focus of the evaluation and the scope. The scope refers to the boundaries of the evaluation activities, e.g. timeframe, discipline area(s), extent of measurement, etc. Figure 1 attempts to bring together aspects of the project and the evaluation activities to identify different focus for the evaluation activities. Most learning and teaching projects will have more than one foci for the evaluation activities.

Figure 1. Learning and teaching project evaluation framework

Adapted from the University of Tasmania Project Evaluation Toolkit website: http://www.teaching-learning.utas.edu.au/elearning/evaluating-projects.

 

---

¹ Formative Evaluation provides information for improvement by identifying aspects of the project that are successful and areas in need of improvement. The study generally focuses on the content and design of the project, with results useful to staff.
Summative Evaluation provides an overall perspective of the project. The study usually focuses on the value or worth of the project and is designed for accountability or continuation purposes.

² These forms of evaluation are derived from the work of Owen published in Program Evaluation: Forms and Approaches (2006).