7. What resources and skills are required to conduct the evaluation?

• Who will conduct the evaluation?
• Should the evaluation be undertaken by an individual or team?
• Should the evaluation be undertaken by insiders or outsiders?
• What are the key issues to consider in engaging an evaluator?
• At what stage of the project should the evaluator become involved?
• What is the budget required for the evaluation activities?

In planning a project, consideration needs to be given to the human, material and other resources needed both for the overall project and for its evaluation component.  Planning the budget in detail for the evaluation component will enable the insertion of an evaluation line item in the overall budget.  Careful costing of each of the proposed project activities and of the project’s operation and management needs to be carried out to ensure that the project is feasible, and that its budget components can be justified.  Reference should be made to the applicable ALTC Grants and Fellowships Program Guidelines that list the items for which the budget can provide, along with certain specified maximum amounts and rates, plus items for which grants may not be used.

The costs of an evaluation of a project or fellowship program should be developed as part of the funding submission.  These should include a rate for the evaluator, travel costs, costs of materials, and costs for any additional expertise or specialised services.  Given the evaluation reports will generally not be published separately, the costs of report production should be included in this line of the project/fellowship program budget.  In general, evaluations have been allocated about 5 per cent of the overall budget in many recent ALTC projects but this may vary depending on the role of the evaluator and any special requirements.

Who will conduct the evaluation?

This is a key question underpinning the budget. This may be considered in terms of two sub-questions:

i. Should the evaluation be undertaken by an individual or by a team?

There are advantages and disadvantages for each option.  Having a team means finding common times for meetings and other activities, developing shared understanding on procedures, data analysis and findings, pulling together possibly different writing styles, and meeting joint deadlines.  Working as an individual removes these constraints.  A team however may enable the evaluation to tap into a range of specialist skills relating to planning, data gathering and analysis, and reporting, which the individual may not necessarily have to the same levels.  The range of skills may also enable allocation of specific tasks to team members, thereby sharing the load and individual time commitment.  A team may also provide a variety of perspectives that can be brought to the evaluation, providing an inbuilt ‘sounding board’.  However, given the amount normally budgeted for evaluation (seldom more than $10,000 in recent ALTC projects), it may not be possible to afford more than one person to undertake the evaluation.

ii. Should the evaluation be undertaken by insiders or outsiders?

An insider is defined here as anyone who is directly involved in the operation of the project being evaluated or who has a direct stake in the project’s outcomes.  An outsider is defined as anyone who is not directly involved in the development or operation of the project being evaluated or who does not have a direct stake in the project’s outcomes.  Therefore members of the project team, the reference group, or staff who are participants in the project should not be considered independent of the project/fellowship. 

Insiders may be located externally as well as internally.  An example of an external insider would be an outside organisation providing professional services of a type that would be needed if the project were deemed to be a success and its operation extended e.g. a professional association.  By the same token, outsiders may be located internally as well as externally.  An example of an internal outsider would be an individual in the project institution from another Faculty or School who has no actual, potential or perceived stake in the project or its evaluation outcomes.

Insiders, particularly internal insiders, carry advantages in that they often will have detailed understanding of the context in which the project is operating, along with ready-made points of contact for information gathering.  Being known to the parties involved, they may be seen as less threatening and thus find it easier to elicit information.  Their involvement may moreover be costed at a relatively low level, to the extent that it is seen as a part or an extension of their continuing substantive role in the institution.

Having a direct stake in the project or the evaluation’s outcomes raises however the issues of actual, potential and perceived conflict of interest.  Essentially the insider cannot be seen as providing an evaluation that is independent because by definition the insider has a stake in the project outcomes.  This raises questions of credibility and reliability, which in turn may diminish the usefulness of the evaluation’s findings in informing future decision-making.  Independence is however largely a matter of professional practice and an evaluator needs to ensure they conduct the evaluation according to high professional standards of ethics, trustworthiness, honesty, analytical rigour, and transparency.  Where this is demonstrated in practice and in the evaluation reports, the extent of institutional independence of the evaluator becomes less of an issue.

Involving an outsider will help in establishing an independent process.  An independent process is desirable in any project evaluation, and for projects receiving ALTC funding of more than $120,000 it stands as a formal requirement.¹ 

Involving an outsider also carries other advantages.  It may enable the acquisition of specialist evaluation expertise and experience that may not be present to the same extent among insiders, or at least not readily available at the time when it is needed.  Outsiders can bring new perspectives and a sense of impartiality to the evaluation, coming with fresh eyes to the project and its operation. 

Consideration might also be given in large or complex studies to establishing an evaluation team that includes both insider and outsider members, to capitalise on the advantages of both sources – detailed understanding of the project’s operating context, ease of access to information, a range of skills and perspectives, and a degree of impartiality.  

Finally, in relation to gaining access to outsider expertise, the role of the Project Reference Group warrants close consideration.  The current Grants Program Guidelines state that

All project teams should appoint a reference group.  The reference group should include some external participants who have appropriate expertise to ensure there is constructive advice on the design, development and ongoing evaluation of the project and to ensure the project has maximum impact within the institution/s engaged in the project and beyond those institutions.

While the Reference Group will not carry out the evaluation, careful selection of its outsider members should provide a useful sounding board and sources of expert advice in relation to the evaluation’s development and implementation.

What are the key issues to consider in engaging an evaluator?

A first step in engaging an evaluator is to prepare an evaluation brief or terms of reference. This is a statement that should give a prospective evaluator sufficient information to prepare an evaluation proposal. It is important not to be too specific in relation to the evaluation methodology in the statement, leaving some freedom for the prospective evaluator to use his/her expertise and experience to propose detailed ways of proceeding.

The evaluation brief or terms of reference would normally include at least the following:

• a brief outline of the project (background, processes, outputs, timeline and intended outcomes)
• the purpose(s), focus and scope of the evaluation
• any preferred or required information gathering sources and techniques
• the roles of the Project Manager, Project Team and the Project Reference Group in relation to the evaluation and to the evaluator
• reporting requirements and timelines
• the budget available for the evaluation (generally 5-10% of the project budget)
• qualities expected of the evaluator.

The qualities expected of the evaluator include those expected of evaluators in general as well as those that relate to the particular project. The qualities would normally include:

• project evaluation experience in higher education, and ideally in the discipline or area of the project
• broad understanding of the discipline or area of the project
• skills in quantitative and/or qualitative data analysis, as appropriate to the project
• high level oral and written communication skills
• independence
• capacity to meet the project's evaluation timelines
• willingness and capacity to work with the Project Manager, Project Team and the Project Reference Group, as required.

The ALTC has developed a database of individuals who have undertaken evaluation of funded projects and fellowships to assist project leaders and fellows to identify potential evaluators.  There is no requirement to select an evaluator from this list if another suitably qualified and experienced individual is available.

At what stage of the project should the evaluator become involved?

The evaluator should be involved in discussions with the project team or fellow as early as possible, and ideally before the project is underway. An internal evaluator may be already known and could be involved in the application or nomination. This allows for critical examination and shared understanding and endorsement of the details of the evaluation. It also enables timely planning of how the evaluation procedures can dovetail with the project and become an integral part of it rather than an add-on.

The evaluator at this stage can act as a sounding board, asking questions that will enable greater clarity and precision to be attached to planned processes and outcomes. Early and continuing involvement of the evaluator along these lines should also deepen the evaluator's understanding of the project or fellowship context, thereby further developing the potential of the evaluation to generate rich insights into the factors influencing the project or fellowhip outcomes.

Costing and evaluation

The costs of a project evaluation are largely to cover the time spent by the evaluator carrying out their activities, including attending meetings and project activities, reading and analysing documents, collecting and analysing data, and writing reports.  In addition, there may be travel costs and the production costs of reports although these may be incorporated in other areas of the project budget.  Therefore, calculating the budget for an evaluation is primarily an exercise in outlining the extent to which the evaluator will be expected to be involved in project activities, the extent to which they will collect and analyse new data, and the amount of time they will need to prepare their reports.  The practice in many recent ALTC projects and fellowships has been to budget between $5000 and $10,000 for the evaluation and then negotiate the evaluation plan within this budget.  Experience suggests that in many cases this funding does not cover the full costs of the evaluator and there is some institutional or personal contribution required to cover the evaluation activities.

---

¹ Current ALTC Grants Program Guidelines state that 'recipients of grants in excess of $150 000 are required to commission a formal independent evaluation of the project. This may be funded from the ALTC grant and should be included in the project proposal budget.'