5. How will the information be collected and analysed?
- What/who are the data sources?
- What types of data are most appropriate?
- What are the most appropriate methods of data collection?
- How will the data be analysed and presented in order to address the key evaluation questions?
- What ethical issues are involved in the evaluation and how will they addressed?
The process of data collection and analysis in evaluation can be termed 'data management' as described in the following diagram.
Figure 2. Data Management Framework
Adapted from Owen, 2006, p.99.
The starting point for this process is the set of key evaluation questions identified in Section 4. The data in an evaluation is collected primarily to address these questions.
The data assembly process comes next and involves identifying the data sources, gaining access to the necessary data and obtaining the data in a useful form. Each of these elements of the process has a number of steps - too many to list here - however, it is useful to address several common questions.
What/who are the data sources?
In many ALTC projects, students and staff will be the primary data sources but documents and other stakeholders may also be useful sources of information. Due to the small size of many ALTC projects, all students and staff participating may be able to be approached to provide data and sampling therefore will not be an issue. If the population of any data source is too large then sampling will be required. Probability sampling (random or some variation of it) will usually be the best approach for quantitative information and explanatory analysis, whereas qualitative information and descriptive analysis are often served better by non-probability (purposive) sampling (see social science research texts for more detailed guidance on sampling). The ready availability of existing data may make it generally preferred in evaluation studies, especially if it is accepted as appropriate and of high quality by stakeholders. However, where existing data is of poor quality or not available then new data must be collected, and this is generally more expensive and time consuming. Issues of the quality of any data used in the evaluation should be explicitly addressed in reporting the evaluation.
What types of data are most appropriate?
The data to be collected will depend on the key evaluation questions. In most evaluations, a combination of qualitative and quantitative information is collected, as required by the different questions being addressed. There is no a priori preference for one type of data over another, and both quantitative and qualitative data have standards of quality (see Guba and Lincoln's Fourth Generation Evaluation for a discussion of the indicators of data quality).
What are the most appropriate methods of data collection?
The process of actually collecting the data is often the focus of most discussion and controversy but if the process of identification and access is properly addressed, the process of obtaining the data is much less problematic. There is a wide range of methods of obtaining data, see the LTDI Evaluation Cookbook for suggestions on different methods and how to analyse the data collected:http://www.icbl.hw.ac.uk/ltdi/cookbook
The objective of the evaluation is to answer each of the key evaluation questions, so a matrix might be developed mapping each question against potential sources of information. A sample for such a matrix is provided in Figure 3. The matrix enables identification of overlaps in data collection and the development of more efficient processes.
Figure 3. Sample Data Source Matrix
According to Owen's model, the second part of data management is analysis and reporting, which has three components; data reduction, data display and conclusion drawing and verification.
The general data analysis process in evaluation is one of reduction- that is, 'the process of simplifying and transforming the raw information according to some logical set of procedures or rules' (Owen 2006: 101). There is a wide range of processes for data reduction for both quantitative and qualitative information. The processes used must be explicitly described when reporting the data analysis results. There are two general purposes for data analysis in evaluation; description and explanation. Both are important because description enables the audience to understand the project, its intended processes and outcomes, and the extent to which these were achieved, whereas explanation provides evidence about the underlying logic of the project and the extent to which it is sustainable, transferable and/or reproducible.
The display of data is a process of organising the information in ways that lead to the drawing of explicit and defensible conclusions about the key evaluation questions. In many evaluations, conclusions are the endpoint, however in others, the evaluators go further to offer recommendations about the project. The former require placing values on the conclusions such as stating the project is successful or not, whereas, the latter are advice or suggestions for courses of action made to decision makers.
Ethical issues often arise in the data management process described above e.g. in the selection of data sources, obtaining the information or reporting results. Data collection activities in ALTC funded projects will usually require approval from the lead institution’s research ethics committee, and may require complementary approval from partner institutions. The main issues which are likely to arise include appropriate methods of collecting, analysing, storing and reporting data from students to protect their confidentiality and anonymity, ensuring students and staff are not impacted unfairly by the evaluation activities (avoiding interruptions to the learning and teaching processes), and unfairly disadvantaging students who are not receiving the project benefits. The Australasian Evaluation Society has produced a Code of Conduct and a set of Guidelines for the Ethical Conduct of Evaluations, which provide useful guidance for evaluation activities. These are widely used in the conduct of evaluations in Australia and New Zealand and are available at: http://www.aes.asn.au/about/.
Wording to ascertain Aboriginal and/or Torres Strait Islander origin
The Australian Institute of Health and Welfare recommends the following wording to ascertain Aboriginal and/or Torres Strait Islander origin:
Are you of Aboriginal or Torres Strait Islander origin?
- Yes, Aboriginal
- Yes, Torres Strait Islander
- Yes, both Aboriginal and Torres Strait Islander
Source: Australian Institute of Health and Welfare (2010) ‘National best practice guidelines for collecting Indigenous status in health data sets, Cat. No. IHW 29,’ AIHW Canberra available at http://www.aihw.gov.au/publication-detail/?id=6442468342&tab=2 accessed 9 February 2012.