Has student learning been improved by the use of online and offline formative assessment opportunities?Mary Peat and Sue FranklinThe University of Sydney For almost a decade we have been providing a large group of first year, undergraduate biology students with both offline (paper based) and online assessment resources to support them in their learning. This paper reports on an investigation of the students‘ use of these assessment resources, as well as their perceptions of the usefulness of these resources to their learning. The research plan enabled us to investigate any correlations between use or non-use of the assessment resources and final performance in the course. The results show that while the majority of students use and find useful both offline and online assessment resources, use has no differential impact on final learning outcomes.
IntroductionThe educational research literature shows that students who make use of every learning opportunity approach the final assessment tasks with a greater likelihood of high performance outcomes (eg. De Vita, 2001; Heffler, 2001). Recent studies are also indicating that current students in the Australian context need more support than their predecessors, in part due to the increasing heterogeneity of the student cohort (McInnis, James & McNaught, 1995). In addition students have higher expectations of support including more early and relevant feedback from assessment tasks (McInnis, James & Hartley, 2000). Fowell, Southgate & Bligh (1999) suggest that students are best served by the provision of a diverse range of assessment methods as individual methods may disadvantage some students. They also suggest that from the teacher perspective, using a selection of methods allows performance from different sources to be related. Seale, Chapman & Davey (2000), who investigated which types of assessment students found most motivating for their learning, found that having a range of assessment opportunities was most motivating. They also found that the relevance and content of the assessment appeared to influence student motivation as well as the enthusiasm of the teachers. Feedback on performance, especially that of a formative nature, has been shown to be a valuable tool in the learning process enabling students to assess their own progress and understanding and remedy any weakness exposed by the assessment (Clariana, 1993; Macdonald, Mason & Heap, 1999; Zakrzewski & Bull, 1999). However, for this feedback to be effective it needs to be provided early in the learning process (Brown & Knight, 1994) and have some degree of prescription about how to improve performance (Wiliam & Black, 1996). Fowell, et al. (1999) argue that the presentation of results also require the provision of effective feedback to students, in both summative and formative assessment tasks, enabling students to identify their strengths and weaknesses in order to improve future performance.Some of these issues have been addressed by the development of online assessment resources in particular when dealing with large student numbers. Such resources are available for use by students any time and any place. In this way large groups of students can be provided with the feedback they are now requesting and this may help them in their final assessment tasks. The development of online assessment resources has a number of advantages over offline, pen and paper, tasks. They can be easily marked, provide instant feedback and can be taken repeatedly by students in order to assess and improve performance. Also online tests can be taken unsupervised in students own time. Clariana (1997) has shown that online assessments allow students to tailor their use to their own learning style, while Zakrzewski & Bull (1999) emphasise the advantages of online assessment in terms of fast feedback to large numbers of students with no staff involvement. They also indicate that the use of online formative assessment prior to summative tests reduces student anxiety. The contribution of formative computer based assessment on improvements in student learning outcomes is documented by Buchanan (2000) who found that undergraduate psychology students who used an online formative assessment package, which provided instant feedback, performed better in the end of course summative assessment than those who did not use the package. In First Year Biology, at the University of Sydney, the cohort of students is large and we are managing the learning environment in a culture of reducing resources. This has led us to develop a mix of online and offline (paper based) assessment activities that are aimed at enhancing student learning. These activities include both formative and summative items some with the provision of extensive feedback. The online assessment materials and the students‘ perceptions of their usefulness in student learning have previously been described in Peat & Franklin (2002). The gateway to our online resources is the first year biology Virtual Learning Environment that allows easy access for students to all available learning resources (http://FYBio.bio./VLE/L1/). This site provides access to both summative and formative online assessment resources, as well as other learning materials and access to synchronous and asynchronous communications. This development has previously been described (Franklin & Peat, 2001). We are interested in how our students have used the learning opportunities we have provided and whether they have helped them in their learning. A previous study has indicated that 15-20% of our first year biology students are choosing not to use online resources (Peat, Franklin, Lewis & Sims, 2002). We also have data indicating that some students are not using all the learning resources and have information that suggests that not using these resources may influence final performance (currently unpublished). In this current study we are investigating what contribution formative assessment activities have towards final grades so that we can inform our students about what strategies we believe they could use to enhance their performance. This paper will report on an investigation of the relationship between the use and perceived usefulness of a variety of online and offline assessment resources to the final performance of students in their first semester first year biology course. The assessment resources are described below in Table 1. Description of assessment resourcesThe assessment resources provided for our first year biology students fall into several classification categories - online versus offline (paper based); summative assessment (compulsory) versus formative assessment (non-compulsory); supervised (done in class time) versus non-supervised (done in students‘ own time). Table 1 identifies these resources.
Weekly summative quizzes consist of eight multiple choice questions randomly selected from a large bank of questions. Student performance is available as soon as the quiz is completed, the mark is recorded on a database, and cohort histograms are available online. A commercial provider, WebMCQ Pty Ltd, is used to deliver and mark the online quizzes. Individual laboratory report - students carry out an experiment in class which is then written up as a scientific report as part of their summative assessment tasks. Students are given the opportunity to have a member of staff critique the report prior to preparing the final report. Both content and literacy skills are assessed. Group work on poster and oral presentation - students work in peer groups of five to choose a topic, do the research required and prepare and present a poster. The poster is presented in class time to the rest of the class and the staff. Students assess the poster presentation by giving formative feedback while the staff member gives summative assessment on the poster and the oral performance. Self assessment modules are designed to draw together related parts of a course to help students make connections between topics in biology and to promote a deeper learning strategy, whilst providing an enjoyable feedback and reinforcement session. They allow students to identify their level of understanding and consolidate learning whilst working at their own pace in their own time. The most recent discussion about the development and evaluation of these modules is in Peat (2000), Franklin & Peat (2001) and Peat & Franklin (2002). The mid course practice examination is paper based, taken in class time and administered under examination conditions in order to give the students as close to the "real" examination experience as possible. The students mark their scripts in their own time from an online version. To gain feedback on their answers students use the online version in an interactive way, by entering their answers and having the program mark their performance and give them feedback where appropriate. The feedback is aimed at helping students identify their understanding of course concepts, which in turn might indicate the need for some remedial action. This also helps reduce the stress about end of course examinations, and hopefully, allow students to achieve at a high level in the final assessment. Students perceived to be "at risk" are encouraged to use our online revision materials, designed to enhance student understanding of major topic areas. Weekly paper based self test quizzes are provided at the end of each week‘s practical notes. Each short test consists of multiple choice and short answer questions and is designed to help students self appraise their performance. The questions are similar to those in the formal examinations and weekly summative quizzes. Answers are provided both online and offline (in a student resources room). MethodsThe research question asked was "Are the various online and offline assessment resources used by the students, and if so is their use correlated with final performance?".The students (n=1300) enrolled in the first year biology courses are randomly timetabled into 21 laboratory sessions by the university timetabling computer. In 2002, quantitative and qualitative evaluation of student perceptions of their experiences of using the assessment resources was carried out, during randomly selected laboratory sessions before the end of the semester. Participation was voluntary. The survey collected student demographics, including university entry score and prior experience of biology at secondary school. Perceptions of the usefulness of resources were investigated using a four point scale, with students classifying statements according to whether they used a resource, found it not useful, useful or extremely useful. Open ended questions asked students why they had not used a resource (if relevant) and in what way the resource helped them in their learning (if they had used it). These open ended responses were thematically analysed and categorised (Denzin & Lincoln, 1994). Student success was measured by their final mark, which included a compilation of continuous assessment and a final examination. The implementation of the survey complied with the University of Sydney‘s Ethics Committee Guidelines for research with humans and this enabled us to seek permission to correlate performance with usage/non-usage of materials and with perceptions of usefulness. Results and discussionThe survey was handed out to 457 students in class time. The response rate was 77%. The students responding to the survey were considered to be representative of the entire cohort (n=1300) in that 66% of respondents were female and 34% male (the total class cohort is 67% female and 33% male); 77% were in science based degrees (81% for the entire cohort); 54% had previously taken biology in their last year at school (50% for the entire cohort) and 88% were school leavers (84% for the cohort). In addition the final mean mark for the respondents was 64% (range of 27-91%) whilst the entire cohort final mean mark was 64% (with a range of 27-94%).Students were asked if they had used the summative and formative assessment materials and how useful they had found them in supporting their learning in the course. Table 2 shows the level of use and perceptions of usefulness for the various assessment materials. Participation rates in all compulsory assessment tasks were high (97-99%). Student participation in the formative assessment opportunities was much lower, with approximately 20% of students not using the SAMs or the weekly self test quizzes. With the mid course practice exam 97% of the students took the exam (it was a supervised timetabled class activity) but at the time of the survey only 28% of the students had marked it in their own time from online or offline materials. This non-use of both offline and online resources mirrors data previously obtained about the use of online resources by Oliver & Omari (2001) and with first year biology students (Peat, et al. 2002) and with the use by our first year students of offline materials (unpublished). Generally speaking, most of the students who had attempted or completed the various assessment resources found them to be at least useful, if not extremely useful. However, students responded less positively to the summative resources (weekly quiz, report and poster presentation) than to the formative resources (mid-course exam, self assessment modules and weekly self test quiz).
Students were asked to indicate which of three categories best represented their use of the formative assessment resources and this is shown in Figure 1. Students mainly indicated that they used these formative assessment resources for revision and consolidating knowledge rather than learning new knowledge.
Figure 1: Perceived usefulness of formative assessment Open ended questions on the survey investigated students‘ perceptions, from students who used the resource, on how using the formative assessment resources helped them in their learning. Each set of responses was independently thematically categorised and is reported in Table 3. The data in Table 3 supports the information presented in Figure 1 which indicates that students found that the self assessment modules and the self test quizzes were most useful for revision and consolidating knowledge/enhancing understanding. In addition the open ended responses indicated that the self assessment modules and the self test quizzes were useful in highlighting an awareness of their understanding/lack of understanding of the course content. It is also interesting to note from Figure 1 and Table 3, that students report the use of self assessment modules is also associated with gaining new knowledge. This was not an overt design feature of these resources (Peat, 2000), but it has previously been reported by students (Peat & Franklin, 2002). Figure 1 indicates that students taking the mid course practice exam (97%) perceived it was most useful for revision and consolidating knowledge, whereas those students who had actually marked the practice exam (27%) found marking it most useful for indicating an awareness of their understanding/lack of understanding of the course and in giving them an idea of the final exam structure and expectations (Table 3). Thus is seems that both taking and then marking the mid course practice exam assists in multiple aspects of student learning. We need to put into place a mechanism that encourages marking it to maximise its benefit to students.
Students responses to the more general question "How did the formative assessment resources help in your learning?" are captured in the following quotes: They test what you do and don‘t know, but in a pressure-free environment.Students who did not use a particular formative assessment resource were asked to explain their reasons for non-usage. Table 4 summarises their responses. Students indicated that time, motivation or awareness of the resources was the primary reason for non-use, however, some indicated that they would use them later for revision. Pearson‘s correlation was used to investigate links between student use and perceptions of usefulness of the various offline and online assessment resources provided in the course and final performance (Table 5).
There is a significant positive correlation between students‘ final mark in the course and their university entry score, which is normally expected for a first year science cohort. There is no significant correlation between the final marks and student use of formative assessment resources. There is a significant correlation between the various formative assessment resources in that students who found any of them useful were more likely to have found all of them useful. The relationship between end of course student performance and use of formative assessment resources was further investigated. Student performance was clustered into three categories - students with a credit or above for the final mark (65% and over, i.e. excelling students), students with a pass (50-64%) and failing students (less than 50%). Table 6 shows the percentage of students within each of the performance categories who used the formative assessment resources. Interestingly a greater proportion of the students who failed the course had taken advantage of the formative assessment resources - more than the students who passed!
related to student performance categories
Within each student performance category the use and non-use of formative assessment resources was compared with the mean mark for each category of students. Table 7 shows that there is no apparent difference in any of the student categories for final performance outcome for students who did or did not use the various formative assessment resources, thus using the formative assessment resources and finding them useful is not a predictor of learning outcomes for any of the three student performance categories. Implications for teaching and learningOur data indicate that at this stage in their degree program, the majority of this cohort of students perceives the use of online and offline assessment resources as beneficial to their learning. However, there appears to be no differential effect of these resources on their performance as measured by their final mark. The current study indicates that the reasons some of the students do not use the assessment resources are not academic but ones associated with how they prioritise their use of time.It appears that higher achieving students used the formative assessment opportunities less frequently than lower achieving students, possibly because they do not need the extra support. It has been shown (Entwistle, Hounsell, Macaulay, Situnayake & Tait, 1989) that an important contributing cause of failure of first year students is an absence of feedback on progress and this is also cited as a reason to discontinue (McInnis, et al., 1995). As we are currently providing a variety of assessment resources with what we believe to be relevant feedback we are concerned that these resources are not having the desired impact on student learning. Thus the worrying aspect of our results is that, although the poorer students are trying very hard and more of them (compared with the more successful students) are using the formative assessment resources provided, we do not appear to be helping them. This is in contrast to the reports of Buchanan (2000) and Zakrzewski & Bull (1999) who showed that the use of formative tests before summative examinations increase the grade point of students. We, as teachers, need to demonstrate to our students how to use our resources to their advantage. Perhaps to do this we may need to review our feedback and ask ourselves is it good enough? We need to stop and reflect on why it is that students believe the online and offline assessment materials help them in their learning, even though our current findings seem not to support this. Is it possible our method of data analysis was not sufficiently sensitive for the data? Perhaps some re-analysis of data using different approaches is indicated. In particular we intend to correlate the mark in the exam with use of resources, instead of the overall mark for the course, which is a compilation of the final exam and a variety of continuous assessment items. Is it possible that our students (the majority in transition from secondary school to university) are not able to fully utilise our materials, perhaps because they have not had sufficient prior experience of such assessment items? Do they need more support in learning how to learn? We need to determine whether transition is an issue, before making decisions as to how to deal with it. Therefore we intend, in 2003, to repeat our survey in both semester one and two, as well as utilising student interviews as part of our methodology, in order to probe deeper into this issue. Has student learning been improved by the use of online and offline formative assessment opportunities? We are not sure we have any answers yet. AcknowledgmentsWe would like to thank Alison Lewis for her input into the design of the survey instrument and Margaret Lindsay for helping with the quantitative analyses.ReferencesBrown, S. & Knight, P. (1994). Assessing Learners in Higher Education. Kogan Page, London.Buchanan, T. (2000). The efficacy of a World Wide Web mediated formative assessment. Journal of Computer Assisted Learning, 16, 193-200. Clariana, R. B. (1993). A review of multiple-try feedback in traditional and computer based instruction. Journal of Computer Based Instruction, 20(3), 76-74. Denzin, N. K. and Lincoln, Y. S. (Eds) (1994). Handbook of qualitative research. Thousand Oaks, CA, USA: Sage Publications. De Vita, G. (2001). Learning styles, culture and inclusive instruction in the multicultural classroom: A business and management perspective. Innovation in Education and Training International, 38(2), 165-174. Entwistle, N. J., Hounsell, C. J., Macaulay, C., Situnayake, G. & Tait, H. (1989). The Performance of Electrical Engineers in Scottish Education. Report to the Scottish Education Department, Centre for Research on Learning and Instruction Department of Education, University of Edinburgh. Fowell, S. L., Southgate, L. J. & Bligh, J. G. (1999). Evaluating assessment: The missing link? Medical Education, 33(4), 276-281. Franklin, S. & Peat, M. (2001). Managing change: The use of mixed delivery modes to increase learning opportunities. Australian Journal of Educational Technology, 17(1), 37-49. http://www./ajet/ajet17/franklin.html Heffler, B. (2001). Individual learning style and the learning style inventory. Educational Studies, 27(3), 307-316. Lim, K. F. and Lee, J. (2000). IT skills of university undergraduate students enrolled in a first year unit. Australian Journal of Educational Technology, 16(3), 215-238. http://www./ajet/ajet16/lim.html Macdonald, J., Mason, R. & Heap, N. (1999). Refining assessment for resource based learning. Assessment and Evaluation in Higher Education, 24(3), 345-354. McInnis, C., James, R & McNaught, C. (1995). First Year on Campus: Diversity in the initial experiences of Australian undergraduates. A commissioned project for the Committee for the Advancement of University Teaching. Canberra: AGPS. McInnis, C., James, R & Hartley, R. (2000). Trends in the First Year Experience in Australian Universities. A project funded by the Evaluations and Investigations Programme, Higher Education Division, DETYA. Canberra: AGPS. [11 Sep 2002] http://www./archive/highered/eippubs/eip00_6/fye.pdf Oliver, R. & Omari, A. (2001). Student responses to collaborating and learning in a web-based environment. Journal of Computer Assisted Learning, 17(1), 34-47. Peat, M. (2000). On-line self-assessment materials: Do these make a difference to student learning? Association for Learning Technology Journal, 8(2), 51-57. Peat, M. & Franklin, S. (2002). Supporting student learning: The use of computer-based formative assessment modules. British Journal of Educational Technology, 33(5), 517-526. Peat, M., Franklin, S., Lewis, A. & Sims, R. (2002). Learning human biology: Student views on the usefulness of IT materials in an integrated curriculum. Australian Journal of Educational Technology, 18(2), 255-274. http://www./ajet/ajet18/peat.html Seale, J., Chapman, J. & Davey, C. (2000). The influence of assessments on students‘ motivation to learn in a therapy degree course. Medical Education, 34(8), 614-621. Wiliam, D. & Black, P. (1996). Meanings and consequences: A basis for distinguishing formative and summative functions of assessment? British Educational Research Journal, 22, 537-538. Zakrzewski, S. & Bull, J. (1999). The mass implementation and evaluation of computer-based assessments. Assessment & Evaluation in Higher Education, 23(2), 141-152.
Print version errata: p.94, Table 4 - last column - last number (15) should be 15%. p.96, Table 6 bottom line 83%% should be 83%. [ AJET 19 ] [ AJET home ] HTML Editor: Roger Atkinson [rjatkinson@bigpond.com] This URL: http://www./ajet/ajet19/peat.html Created 10 Mar 2003. Last revised 12 Mar 2003. 
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
|
|
