Using item analysis to evaluate the validity and reliability of an existing online information literacy skills assessment instrument
Abstract
At the end of 2012 a short course, the Certificate of Information Literacy, was registered by the Cape Peninsula University of Technology Libraries and was first introduced in 2013. The course consisted of five modules and a summative assessment. The assessment, accessed via the Learning Management System, comprised 100 multiple-choice questions of which students received a randomly selected fifty to answer in ninety minutes. This article reports on the effectiveness and validity of the online assessment for the Certificate of Information Literacy: whether the assessment instrument accurately measures the information literacy skills of the students and whether the test can be considered valid and reliable. There are two parts to the study, but this paper focuses on the first part which is an item analysis on the assessment data of the students who completed the test in 2013. The results highlight weaknesses and problematic areas in the test. The second part of the study will focus on comparing the results students received in the test with the results of a subject-specific essay assignment to measure the level of application of their information literacy skills.Downloads
References
Cape Peninsula University of Technology (CPUT). 2009. Information literacy policy. (Unpublished).
Dunaway, M.K. and Orblych, M.T. 2010. Formative assessment: transforming information literacy instruction. Reference Services Review, 39(1): 24-41. DOI:10.1108/00907321111108097.
Fain, M. 2011. Assessing information literacy skills development in first year students: a multi-year study. The Journal of Academic Librarianship, 37(2): 109-119. DOI:10.1016/j.acalib.2011.02.002.
Kermani, R.A. 2013. Survey of the relation between organizational structure and informational overload in case study: Payame Noor University. Research Journal of Recent Sciences, 2(6): 52-57.
Mery, Y. and Newby, J. 2010. Assessing the reliability and validity of locally developed information literacy test items. Reference Services Review, 39(1): 98-122. DOI:10.1108/00907321111108141.
Nunnally, J. C. 1972. Educational Measurement and Evaluation. 2nd ed. New York: McGraw-Hill.
Oakleaf, M. 2006. The right assessment tool for the job: seeking a match between method and need. Proceedings of the Library Assessment Conference: Building Effective, Sustainable, Practical Assessment. 25-27 September 2006. Charlottesville, Virginia. 201- 213.
Oakleaf, M. 2008. Dangers and opportunities: a conceptual map of information literacy assessment approaches. Project Muse: Libraries and the Academy, 8(3): 233-253. DOI:10.1353/pla.0.0011.
Oakleaf, M. 2009. The information literacy instruction assessment cycle: a guide for increasing student learning and improving librarian instructional skills. Journal of Documentation, 65(4): 539-560. [Online]. http://meganoakleaf.info/iliac.pdf (18 June 2013).
Oakleaf, M. 2010. Value of academic libraries: a comprehensive research review and report. Chicago: Association of College and Research Libraries. [Online]. www.acrl.ala.org/value (15 July 2013).
Ondrusek, A. 2005. A longitudinal study of the development and evaluation of an information literacy test. Reference Services Review, 33(4): 388-417. DOI:10.1108/00907320510631544.
Sonntag, G. 2008. We have evidence, they are learning: using multiple assessments to measure student information literacy learning outcomes. Proceedings of the 74th IFLA General Conference and Council. 10 – 14 August 2008. Quebec, Canada: IFLA. [Online]. http://archive.ifla.org/IV/ifla74/papers/134-Sonntag-en.pdf (14 June 2013).
Venter, I. 2006. Development of a valid and reliable test for higher-educated young adults measuring dietary fiber food source and health-disease association knowledge. Journal of Family Ecology and Consumer Sciences, 34: 10-19. [Online]. http://hdl.handle.net/10520/EJC33678 (12 June 2013).
Venter, I. 2008. Construction of a valid and reliable test to determine knowledge on dietary fat of higher-educated young adults. South African Journal for Clinical Nutrition, 21(3): 133-139. [Online]. http://www.ajol.info/index.php/sajcn/article/view/34792 (18 February 2014).
Walsh, A. 2009. Information literacy assessment: where do we start? Journal of Librarianship and Information Science, 41(1): 19-28. DOI:10.1177/0961000608099896.
This journal is an open access journal, and the authors (copyright owners) should be properly acknowledged when works are cited. Authors retain publishing rights without any restrictions.
South African Journal of Libraries and Information Science is an Open Access journal which means that all content is freely available without charge to the user or his/her institution. Users are allowed to read, download, copy, distribute, print, search, or link to the full texts of the articles, or use them for any other lawful purpose, without asking prior permission from the publisher or the author. This is in accordance with the BOAI definition of Open Access.
Creative Commons Attribution-ShareAlike 4.0 International License
