Open educational resources (OER) have been on the horizon in higher education for over a decade, but only recently have they become part of the discussion of many institutions’ strategic plans. According to Inside Higher Ed’s 2018 survey of college and university presidents, 85% of presidents agreed that colleges should embrace OER.¹ In many ways, this is not surprising given the perceived benefits of OER—low or no-cost resources that are openly licensed and available for distribution. Seemingly, OER are good for faculty, students, and college administrators; so why isn’t every college implementing OER as a priority project? The short answer is that OER cost is just the beginning. The planning, selection, management, and maintenance of OER can propel even the best of planning teams into a quandary of unknown variables and decisions that can consume considerable resources for a potentially favorable outcome. This article examines the decisions, challenges, and lessons learned surrounding the implementation of OER. Although users are permitted to revise OER, which brings other considerations, the amount of research in this area is limited, and we focus here on costs, selection, and maintenance of OER.

OER can be defined as “freely available materials that can be downloaded, distributed, adapted, and openly shared to better serve all students,” and OER include “a broad range of high-quality learning materials, resources, and lesson plans that deepen the learning program.”² This can include texts, peer-reviewed and non-peer-reviewed articles, curriculum, lesson plans, learning objects, and multimedia. OER can be used as stand-alone content or as a supplement to traditional textbooks. Most OER content has a Creative Commons license assigned, which sets the conditions under which the content can be distributed and reconfigured.³ Some OER have varying levels of licensing, which can cloud the use and management of the content.

Literature Review
Although OER are widely discussed in academia, there are relatively few studies on the cost savings, implementation, experience, or satisfaction among students or faculty. Hilton’s review of sixteen studies, nine of which were related to OER efficacy, found that “utilizing OER does not appear to decrease student learning.”⁴ But Hilton cautions readers to interpret at ease due to the varying research methodologies and less than adequate research designs in some studies. Only a year later, Hilton found just seven additional studies, many of which were conducted at international universities.⁵ The prevalence of OER at international universities should be no surprise given the origins of its conception at the 2002 UNESCO Forum on the Impact of Open Courseware for Higher Education in Developing Countries.⁶ It was there that the members developed a declaration, known as the Paris Declaration, “to develop together a universal educational resource available for the whole of humanity, to be referred to henceforth as Open Educational Resources.” In many ways, OER was a global initiative that gained traction faster in countries other than the United States. It has been more than fifteen years since the Paris Declaration, and in some ways OER is just gaining more attention.

On a more local level, the Community College Open Textbook Project (CCOTP) was started in 2007 as part of the Community College Consortium for Open Educational Resources (CCCOER).⁷ The inaugural project of CCOPT was the development of an open textbook, Collaborative Statistics, which was piloted and subsequently used by faculty and students at multiple community colleges that were included in the study.⁸ Findings from the research indicated that “while cost savings and ease of use initially attracted both faculty and students to adopt open textbooks and use them to facilitate existing teaching and learning practices, opportunities nonetheless exist for engaging and building upon open textbook use to increase interactivity and enhance teaching and learning for users.”⁹ CCCOER has continued to move its mission forward and now has eight case studies of community colleges that have implemented OER, documenting the findings on its website.

A few other studies at the community college level but not connected to CCCOER revealed that more research is needed to understand the benefits of OER beyond cost savings. A study conducted within the math department of a community college in Arizona included multiple instructors. The instructors employed varying methods to adopt OER over a span of several years.¹⁰ The study looked at cost, student retention and success, and faculty perception of OER quality. The authors concluded that while there was no notable correlation between educational outcomes and OER, there were some cost savings to students. Faculty comparisons of OER quality to traditional texts were mixed. Winitzky-Stephens and Pickavance examined OER adoption across multiple general-education disciplines and found no significant difference between courses using OER and traditional textbooks for continuing students and a minimal benefit for new students.¹¹ Although community colleges have much to gain in the potential cost savings and access factors that initially drive them toward OER, there is much to learn about the impact to teaching and learning.

A few studies beyond the community-college level have confirmed the need for further research and understanding of how OER affect students, faculty, and institutions. In a multi-institutional study by Fisher, Hilton, Robinson and Wiley, students in courses with OER “enrolled in a significantly higher number of credits in the next semester”¹² but had only moderate differences in completion rates and final grades as compared to the control group. A qualitative study conducted at New York College of Technology about the impact of OER in a health psychology course revealed that the majority of students preferred OER to traditional materials.¹³ Cost, access, and ease of use were found to be common themes for student preference of OER. In a mixed-methods study using an OpenStax textbook for introductory biology courses at the University of Georgia, researchers found that students were satisfied with the cost savings of OER.¹⁴ Additionally, students and faculty thought the quality of the OER was comparable to that of traditional texts, but faculty experienced less-than-favorable perceptions from some of their colleagues for using OER. While the jury is still out on the benefits of OER beyond cost savings to students, adoption is still moving forward.

Strategies and Methods for Selection
Strategies and rationale for OER adoption can vary based on institution type and resources. In some instances, individual faculty or small groups of faculty interested in innovative teaching and learning practices have initiated action toward OER. Groups of instructional materials or library initiatives have also moved toward OER adoption, often assisting faculty. Individual departments of some colleges and universities have also initiated OER adoption. For example, a math department of a community college in Arizona converted five math courses to OER.¹⁵ As of this writing, there are not any known colleges or universities that have fully converted all programs to OER. While there is no formula or methodology that exists to select courses for OER, some variables are worthy of note. The first is cost and counts. Considering that cost savings to students is a driver to adopt OER, courses with higher seat counts—generally in the general-education disciplines—would naturally produce more cost savings to both students and the institution. There are exceptions to this, in that if an institution has more students at the graduate level then the undergraduate level, seat counts should be considered from that perspective. Another way to look at the cost variable is to do a cost analysis of texts and select courses that have the most expensive textbooks.

The next variable to consider when adopting OER is discipline or subject matter. Disciplines that are more vulnerable to industry, market, or historical interpretation of knowledge will require more upkeep in the maintenance of OER. For example, an undergraduate course on health education today would likely have different resources from a course five or ten years ago due to advances in the health sciences and an overall increase in consumer awareness about health. On the other hand, the principles of algebra have not changed, so selecting OER for subjects that fluctuate less over time is worth consideration. Additionally, if doing multiple courses at a time, it may be more efficient to stay within a discipline, such as sociology, given that the procurement of OER within a similar subjects is less time-consuming than for multiple subjects.

Another variable to consider when selecting OER is whether materials may be more or less appropriate for undergraduate versus graduate courses. From one perspective, graduate students might more easily assimilate information from multiple sources, but there is no research that looks at student satisfaction of OER with regard to level of education. There are, however, students involved in the contribution and creation of OER. Both at the undergraduate and graduate levels, students at Plymouth State University and universities in Canada and Scotland are assisting in the curation and creation of OER.¹⁶ Given the time constraints that can be involved in curating OER, involving students in the selection of content that is engaging has promise.

Management and Assessment
Although OER are available for little or no cost, there are costs involved in the selection, management, and maintenance of OER. The selection of OER is time-consuming. Even for a faculty member who is an expert in his or her field, it takes time to ensure that content—whether it be an article, open text, videos, or other materials—supports outcomes for the course and is relevant to the level of course. There is no central database, either for higher education generally or for any discipline, that faculty can source for OER content. The authors of this article have created several OER courses at both the undergraduate and graduate levels, and the time involved can be substantial. Obviously, the selection of a single open text for a course would take less time. Some institutions involve librarians or instructional designers to help curate content, but that also requires time of the designated personnel. Teams of faculty collaborating on one course is a way to decrease the workload on one individual but requires coordination of goals and tasks.

Completion of the selection of OER for a course is just the beginning. Similar to selecting traditional instructional materials for a course, quality matters. We recommend that either multiple faculty or a discipline-specific or curriculum committee review the OER for adequacy. The reason for this is that individuals tend to hold distinct biases and preferences for how they understand and interpret knowledge. If six faculty were provided three different OER texts for a sociology course, they would likely not all choose the same resource. Coolidge and DeMarte recommend that faculty or curators of OER research use rubrics to evaluate quality.¹⁷ Ideally, some students should also be involved in at least the review of the OER. The permissions to edit and remix OER that fall under some Creative Commons licenses introduce an additional layer of quality assurance that must be considered.

Similar to the procurement of OER, the management and maintenance of the content also require planning and resources. The level of management and maintenance varies depending on whether the course and OER are offered online within a learning management system (LMS) or offered on ground with access to OER online and/or in print version. This discussion is limited to online courses. Other considerations include access to OER through an institution’s online bookstore and/or its course materials management and delivery partner. It also is important to know what file formats are supported by the institution’s partners. Some platforms have limitations for type and size of file. It is equally important to know whether the OER are accessible on a variety of platforms and devices. Storage is another factor to consider with OER. If the platform has a content-creation database that is able to assign OER to a specific course and export for consecutive terms, this can be an efficient way to store OER. Otherwise, a database can be designed to store and access OER.

The maintenance of OER can require more oversight than traditional textbooks or articles, and therefore the assignment of responsibility should be clear. Copyrights can change, so OER should be checked periodically for copyright clearance. If the content is provided through the internet, web links should be confirmed. Faculty may edit or remix OER content, and depending on the institution’s curricula policies, versioning may be required. The reconfiguration of OER may also impact the support of learning outcomes and assessment for a course. A few studies have examined the reconfiguration of OER by faculty and found low remix or revision rates,¹⁸ but that may change in the future as fluency with OER increases among faculty. Even if the OER are not changed, maintenance and assessment are critical components to consider when integrating OER.

The assessment of learning outcomes of a course with OER should ideally be planned in conjunction with the OER. One of the main drivers of converting to OER is the cost of traditional texts. If educators want to ensure that students are using OER within courses, the content should be tied to assessments to demonstrate learning. Because they are typically copyrighted by publishers, many traditional textbooks come with corresponding assessment items. Most OER content has not evolved to the point of including accompanying assessments, although some publishers are moving toward developing OER content with assessments for a fee.¹⁹ Otherwise, the assessment is typically created by either faculty or instructional designers.

Conclusion
Frameworks for assessing the cost and cost savings, effectiveness, quality, and impact to learning outcomes of OER are in early stages, but a few lessons can be observed from the studies reviewed in this article. The first and perhaps most significant is that although OER content is free or low-cost, the curation, management, and assessment are not, and the costs can be significant. It is important to start with a plan that provides resources, systems, training, and quantitative and qualitative measures of success. Next, the quality of OER varies, so having standards or rubrics by which content is reviewed will decrease curation time. Lastly, as with any revision to curriculum, there are pedagogical considerations. The conversion to OER should be integrated with course-level assessments and a broader outcomes-assessment plan at the institutional level.

There are several plausible reasons for the timid adoption of OER despite widespread interest. First, although awareness, knowledge, and research are increasing, they need to happen on a more widespread basis. Implementation of OER often happens in silos or well-intentioned projects that get derailed due to other academic priorities like research or teaching. Additionally, OER materials vary in quality, so shifting from an established textbook that has been vetted by a publisher to an unknown text or set of materials comes with risk. Lastly, while the cost savings to students can be significant, little is known about the impact of OER on educational outcomes.

It is important to remember that OER is a starting point, not an ending point. As institutions plan and integrate some OER into their futures, give consideration to the challenges and lessons learned from the pioneers. Most importantly, whether it be through an article, website, blog, or conference, share experiences and best and worst practices learned to impact the education of future generations of students. 

1. Doug Lederman, “What Presidents Think About Digital Learning,” Inside Higher Ed, March 14, 2018.
2. Lisa Petrides, “Open-Sourcing Education to Bolster Engagement and Educator Collaboration,” The Source, Fall 2017.
3. See “Share your work,” Creative Commons.
4. John Hilton III, “Open Educational Resources and College Textbook Choices: A Review of Research on Efficacy and Perceptions,” Educational Technology Research and Development 64, no. 4, 2016: 573–90.
5. John Hilton III, “Special Issue: Outcomes of Openness: Empirical Reports on the Implementation of OER,” International Review of Research in Open and Distance Learning 18, no. 4, 2017: i–v.
6. “Forum on the Impact of Open Courseware for Higher Education in Developing Countries,” UNESCO, 2002.
7. See Community College Consortium for Open Educational Resources.
8. Lisa Petrides, Cynthia Jimes, Clare Middleton-Detzner, Julie Walling, and Shenandoah Weiss, “Open Textbook Adoption and Use: Implications for Teachers and Learners,” Open Learning 26, no. 1, 2011: 39–49.
9. Ibid.
10. John Levi Hilton III, Donna Gaudet, Phil Clark, Jared Robinson, and David Wiley, “The Adoption of Open Educational Resources by One Community College Math Department,” International Review of Research in Open and Distributed Learning 14, no. 4, 2013.
11. Jessie Winitzky-Stephens and Jason Pickavance, “Open Educational Resources and Student Course Outcomes: A Multilevel Analysis,” International Review of Research in Open and Distributed Learning?18, no. 4, 2017: 35–49.
12. Lane Fischer, John Hilton III, T. Jared Robinson, and David A. Wiley, “A Multi-Institutional Study of the Impact of Open Textbook Adoption on the Learning Outcomes of Post-Secondary Students,” Journal of Computing in Higher Education 27, no. 3, 2015: 159–72.
13. Cailean Cooney, “What Impacts Do OER Have on Students? Students Share Their Experiences with a Health Psychology OER at New York City College of Technology,” International Review of Research in Open and Distributed Learning 18, no. 4, 2017: 155–78.
14. C. Edward Watson, Denise P. Domizi, and Sherry A. Clouser, “Student and Faculty Perceptions of OpenStax in High Enrollment Courses,” International Review of Research in Open and Distributed Learning?18, no. 5, 2017: 287–304.
15. Hilton, Gaudet, Clark, Robinson, and Wiley, “The Adoption of Open Educational Resources.”
16. Christina Hendricks, “Students’ Vital Role in OER,” Inside Higher Ed, December 13, 2017.
17. Amanda Coolidge and Daniel DeMarte, “College and University Open Educational Resources (OER) Policy Development Tool,” Version 1.0, June 2016.
18. S. M. Duncan, “Patterns of Learning Object Reuse in the Connexions Repository,” Utah State University, Graduate Theses and Dissertations, paper 423, 2009; and John Hilton III and David A. Wiley, “Examining the Reuse of Open Textbooks,” International Review of Research in Open and Distributed Learning 13, no. 2, 2012.
19. Lindsay McKenzie, “A Guide to Good OER Stewardship,” Inside Higher Ed, March 5, 2018.

Authors
Nichole Karpel and Bruce Schneider

Original Publication
er.educause.edu/articles/2018/12/oer-cost-assessment-strategies

Nichole J. Karpel is a higher education consultant and advisor on curriculum and online learning. Bruce Schneider is Vice President of Business Engineering at Ambassador Education Solutions. © 2018 Nichole J. Karpel and Bruce Schneider. The text of this work is licensed under a Creative Commons BY-NC-SA 4.0 International License.