Task Force 2014-2015
SUMMARY
The University Community has had an ongoing conversation concerning the end-of-course (EOC) student evaluation survey (currently SAIS) as it fits with UTK faculty and student culture, practices, and needs. (Click here for background on SAIS.) In the past two years, several committees have addressed issues of course evaluation. A Task Force on Teaching, formed in 2013-2014, considered the SAIS as part of their charge to examine all aspects of the evaluation of teaching at UTK. (Click here for the report on the Task Force on Teaching Evaluation) An SAIS Task Force, formed in 2013-2014, examined the SAIS in the context of surveys used by research universities around the country and companies providing evaluation software. (Click here for the Task Force on SAIS/End of Course Evaluation.)
The 2014-2015 SAIS Task Force was formed by the Provost, on recommendation from last year’s committee, to revise the SAIS questionnaire and delivery system. Specfically, this group was asked to recommend a more streamlined and customizable EOC survey form in response to student and faculty complaints, as well as the findings of the previous committees.
The commitee met in the fall on a bi-weekly basis and addressed the concerns and recommendations of the previous committee, reviewed research on EOCs and examined the practices of other research institutions. The committee also examined all of the SAIS questions on the basis of the research on EOC questionnaires. In the spring, the committee has met weekly, adopting a research-based model for EOC’s and creating and revising questions on the basis of the research and the advice of experts in assessment.(Click here for committee member information.)
The chair wishes to extend her thank you to the committee members for their work, which was extensive, and for the commitment that they have shown on behalf of the UT community. The group has worked diligently, serving to help the university move forward to a better system of course evaluation. Much of the work of the committee has been beyond that required of typical committees – the members have worked to create a literature review, assess company software, and to create a model through their own original research. Through the committee’s hard work, the process of validating a new survey is already underway.
The committee has produced several products, including:
- Working assumptions regarding the needs of UT for a course evaluation system
- Literature review
- Recommendations for an evaluation software system, including items for an RFP
- Benchmarking on the practices of other research universities
- Close consideration of the forms, and all the questions within, of the current SAIS survey (over 100 individual questions which were rated by committee in a Qualtrics survey on the TennTLC account). Results available on request.
- A research-based model construct for a new course evaluation
- A multi-tiered structure for core, unit, and instructor questions
- Options for response scales
- Finalization – for review – of a core set of questions
- Draft proposal for a validity and reliability study (several steps already underway, given the work listed above)
- Discussion piece on policies and procedural questions
- Recommendations for an OIRA informational web page
- Recommendations from Dr. Skolits and the committee regarding the piloting study
RESEARCH
The committee’s work was researched based, not only in the use of a construct by Marsh (1993, 1997), but also research into “end-of-course” evaluations, including the ebook, The Effective Evaluation of Teaching. For further references, see the attached list.
SURVEY CONSTRUCTION
The committee uses a research construct created by Marsh and Roche that was instrumental in the development of the SEEQ: Student Evaluation of Educational Quality. The SEEQ is used worldwide to evaluate teaching on the university level. Because the SEEQ instrument contains the same number of questions as the current SAIS instruments, and there was a distinct charge to streamline the survey, the decision was made by the committee to use Marsh and Roche’s construct to develop a set of core questions to be used at UT.
| Marsh’s 9 Factors in student evaluation of teaching | Construct |
|---|---|
| Rapport/Enthusiasm for Subject | The perceived ability of the instructor to reach out and connect with students. The perceived level of availability of the instructor beyond regularly scheduled meeting times. |
| Breadth of Coverage Course Delivery Organization/Clarity Grading (feedback) Course Resources |
Course dimensions – The perceived organization and effectiveness of the course as delivered. |
| Group Interaction | The perceived opportunity for student-to-student to practice or actively engage course content. |
| Learning/Value | The perceived opportunity for student to practice or actively engage course content. The perceived level of value of the course experience |
| Workload/Difficulty | The perceived level of academic challenge the course presented to the student. |
In addition to the core questions, there will also be a small subset of questions for particular types of course, e.g. labs, discussion sessions, online, studio/performance. College and departmental level questions and individual instructor questions will also be an option.
Core questions, each with an open ended comment box.
↓
Subset of questions for online, labs, discussion sessions, studio and/or performance sessions.
↓
Departmental questions
↓
Instructor questions (only reported to the instructor)
↓
Background information – student level, grade expected, is course in or out of major, etc. (Similar to current SAIS questions.)
FEEDBACK
Input from the university community is an important part of this process. Please use the comment box to provide any feedback that should be considered by the task force.