Sample Study Abstract
Factor Analysis Applied to Obtaining Validity and
Reliability Evidence for Student Evaluation Forms
Statemen of the Problem The purpose of the present study was to estimate the reliability and validity of the twenty-eight item Student Evaluation Form (SIF). Three research questions were posed: How reliably is the item set? What factors underlie the construct, “institutional effectiveness” and what are the relationships among similarly derived factors from similar types of forms?Procedures and Findings In order to answer the three research questions, edited student response data (n-661) resulting from the completed SIF Fall, 2019, UNC SCAP and Outreach course evaluations (N=3686) was utilized. A preliminary axis factor solution resulting from input commonality estimates was rotated to varimax, simple structure and pattern factor solutions. Ten factors resulted from the total group (n=661) primary pattern factor solution. Two literature studies were utilized in the factor analysis relating procedure. Varimax factor solutions from each study were entered in the analysis of the Student Information Form factors. High and moderately high across-instrument factor correlations were found. Three of the four Course Evaluation items, the three instructional Evaluation items and the two General Evaluations all load heavily on one common factor and do not appear to support separate domains. The Self Evaluation item subset appears to have been justifiably identified as a separate domain. The Use and Value of instructional techniques, in general, each represent separate factors. The following perceived item subset had acceptable levels of internal consistency (coefficient alpha):Instructor EvaluationSelf-EvaluationGeneral EvaluationTotal Instrument The Course valuation subset and the Instructional Techniques subsets (Use and Value) had lower than acceptable alpha levels.
RecommendationsSeven recommendations were offered to improve the Student Information Form. These recommendations included the removal of item four from the form, elimination or modification of the instructional Techniques section, the grouping of common items, requesting respondents to answer all items, an overall revision of the form format, and validation of the present findings though reanalysis of data resulting from administration of the new form.
TABLE OF CONTENTS FOR THE ENTIRE TECHNICAL REPORT
I. INTRODUCTION Need for the Study Statement of the Problem General Procedures Definitions Limitations of the Study
II. REVIEW OF RELATED LITERATURE Introduction General Summaries Studies Specific to The Present Study Literature Studies
III. PROCEDURES Introduction Data Collection Instrumentation Analysis of Data
IV. FINDINGS Introduction Data Edit Research Question one Research Question Two Research Question Three Literature Study 1 Literature Study 2
V. SUMMARY, CONCLUSIONS AND RECOMMENDATIONS Summary Conclusions Recommendations
TABLES IN THE FULL TECHNICAL REPORT (SAMPLE)
1. TOTAL (N=3686) ITEM CHOICE DISTRUTIONS (Percentages)2. TOTAL GROUP EDITED DATA (n=661) ITEM MEANS AND STANDARD DEVIATIONS FROM THE STUDENT INFORMATION FORM3. INTER-ITEM CORRELATION MATIX FROM STUDENT RESPONSES TO THE STUDENT INFORMATION FORM4. VARIMAX ROTATION OF FACTORS FOR THE TOTAL GROUP ANALYSIS OF THE STUDENT INFORMATION FORM (n=661)5. PRIMARY PATTERN FACTOR SOLUTION FOR THE TOTAL GOUP ANALYSIS OF THE STUDENT INFORMATION FORM (n=661)6. TOTAL GROUP PRIMARY PATTERN FACTOR INTERCORRELATION MATRIX7. BETWEEN-CLASS PRIMARY PATTERN ACTOR SOLUTION FOR THE STUDENT INFORMATION FORM (n=661)8. BETWEEN-CLASS PRIMARY PATTERN FACTOR INTERCORRELATION MATRIX9. TOTAL GROUP (n=661) ITEM CHOICE DISTRIBUTIONS10. ITEM MEANS, STANDARD DEVIATIONS AND POINT-BISERIAL CORRELATIONS WITH THE TOTAL TEST AND SCALE SCORES11. STUDENT INFORMATION FORM SCALE AND TOTAL INSTRUMENT MEANS, STANDARD DEVIATIONS AND RELIAIBLITY ESTIMATES12. STUDENT INFORMATION FORM SCALE AND TOTAL INSTRUMENT MEANS, STANDARD DEVIATIONS AND RELIAIBLITY ESTIMATES WITH INSTRUCTIONAL TECHNIQUES (USE AND VALUE) AND THE COURSE DUPLICATION ITEMS REMOVED13. VARIMAX FACTOR MATRIX FOR PART 1 OF THE IIQ (Instructor)14. VARIMAX FACTOR MATRIX FOR PART II OF THE IIQ15. VARIABLES COMMON TO THE STUDENT INFORMATION FORM AND THE IIQ (Parts I and II)16. RELATIONSHIPS AMONG FACTORS: STUDENT INFORMATION FORM AND IIQ (Part I)17. RELATIONSHIPS AMONG FACTORS: STUDENT INFORMATION FORM AND IIQ (Part II)18. VARIMAX FACTOR MATRIX OF THE STUDENT EVALUATION OF TEACHING (SET) FORM19. VARIABLES COMMON TO THE STUDENT INFORMAION FORM AND THE SET20. RELATIONSHIPS AMONG FACTORS: STUDENT INFORMATION FORM AND THE SET
FIGURES IN THE FULL TECHNICAL REPORT1. ORIGINAL STUDENT INFORMAION FORM 2. REVISED STUDENT INFORMATION FORM
RecommendationsSeven recommendations were offered to improve the Student Information Form. These recommendations included the removal of item four from the form, elimination or modification of the instructional Techniques section, the grouping of common items, requesting respondents to answer all items, an overall revision of the form format, and validation of the present findings though reanalysis of data resulting from administration of the new form.
TABLE OF CONTENTS FOR THE ENTIRE TECHNICAL REPORT
I. INTRODUCTION Need for the Study Statement of the Problem General Procedures Definitions Limitations of the Study
II. REVIEW OF RELATED LITERATURE Introduction General Summaries Studies Specific to The Present Study Literature Studies
III. PROCEDURES Introduction Data Collection Instrumentation Analysis of Data
IV. FINDINGS Introduction Data Edit Research Question one Research Question Two Research Question Three Literature Study 1 Literature Study 2
V. SUMMARY, CONCLUSIONS AND RECOMMENDATIONS Summary Conclusions Recommendations
TABLES IN THE FULL TECHNICAL REPORT (SAMPLE)
1. TOTAL (N=3686) ITEM CHOICE DISTRUTIONS (Percentages)2. TOTAL GROUP EDITED DATA (n=661) ITEM MEANS AND STANDARD DEVIATIONS FROM THE STUDENT INFORMATION FORM3. INTER-ITEM CORRELATION MATIX FROM STUDENT RESPONSES TO THE STUDENT INFORMATION FORM4. VARIMAX ROTATION OF FACTORS FOR THE TOTAL GROUP ANALYSIS OF THE STUDENT INFORMATION FORM (n=661)5. PRIMARY PATTERN FACTOR SOLUTION FOR THE TOTAL GOUP ANALYSIS OF THE STUDENT INFORMATION FORM (n=661)6. TOTAL GROUP PRIMARY PATTERN FACTOR INTERCORRELATION MATRIX7. BETWEEN-CLASS PRIMARY PATTERN ACTOR SOLUTION FOR THE STUDENT INFORMATION FORM (n=661)8. BETWEEN-CLASS PRIMARY PATTERN FACTOR INTERCORRELATION MATRIX9. TOTAL GROUP (n=661) ITEM CHOICE DISTRIBUTIONS10. ITEM MEANS, STANDARD DEVIATIONS AND POINT-BISERIAL CORRELATIONS WITH THE TOTAL TEST AND SCALE SCORES11. STUDENT INFORMATION FORM SCALE AND TOTAL INSTRUMENT MEANS, STANDARD DEVIATIONS AND RELIAIBLITY ESTIMATES12. STUDENT INFORMATION FORM SCALE AND TOTAL INSTRUMENT MEANS, STANDARD DEVIATIONS AND RELIAIBLITY ESTIMATES WITH INSTRUCTIONAL TECHNIQUES (USE AND VALUE) AND THE COURSE DUPLICATION ITEMS REMOVED13. VARIMAX FACTOR MATRIX FOR PART 1 OF THE IIQ (Instructor)14. VARIMAX FACTOR MATRIX FOR PART II OF THE IIQ15. VARIABLES COMMON TO THE STUDENT INFORMATION FORM AND THE IIQ (Parts I and II)16. RELATIONSHIPS AMONG FACTORS: STUDENT INFORMATION FORM AND IIQ (Part I)17. RELATIONSHIPS AMONG FACTORS: STUDENT INFORMATION FORM AND IIQ (Part II)18. VARIMAX FACTOR MATRIX OF THE STUDENT EVALUATION OF TEACHING (SET) FORM19. VARIABLES COMMON TO THE STUDENT INFORMAION FORM AND THE SET20. RELATIONSHIPS AMONG FACTORS: STUDENT INFORMATION FORM AND THE SET
FIGURES IN THE FULL TECHNICAL REPORT1. ORIGINAL STUDENT INFORMAION FORM 2. REVISED STUDENT INFORMATION FORM