Understanding Survey Results

CCSSE in Action: Understanding Survey Results

CCSSE recommends that college leaders familiarize themselves with CCSSE findings before communicating about the results. The following are some things to consider:

CCSSE Benchmarks[1]

Benchmarks are groups of conceptually related items that address key areas of student engagement, learning, and persistence. CCSSE's five benchmarks denote areas that educational research has shown to be important in quality educational practice. The five benchmarks of effective educational practice in community colleges are active and collaborative learning, student effort, academic challenge, student-faculty interaction, and support for learners. These benchmarks are tools that can be used to compare college performance across benchmarks, to similarly sized institutions, and to the full CCSSE population of community colleges.

Enrollment Status

Enrollment status (part-time versus full-time) receives special attention in CCSSE reports; all results are either presented separately for part-time and full-time students or are weighted by enrollment status. In the CCSSE sampling procedure, classes are selected, not students. Accordingly, full-time students, who by definition are enrolled in more classes than part-time students, are more likely to be sampled. As a result, though approximately two-thirds of the students enrolled at the participating institutions are part-time students, the proportion in the CCSSE sample is nearly opposite. In the data analysis process, therefore, CCSSE assigns weights to responses based on respondents' enrollment status, thereby producing more accurate measures of student engagement.

Weighting is a technique that proportionally adjusts an individual respondent's contribution to a statistic, such as a mean or frequency; thus, some responses are weighted more heavily than others. If subgroups (e.g., part- versus full-time students) differ in their responses, then aggregate results will be biased in favor of the larger subgroup. Bias occurs, for example, when a disproportionate number of full-time students complete the survey as compared to the population. With the assignment of weights, subgroups (part-time) that are disproportionately small in the sample relative to the population have larger weights that increase their impact on summary statistics; the converse is true for subgroups (full-time) that are disproportionately large in the sample relative to the population.

There are several other individual characteristics, such as race, sex, or credit hours completed, where there could potentially be differences in subgroups. This observation begs the question: Why does CCSSE weight data on enrollment status and not on other individual characteristics? The answer is simple: there is no reason to do it. The only systematic bias that occurs is with enrollment status.

Effect Size as a Measure of Notable Differences

Effect size is a measure of group differences. In the CCSSE results, it refers to mean differences between your institution and the group of colleges to which your institution is being compared divided by their standard deviation. This procedure rescales all effect sizes to the same scale (differences in standard deviations) and thus allows for comparisons.

CCSSE uses both statistical significance and standardized effect sizes to identify items on which a college's performance differs from comparison groups. An asterisk (*) highlights items for which students' responses differ at a statistically significant level (p < .001) and have standardized effect sizes equal to or greater than (.2). Statistical significance is based on the effect size, the number of respondents, and the variability in their responses; as a single number, it also is the probability that the observed difference between outcomes would occur where there is truly no difference. While this is a useful guideline for identifying differences between groups, very small differences can be statistically significant in very large sample sizes such as the CCSSE national data set. Thus, items where notable differences occurred were identified as standardized effect sizes of (.2) or greater.

Statistical Significance Meets Practical Significance

In addition to focusing on items meeting the criteria highlighted above, look for patterns in students' responses. For example, are students consistently above or below the mean of the comparison group in certain areas of engagement? Are the differences explainable in terms of a college's mission, the nature of the undergraduate program, or certain students' characteristics? Also, do not rely exclusively on statistical significance tests to identify areas that warrant attention. A consistent pattern of scoring above the mean, even though all the items may not reach statistical significance, may indicate the institution is doing the right things in terms of good educational practice. At the same time, some institutions have very high expectations for student engagement and may fall short of their own aspirations even though comparisons with other institutions are favorable. And in some cases, of course, it may be that the national mean is itself unacceptably low.

CCSSE Consortia Results

CCSSE consortium colleges [2] that added questions to the survey instrument will find their corresponding frequency results in the Frequency Distributions tab. In addition to a college's comparison to its consortium group and the 2004 CCSSE population, a consortium college also will receive a comparison to other colleges in its size category, provided on the institutional report cd.

Over-sampling

CCSSE's sample sizes are determined by institutional size, as reported in IPEDS. Colleges can elect to over-sample in order to examine results for specific groups (such as students enrolled in developmental courses or students attending particular campus sites) or in order to increase overall sample size. The over-sample dataset is included on the college's institutional report CD.