Generalizability, Reliability, and Validity Analysis

Developed by an interdisciplinary team of college health professionals, the ACHA-NCHA was pilot tested in 1998-1999 and systematically evaluated with reliability and validity analyses comparing common survey items with national studies such as the National College Health Risk Behavior Survey (CDC).

Generalizability

Only schools that randomly selected students, or surveyed students in randomly selected classrooms, are part of the national databases. Because the schools are self-selecting, the ACHA-NCHA databases cannot be said to be generalizable to all schools and students in the United States. However, the generalizability of the three ACHA-NCHA Pilots and the Spring 2000 database was evaluated by comparing the results to other surveys of the same population that have been sampled to represent all students in the United States.

Reliability and Validity Analysis

These analyses include:

  • comparing relevant percentages with nationally representative databases,
  • performing item reliability analyses comparing overlapping items with a nationally representative database,
  • conducting construct validity analyses comparing ACHA-NCHA results with a nationally representative database, and
  • conducting measurement validity comparing results of the ACHA-NCHA with a nationally representative database.

The data sets used for evaluation of reliability and validity are:

  • National College Health Risk Behavior Survey CDC 1995
  • Harvard School of Public Health 1999 College Alcohol Study (CAS)
  • United States Department of Justice: The National College Women Sexual Victimization Study 2000 (NCWSV)
  • ACHA-National College Health Assessment 1998, Spring 1999 and Fall 1999 Pilots, ACHA-NCHA Spring 2000

The series of comparisons and statistical analyses, in a sense, used triangulation, in that information from various resources were independently used to achieve the goal of demonstrating the reliability and validity of the ACHA-NCHA, and thus its utilization and its ability to represent the population of students. The analyses employed different national databases, covered different approaches, and utilized different statistical procedures to accomplish the evaluation. All in all, the ACHA-NCHA appears to be both reliable and valid and of empirical value for representing the nation's students.

Through this evaluation and statistical endeavor is it possible to say that the ACHA-NCHA is generalizable to the college and university students in the United States? The answer would be no even in light of the consistent and divergent means of demonstrating consistency and replication. To ensure that other researchers, educators, administrators and clinicians are aware of this, the name chosen to describe the ACHA-NCHA is Reference Group. It is simply a set of schools with which one can compare data. This in turn helps in the decision-making process for planning programs and understanding the extent of health as well as unhealthy behavior that occurs on campus.

A brief description of each data set used for evaluation of reliability and validity is provided below.

National College Health Risk Behavior Survey, CDC 1995

The NCHRBS, developed by the Centers for Disease Control and Prevention in collaboration with representatives in academia, national health organizations, and federal agencies, was a nationally representative sample conducted in 1995 of undergraduate college students aged 18 or older. Risk behaviors associated with injuries, the consumption of alcohol, drug and tobacco usage, sexual behaviors and physical inactivity were assessed. There was an adequate representation of minorities, sex, and age groups. Sampling was conducted at both two- and four-year institutions of higher education.

Harvard School of Public Health 1999 College Alcohol Study (CAS)

The CAS was conducted in 1999, surveying students in 130 colleges. Sixteen schools were dropped due to low response rates. The remaining 116 schools were located in 39 states. They represent a cross section of U.S. secondary education. The schools were stratified by size (less then 5,000; 5001 - 10,000; 10,001 or more) and location (urban or suburban and rural). The self-administered instrument surveyed students regarding alcohol, tobacco, drug use, student life-style and demographics. Surveys were mailed to 26,508 students and 14,521 students returned the surveys yielding a response proportion of 60%.

United States Department of Justice: The National College Women Sexual Victimization Study 2000 (NCWSV)

The NCWSV was conducted between February and May 1997. The study is a nationally representative sample of 4,446 women who were attending 2- or 4-year colleges or universities during fall 1996. Schools included in the sample had at least 1,000 students. The sample was stratified by the size of the total student enrollment (1,000-2,4999; 2,500-4,999; 5,000-19,999; 20,000 or more) and school locations (urban, suburban and rural). Schools were randomly chosen using a probability proportional with the size of the total female enrollment.

ACHA-National College Health Assessment 1998, Spring 1999 and Fall 1999 Pilots

The first pilot was conducted in 1998 at nine campuses with a combined sample size of 2,007. A second pilot was conducted in the spring of 1999 at 10 campuses. The combined sample size was 3,531. A third ACHA-NCHA pilot was conducted in Fall 1999. Seven campuses participated with a sample size of 3,649. One school used a web-based surveying method. Reports of the results have been mailed to each participating school. Preliminary reliability and validity analyses were conducted comparing the CDC National College Health Risk Behavior Survey and the Harvard School of Public Health College Drinking Study with the NCHA pilots. These analyses demonstrated robust statistical results, which were presented in a session at the 2000 ACHA Annual Meeting in Toronto, "Update on the ACHA National College Health Assessment."

ACHA-NCHA Spring 2000 -- 28 Schools, 16,024 Participants

Thirty-five US postsecondary institutions self-selected to participate in the Spring 2000 ACHA-NCHA and 20,164 surveys were completed by students on these campuses. For the purpose of this study and the formation of the reference group database, only those institutions that used a random sampling technique were included in the analysis, yielding a final data set consisting of 16,024 students on 28 campuses.

Among the demographic characteristics of the 28 campuses surveyed, 20 were public college/universities and 8 were private. The vast majority, 27, were 4-year institutions. School size varied, with 1 school having less than 2,500 students, 2 schools 2,500-4,999, 7 schools 5,000-9,999, 10 schools between 10,000-19,999 and 8 schools 20,000 or more. The geographical location around the country also varied, with 5 schools in the northeast, 11 schools in the Midwest, 5 schools in the south, and 7 schools in the west. There was also variation in campus settings, with 6 schools located in urban areas greater than 1,000,000. Ten schools were located in urban settings between 100,000 and 1,000,000. Six schools were located in suburban settings and 4 in rural settings.

The Carnegie classification was as follows: Doctoral Research University Extensive were represented by 14 schools, Doctoral Research University Intensive included 5 schools, Masters College and University I comprised 7 schools, Baccalaureate College had 1 school as did Associates Colleges.

There was a mix of randomized classrooms, randomized mailing or sampling of the universe. The overall response proportion was 54%.