Analyzing Student Growth

In Features, Winter/Spring 2016 by Aleisa Dornbierer-Schat

Each year in August, before a single class has been held or a single syllabus distributed, the new crop of first-year Dordt students spends two hours taking a test. Anyone passing the classroom would see students in costume—cartoon characters and superheroes—gripping their number two pencils and filling in bubbles, trying to keep their minds on the test (next in their tightly scheduled Week of Welcome agenda: the Freshman Olympics).

The test, taken by some incoming students since 2006 and all students since 2010, measures students’ aptitude for college work by assessing their skills in four “core” areas: critical thinking, reading, writing, and math. Once the two hours are up and they’ve laid their pencils down, students resume more active WOW activities and, for all intents and purposes, forget about the test.

In April of their junior years, these same students take the test again—this time in plainclothes and with most of the core curriculum under their belts. Here is where it gets interesting: on the whole, Dordt students—who already outrank most first-year students across the country when they start college—perform much better the second time around.

The ETS Proficiency Profile doesn’t have the name recognition of the SAT or GRE, but it’s an important tool for measuring learning outcomes at the college level. Developed by the same nonprofit that developed the GRE, the freshman-to-junior-year skills-based assessment is being used by colleges and universities across the country to gauge how well they’re equipping students for life beyond school. Dordt’s results indicate that the college has been remarkably successful in fulfilling this purpose. Already testing in the 82nd percentile at the outset of their college careers, by the time students leave Dordt, they outperform 98 percent of students at comparable institutions nationwide. That number drops only slightly, to 96 percent, when all schools using the test—from state schools to the Ivy League—are lumped into the pool (see graph 17). The percentiles reflect five years’ worth of nationally normed data from 77,500 students, and crunching the numbers in this way helps illuminate the significance of Dordt’s results each year by pitting them against the results of students nationwide.

“That growth that we see between the freshman and junior years at Dordt is off-the-charts bigger than what other institutions are showing,” says Dordt Registrar Jim Bos. He identifies this as “‘Aha’ moment number one” as he interprets the test’s results. “We are moving our students up the scale,” says Bos.

Amid calls for increased accountability in higher education, more than 400 institutions in the past five years have adopted the ETS Proficiency Profile to help them assess educational quality. Over that time, nearly 450,000 students across the country have taken the test. To generate comparable results, schools are grouped according to size and existing degree programs. More than 80 institutions in Dordt’s cohort use the survey, including Taylor University, Concordia University, Union University, and Covenant College.

Leah Zuidema, Dordt College associate provost and dean for curriculum and instruction, sees the test as a helpful way to assess how Dordt students stack up against students at similar institutions.

“Using a nationally normed instrument allows us to have a more objective picture of our results,” Zuidema says. “We tend to have a bias to think we’re doing a good job, but we want to be able to compare that to other schools. This test helps us do that.”

The test was conceived as a way to measure the effectiveness of an institution’s general education curriculum. In Dordt’s case, that program, known as the Core Program, is one of the things that makes a Dordt education distinctive. The core is the curricular expression of Dordt’s commitment to Christ-centered learning in all areas of life and reflects a cross-disciplinary consensus about what a broad foundation of study should look like for each student at Dordt.

Using the tool to ask “Is the core doing what it’s supposed to be doing?” led to what Bos calls “‘Aha’ moment number two.” The test results reveal that students who come to Dordt with strong math skills stay strong in math, but they also boost their reading and writing skills by a significant margin. The reverse is also true: students arriving at Dordt with strong reading and writing skills continue to test well in those areas, but show remarkable gains in the area of quantitative reasoning by the time they graduate. This achievement is striking when compared to the dismal picture of math education at institutions across the country: test results indicate that students, on average nationally, leave college with the same level of math aptitude they came in with (see graph 17.4).

Dordt’s results paint a much brighter picture—not just in math, but in all four “competencies” measured by the test. Bos says that speaks to the strength of the Core Program. “We are already taking good students and pushing them farther. But just as importantly, we’re doing that by rounding out the places where they have room to grow,” Bos says.

Junior Public Relations major Luke Venhuizen recalls taking the ETS test shortly after he arrived on campus his freshman year. In preparation for Freshman Olympics, his face was painted yellow and he was dressed as Spongebob Squarepants. While he doesn’t remember much about the content of the test, he says Dordt’s results reflect his experience of academic growth as a student.

“Taking classes in a lot of different disciplines has definitely made me a more well-rounded student,” Venhuizen says. While he came to Dordt with AP credits in English, he soon realized public relations required more than just good communication skills. “PR uses a lot of statistics, so strong skills in math are really important,” Venhuizen says. “Statistics can help you figure out who your audience is, and that’s the most important part of building relationships with different publics.”

The ETS Proficiency Profile is designed to assess the growth of individual students in order to discern patterns of growth across the student body. That’s made possible by the way the test is scored. Analysis proceeds from a data set that correlates an individual student’s performance on both the freshman and junior year tests. Those linked scores are used to draw conclusions about how students grow in particular skill areas during their time at Dordt, and to determine how that growth is tied to other factors, such as declared major or certain demographic categories.

This kind of targeted assessment not only helps isolate specific areas of weakness and strength in the core, but can drive curricular change, says Bos. In fact, core requirements at Dordt have already been revised in light of students’ performance on the ETS assessment. In 2011, to further boost students’ growth in math, the core was redesigned to include two courses in mathematics and quantitative reasoning rather than one. That change has improved results, with last year’s juniors showing the most

dramatic gains in math skills since Dordt started using the test. This kind of test-directed fine-tuning is becoming increasingly important as the student body grows in diversity—whether geographic, denominational, or otherwise, Zuidema says. Tools like the ETS assessment can help Dordt better meet the needs of a changing student body.

As useful as the tool is, Zuidema acknowledges that standardized tests have their limits. At Dordt, the test is meant to complement other, qualitative measures used to assess learning outcomes at Dordt. For example, at the end of each semester, students have a chance to fill out surveys in courses across campus. This gives administrators a “pulse” on how well the core is serving students, and it helps professors adapt their courses and teaching methods to better meet student needs.

Ultimately, Zuidema locates the test’s value in how it fits into a bigger picture of Dordt’s educational mission. “While Dordt’s results speak to the rigor of our academic program, ultimately, this test can’t measure intangible things, like the way a biblical perspective is woven through our curriculum or the extent to which the core is shaped by a commitment to Christ-centered learning,” says Zuidema. “This test is one important measure of how students grow during their time at Dordt, but it isn’t the only measure.”

Senior Renee Mahaffy, who took the junior portion of the assessment last April, agrees. While she says she’s “definitely grown academically while at Dordt,” she has also been challenged to grow in ways that can’t be reduced to numbers or plotted on a colorful graph. “Academic knowledge is important, but I will be taking so much more away with me when I graduate,” Mahaffy says.

One thing she’ll leave here with are lifelong friendships. Mahaffy recalls taking the test during her first week on campus, but what she remembers most clearly about that day three years ago is meeting her roommate for the first time that afternoon. “I was dressed as Super Grover. I was covered in water, wearing a cape. Our other roommate, Corrina, was next to me, dressed as a nerd,” Mahaffy says. “I was like, ‘I promise, we’re not actually this weird!’”