From Edutopia, the website of the George Lucas Educational Foundation, an excellent interview with James Paul Gee, a linguist who has become the leading authority on video games and pedagogy and who gave a great talk at the CUNY Grad center last year. Enjoy.
Archive for the 'Assessment' Category
Has anyone else noticed the new signs on the subway? For the second time in two years, the MTA is conducting a survey of its riders. I don’t remember seeing the signs when they were doing the survey the first time around, but it was apparently some time in 2007, and they wanted to know what suggestions we had for making the subway system better. You can go to to their website and see the results — what they call the “Rider Report Card.”
Now the MTA wants to know exactly how and why we New Yorkers get around the city. When I first saw the advertisement for the survey I was skeptical. I couldn’t help but wonder if they were actually going to take our feedback seriously or if this was just a public relations move to get us all to feel a little more hopeful that better commuting days are ahead. When I got home, I went online to learn more about the survey. As it turns out, the MTA has contracted an outside firm, Nustats, to gather this information for them. Somehow, the fact that they are investing money to do this made me feel a little more confident that the MTA is actually making an attempt at genuine communication with its customers. However, I found two things rather peculiar: The MTA does not actually mention this current survey on their web page; I actually had to google “MTA Survey” to find it. Also, the survey is not being made available to the public via the internet. In order to participate you need to either download a paper form from a PDF file or call a toll free number and take the survey over the phone. I’m curious about those choices. I’m also curious about the $500 prize they are giving out weekly to one survey participant who is to be chosen at random from a drawing. If you’re interested, go to:
As the presidential election approaches I find myself thinking a lot about communication between institutions and individuals and wondering how much weight does the individual voice carry. But also, how important is it that individuals feel their voices are being heard? Will the chance for $500 entice subway riders to actually pick up the phone or download the file and participate in this survey? How sincerely does the MTA actually want us to?
I just read an interesting article by Mark Bauerlein in The Chronicle about how students’ approaches to reading and interacting with information online seem to be hindering their ability to read and learn from texts in more traditional settings. Specifically, he contends that:
The inclination to read a huge Victorian novel, the capacity to untangle a metaphor in a line of verse, the desire to study and emulate a distant historical figure, the urge to ponder a concept such as Heidegger’s ontic-ontological difference over and over and around and around until it breaks through as a transformative insight — those dispositions melt away with every 100 hours of browsing, blogging, IMing, Twittering, and Facebooking.
This brings up a lot of interesting questions as educators are increasingly trying to incorporate some of these technologies into the classroom and publishers are pushing textbook content into more profitable eBooks. Are we actually helping students by doing all of this? Some initial studies of middle and high school students suggest that technology-intensive curricula do not improve student achievement.
Bauerlein has many interesting points in the article and makes a good case for “unplugging” some aspects of teaching and learning. However, in my opinion, the question of whether or not technology in general improves/impairs student learning is not that interesting. Instead, we should be focusing our assessments on understanding which technologies can be usefully employed in which aspects of the curricula. Finding pedagogical fit for relevant technologies seems to be what we are striving towards at BLSCI. Thus, as an institute, we undoubtedly have much to contribute to this important discussion.
In BLSCI’s application for the TIAA-CREF Hesburgh Award, we made use of the writing diagnostic assessment data to demonstrate the many ways the Institute has grown over the past 10 years.
As Figure 1 and 2 below show, BLSCI fellows support faculty teaching a number of distinct Communication Intensive Courses (CICs) across a variety of disciplines. As Figure 2 shows, the largest representation of faculty teaching CICs is in departments that have traditionally placed a heavy emphasis on both written and oral communication, such as English, Modern Languages, Marketing, Management, Performing Arts, Sociology and Anthropology. However, the institute has also supported CICs in departments that have not traditionally incorporated communication intensive elements into their curricula, such as Accountancy, Natural Sciences (e.g., biology, chemistry, and environmental sciences), and Computer Information Systems.
When we look at these data and hear about all of the great work going on at the Institute during our staff meetings, what we often don’t take into consideration is the amount of expansion that has taken place over the past ten years. As Figure 3 demonstrates, the number of faculty supported by BLSCI has steadily increased, reaching a peak of 144 last year. The number of faculty currently teaching CICs is nearly three times what it was ten years ago. Despite some minor fluctuations, the number of sections of CICs has also increased dramatically. Specifically, as shown in Figure 4, the number of sections of CICs offered last year is nearly five times as many as there were in 1997.
There is no doubt this kind of institutional growth contributed to BLSCI’s being awarded the Hesburgh award. However, the most interesting growth going on at the institute is arguably what happens on a more micro level among students, faculty, and fellows throughout continued mentorship and collaboration. Although we all get to observe this in our individual work, it’s often hard to demonstrate this kind of growth across the institute. As we keep on thinking about and celebrating growth at BLSCI we continue to think about ways to assess it. It’s my hope that this post will spark some ideas among readers on how we might approach this kind of assessment next semester.
Assessment, assessment, and assessment; this seems to be all I am hearing these days. Editors at academic journals inform me that if I have an article on assessment, they would be happy to publish it… Even my daughter’s 2nd grade teacher had a meeting to explain to the parents how our 7 year olds would be assessed over the school year. In my new position as Deputy Director of the Bernard L. Schwartz Communication Institute, I am currently overseeing two major assessments: one of the Institute’s programs and another of student writing in CICs. Surprisingly I find myself enjoying it immensely.
As assessment seems to be the 21st century companion to the education field I guess it is time to just jump in. I find myself fascinated and passionate about looking at and assessing students’ writing over a ten-year period and all of its infinite possibilities. However there is a small part of me that feels there is a fine line that is frequently crossed in assessment. It is when educational institutions, or even individual educators, over-invest in the assessment of whether students attain pre-established learning goals to demonstrate that students have learned. When student outcomes, in relation to pre-set learning goals, are the main goal of an assessment, outcomes can easily become the dominant product of education rather than the messy but profound experience of learning itself, which does not always produce a clear outcome. And if this becomes the assessment norm, to measure outcomes rather than transformative experience, than education runs the risk of merely accumulating material and compromising its fundamental role. This is a line I feel is dangerous for any educator to cross.
Ken Buckman, professor of Philosophy at the University of Texas, wrote in the fall issue of Thought and Action that, “the primary character of education is its transformative influence…” Therefore, experiencing the process of learning is as important if not more than the final goal of having attained an acceptable outcome or learning goal. Yet this is one of the hardest areas of learning to assess: the transformative experience.
However, there might be a way of examining student learning as a process and not an outcome. Follow the same students for several years and see what parts of their learning process have become integrated into the way they talk or write. In a sub-sample set of the written diagnostic data we are now examining the same set of students writing over a period of 3-4 years. Their vocabulary, their expectations and the complexity with which they express themselves can be analyzed as well as how this has been transformed over time. This data will not tell us whether the students did well on the final exam or whether they were able to write an “A” term paper. But we might be able to note when and if the student was going through a transformative process between the times they started writing in their first CIC course and the 3rd or 4th CIC.
There is no doubt that any educational institution needs to be able to demonstrate that its students on an aggregate level are reaching the goals and outcomes that have been put forth. However, students should also be studied as distinctive learners with unique goals and experiences. As for my part, I can’t get enough of knowing that my students might have been transformed.
As many of you know, last month BLSCI applied for external funding from an organization that recognizes exceptional faculty development programs focussed on enhancing undergraduate teaching and learning. In order to make our case for the award, we included some preliminary results from the Writing Diagnostic Assessment data. I’d like to use this post as an opportunity to share some of these results with readers to demonstrate the effectiveness of the work that many have been doing over the years and get some feedback regarding ideas for future analyses.
When looking at the data, on average, students start the semester with scores on both the expectations and writing quality variables in the “middle” range (scores around 3). When we then looked to see if students’ scores significantly improved over the course of one semester in a CIC, there were no major findings. This was because many students started the semester out scoring high (scores from 4 to 5) on many of the variables, and thus did not have any room for improvement (as measured by the diagnostic scoring criteria).
However, when we looked at students who scored in the “low” to “middle” range on all of the variables (thanks Diana for this suggestion!) we observed statistically significant increases from the beginning to the end of the semester on all variables. These increases were consistent across disciplines and schools as well. The figures below illustrate the changes we observed in the data separately for the Weissman and Zicklin Schools.
Although these results are based only on a subset of the data we’ve been able to clean, match, and analyze (~ 5,000 students), they nonetheless illustrate that the work of BLSCI in creating and implementing CICs seems to be paying off for students across the board. Although most probably knew or were able to sense this already, it’s always great to have “hard data.” We would love to hear readers’ thoughts on these findings and how you see these data stacking up next to the work you’re doing with students in your own classes. Also, as always, any general thoughts and/or questions on the assessment data are welcome.
As Mikhail noted at the first BLSCI sandwich and cookie-fest, we’re being encouraged to develop new assessments of our work. I thought it would be good to try to get a discussion going here on Cacophony where we could share our thoughts about assessment.
At the CUNY Writing Fellows Orientation, I attended a breakout session where we looked at a compilation of the surveys given to all of the WAC-WID Coordinators at CUNY. We learned a few lessons. Each program is drastically different in structure, oversight, and activity. In some ways these differences reflect the particularities of the CUNY campuses; in other ways, they are just the products of local bureaucracies. Nearly every program has some element of faculty development, though with varying incentives for participation. Most programs have a web presence, and a few are experimenting with new media as an instructional tool. The writing-intensive requirements vary wildly across the campuses, and nearly every campus expressed concern about constant change and limited resources. Finally, a question on “How do you measure the work you are doing?” garnered responses that effectively said, “we assess,” without much exploration of what that meant.
I reference this survey because I think that when doing assessment, it is necessary to first understand the role of your program within the whole institution. The BLSCI (of which the Writing Fellows are only a small part) has a unique challenge because while we are committed to improving communication-intensive instruction, this means very different things in each of the disciplines. Such a situation complicates assessment, not least of all because it depends wholly on the input of a group of individuals who are not necessarily the best-positioned to design and perform assessment: fellows. We’re mostly temporary employees who are completing our degrees and applying for full-time gigs. At the same time, we work for an Institute that is the closest thing to a Teaching and Learning Center that Baruch has, so we should have something to say about how assessment works.
This situation exists in tension with some of the “Nine Principles of Good Practice for Assessing Student Learning,” as outlined by the Center for Teaching, Learning, and Assessment at Indiana University-Kokomo. I found the following points of particular interest: “Assessment works best when it’s ongoing, not episodic;” “Assessment fosters improvement when representatives from across the educational community are involved;” and “Assessment is most likely to lead to improvement when it is part of a larger set of conditions that promote change.” In addition to exploring our work within our narrow assignments, then, assessment is needed of the environments within which that work takes place: the curricula of the Weissman and Zicklin schools, the functioning of the Institute, and the mission of CUNY. Assessment should be done of administration as well as of courses and special programs, and our input, as fellows, should be integrated with a larger, systematic approach where the utility of assessment is clearly stated. Too often I fear that assessment is done merely for self-justification. An assessment that maintains the status quo is difficult to get on-board with, while an assessment that will yield improvements at the college could be exciting.
One idea dominated the breakout session I mentioned above: that Writing Fellows and WAC/WID programs at CUNY would benefit greatly from the centralization of information (though not of the administration of WAC/WID programs). The Office of Undergraduate Education is making strides in structuring support for the provincial programs that can be shaped to local concerns, particularly by promising a central digital archive that organizes and distributes the knowledge being generated on the campuses. What other campuses do for and with assessment would be a component of this, which is great.
What other questions might we ask and information might we seek as we begin to assess our path to assessment?
Recent Comments