Archive for the 'Assessment' Category

Testing As a Weapon

Photo credit: Robert King/Getty Images

A bill that will link individual teacher’s salaries to student performance and effectively destroy teacher job security was passed by the Florida state Senate last week.

Besides its obvious anti-unionism (pretty much business as usual for Florida politics), this bill will most likely serve to punish rather than help schools that are facing a number of difficult obstacles while rewarding those that are already relatively successful. It’s unfortunate that the war against public education in recent years is so often waged using the tools of accountability and evaluation, both concepts that might actually be put to good use. Even historian and former assistant secretary of education under Bush Diane Ravitch, a long-time supporter of standardized testing and No Child Left Behind, seems to be reversing her position on the matter in her newest book, The Death and Life of the Great American School System: How Testing and Choice Are Undermining Education.

If the bill passes (as it most likely will given the comfortable Republican majority in the House and the willingness of Florida Governor Charlie Crist) notoriously inaccurate standardized testing outcomes will be used to evaluate teacher salaries and job security, essentially using one inaccurate form of evaluation as a foundation for another; however, it will also have a much more direct effect on learning. When Florida teachers begin “teaching to the test” in a desperate attempt to hold onto their jobs and a decent standard of living, it seems inevitable that the teaching of many important written and oral communication skills will quickly drop out of the curriculum.

The Stressful CPE

184; Stress level: Midnight (please read description!)
Creative Commons License photo credit: Sara. Nel

After doing several workshops for students planning to take the CUNY Proficiency Exam (CPE) I’ve been thinking about some fairly basic questions about standardized testing that are nonetheless important ones. I’ve come to realize (as have other Fellows at Baruch) that one of the most important functions of these workshops is to alleviate student anxiety. While some students do not seem to worry too much about the exam, many (some of them excellent students) become rather anxious especially in regards to the time constraints. This raises a number of questions for me regarding the effectiveness of this form of assessment. Are we really setting up a situation that accurately measures student performance of these skills given the stress of the testing situation? According to this article, we aren’t.

As health blogger Laurie Pawlik-Kienlen points out, “Scientists have long known that long-term stress impairs brain cell communication, but they’re just now learning that even short-term stress – such as a few hours of anxiety – can negatively affect cognitive skills.” Pawlik-Kienlen cites research from the University of California (Irvine) School of Medicine as well as the Laboratory of Stress Research at Douglas Hospital Research Center to make this point. Given this negative affect of stress on memory it would seem that we are setting up students for failure. Of course, it could be argued that the anxiety-producing test situation is preparation for stress soon to be experienced by students in the work world. If this were the case, why wouldn’t we coach students on ways to manage this type of stress early in their educational careers? In general I understand the need for assessment of student learning; however, I wonder if it isn’t time for us to start thinking about some different ways of accomplishing this goal outside of the traditional timed exam.

Of Student Debates and Other Demons

20090419_EUD_045
Creative Commons License photo credit: mhonpoo

I finally figured out what to write about for Cacophony! Following the advice of my colleagues at the Bernard L. Schwartz Communication Institute, the best way to approach this was to write about something I am familiar with in the context of my work.  As a professor myself, I set specific guidelines and objectives when giving assignments to my students in order to avoid writer’s block because of the openness of possibilities. I don’t want to curtail, however: Cacophony’s open posting policy makes it versatile and unique.

I hope this post gives some basic guidelines for anyone out there interested in organizing debates as a classroom assignment.  The topics of the debates I am coaching are in the 12th Edition of the Management and Society textbook issued by the Department of Management at Baruch College. But you can device your own and have students do a little research to defend their positions.

The first step is to assign students to groups and divide the groups into PRO and CON sides of a given topic.   Then, provide precise instructions about the format of the debate.  For example, one format consist of a ten minute opening presentation, followed by a five minute period for rebuttal, and three minutes for conclusions, going back and forth between the PRO and CON side.  Ten minutes for the PRO, Ten for the CON; five minutes for the PRO, five for the CON; and three minutes for the PRO, and three for the CON. You can make them longer depending on the number of participants and the time available.

Make sure students understand that the objective is to persuade the audience that their point of view (in the debate) is the most valid: they need to make arguments.

In the beginning, they should introduce themselves, the issue, the point they are defending and any terms that might be unfamiliar or that might take a particular meaning in the context of the debate.  For example, in a debate that deals with whether genetically modified foods should be labeled, it is necessary to know from the beginning what constitutes a genetically modified food product.

Encourage them to read the materials a couple of times (in the management course I coach these are organized in chapters), even the reading for the opposite team.  In that way they can figure out a strategy to organize their presentation as well as anticipate the points are going to be brought up against their arguments.  It’s also important for students to practice their entire presentation out loud so they have an idea of time management as they become familiar with public speaking.  In terms of oral presentation skills,  you should emphasize to the debaters that they should not read, and should maintain eye contact with the audience,  which is a non-verbal way of engaging their attention.  Index cards are an acceptable way of keeping track of the order of the arguments they will stress, but in order to avoid reading too much from them,  suggest they write bullet points, rather than entire sentences.

If they are using numerical data such as statistics and/or percentages, remind your students that if they are hard to understand, the audience will just glaze over them.  Quantitative data should be easy to read and understand and should make a strong point.  If they are quoting textbooks or the internet, make sure they cite valid sources and not just random articles (especially online),  and that they have those sources (author’s names particularly) readily available during the debate, in case someone asks.

Time does not have to be equally split, but all students in a team must participate.  Have students dress professionally (although this is not a strict requirement).  Attire is a non verbal language that reveals many things, and it is difficult to find credible someone wearing an oversized sweater whose sleeves are longer than the arms. Lastly, remind students to keep their language appropriate and to keep their composure.   Debates can get heated,  but for as much as a Jerry Springer fight will definitely engage the audience, the loudest people are usually revealing insecurity.

The end of each debate could be marked by an open Q&A period where the audience can participate and ask questions or comments to the presenters.  Here you can explain how the topic is still current and give an informal assessment of the students’ participation.

Assessment: the dirty word

Now seems like as good a time as any to reflect on something that’s been on my mind for a while: assessment. While maybe not the most exciting topic, I think it’s a really important and prevalent one. To be clear I’m referring to program assessment here, not assessment of student writing. Until last year my only experience with and training in assessment was through working at community-based organizations, specifically programs for youth that incorporated education and work readiness as well as several other elements. While this experience had its ups and downs, last year I figured out pretty quickly that assessment means something very different in the university context. I, of course, saw assessment and the implementation of Writing Across the Curriculum at CUNY as a great marriage. Faculty in different disciplines trying out different pedagogical tools? Lots of written products, i.e. data? Opportunities for different people to get together and talk about their teaching experiences, what works and what doesn’t? Great! I really did not expect the resentment and lack of cooperation I received when I began to talk to faculty about these issues.

Rather than focusing on all of the problems and tensions around these issues within some (not all) universities, I thought I might mention a few basic elements often emphasized by community-based organizations:

First, assessment should be truly collaborative or it can quickly become extremely divisive. Transparency seems really important here. Asking for all kinds of information about someone’s classroom, students, and teaching without being clear about how that information will be used can be a great way to alienate faculty members.

This leads to the second point, which is that assessment should serve as a means of improving the overall quality of education in a particular department or discipline or university rather than as a policing mechanism. While it’s important to be aware of areas that need improvement, highlighting best practices is equally, if not more, important.

Finally it seems important to start and finish with the people actually doing the work, in this case, faculty members teaching writing and using writing as a teaching tool. Being aware of the needs of these folks allows the assessment to be more than charts and graphs. This way the information gleaned from this assessment project can be put to practical use. This is also a good incentive for faculty members to cooperate and provide useful data. It can even make it possible to enlist their help more directly. While faculty and administration often have different priorities, they don’t have to conflict. I think both groups have some stake in assessment and, if designed and implemented properly, it can help both meet their goals.

VOCAT Switcheroo: Assessing the Assessor

A few weeks ago, I logged on to VOCAT for the first time, and as watched the video of a student’s rehearsal for their presentation, I was surprised to hear my own voice. I was sitting near the camera, and focused on the students as they went through their Powerpoint slides. Maybe because the camera was pointed towards them, at the time of the recording I was unaware that I was also recording myself.

And this made for a kind of unexpected self-assessment, along with the student assessment I was prepared to do. I’ve often wondered if my voice is too low, if I repeat myself too much, if what I’m saying makes any sense, if what I’m saying is more helpful than confusing to my students. And I realized, listening to myself talk to a student on the VOCAT video, that I’ve spent six years of graduate school trying to get better at absorbing what I read, and better at writing clearly. But I haven’t put any sustained or rigorous effort into getting better at speaking.

For me, the VOCAT incident, the unexpected switch of the assessment tool back on the assessor, made me realize how alone I have felt with this part of teaching. The first day of your adjunct job: the door shuts behind you, it is just you and students. A professor visits my class for one session during the semester, sometimes they don’t stay for the whole class. Their written assessment is usually generous and they’ve all talked with me after the class to offer encouragement and the wisdom of their experience. But, you know, the rest of the time, it is just you in there. Talking and talking. Wondering if the students are falling asleep because they’ve just eaten lunch, or is it the lulling drone of my voice? I know there are books and articles out there I could be reading on how to effectively engage a class. And I’ve sat in on other professor’s classes to see what I pick up from the way they engage a class. George Shulman at NYU Gallatin showed me how effective it is to value every student’s contribution, repeating it, rephrasing it, writing it on the board. Heidi Kruger at the New School held me spellbound with her intense, low whisper. Sekou Sundiata at the New School moved around the class like we were the orchestra and he was conducting us.

But, what works for me, and for my students, on this particular subject? I hadn’t really focused on that so much. Which is weird, given how, you know, important oral communication skills are in teaching.  Should the VOCAT assessment tool be turned on teachers? Well, I wouldn’t volunteer. But, when confronted with it, I thought it showed me some things that I should be aware of.

This brings me to the connection between writing and speaking. At the recent WAC conference, several people brought up the fact that writing often, in different forms, helps people become better writers. Speaking about writing also improves writing.  We talk about students ‘finding their own voice.’ One impediment to that might be that students are reading authors whose voices are quite different than their own. Often when I’m working with students on their presentation, I’ll ask them to summarize or draw a conclusion from their research. They articulate clear, original, logically organized claims aloud. But, when it comes to the formal work, they leave this out. Why? The answer I’ve heard more than once was, “But, that is just my opinion.”

What I want students to do, what I’ve heard other teachers say they want students to do, is enter a conversation with the authors they cite. What I’ve seen happen too often is a student articulating their own view, then summarizing an author’s view, using the author’s own style. How can we yoke them together?

One possible way might be to value thought when it is articulated aloud, not just in print. And one way to do this might be to film it, to actually turn the light and focus on recording speaking a thought, the way writing records a thought.

At the WAC meeting, Thomas Meechum and Karen Gregory’s documentary about the writing process in professor Michele Pacht’s class showed students responding to questions about their opinions about graffiti. I wondered if the heightened attention of the camera on the spoken thoughts helped the students to value their thoughts enough to commit them to print. I wonder if I should review the recording of my voice, talking to my students, as many times as I am reviewing the drafts of my dissertation proposal. I kind of think I should.

Against Grades and Grading

The majority of students from the Business school who come to the Schwartz Institute to rehearse their company or industry analysis powerpoint presentations seem to look at the rehearsal process as an opportunity to improve a necessary skill. This has been one of the most rewarding aspects for me of my work as a Communication Fellow: the students are always grateful for the help in improving their public-speaking skills. They are motivated by the idea that they are helping themselves. I like that I do not have to grade their work for them to see it as important.

The institution of grading students on an A through F scale has done a horrible disservice to education. It has falsely given the impression to generations of students that the teacher or the professor has some ultimate authority over the value of their work, as if their own assessment of what they were doing was somehow secondary. The result of this institution is a division among most students into two groups — a group motivated by competition and the drive for the teacher’s approval, and a group lacking in motivation with little interest in the teacher’s assessment. What is missing all too often among students in both of these groups is the sense that their education is their own.

I have found several methods of correcting this problem that work within the extant system. By far the best of these methods is to ask students to write self-evaluations. All teachers who have ever taught a graded course know that students approach them to apologize for not having completed an assignment — the proverbial “my dog ate my homework” moment. The self-evaluation taps into the students’ innate authority over their work which is too often evident only in their apologies. If you ask students to write about how they have approached the assignments of the class and you ask them to write about their own perceptions of their strengths and weaknesses, they very quickly begin to realize their own agency in the learning process and to begin take responsibility for their own education.

Of course the best thing, I think, would simply be to do away with grades and grading altogether. I know that for many people this suggestion amounts to advocating “mere anarchy.” Without the carrot and stick, there would be no motivation anywhere among students, no assessment, no accountability. It’s true that in all likelihood, the students who come to me to rehearse their powerpoint presentation are not motivated purely by their own desire to improve. Their presentations are graded and they want to get a good grade. Well, perhaps this is true. But in a time when the movement for standards has taken over every level of education, I find some comfort in recollecting a different ideal.

The Video Oral Communication Assessment Tool and the Question of Openness

It recently occurred to me that in the almost 4 years of this blog’s existence very little has been said about the Schwartz Communication Institute’s most ambitious and potentially most promising project, our Video Oral Communication Assessment Tool, or VOCAT. I have presented on VOCAT a number of times over the years (most recently at the 2009 Computers and Writing conference in June), but have not yet written about it here. So it’s high time to remedy that.

VOCAT is a teaching and assessment web application. It is the fruit of a collaboration between the Schwartz Institute and mad genius code-poets at Cast Iron Coding, Zach Davis and Lucas Thurston. It is still very much in development (perpetually so) but is already in use in introductory speech communication and theater courses as well as a number of assessment projects. Our career center used it effectively a few semesters ago as well. To date, approximately 3200 Baruch students have used the tool.

VOCAT was developed in recognition of the principle that careful, guided review of video recordings of their oral presentations (or of any performance, for that matter) can be remarkably effective for aiding students in becoming confident, purposeful and effective speakers. It serves as a means for instructors to easily provide feedback on student presentations. It enables students to access videos of their performances as well as instructor feedback and to respond to both. It likewise aggregates recorded presentations and instructor feedback for each user and offers an informative snapshot of a student’s work and progress over the course of a given term or even an entire academic career. Presentations can be scored live, as students perform, or asynchronously once the videos have been uploaded. (Our turnaround time at this stage is between one and seven days depending on how many sections are using the tool at once — once some of the key steps happen server-side, turnaround time will not be as much of a concern.) Built on the open source TYPO3 content management system, it is a flexible, extensible and scalable web application that can be used at once as a teaching tool and as a means of data collection for research or other assessment purposes. (Screenshots are available here. I am also happy to share demo login info with anyone who would like to take a look — please email me at mikhail [dot] gershovich [at] baruch [dot] cuny [dot] edu.)

While VOCAT is quite feature-rich at this early stage, especially when it comes to reporting, data export, and rubric creation, we are always thinking about ways in which the tool can be made more robust and flexible. Currently, we are playing around with adding a group manager feature for group presentations, tagging for non-numeric assessment, moving from QT to Flash video, video annotation, as well as server-side video processing and in-line video and audio recording. We are also considering allowing users to choose to enable social functionality to take advantage of web 2.0 tools for sharing and commenting on one another’s work. And since, at its core, VOCAT is a tool for aggregating and responding to anything that can be uploaded, we’re thinking about other uses to which it could be put. It could easily, for example, be adapted for writing assessment. And someone once suggested that it could be useful for teaching bedside manner for medical students. Adapting VOCAT for these purposes is hardly a big deal.

The platform on which VOCAT is built is open source but the tool itself is not yet open. Right now, it is Baruch’s alone. Whether it should stay that way is a question much discussed around here. Here at the Institute we face several critical issues around open education, not the least of which is conflicting views on student access of Blogs@Baruch. In regards to VOCAT, however, the one thing constantly on my mind is the tension between an internal drive to share the tool as an open-source web application and build a community around it (there are no shortage of interested parties) and the pressures (or maybe a pernicious institutional common sense) that seem to compel us to keep VOCAT proprietary and use it to generate as much revenue as possible. I have heard arguments that VOCAT should be Baruch’s alone — that we should charge for its use and seek private funding for its deployment and development. This is a business school, after all, and I’m sure promoting and marketing VOCAT could be an interesting project for an upper division Marketing course.

Yet, I am inclined to believe that VOCAT should be shared freely and widely with other institutions and that other developers should be encouraged to develop for it. A great many more students would benefit and development would certainly be accelerated as more and more schools add features they need that could then be adopted for use here. Were VOCAT open, in other words, it would evolve quickly and probably in ways we haven’t even imagined. And that is very exiting.

In the coming months, I hope to continue to present on VOCAT and to gain insights into the roles it can play in communication intensive courses or in a communication-focused curriculum of any sort. More importantly, I would like to move towards opening it up and will work with our developers on the features and functionality that facilitate sharing. I hope also to draw upon the tremendous expertise of my friends and colleagues involved in the open education movement and learn from those who have worked with and developed various open source tools for teaching and learning. Listening to others’ ideas for VOCAT has been invaluable to thinking through what this web app could ostensibly do with the right sort of development. could be and how to best realize its full potential as a teaching tool — both in terms of deployment, training, and development.

Reading, Assessment and Great Works of Literature

charles-dickens-caricature

I am currently working with the Great Works of Literature faculty at Baruch on an assessment for the Great Works course. The faculty is interested in evaluating the learning goals for the course. The first step was to talk with the faculty about what they teach, how they teach it and what they feel about it. These are always great discussions and I believe fundamental to making a good assessment. At one point a faculty member stated that reading was a central part of the course and that she was, among other things, teaching in-depth reading. I was quite struck by this thought of reading and how the Great Works of Literature course taught students to engage with different texts and make inferences to the world around them through reading.

David Frost and I thought a lot about how to incorporate reading as part of the assessment process and how to design a prompt to fit with the specifics of reading for a Great Works of Literature course. The obvious was to ask students to read a short text and respond to it. But the difficulty was to find a reading experience that mirrored what goes on in the course.  Reading literature from multicultural environments and then exploring the relationship between the different genres and cultures is an essential part of the course.  But this is not easy to assess, and as most faculty say, even to grade.

For this assessment we are going to try an experiment; a pool of short texts will be available for the faculty to choose for the prompt. The students will be reading pre and post prompt texts that might be different in author or culture but the same in length and complexity and genre. The texts will also relate to authors or literary periods that the students studied during the semester.

The hope is students will be reading, and responding to the reading, in the same way they do in the course. The second hope is we will be able to draw out meaningful information about the students experience in the course as well as any increase in comprehension and knowledge. Everyone involved in this assessment is pretty excited about this experiment and its creative use of texts for the prompt.

I am too, as I hold in my breath to see if it really works.

Draft Learning Goals for Writing and Speaking

I was reminded today that I once drafted a set of learning goals for writing and speaking at the undergraduate level for a project headed up by our office of advisement and orientation. While these goals implicitly inform the curricular support and development work of the institute, they have not been codified beyond the document I created in 2006 (before I learned about Bloom’s taxonomy). These goals have not seen the light of day beyond their very limited original context. With that, I thought I’d post them for discussion. Take a look and let us know if you find these useful and/or whether you’d recommend revisions. Here we go:

By the end of their undergraduate experience students should be able to:

  • comfortably pose pertinent questions to faculty both in and out of class
  • demonstrate proficiency in a number of everyday written genres (email, letter, etc.)
  • demonstrate sensitivity to audience in oral and written communication – write and speak in a manner appropriate to audience – articulate similarities and differences in addressing different audiences (email to peer vs. email to faculty, conversation with parent vs. conversation with prospective employer)
  • demonstrate awareness that all communication is purposeful – each individual communication is meant to accomplish a particular goal or set of goals – sensitivity to purpose
  • grasp rhetorical purpose of own written work (what is this paper, email, memo, etc. meant to accomplish? What do I need it to do? What should it accomplish?)
  • articulate how they might go about accomplishing purpose of given communication (in order to accomplish X in my email to my professor, I need to make clear that Y and establish Z before making the argument that A)
  • work responsibly and productively as a member of a group – to communicate appropriately with all group members
  • comfortably speak before an audience – impromptu and prepared presentations
  • articulate own understanding of how they can become better communicators (what do I need to work on to become a better writer/speaker?)

Discuss.

Lessons from a First-Time Course Blogger

I’m finally looking back to Spring ’09, when I had my first experience using Blogs@Baruch in two sections of COM1010, Intro to Speech Communications. I used the blog for the midterm, in which students write critiques of speeches they’ve found online. In past semesters, students have been inventive in their speech choices and committed in their critiques. But the question of how to best enable their classmates to see these videos still lingered. Curious about Blogs@Baruch, I decided to migrate this assignment onto a blog, allowing students to watch (and comment upon) each other’s videos and share their critiques of the speeches. Having learned from the adventure, here are a few words of advice to potential Blogs@Baruch-ers.

1. It’s not difficult. Considering the gong show of Blackboard’s tech problems this semester, it was almost comical how smoothly the blog functioned. A handful of students ran into some problems accessing it at certain computers, but often I found that problems encountered by students were frequently due more to lack of time and preparation on their part than any issue with the blog itself.

2. Don’t be conservative! I was. As one of my students told me at the end of the semester, “the blog was just there.” It wasn’t as dynamic as it could have been, in part because I didn’t use it to capture anything in progress. Students cut and pasted their work onto the blog, and then made the requisite comment on a post, creating a static space outside of the classroom, not a particularly engaging one. While it was satisfying to see this vast collection of interesting video clips assembled in one place—along with frequently cogent, in-depth analyses of them—I see now that I used the blog to solve a problem (that of my midterm assignment) rather than tailoring it for uses that would really suit the nature of the blog. Recent conversations with my students and others have highlighted a range of ways that it could be used in an Introductory Speech course– sharing audio files or outlines of student speech drafts that could be revised as the “audience” comments. On a related note, the public forum really does elicit strong work. When students feel the watchful eyes of their peers, the bar is set somewhere different. This makes my mouth water for the possibilities of the course blog—like facilitating peer review, for example—that I didn’t explore.

3. Be forewarned: out of sight, out of mind. In part due to #2 above, the blog can feel like that side dish you ordered but weren’t quite hungry for. It’s easy to lose track of the blog, and its implementation should be planned with an eye towards avoiding this. Usually, the material nature of grading compels you to eventually plop down on a long train ride and hit it out of the park. With the blog, not so easy. I had good intentions—I wanted to comment on posts frequently, but commenting is time-consuming, especially if students are posting 40-minute inauguration speeches. This in turn leaves less time to evaluate the work for grading purposes. From the student side, they were assigned a date for one post; once students posted, they didn’t have a strong incentive to return, which would leave me begging them to “visit the blog!” when I myself was embarrassingly behind on reading their old posts.

4. Students might be less excited about instructional technology than you are. (…How to get them more excited is part of the task.) Take ‘tagging,’ for example—it was harder than I might have imagined getting the ‘tagging’ to happen. Some assume that the ‘Sidekick generation’ will tag as if it were natural as breathing. Not so– every nineteen-year-old might know how to search YouTube, but they’re not all writing Facebook applications or even their own blogs. Making some class time available to teach students the rhyme and reason behind some aspects of the blog is arguably essential, and yet somehow easy to overlook.

The Com1010 Public Speaking Award Goes To...

The Com1010 Public Speaking Award Goes To...

5. Students love Pacino. As in past semesters, his speeches were cited with a remarkable frequency, rivaled only by Randy Pausch. This is perhaps not a surprise, since the first hit from googling “inspirational speech” is Pacino’s “peace by inches” monologue from Any Given Sunday, but still. City Hall has a less predictable—and arguably far better—dramatic monologue that I’m glad one of my students spread around.

I’ll end here with a question. As Luke articulated so well in his WordCampEd post, these open source technologies are blessedly DIY. But I can’t help feeling a little protective of the adjunct in this discussion– don’t adjuncts “do it themselves” enough? Can the full potential of Instructional Technology really be unleashed with the real limitations of the adjunct labor force operating in higher education? I’m in a distinctly lucky position as a dual-hatted Communications Fellow and adjunct; working with people jazzed and knowledgeable about these technologies has taught me tremendous amounts about how to use it and why. But how will Jane Q. Adjunct learn about the potential of a course blog, after tearing her hair out over Blackboard for months and missing the departmental meeting that announced a later workshop about blogs, all time she’s not paid for? How will Jane Q. Adjunct get excited about the potential of these tools, and why will she motivate to prioritize the time required to integrate them thoughtfully and productively in her course?