Assessment and the Transformative Experience

images.jpeg

Assessment, assessment, and assessment; this seems to be all I am hearing these days. Editors at academic journals inform me that if I have an article on assessment, they would be happy to publish it… Even my daughter’s 2nd grade teacher had a meeting to explain to the parents how our 7 year olds would be assessed over the school year. In my new position as Deputy Director of the Bernard L. Schwartz Communication Institute, I am currently overseeing two major assessments: one of the Institute’s programs and another of student writing in CICs. Surprisingly I find myself enjoying it immensely.

As assessment seems to be the 21st century companion to the education field I guess it is time to just jump in. I find myself fascinated and passionate about looking at and assessing students’ writing over a ten-year period and all of its infinite possibilities. However there is a small part of me that feels there is a fine line that is frequently crossed in assessment. It is when educational institutions, or even individual educators, over-invest in the assessment of whether students attain pre-established learning goals to demonstrate that students have learned. When student outcomes, in relation to pre-set learning goals, are the main goal of an assessment, outcomes can easily become the dominant product of education rather than the messy but profound experience of learning itself, which does not always produce a clear outcome. And if this becomes the assessment norm, to measure outcomes rather than transformative experience, than education runs the risk of merely accumulating material and compromising its fundamental role. This is a line I feel is dangerous for any educator to cross.

Ken Buckman, professor of Philosophy at the University of Texas, wrote in the fall issue of Thought and Action that, “the primary character of education is its transformative influence…” Therefore, experiencing the process of learning is as important if not more than the final goal of having attained an acceptable outcome or learning goal. Yet this is one of the hardest areas of learning to assess: the transformative experience.

However, there might be a way of examining student learning as a process and not an outcome. Follow the same students for several years and see what parts of their learning process have become integrated into the way they talk or write. In a sub-sample set of the written diagnostic data we are now examining the same set of students writing over a period of 3-4 years. Their vocabulary, their expectations and the complexity with which they express themselves can be analyzed as well as how this has been transformed over time. This data will not tell us whether the students did well on the final exam or whether they were able to write an “A” term paper. But we might be able to note when and if the student was going through a transformative process between the times they started writing in their first CIC course and the 3rd or 4th CIC.

There is no doubt that any educational institution needs to be able to demonstrate that its students on an aggregate level are reaching the goals and outcomes that have been put forth. However, students should also be studied as distinctive learners with unique goals and experiences. As for my part, I can’t get enough of knowing that my students might have been transformed.

cartoon_large_intro.gif

Writing Diagnostic Assessment: Some Preliminary Findings

As many of you know, last month BLSCI applied for external funding from an organization that recognizes exceptional faculty development programs focussed on enhancing undergraduate teaching and learning. In order to make our case for the award, we included some preliminary results from the Writing Diagnostic Assessment data. I’d like to use this post as an opportunity to share some of these results with readers to demonstrate the effectiveness of the work that many have been doing over the years and get some feedback regarding ideas for future analyses.

When looking at the data, on average, students start the semester with scores on both the expectations and writing quality variables in the “middle” range (scores around 3). When we then looked to see if students’ scores significantly improved over the course of one semester in a CIC, there were no major findings. This was because many students started the semester out scoring high (scores from 4 to 5) on many of the variables, and thus did not have any room for improvement (as measured by the diagnostic scoring criteria).

However, when we looked at students who scored in the “low” to “middle” range on all of the variables (thanks Diana for this suggestion!) we observed statistically significant increases from the beginning to the end of the semester on all variables. These increases were consistent across disciplines and schools as well. The figures below illustrate the changes we observed in the data separately for the Weissman and Zicklin Schools.

[click to enlarge]

[click to enlarge]

Although these results are based only on a subset of the data we’ve been able to clean, match, and analyze (~ 5,000 students), they nonetheless illustrate that the work of BLSCI in creating and implementing CICs seems to be paying off for students across the board. Although most probably knew or were able to sense this already, it’s always great to have “hard data.” We would love to hear readers’ thoughts on these findings and how you see these data stacking up next to the work you’re doing with students in your own classes. Also, as always, any general thoughts and/or questions on the assessment data are welcome.

Writing Diagnostic Assessment: Preliminary Research Questions

After meeting with the Associate Provost and faculty representatives of the Zicklin School of Business and the Weissman School of Arts and Sciences, we have now settled on some initial research questions to ask of the Writing Diagnostic Data. I share them here in hopes of generating discussion around the assessment and further ideas for analysis.

The first and most general research question is “Does the quality of students’ writing improve over the course of a semester in a CIC?” Related to this question, we are interested in whether improvement occurs across the college or only within specific classes or disciplines. We will also explore whether we see improvement for all students, or only among students who are scoring in the low or middle range to begin with. It will also be worthwhile to see what happens over the course of a semester with students who score high on their initial diagnostics.

Second, we hope to answer the questions of “What happens to students’ who are enrolled in multiple CICs throughout their career at Baruch?” and “Is there consistent improvement among these students from one CIC to the next?” These related questions are of particular importance within the Zicklin School, as students there are required to take 4 CICs before graduation.

We hope to have answers to these questions in the form of a preliminary report by the end of the semester. In the meantime, we are continuing to think about how we can use the data to address issues related to changes in admissions requirements, differences between lower level vs. upper level classes and ESL students vs. native English speakers, and how the outcomes of this assessment may correlate with other academic outcomes, like GPA.

As most readers are very familiar with the data (both its strengths and weaknesses) we welcome your comments, questions, and suggestions regarding these preliminary ideas for analysis as well as other research questions you think are important and would like to see addressed.

Writing Diagnostic Assessment Project

As most readers of CAC.OPHONY already know, we are in the process of delving into the Writing Diagnostic data that BLSCI has been collecting over the past ten years. My work, as a first-year Writing Fellow, is to help organize and make sense of this data. In order to keep the dialogue about this project ongoing and public, I will be posting periodic updates and I welcome feedback in the form of questions, comments, and suggestions. In this first post, I would like to provide readers with an overview of the data and the project more broadly as well as some initial ideas Mikhail, Suzanne, and I have for analysis.

Starting back in 1997 and ending last spring, fellows at BLSCI have done a tremendous amount of work collecting writing samples from students at Baruch enrolled in Communication Intensive Courses (CICs) across a variety of schools and disciplines. As part of an ongoing assessment of the effectiveness of CIC curricula in improving students’ writing, fellows scored students’ writing samples at the beginning and end of each semester. Specifically, fellows scored each writing sample in terms of students’ expectations for the class (i.e., development, tenor, and range of expectations) as well as the quality of their writing (i.e., ideas and development, organization and coherence, spelling and vocabulary, syntax and punctuation, and grammar).

We are currently in the processes of organizing and cleaning the data. So far, we have data from 1,395 CICs and 31,408 students entered into electronic format and (almost) ready for analysis. These data are primarily from 2000 – 2006, so we can expect the size of the sample to almost double when we are finished entering the early data and as fellows continue to enter remaining diagnostic data.

I can honestly say that I have not yet had the pleasure of working with such an extensive and impressive sample, despite having been a part of several large-scale research and assessment projects. The data are very exciting, because of the many lines of analysis we can potentially follow. For example, we have the ability to look across all of the students’ work to see if their writing has been improved by CIC curricula. We can also focus on the effectiveness of CICs within specific student populations (e.g., ESL students) and compare the effectiveness of CICs across schools and disciplines. We can also follow students who have been enrolled in multiple CICs, to see if and how their writing has changed over their entire career at Baruch, not just during one semester.

As with any “good” assessment project (see Luke’s posting from 9/12/07), we will be meeting with representatives from the Zicklin School of Business, Weissman School of Arts and Sciences, and the School of Public Affairs early next week to determine how to best use the data. Our goal is to identify each school’s unique concerns and questions and push forward with an analytical approach that satisfies the needs of each stakeholder as well as our internal curiosities as BLSCI.

I will continue to post updates as we make progress with this project. We welcome your thoughts and suggestions at any stage in the process.

On Assessment

As Mikhail noted at the first BLSCI sandwich and cookie-fest, we’re being encouraged to develop new assessments of our work. I thought it would be good to try to get a discussion going here on Cacophony where we could share our thoughts about assessment.

At the CUNY Writing Fellows Orientation, I attended a breakout session where we looked at a compilation of the surveys given to all of the WAC-WID Coordinators at CUNY. We learned a few lessons. Each program is drastically different in structure, oversight, and activity. In some ways these differences reflect the particularities of the CUNY campuses; in other ways, they are just the products of local bureaucracies. Nearly every program has some element of faculty development, though with varying incentives for participation. Most programs have a web presence, and a few are experimenting with new media as an instructional tool. The writing-intensive requirements vary wildly across the campuses, and nearly every campus expressed concern about constant change and limited resources. Finally, a question on “How do you measure the work you are doing?” garnered responses that effectively said, “we assess,” without much exploration of what that meant.

I reference this survey because I think that when doing assessment, it is necessary to first understand the role of your program within the whole institution. The BLSCI (of which the Writing Fellows are only a small part) has a unique challenge because while we are committed to improving communication-intensive instruction, this means very different things in each of the disciplines. Such a situation complicates assessment, not least of all because it depends wholly on the input of a group of individuals who are not necessarily the best-positioned to design and perform assessment: fellows. We’re mostly temporary employees who are completing our degrees and applying for full-time gigs. At the same time, we work for an Institute that is the closest thing to a Teaching and Learning Center that Baruch has, so we should have something to say about how assessment works.

This situation exists in tension with some of the “Nine Principles of Good Practice for Assessing Student Learning,” as outlined by the Center for Teaching, Learning, and Assessment at Indiana University-Kokomo. I found the following points of particular interest: “Assessment works best when it’s ongoing, not episodic;” “Assessment fosters improvement when representatives from across the educational community are involved;” and “Assessment is most likely to lead to improvement when it is part of a larger set of conditions that promote change.” In addition to exploring our work within our narrow assignments, then, assessment is needed of the environments within which that work takes place: the curricula of the Weissman and Zicklin schools, the functioning of the Institute, and the mission of CUNY. Assessment should be done of administration as well as of courses and special programs, and our input, as fellows, should be integrated with a larger, systematic approach where the utility of assessment is clearly stated. Too often I fear that assessment is done merely for self-justification. An assessment that maintains the status quo is difficult to get on-board with, while an assessment that will yield improvements at the college could be exciting.

One idea dominated the breakout session I mentioned above: that Writing Fellows and WAC/WID programs at CUNY would benefit greatly from the centralization of information (though not of the administration of WAC/WID programs). The Office of Undergraduate Education is making strides in structuring support for the provincial programs that can be shaped to local concerns, particularly by promising a central digital archive that organizes and distributes the knowledge being generated on the campuses. What other campuses do for and with assessment would be a component of this, which is great.

What other questions might we ask and information might we seek as we begin to assess our path to assessment?

Outsourcing grading and written feedback

I was surprised today to find an ad on InsideHigherEd to a company called EduMetry, which offers various outsourcing services to colleges and professors, among them, a 24/7 virtual writing center, and “Virtual TA.”

Virtual TA is a program where the grading of papers, and the writing of “Rich Feedback” is outsourced.

According to “The Case for Rich Feedback,” on the EduMentry site,

One area crying out for attention is the extent of feedback students receive on their ongoing written assignments. Almost universally, assignments come back with a score or letter-grade and a few scrawls that are too brief, general, vague and or otherwise too minimal to make a difference. The student is left with little guidance on what to do differently.

The same page attributes this lack of rich feedback being given to students to a a lack of support that faculty members receive from the university.

But the solutions to that problem are smaller and fewer classes, and faculty learning how to give useful feedback, and students learning how to use that feedback.

I realize this kind of service would best appeal to those running the kinds of huge lecture classes in which “non-virtual” TAs currently do the grading and feedback writing for professors.

But the name “Virtual TA” is kind of a misnomer. They’re not virtual TAs; they’re real TAs you can’t see or visit with, who have no connection with the culture of the college or its student body.

Leaving aside the large lecture course, which I think is not a good idea pedagogically except in certain fields, I fear that this Virtual TA system might be employed in more standard college courses. And if it were, I can’t imagine it being a good idea for the professor, who won’t be reading the papers, and who won’t be learning all s/he can about the students, who themselves won’t be benefitting from the professor’s own feedback.

In response to that concern, EduMetry says (on their FAQ):

Wouldn’t you be interfering in the instructor-student relationship?

We have no direct contact with students. We are at the service of the institution and its faculty members. We provide instructors with a student-ID-scrambling utility that ensures a double-blind grading process. As former academics ourselves, we do everything possible to keep the professor in the loop (and not in the dark, as delegating grading might appear at first blush). In addition to having access to all the individual-student feedback, professors receive a summary report for the class that points out the highlights from the grading of that assignment. This report further ensures that faculty members are aware of how students did, adjust their teaching (content, pace, style, design of assignments) based on the summary feedback.

Truly rich feedback can be provided only by those who know both the subject, the students, and their work, as intimately as is possible given the circumstances.

It’s not the interference between teacher and student, mentioned by EduMentry above, that I fear.
Instead, it’s the gap between teacher and student that this creates.

I guess you might say I “mind the gap.”

Technology-across-the-curriculum, or “Why can’t Johnny sort his email into appropriate folders?”

I read with interest today’s report on Inside Higher Ed that the Educational Testing Service has a test of Information of Communication and Technology Literacy. Here’s a web demo.
Inside Higher Ed reports that Cal State is contemplating requiring the test of its students:

The California State University system … [is] putting the finishing touches on a test — developed in conjunction with Educational Testing Service — that they believe accurately gauges students’ technological literacy. And they are contemplating making the test a requirement that students would have to pass to move on to higher level courses, much like they do now for writing proficiency.

“People are good at learning technologies, but they are not so good at applying them,” said Barbara O’Connor, a professor of communications at California State University at Sacramento. O’Connor has become a strong advocate for increasing technological literacy.

My first instinct was to cringe at the idea that Cal State would make “the test a requirement that students would have to pass to move on to higher level courses, much like they do now for writing proficiency.” I hope that CUNY would not turn the technology skills test into a stick with which to beat students.
But I am also a strong advocate for increasing technological literacies– and I know all of you are too.

Don’t we all, already discover which skills are lacking and help students to acquire them? The first day of many classes using Blackboard or a blog is often the day students are given instructions to log on and post; if they have trouble, they’re given extra help by the professor or asked to get someone at the computer labs to walk them through. Skill building in the context of the course, with attention paid to which skills are needed and when, seems a no-brainer to me.

There are seven proficiencies tested in the 75 minute test, here are three examples:

Under “Manage” information, activities include:

  • Sorting e-mails into appropriate folders
  • Re-ordering a table to maximize efficiency in two tasks with incompatible requirements
  • Documenting relationships using an organization chart

Under the “Evaluate” header, activities include:

  • Selecting the best database for an information need
  • Determining the sufficiency (or lack) of information in a Web site, given the information need
  • Ranking Web pages in terms of meeting particular criteria
  • Determining the relevance of postings on a Web discussion board

Activities under the heading “Communicate” include:

  • Formatting a word processing document
  • Recasting an e-mail
  • Adapting presentation slides
  • Preparing a text message for a cell phone

Those are mostly really useful things for students to be able to do in some educational or work settings. Some skills are useful for all. (Some not so much.) I don’t think many Baruch students need much help on “preparing a text message for a cell phone,” but that’s another story.
My gut reaction to this is that students learn technological skills by using technological skills. And they all have different proficiencies.

First year composition teachers know that lecturing to a class of students about grammar doesn’t do much. Each student has their own patterns of error: they don’t all have the same skills that need work. You can give some brief targeted lessons about the most common patterns of error, but they have to be brief and targeted.

I think technology skills are similar. If most of the class does not know how to post to the new blog, a brief lesson and a handout with the details for reference, is in order. But I don’t think the skills noted in the brief snippet above (and those in the other 4 areas tested) can be easily and quickly taught except where integrated into content-based courses. We have to continually teach (and test) these skills in courses where they are needed and used.
Sure, we could use some more technology workshops and maybe even a test that helps students decide which of those workshops to attend.
What we can’t do is teach all technology/writing/critical thinking skills at once. Non-context-specific technology education is boring and does not work.

The 6th Annual Symposium: Afternoon Discussions

Remember that Symposium of ours? Below is a summary of the afternoon discussions as reported by participants at each table during the plenary session. Take a look too at the summary of the morning discussions. To watch a video of the plenary, click here and then click “Launch Media” and select “Part V: Afternoon Discussion and Report Back” on the following screen.

The 6th Annual Symposium Afternoon Discussion: Challenges and Proposed Solutions* (By Table)

Table 1: Reported by Ruth-Ellen Simmonds, Executive Director, One Stop Senior Services

Challenge: How can we teach students the kind of flexibility needed to communicate effectively in business?

Recommendation: Create a coaching culture–either through an experience mentor from another division or a team of people with whom they can practice and learn and grow. In terms of some of the activities that might engage students we suggest client-based projects. E.g., a sample activity where students have to learn to sell their ideas.

Table 2: Reported by Jody Rosen, Communication Fellow, Bernard L. Schwartz Communication Institute, Baruch College

Challenge: How do you diagnose yourself as a communicator?

Recommendation: Ask ourselves, our colleagues, and our students the following questions as we communicate:

(1) Am I clear, concise? complete? correct?
(2) What decisions do I need to make in communicating?
(3) How do I get the data?
(4) Is the person that I’m communicating with receptive to my communication?
(5) How can I better engage my audience?
(6) Why should someone take the time to listen to the message that I’m giving?
(7) How is time a factor in communication? If I’m sending an email at 3 a.m. does the time of sending have as much effect on how it’s received as what I say?

After communicating, ask yourself the following questions:

(1) What worked?
(2) Where did I get stuck?
(3) What can I take away from this that can be applied next time?

Table 3: Reported by Deborah Bosley, Director, Center for Writing, Language, and Literacy, University of North Carolina, Charlotte

Challenge: Money to support education. Business should value communication enough to underwrite CAC the same way they underwrite math and science. The College Board did a report showing that over $2 billion is spent by companies to train people in communication skills that they don’t already have out of college.

Recommendation: Seek funding from companies and other private sources for better communication training and curricular support. The BLSCI is a perfect model. How do we replicate it across the country?

Table 4: Reported by Diane DeFilippo, Assistant Vice President of Distribution and Service, AXA Financial, Inc.

Challenge: What are the conditions that make communication more likely to be successful or effective?

Conclusions:
(1) The first thing we decided is that communication is not a one at a time thing. It’s a continuum. Even when you think it’s over, it’s not over.
(2) You have to have communication strategy. And a key component to most strategies is that we are not all the same. I am visual; some are auditory. What works for one does not necessarily work for others.
(3) You need a clearly defined purpose for communicating. You have to know WHY you’re creating a communication.
(4) Your audience is not just someone on the other side. This works whether it’s email or a meeting. Ongoing dialogue is needed. Collaboration is needed. You want to be able to adjust and inspire communication.
(5) After you think you’ve been successful, you need confirmation. Ask if you explained your message clearly. Have people paraphrase. Have them play back the information.

Table 5: Reported by Bob Garland, National Managing Director, Assurance and Advisory Services, Deloitte & Touche LLP

Challenge: Educators must develop the whole person in order to train good communicators.

Recommendations:
(1) Practice, practice, practice. Students should start with small, less formal, low-stakes presentations and build over a four-year academic career to give several much higher stakes presentations. A key thing is overcoming fear. If you can’t overcome that in a low stakes situation, you won’t be able to overcome it. Students also need the opportunity to fail. College is more forgiving than a business environment.
(2) Develop students’ listening skills.
(3) Enhanced, continuous feedback for students from professors as well as outside visitors to the classroom, including career center staff.
(4) More mentoring; for instance, expansion of Baruch’s Executives on Campus program.

Table 6: Reported by Norm Brust, Vice President, Corporate Communications (retired), Contel Corporation

Challenge: Measuring the effectiveness of communication in a timeframe early enough in the process so students and employees can change their course and maximize their effectiveness.

Recommendation: Preview all types of communication. Get some representative member of your audience (who you trust to give honest feedback) to sit with you before your presentation–before you write the memo, before you write the talk–and in effect preview your communication.

Table 7: Reported by Wendy Ryden, Assistant Professor of English, Coordinator of Writing Across the Curriculum, Long Island University, CW Post Campus

Challenge: How can communication be reconfigured as a relationship–a more holistic model that includes visual and ethical dimensions of the situation?

Recommendations:
(1) Teach and practice communication as story telling. Use role-playing for problem solving. Be more attuned to what’s going on visually, how you’re presenting yourself. It is important that there be a meta-cognitive dimension to this. Participants should move from speaker to spectator, asking “Why do I make the choices that I’m making?”
(2) Use PowerPoint in a better, more narrative way.
(3) When we’re teaching communication we’ve been focusing on what the communicator can do. But we can’t forget that the audience also has an obligation. What can we teach audience members to do in a responsible communication? Why do we react the way we do? What obligation do we have to teach the audience to understand what’s trying to be communicated?

Table 8: Reported by Virginia Malone, President, ILM, Inc.

Challenge: What’s the standard for evaluation of communication?

Recommendations:
(1) Peer evaluation, e.g., using a questionnaire.
(2) Self-evaluation: Have students look at a performance review from a company so they can see what employers use to evaluate them. 70% of the evaluation is about communication. Let them see that in their future pay raises will be affected by this.

*Many thanks to Elizabeth Busch for transcribing the discussion and to Tom Harbison for creating this summary.

6th Annual Symposium: Steve Kerr’s Slides

By popular request, here are the slides from Steve Kerr‘s amazing keynote from the Institute’s 6th Annual Symposium on April 28th. You can download the orignial PowerPoint slides here.




As soon as it is ready, we will post a link to a video of Steve’s keynote from Baruch’s Digital Media Library.

The 6th Annual Symposium, Part I

Each year, the Schwartz Communication Institute at Baruch College hosts the Symposium on Communication and Communication-Intensive Instruction which brings together about 100 educators and business people to talk about communication. This year’s Symposium, entitled “What is ‘Effective’?: Assessing Communication in Education and Business,” saw lots of discussion on the way we evaluate communication in academic and business contexts. During the morning roundtables, participants identified salient challenges or problems related to teaching and learning effective communication. Here’s what they came up with:

TABLE 1
The differences in the communication in business and industry v. the communication in the academy is that, in business, one has to be able to think on one’s feet and address an audience who wants to know: “What’s in it for me?” In the academy, students often write or speak to and for a professor and their message is: “This is what I
know.”

Making that transition often means thinking outside the box: but do students understand what defines the box? How can we teach students the kind of flexibility needed to communication effectively in business?

TABLE 2
To make communication more effective, we need to build a pre-assessment tool into the process. As you communicate ask yourself: “How do I find out if this will be effective? Diagnose yourself as you communicate. Is the core message is coming through?

TABLE 3
Learning is a life-long activity and universities do not turn out finished products. How can business and academia create an ongoing partnership to continue education as students move from the classroom into the workplace?

TABLE 4
The challenge is communicating more effectively about what effective communication is.

Two main positions keep coming up in answer to the question, What constitutes effective communication?

“It depends on the context.”
“Everybody should know—we know it when we see it.”

We want to define it. Articulate it. Deal with the variance in standards.

TABLE 5
What does “effective communication” mean and how do we as educators cultivate it?

To make effective communicators—people who are going to be hirable and desirable—we have to cultivate the whole person – to transform the students in fundamental ways: maturity, reliability, skills, knowledge, and creativity.

TABLE 6
Audience is the central concern. We are interested in the various audiences to whom effective communicators have to communicate. Regardless of genre, how does feedback—both overt and covert—effect the communication? How does evaluation enter into the planning process? How can students learn to balance fulfilling of expectations and creative thinking? How can we make sure students have contact with those who can demonstrate effective communication?

TABLE 7
Effective Communication is about reconfiguring the model of what communication is. It’s really about forming relationships. It’s about taking into account all the factors. — How can we foster this idea as opposed to an academic, linear, individualistic model of communication?

TABLE 8
What is the standard of evaluation? If we’re talking about a population that speaks 100 languages, and many more cultures, do we have a standard of evaluation that is appropriate for a diverse population?

To be continued . . .