Thursday, September 06, 2007 • 8:45 AM Comments (7)

Evaluating the Ephemeral

posted by Hayley Wood

Cultural organizations in general and state humanities councils in particular have long struggled with the question of how to evaluate what they do. Is a good head-count at an event sufficient? Is a high level of engagement during a post-performance talk-back adequate proof that there’s thinking going on? How can certain program concepts be packaged for funders so that they will be perceived as compelling and important? Sometimes it seems so difficult to define and measure internal change that one wonders if any attempt at evaluating the effect of a project that aims to present intellectual content to a group is worth the effort.

And yet we persist; grant makers haven’t stopped caring about measurable goals and descriptions of meaningful impact, and staff members at organizations like mine haven’t stopped trying to define success and describe why we know we’re successful—or not.

Increasingly, we aim to define specific changes that our programs effect in participants, and the more that change is described in practical terms the better. Perhaps a concrete example is in order, such as the Massachusetts Foundation for the Humanities’ evaluation efforts for its flagship program, the Clemente Course in the Humanities, for which fellow Public Humanist Jack Cheng is an art history teacher.

The Clemente Course is unique among MFH offerings and has been the most successful of all its programs at attracting donations, probably because it provides a universally recognized gain (college level courses and transferable college credits) for underserved adults in poor Massachusetts communities. Interns from the Center for Public Policy at the University of Massachusetts Amherst designed the longitudinal evaluation study, and MFH pays a consultant to conduct phone interviews with current and former Clemente students, interpret data, and keep up with the changing yearly tasks involved with tracking people’s lives after they graduate from the program. Both qualitative and quantitative information is sought with the study. What MFH is hoping to be able to demonstrate is that a significant percentage of Clemente Course graduates go on to enroll in college and that the positive effects of the Clemente Course are lasting for the students and others in their lives.

Compared to other kinds of programs we launch and support, none of which approaches the depth or expense of the Clemente Course, that one’s easy, and I can attest to the fact that it’s been laborious, time consuming, and expensive to develop a specific evaluation tool that we believe will provide meaningful hard data for our grant proposals seeking outside funding for the program. Increasingly we have learned that what we often think of as the self-evident value of the humanities is lost on those in a position to fund it.

Our organization, like many other cultural organizations and institutions, is up against the idea that public opportunities to deepen one’s engagement with ideas are non-essential frills in a world full of people with intense basic needs.

As a result, we face mounting pressure to think about devising and supporting programs that will yield easy-to-digest results to people in a position to give money. And while it’s certainly not a bad thing to think hard about defining a change (what all evaluation tries to do) that can be measured, certain other aspects of the humanities—say, the pleasure principle—become secondary concerns. I have yet to add “joy quotient” to a logic model table, or ask the question: “On a scale of 1-10, how much happiness did this program inspire in you?” in a survey, or ask our own grant application review committee to rate the likelihood that any given program will promote a sense of well-being and happiness for participants and audience members.

But I really can’t end on a note that dismisses the rigor and legitimacy of evaluation efforts. It is a worthy exercise to ask oneself, “what change do we want to this project to promote?” and “how can we measure that change and present it in a persuasive manner?”

One of the main problems with the scenario I’ve described is that philanthropic organizations with large sums of money to distribute want the kind of data that only expensive, professional evaluation processes can yield. I for one, would welcome a chance to get better at designing evaluation tools, and I’d like to be met with a general acceptance of the premise that efforts to create public learning opportunities and promote wisdom and pleasure are self-evidently valuable and worthy of support.

--Hayley Wood, Program Officer, Massachusetts Foundation for the Humanities

Comments (7)
Post a Comment
I have never heard the importance of evaluation put so persuasively! It's exciting to hear pleasure being talked about as something to value for it's own sake, and perhaps even pleasure as a value, and to hear it talked about in public at all.
Posted by Karen Blazer on 9.6.07 at 9:05
is up against the idea that public opportunities to deepen one?s engagement with ideas are non-essential frills This just makes baby Plato cry.
Posted by Ea on 9.6.07 at 15:58
This is such a well-constructed think piece! Wonderfully made. In addition to "satisfaction," we might also look for "transformed practices." A liberal education transforms a human being. The stuff of public humaities programs is made of the same cloth as liberal education. The Clemente Course changes the habits of thought and action of individuals, right? A family reading program hopes to transform the practices of parents. Every program worth funding hopes to transform and reshape practices of thought. Teacher institutes reshape teaching practice. Good programs do more than "expose" people to something; they shape people's thinking ability and motivation to actions.
Posted by Michael Bouman on 9.7.07 at 8:59
College and universities have the same challenge as humanities councils when it comes to outcomes assessment of humanities learning. The process can be expensive and time consuming as Hayley Wood testifies. She also notes that assessment is essential to convince funders and even boards that programs are worth continuing. My experience as a liberal arts dean, member of a regional accreditation staff, and board member of two state humanities councils leads me to the following: First, it is difficult to establish the goals for humanities assessment. The real challenge is to define what are we expecting? I do think that there are some ?skills? and some content goals one could set. However, I also believe that the humanities properly presented stimulate experiences which are valued in themselves because of the way they open our eyes, warm our hearts, give us the joy of which Hayley speaks. My conclusion is that we should always assess a humanities program by measuring the skills and content it provides and by evaluating the degree to which is has, directly or indirectly, delivered a valued human experience. The humanities are both means and ends. Second, we can and should find relatively straight forward ways to measure skills, content, and the intrinsic value of the humanities. Content is always the easiest to measure. If the goal of the program is a better understanding of the Constitution or an ethical theory or cycles in history, we can ?test? audiences following performances with short content questions. We can find out if they ?understand.? However, it is more difficult to define and then evaluate the skills which the humanities are expected to teach. I once proposed that the humanities were a valued part of any liberal arts education because they impart empathy, elegancy in the use of language, and ethical norms. If this list or one like it can be agreed upon, it may be possible to evaluate the usefulness of a particular program by giving audiences or a sample of them a practical test. Audiences could be asked to analyze ?cases? in such a way to reveal their empathy, their facility at expression, and their appreciation of ethical standards. In other words, examiners would apply a rubric to short essays provided by audiences, examining the degree of sophistication audience members bring to the examination of real life experiences following a humanities program. Such case-based examinations are being employed to evaluate general education programs at universities and have shown considerable promise. The most difficult to assess is the most important, the degree to which a program has ?moved? audiences and presented a fresh experience to them which they need time to unpack. How do you measure epiphanies? I doubt that this can be done directly. It is hard to be sure that people themselves can honestly testify to such a happening immediately after it occurs. However, I would suggest that epiphanies have consequences and the measurement of these consequences may be easier to pursue. If an audience member has had a humanities moment of significance, one might expect that the reflective habits of the person will change to include new literatures (books, articles, media) related to the moment they experienced. In other words, shouldn?t we be expecting that humanities council events will stimulate audience members to change the amount and type of reading, viewing, and attending they do? I recall that Billy Graham measured his success by the number of ?converts? that attended local churches of any denomination after his tent meetings. Could we measure the degree of movement an audience has experienced by the number and type of additional humanities experiences they seek subsequently? I am not sure that I have a found the best measure here, but I am convinced that we should never give up the insistence that humanities be measured by the experiences they create as well as the content and skills they impart. They should to be understood as more than utilitarian ?tools? for objectives beyond themselves. They are singular joys to be sure, and we should get ever better evaluating when they are, in fact, occurring.
Posted by Bob Benedetti on 9.7.07 at 19:37
Something that I allude to in my piece but that bears repeating is the Clemente Course in the Humanities is easier than other kinds of humanities programs--one-off events for example--because the course requires a nine month commitment from its students and the impact is profound on individuals. Another program of ours that we've successfully assessed as valuable in language that most people can relate to is our Literature and Medicine program (the model program was created by the Maine Humanities Council), which is another comparatively long-term commitment for participants, who meet monthly with hospital colleagues of all stripes to discuss literature about the experience of illness and the ethic of care. Defining meaningful, measurable and realistic goals is, indeed, a big challenge. Implementing the systems required for capturing data (and I think that thought transformation would be very diffcult to define and measure, never mind the logistical concerns) is, on top of the original intellectual exercise, difficult.

I wonder if one could actually pay a group of people to attend and provide feed back for a year's worth of humanities council projects. Would the results provide any reason to believe that these people had experienced transformation or epiphanies?

It may be that the most worthy humanities programs promote internal changes besides the basic acquisition of knowledge. The more one presses this idea, the closer one comes to prescribing specific ethics. I'm not opposed to that, but it's sensitive to describe what we see as right behavior, and right thinking. And I DO think that we are in game of defining what we see as "right" and "good."
Posted by Hayley on 9.8.07 at 9:16
It may be worth noting that even in the Clemente Course we've found that it's good to put a dollar value on the program so that the students don't perceive the class as "free." Why? Because generally in life you get what you pay for. So to sell the program to its beneficiaries we have to frame ourselves as in terms of pragmatic and financial incentives and not just because it's enjoyable and rewarding.
Posted by Jack Cheng on 9.10.07 at 10:36
The fact that the students benefit from knowing the dollar value of their courses and credits is interesting. I see it as relevant information for them to be aware of. Most, no doubt, view college as expensive, most likely prohibitively so. It's sensible to emphasize that enrollment in the Clemente Course is a financially valuable opportunity as well as an intellectually valuable one.
Posted by Hayley on 9.10.07 at 13:21
Comment:

Name:

Password:

New User/Guest?

Find it Here:
keyword:
search type:
search in:

« Previous   |   Next »
« Most Recent Post
« Permalink
Print Email RSS feed

Photo Galleries
Archives
SEPTEMBER 2007
S M T W T F S
            1
2 3 4 5 6 7 8
9 10 11 12 13 14 15
16 17 18 19 20 21 22
23 24 25 26 27 28 29
30            
Philosophy in Pop
The Rest of the World
Links
Copyright © 2014 by The Valley Advocate.