Prepared by EFFECTIVE CHANGE Pty Ltdfor Arts Victoria; Darebin City Council; City of Whittlesea and VicHealth
EvaluatingCommunityArts
CommunityWellBeing
AN EVALUATIO N GUI DE FOR COMMUNITY ARTS PRACTITIONERS
Thisguideisavailablefromthefollowingwebsites:
ArtsVictoria:
VicHealth:
EffectiveChange:
DarebinCityCouncil: :
EvaluatingCommunityArtsandCommunityWell-Being
AnEvaluationGuideforCommunityArtsPractitioners
Evaluationresearch;Evaluationindicators;Communityarts;Communitywell-being;Communitydevelopment;Communitycapacitybuilding
Researchedandwrittenby:ClareKeating
EffectiveChangePtyLtd(03)93881661
©StateofVictoria2002
Thisworkiscopyright.Itmaybereproducedinwholeorpartforitsintendeduseincommunityartsprojects,forstudyortrainingpurposes,subjecttotheinclusionofanacknowledgementofthesource.Commercial
usageorsaleisnotpermitted.
Acknowledgements
Theauthor wishestoacknowledgethesupportoftheorganisationswhichcommissionedthisworkandthecommitmentofthestafffrom thoseorganisationstothisguide:
•MariaKatsonisandJudyMorton(ArtsVictoria)
•SusanBall(VicHealth)
•MarkWilkinson(DarebinCityCouncil)
•JudyTurner,KathMelbourneandSarahPoole(CityofWhittlesea)
Ialsowishtoacknowledge andthankofthebroadrangeofcommunityarts practitioners,
managersandparticipantswhowereconsultedin thedevelopmentstages;commentedondraftsoractuallypilotedtheguidewiththeirowncommunityartsprojects.Yourcontributionshavebeeninvaluable.
A MESSAGEFROM THE MINISTER FOR THEARTS
Theartsengage,provoke,amuseandexciteusbut,aboveall,theartsreflectandshapeoursenseofcommunityidentityandhelpsbuildsocialcohesion.
AsMinister fortheArts,Iampassionateaboutspreadingthewordthatparticipatingintheartsandculturalactivitiesisarewardingandcreativeexperience,onbothanindividualandcommunitylevel.
Weneedabalancedunderstandingoftheroleandworthoftheartsinoursociety–onethatembracestheiraesthetic,cultural,economicandsocialvalues.
Oneofthechallengesfacinggovernmentsandtheartssectorisfindingrigorousresearchtodemonstratethepositivesocialimpactofartsinvolvement,ratherthanrelyingonanecdotalevidence.
WiththereleaseofEvaluatingCommunityArtsCommunityWellBeing,wehaveauser-friendlytoolthatallowscommunityartspractitionerstodocumentandassessthesocialoutcomesoftheirprojectsandprogramscomprehensivelyandsystematically.
ThewidespreaduseofthisEvaluationGuidewillincreaseknowledgeandunderstandingofthebenefitsofartsinvolvement,leadingtomore,andmoreeffective,artsprogramsinVictoria’s
diversecommunities.
Istronglycommendtheguide.
TheHonMaryDelahuntyMPMinisterfortheArts
FOREWORD
Iampleasedtocommendthisevaluationguidetoyou. Evaluationisimportant,notonlyforprovingwhatyoudoiseffectivebutalsoforimprovingprogramsandcontributingtobetteroutcomes.
VicHealthisanactivesupporterofcommunityartsthrougharangeoffundingschemes
developedwithinourArtsforHealthProgram. Weregularlyobserveimportanthealthandcommunitywell-beingoutcomesthroughcommunityartsprojects,however,wefindthat
organisationshavelimitedcapacitytoprovideevidenceofthisbeyondtheanecdotal.Thisinturnlimitsallofourabilitytoshowcasesuccesses;tolearnfrom failures;andtoreflectontherangeofachievements.
Iunderstandthatmanypeoplefaceevaluationwithsometrepidation. Itishopedthisevaluationguidewilleasethatfearandbeawelcomeresourcewhichprovidestheframeworkandthenecessarytools. Ithasbeendevelopedincloseconsultationwiththesector,respondingtotheneedforbothashort‘Express’guideandamorecomprehensiveversionwhichhasthecapacitytobeusedasis,oradaptedtoyourspecificneeds. Wehaveensuredthattheresourceis
widelydistributedandreadilyavailable–thedocumentisonthewebsitesofthefiveprojectpartners.
Wethankourprojectpartners,ArtsVictoria,DarebinCityCouncilandtheCityofWhittleseafortheircommitmenttotheproject.We thankClareKeatingofEffectiveChangeforproducingsuchahighqualitydocument.Wehopeithelpsyoutoshowthedifferencecommunityartscanmaketohealth.
DrRobMoodie
ChiefExecutiveOfficer
CONTENT S
AMessage FromTheMinisterForTheArts...... ii
Foreword...... i
Preface...... v
Introduction...... v
EvaluationOverview...... x
Stage1:PrepareForTheEvaluation...... 1
Stage2:PlanTheEvaluation...... 2
Stage3:DetermineEvaluationIndicators...... 3
Stage4:CollectData...... 4
Stage5:AnalyseData...... 5
Stage6:ReportAndImprove...... 6
Stage1:Preparing ForAnEvaluation...... 7
Stage2:PlanningTheEvaluation...... 10
Stage3:DeterminingEvaluationIndicators...... 12
Stage4:CollectingThe Data...... 15
Stage5:AnalysingTheData...... 22
Stage6:ReportingOnThe DataAnd Improving On CurrentPractice...... 24
FurtherReading...... 26
TOOLS:
SampleTool1.1:Setting ProjectAims...... 29
SampleTool2.1:PlanningThe Evaluation...... 30
SampleTool2.2:IndicativeEvaluationTimeline...... 31
SampleTool3.1:DeterminingEvaluationIndicators...... 32
SampleTool 3.2–3.4:SampleEvaluationIndicators...... 33
SampleTool4.1:ParticipantEvaluationSurvey...... 36
SampleTool4.2:GenericInterviewStructure/FocusGroupOutline...... 37
SampleTool4.3:ProjectJournal Guidelines...... 38
SampleTool5.1:Analysing EvaluationDataTemplate...... 39
SampleTool6.1:DraftEvaluationReportStructure...... 40
SampleTool6.2:ImprovingProjectAimsAndOutcomes...... 42
PREF ACE
COMMUNI T Y ARTS PRACT ITIO NERS…
Wouldyouliketohaveyournextfundingapplicationreadybeforeyou’vefinishedyourcurrentproject?
•Wouldyouliketobeabletoplanfutureprojects?
•Wouldyouliketodevelopstrongerpartnershipsinthecommunity?Evaluationisaprocesswhichcanhelpinalloftheabove.
•Haveyoueveraskedparticipantswhethertheyenjoyedacommunityartsproject?
•Haveyoueveraskedapartnerinacommunityartsprojectwhattheythoughtoftheexperience?
•Haveyoueveraskedthepublicwhattheythoughtofaperformanceorpiece?
Ifyouansweredyes,thenyouhaveundertakenatleastabasiclevelofevaluation.Let’sfaceit,mostcommunityartsworkersdothisallthetime–it’sanaturalpracticeformostpeopleinvolvedinthe
communityarts.It’seasy,it’sfun,it’sinterestinganditkeepsyouinvolvedandstrivingfornewgoalsandnew projects.
•But,doyouthensitdownandconsiderthefeedback?
•Doyouthinkaboutwhatyouneedtodobeforecommencinganotherproject?
Yes?Then,youareundertakingsomelevelofcontinuousimprovement,throughevaluation.Thismightbeabitharderandhavesomepitfalls.Often,thereisnoincentivetodothisandnotimetodoitsystematically.Youprobablyhavealreadyexperiencedsomedirectbenefitsfromthisapproachthough,andyou’rekeentokeepgoing.But…
•Doyouhaveaplanreadynowtoevaluateyournextproject?
•Haveyouthoughtaboutwhotospeakto?When?Whatwillyouaskthem?How?
•Howwouldyourecogniseandpreventaloomingdisaster?
•Ifyoudocumentedtheprojectoutcomes,howwouldyoureport,andtowhom?
Ifyoudon’thaveaplanreadyNOWforevaluatingyournextcommunityartsproject,thisguideisforyou.Recognisingtimeconstraintsinthecommunityarts,theguideissimple,straightforwardandreadytouse.
Theguideisorganisedaroundthestagesofevaluation.Atthebackoftheguidearetoolsforeachstageoftheevaluationwhichcanbeusedastheyare,oradapted.Theevaluationmodelrecognisestheimportanceoftheprocessaswellastheoutcomeincommunityartsandhelpsyoutoidentifyachievementsineach
area.Themodelisdesignedasaresource,withallthetoolsprovidedtohelpyouevaluate,reportand
improve.Hopefully,withtheevaluationplannedoutforyou,communityartspractitionerswillhavesupporttoevaluate,reflectandimprove.Theguideisintwoformats–asummaryversioncalledtheExpressGuide
andacomprehensiveversion,whichincludestheExpressGuide.
Bothversionsoftheguideareavailablefromanyofthefollowingwebsites:ArtsVictoria
VicHealth
EffectiveChange
DarebinCityCouncil ebin.vic.gov.au/arts/publications .html
CityofWhittlesea
Itisrecommendedthatyouaccessbothversions.Goodluck!
INT RODUCT ION
PURPO SE OF THE EVALUATION GUIDE
Thepurposeofthisguideistoprovidearesourceforcommunityartspractitionerstousewhentheyevaluatecommunityartsprojects.Itisdesignedtobeasflexibleaspossible,sothatitcanbeusedbythewidevarietyoforganisationsinvolvedinthecommunityartsandsothatitcancovertherangeofcommunityartsprojects.Theguideprovidesacomprehensivebasefor
undertakingevaluations.Theprocessoutlinedintheguidecanbefollowedthroughsequentially.
Alternatively,theprocess,toolsandsuggestionscanbeamendedandrefinedtosuitthe
specificneedsofvariousprojectsandorganisations.Theguidealsoaimstohelpbuildabodyofevidenceabouttheworkofthecommunityartssector,byusingaconsistentandstructured
researchframework.
B ACK GR OU ND
Aconsortiumoforganisationshasbeeninvolvedinthedevelopmentoftheguide.TheprojectwasfundedbyArtsVictoria;DarebinCityCouncilandtheCityofWhittlesea.ArtsVictoriaandVicHealthhavefundedthefinalproductionoftheguideandmakingthemavailableon-line.
Togethertheprojectpartnersrecognisedthat:
•withinthecommunity,increasingvalueisbeingplacedoncommunityactivitieswhichhavepositiveimpactsonthecommunity;
•thereisagrowingrecognitionofthesocialandcommunitywell-beingoutcomesofcommunity
artsprojects,yet
•thereisagapintheresourcesavailabletocommunityartspractitionerstoassistthemin
identifying;collatingandreportingonthepositiveoutcomesachievedthroughcommunityartsprojects.
Thepurposeofthisguideisthereforetoaddressthatgapandtoserveasaresourceforcommunityartspractitioners.
TheguidehasbeenpreparedanddevelopedbyEffectiveChangePtyLtd,aMelbourne-basedconsultancycompany.Inresearchingtheguide,relevantliteraturewasconsultedandarangeofkeyinformantswereinterviewed.
WHO I S THE G UIDE FO R?
Theguideisforcommunityartspractitioners,ratherthanspecificorganisations,ascommunityartsprojectscanbelocatedacrossavarietyoforganisations.
HOW SHOULD I T BE USED?
Theguideessentially‘talks’toagenericcommunityartsprojectworkerandassumesthatthispersonwilldrivetheevaluation.However,itcanbeusedinarangeofways.Forexample,allparticipantsinacommunityartsprojectcouldsharerolesinundertakinganevaluation.Oritcouldbeusedanexternalstudentorconsultanttoundertakeanindependentevaluation.
COMMUN ITY ARTS – N OT JUST COMMUN ITY, NOT JUST ARTS
Thisguiderecognisestheuniqueplaceofcommunityartswithincommunityandculturalactivity.
Casestoriesaboundofcommunityartsprojectsproducingstunningandlastingcreative
achievements.Therearealsonumerousexampleswhereextraordinarylifechangesoccurredforparticipantswhilethecreativeoutcomeswerelessmemorable.Othertimes,communityarts
projectshavecontributedenormouslytotheirlocalcommunity,througharangeofstrategies,suchasengagingdisenfranchisedmembersofthecommunity,orproducingvaluableresourcesforaparticularcommunity.Sometimesexceptionalcommunityartsprojectsproduce
extraordinaryresultsacrossallareas.Othertimes,peoplejustfeelgoodbybeinginvolvedandtakingrisks.Thisguidehasbeendevelopedtoworkacrossallareas–fromthecreativeoutput,tothecreativejourney,totheindividualsandcommunitiesinvolved.Thismodelofevaluation
willallowyoutoidentify theareasofachievementandsuccess.It worksevenwhen,inconventionalterms,theprojectmaynotappear tobethatsuccessful.Itwillalsoallowyoutorecogniseextraordinaryachievementsacrossthisrangeof factorswhentheydooccur.
Thediagrambelowshowsthekeyelementstobeevaluated,usingthismodelofevaluation.
Diagram1:KeyElementsofCommunityArtsProjectstobeevaluated
participants / project/organisation / community
process / howparticipantsareinvolvedinthe
project / howtheprojectismanaged / howthecommunityisinvolvedinthe
project / process
impact / whathappensto,orfor,participantsthrough theproject / whathappensasresultoftheproject / whathappenstoorforthecommunitythrough theproject / impact
outcome / whathappensto,orfor,participantsinthelongtermasaresult
oftheproject / whathappensinthelongterm,asaresultoftheproject / whathappensto,orfor,thecommunityinthelongtermasaresultoftheproject / outcome
participants / project/
organisation / community
Rememberthatyouprobablywon’tbeabletoconsideralltheabovefactors,allofthetime.Thisguide providesacomprehensiveapproach–itisuptoyoutoselectthepartsthatarerelevant
toyourproject,yourorganisationandyourresources. Youmayonlyhavetheresourcestofocusononekeyaspectoftheproject.Mostcommonly,youwillfocusontheimpactoftheproject-Whathappenedasaresultofthisproject?Positiveanswerstothisquestioninparticularyieldsusefulinformationforpromotingtheproject;promotingtheorganisation;thattheparticipants canuse;thatcanbeusedinfundingapplicationsorprogramexpansionapplicationsandsoon.Andwhenyouwanttoevaluateaprojectingreaterdepth–whenyou’reclearthatyouhavetheresources;there’sagoodcaseforevaluatingtheprojectandtheorganisationsupports the
evaluation-theguide’ssimpleframeworkandreadyresourceswillenableyoutocollectthe
informationandhelpyouarticulatetheelementsthatgototheheartofcommunityartsprojects,butaretoooftenleftunsaid.
STRUCTURE O F THE GUI DE
Theguideisstructuredchronologically.Itguidestheevaluatorthroughthestagesofevaluationfrompreparingthegroundthroughtoreportingontheevaluation.Thesixstagesare:
•Stage1:Prepareforanevaluation
•Stage2:Plantheevaluation
•Stage3:Determineevaluationindicators
•Stage4:Collectthedata
•Stage5:Analysethedata
•Stage6:Preparetheevaluationreportandimproveoncurrentpractice
Eachstageispresentedasaworksheetforyoutocompleteand:
•takesyouthroughthedecisionsyouneed tomake
•askstriggerquestionstofocusyourthoughts
•highlightstheissuesyouarelikelytoencounter
•providesachecklist
•refersyoutotoolsyoucanuseandadaptifyouwish
•refersyoutothefulltextforfurtherdetail.
TheOverviewdiagramonthenextpageisasummaryofthecompleteprocess.Inthecomprehensiveguide,eachStagecontains:
•anintroduction;
•discussionontherationalebehindtheparticularstepoftheprocess;
•discussiononresearchmethodsandissues;
•practicalissues–outliningthetaskstobedone,and
•refersyoutothetoolstouse.
Therearesampletoolsattheendoftheguide,rangingfromsamplesurveystoadraft
evaluationreportoutlinewhichcanbeadaptedforyourownproject.Thetoolsarenumberedtocorrespondwiththerelevantstageoftheprocess.
EVALUAT IONOVERVIEW
STAG E 1
PreparefortheEvaluation
STAG E 2
PlantheEvaluation
STAG E 3
DetermineEvaluationIndicators
STAG E 4
CollectData
STAG E 5
AnalyseData
STAG E 6
ReportandImprove
STAGE1:PREPAREFORTHEEVALUAT ION
SETProjectaimsandobjectives
Whyarewe doingthisproject? Whatareourshort,mediumandlongtermaims?
Whataretheaimsforparticipants?
Whatareourorganisation’saims?
Defineyour‘community’targetforthisproject–thelocalcommunity?aparticulargroupinthecommunity–youngpeople?men?women?
Whataretheaimsforourcommunitytargetgroup?
Howwillwedotheproject?
I SSU E SLookatyouranswers–
Whatdotheymeanfortheevaluation?Haveyouplannedtheprojectwell?
Doyouhaveenoughresources(peopleincluded!) tomanagetheproject?Howimportantwilltheevaluationbe?
Investingabout10%ofaprojectbudgetonanevaluationisacommonpractice.
Whatsortofinvestmentcanyourorganisationrealisticallymake?
STAG E 1
CH ECK LI ST
MORE
INFORMATION?
Haveyou:definedaimsandobjectives fortheproject?D
consideredaimsandobjectives forparticipants?D
theproject/organisation?D
andthelocalcommunity?D
consideredtheprocess(howyouwilldotheproject);
theimpact(whatyou wouldliketohappeninthemediumterm)D
andtheoutcome(whatyouwouldliketohappeninthelongterm)?
Refer to:Stage1:Preparingforanevaluationin thefulltextversion
SampleTool1.1:Settingproject aims
[
STAGE2:PLANTHEEVALUATION
DEFI N Ethepurposeoftheevaluation
Whyarewe evaluatingthis project?Doweusuallyevaluateourprojects?Willthisevaluationbeanydifferent?How?Why?
DEFI N Etheaudiencefor theevaluation
Whowillwereportto?(afundingbody?projectpartners?thecommunity?ourboardofmanagement?)Dowehaveaninternalorexternalfocus?orboth?
Howcanweusetheevaluationresults?(Consider:futurefunding
applications;improvingpractice;keepingthisprojectontrack;feedbacktoparticipants…)
DETERMI NEtheprocessforevaluation
Howwillwemanagetheevaluation?Whatsuitablestructuresexist?Doweneedacommittee/group–oranindividual?
Willweneedaninternalorexternalevaluator? Whoisavailable;ableandcapableofmanagingtheevaluation?Are thereanyconflicts?
DETERMI NEtheresourcesavailablefortheevaluation
Whatresources(time;money;materials;equipment;records)can we
allocatetotheevaluation?Canweafford/justifyanexternalevaluator?
I SSU E SLook atyouranswers–
Areyouclearaboutwhyyouwantto dotheevaluation?
Areyouclearabouthowmuchtimeandeffortcanbeputintotheevaluation?Remember,ifyoureflectonyourpracticenow,youalreadyevaluate!
Whatarethepotentialbenefitsofanevaluation?e.g.anewfundingapplication,leading tomoremoney,moretime,morepeople?
Apartfrom money,whatarethepotential coststo consider?e.g.politicalissues?Canyouwriteoneevaluationreportandmodifyitfor variousaudiences?
Howcanyoubecreativewithresources?e.g.Aretherelocaluniversitystudentswilling/appropriate/able totakeontheevaluationfortheircoursework?
Howcanyouintegratestandardpractices?e.g.projectphotos?projectjournals?
STAG E 2:
CH ECK LI ST
NEED MORE
INFORMATION?
Haveyou:clarifiedthereasonsforundertakingtheproject?D
decidedhowtheevaluationwillbemanaged?D
decidedonthelevelofresourcestobeallocatedtotheevaluation?D
Refer toStage2:Planninganevaluationin thefulltextversion
SampleTools 2.1:Planningtheevaluation
2.2:Indicativeevaluationtimeline
STAGE 3: DETERMINE EVALUATION INDICATORS
DETERMI NEKeyevaluationindicators
Broadly,whatwilltellyouthattheaimsandobjectivesoftheprojecthavebeenachieved?(e.g.participantswererecruited fromthetargetgroup;
participantswereinvolvedinthecreativeprocess;weaimedto developaplayandwereachedplayreadingstage…)
Usingthemodelofevaluationinthis guide, itisrecommendedthatyoubreakdowntheprojectintokeyparts.So,firstlylistthestakeholders
involved–e.g.:
participants / organisation / target /communityThen,breaktheprojectdownintoits partsandhowfaryouare‘thinking’
intothefutureandlistbelow.Forexample,processisparticularlyimportantforcommunityartsprojects.Doyoualsohavemediumorlongtermaims?
Then,thinkaboutwhatyouhopetoachieveforeachgroupofstakeholders
–bothintermsofdoingtheproject,andas aresultoftheprojectandputthetwopartstogether inagrid.Keepthissimple–forexample:
I SSU E SLookatyouranswers–
Areyouclearaboutwhatwilltellyouyouhaveachievedyouraims /doneagoodjob? Theseareyourevaluationindicators.
Evaluationindicatorsareatool–notanaim!Youraimistodothecommunityartsprojectwell–challenging;mentoring;supporting;skillingparticipantsin acreativeprocessthattheyenjoyandthattheycanbeproudof!
Effectiveevaluationindicatorsareatooltokeepyouontrack.Usethemtohelpyouarticulatetheachievementsoftheproject.
STAG E 3:
CH ECK LI STHave you:decidedonevaluationindicatorsfortheproject?D
NEED MO R EINFORMATION?
Referto Stage3:Determiningevaluationindicatorsinthefulltextversion(recommended)
SampleTools3.1:Determiningevaluationindicators
3.2–3.4:Sampleevaluationindicators
STAGE4:COLLECTDAT A
DESIG Ndatacollection methods
Whatarethekeyquestionstobeansweredthroughtheevaluation?Tip:turnyourevaluationindicatorsintoquestions.
Whatdata collection methodswillyouuse? (e.g.focus groupdiscussions;written surveys;structured interviews).Matchmethodstokey researchquestions;availabletime;resourcesandtothestakeholders involved.Thinkcreatively e.g.projectjournals;surveyswithpictograms;videos;photos
Whenwillyou collectdata?throughoutthe project?attheend?monthsafter
theprojectfinished? Listbelowhowoftenyou willcollectdata,fromwhom.
DESIG Ndata collectiontools
Whatdatacollectiontools doyouneedtodesign?e.g.surveyusing
pictograms;focusgroupoutline.Tip:designeacharoundyourkeyresearchquestions.
COLLECTevaluation data
Whatexistingdatacanbeusedintheevaluation?e.g.originalfundingapplication;waiting lists;projectarchives…
Whatissues doyouneedtoconsider incollectingthedata?e.g.research
ethics;keepingpeopleinformed;planningbestuseofeverybody’stime;storageofdata
I SSU E SLook atyouranswers–
The keytoefficientdatacollection isfocussed keyresearchquestions–limityourselfto aboutten.Whenthisbasicframeworkisright,designingsurveysorfocusgroupsto suitdifferentgroupsofpeopleisstraightforward.Don’ttrytocollecttoomuchinformation–makeitmanageable.
Considertheethicsoftheresearchbeforeaskingquestions.
Thinkaboutcollectinginformationefficientlyasyougo-itmayonlytake15
minsextraonrehearsalormeetingsfora discussionorto completeasurvey.
STAG E 4:
CH ECK LI ST
NEED MO R EINFORMATION?
Haveyou:consideredtheethicsoftheresearch?Didentifiedtheexistingrecordsandmaterialtobecollected? Didentifiedthenewinformationtobecollected? D
decidedonyourkeyresearchquestions?Ddecidedonthemostappropriateresearch methodsfortheevaluation? Dplannedhowoftenpeoplewillbesurveyed/interviewedetc? D
developedappropriatecommunicationstrategiesfortheproject?D
RefertoStage4: Collectingthedatain thefulltextversion
SampleTools4.1:Participantevaluationsurvey
4.2:Genericinterviewstructure/focusgroupoutline
4.3:Projectjournalguidelines
STAGE5:ANALYSEDAT A
ANALYSEevaluation data
Howcanyoureducethedataintomanageablechunks?Readthroughthe
dataandidentifythekeythemescomingthrough;usetablesandgraphsforquantitativeresponses
HowdoIcheckthedataagainsttheevaluationindicators?Testthedataagainsttheevaluationindicatorse.g.Iftheevaluationindicatoris:
Participantswererecruitedfromthetargetgroup–analysethedatatoseeifthisisthecase.
Whoshouldthedatabediscussedwith?Considerconfidentialityorothersensitivities.
Whatarethekeyfindingsatthisstage?
Whoshouldwereportdraftfindingsto?
I SSU E SLook atyouranswers–
Are youclearabouthowto reducethe dataintomeaningfulchunks?
Are youclearaboutwhoisinvolved incollating;reducing andanalysingdata?Considerconfidentialityissueswhenanalysingdata.
Considerwhetheryouwill beproducinganinternal orpublicdocument.
Ifappropriate,itisoften usefulto discussdraftfindingswitharangeofpeople,preferablyin agroup.Youcantesttheclarityofthereportwiththesepeople.Whatwillyoudoifthefindingsareeithernotverygoodormediocre?Howwillthisimpactonthe organisationoronindividuals?
STAG E 5:
CH ECK LI ST
NEED MO R EINFORMATION?
Haveyou:
collatedthedatayoucollectedintomanageableandlogicalchunks?D
checkedthedataagainsttheevaluationindicators?D
discussedthedatawithsomerelevantpeople?D
developeddraftkeyfindings?Dpresentedordiscussedthekey findingswithappropriatepeople? DRefer toStage5:Analysingthedatain thefulltextversion
SampleTool3.2–3.4:Sampleevaluationindicators
5.1Analysingevaluationdatatemplate
STAGE6:REPORTANDIMPROVE
R EPO RTevaluationfindings
Whatdidyoufindinrelationtotheprocess?(forparticipants;fortheproject,ortheorganisation;forthecommunity)
Whatdidyoufindinrelationtotheimpact oftheproject?(forparticipants;fortheproject,ortheorganisation;forthecommunity)
Whatdidyoufindinrelationtotheoutcomesoftheproject?(for
participants;fortheproject,ortheorganisation;forthecommunity)
I DENTI F Yprojectoutcomes
Whatdidweachieveinrelationtothisproject?Thinkaboutachievementsfromtherangeofstakeholderperspectives.
Whodoweneedtotellabouttheprojectoutcomesandachievements?
Thinkaboutparticipants;fundingbodies;thecommunity;partnershiporganisations
potentialimprovements
Whatdowe needto improve?
I SSU E SLook atyouranswers–
Afteranevaluation,youmayneed totake care thattheinformationisinterpretedappropriatelybyall–somestaffmayfeeltheirworkhasbeenhighlightednegatively.Ensurethatyou usethe evaluationas anopportunitytoturnthesituation aroundto ensureeverybodylearnsfromtheexperienceandimprovementsareimplemented.
Iftheevaluationhashighlightedareasofseriousconcern,e.g.underperformance;inappropriateperformanceetc–ensurethattheissuesarediscussedattheappropriatemanagementlevel oftheorganisationand yourdecisionsandstrategies are carefullythoughtthroughanddocumented.
Youalsoneed toplanhowyouimproveareasthatwereidentifiedasrequiring
improvement.andevento startthinkingaboutyournextevaluation!
STAG E 6:
CH ECK LI ST
NEED MO R EINFORMATION?
Haveyou:developedadraftevaluationreport?D
answeredthekeyresearchquestionsintheevaluationreport?D
collectedfeedbackonthedraft?Dfinalisedthereportandpreparedanexecutivesummary? Ddistributedtheexecutivesummaryto theappropriateparties? Dlinkedtheevaluationresultsintoanimprovementprocess? D
RefertoStage6:Reportingonthedataandimprovingcurrentpractice
inthefull textversion
SampleTools6.1:Sampleevaluationreport
6.2:Improvingcurrentpractice
STAGE1:PREPARINGFORANEVALUAT ION
INTRODUCTIO N
Thissectionisaboutpreparingthegroundforanevaluation.Itconcentratesonhowtosetaimsandobjectivesforthecommunityartsproject,andwhyyouneedtodothisbeforeattemptingtoevaluatetheproject.
WHY EV ALUATE I N THE FIRST PLACE?
Thereisenormousvaluetobegainedbycommunityartspractitionersthroughevaluatingcommunityartsprojects.Anevaluationprovidestheopportunityforallparticipantstomeasureandcomparewheretheywerebeforetheprojectstarted,andwheretheyreachedbythe
project’send.Evaluationresultsalsoprovidevaluableinformationforprojectstaff,includingcreativestaff,volunteers,mentorsandsoon.
Evaluationinformationcanbeusefulacrossarangeofareas.Forexample,aprojectevaluationcanhelpfeedintotheorganisation’sownplanningandevaluationprocesses.Itcanprovide
usefulinformationforfuturefundingapplications.Itcanprovideausefulrecordformeasuringprogresswithcommunityartsprojects,andhelpingtodevelopandextendtheworkofthe
organisation.Theinformationcanhelpshapefuturepromotionandrecruitmentstrategiesfornewprojects.
Onabroaderscale,whencommunityartsprojectsevaluateanddocumenttheirachievements,thecommunityartssectorbeginstoaccumulateabodyofevidencethatprovidesmeaningful
informationaboutthecontributionofthesectortothewidercommunity.Ontheotherhand,iftheoutcomesofcommunityartsprojectsarenotdocumentedwithrigourandconsistency,the
sectorisleftwithoutarecordofitshistoryorachievements.Inaddition,projectevaluationscontributetothecollectivepracticewisdomofthesector.Evaluationresultscanprovide
valuablelessonsandinsightsforpractitionersinotherplaces,neworganisationsandsoon.Buildingsector-wideinformationalsosupportsopportunitiesforcross-sectorwork.Themorethatisknownaboutcommunityartsanditswork, thegreater theopportunitiestoform
partnershipsandlinkswith,forexample,theyouthsector;thehealthsector;agedcaresectorandsoon.
Oneofthekeyaimsofthismanualistothereforesupporttheundertakingofevaluationsinthecommunityartssector,usingaconsistentandstructuredresearchframework.Thisinitselfwillcontributetothesectordevelopingabodyofevidencewhichcanhelpwiththesector-wideandcross-sectordevelopments.Thetablebelowprovidesasummaryofthebenefitsofcommunityartsprojectsevaluations.
SOME BENE F ITS O F COMMUNI T Y ART S PROJECT EVAL UATIO NS
PRACTICAL ISSUE S:
If you don’t know where you’re going, how will you know when you get there?
Evaluationisnotsomethingtothinkaboutattheendoftheproject. Itneedstobeconsideredatthesametimeasplanningtheproject.Havingprojectaimsandobjectivesisfundamentalto
beinginapositiontoevaluatetheproject.As theheadingsays,ifyoudon’tknowwhereyou’regoing,howwillyouknowwhenyougetthere?Settingaimsfortheprojectprovidesagoalto
worktowards,anditprovidesabenchmarkthatyoucanuseatthecompletionoftheprojecttomeasurewhetheryouachievedwhatyousetouttoachieve.Ifyoudidn’t,youhavethechancetoreflectonthat–
•whydidn’tyouachieveyourgoal?
•didyouachievesomethingunanticipated?
•didyouhaveunrealisticexpectations?
•didsomethingoutsideyourcontrolpreventyou achievingit?
•didyoufailormismanagetheprojectinsomeway?
•andifso,howcanthisbeavoidedinthefuture?
Insettingprojectaimsandobjectives,itisusefultoconsideralltheelementsoftheproject,andfromtheperspectiveofthemainstakeholders.Thetablebelowsummarisesthese
considerations.
participants / project/organisation / community
process / whatdowewanttohappen indoing theproject? / whatdowewanttohappen indoing theproject? / whatdowewanttohappen indoing theproject? / process
impact / whatdowe wanttohappen,asaresultoftheproject? / whatdowe wanttohappen,asaresultoftheproject? / whatdowe wanttohappen,asaresultoftheproject? / impact
outcome / whatdowewanttohappen inthelongterm,asaresultof
theproject / whatdowewanttohappen inthelongterm,asaresultoftheproject / whatdowewanttohappen inthelongterm,asaresultoftheproject / outcome
participants / project/
organisation / community
TOOLS TO USE
Toassistyouinsettingprojectaimsfor:
•theprocess–from theperspectiveofparticipants;theprojectandthecommunity;
•theimpact–from theperspectiveofparticipants;theprojectandthecommunityand
•theoutcome–fromtheperspectiveofparticipants;theprojectandthecommunity.refertoSampletool1.1:Settingprojectaims.
CH ECK LIST:
Haveyou:
•definedaims andobjectives fortheproject?D
•consideredaimsandobjectives forparticipants;theproject/organisationandtheD
localcommunity?
•consideredtheprocess;theimpactandtheoutcomes?D
STAGE2:PL ANNINGTHEEVALUATION
INTRODUCTIO N
Nowyou’vedecidedtoevaluateyourcommunityartsproject,thissectionhelpyouplanthesteps.Thisisprobablythemostimportantphaseofyourwholeevaluation,anditisworthspendingthetimeplanningtheprocessproperly.
I SSUES TO CO NSIDER
Therearenumerousissuestoconsiderintheplanningphase.Yourorganisationwillneedtobeclearonissuessuchasthefollowing:
•whyare weundertakingtheevaluation?
•whowillmanagetheevaluation?
•whatresourcescanbeallocated?
•howlongwilltheevaluationtake?
•howwilltheevaluationworkin withtheactualproject?
•howwilltheevaluationintegratewith othersystemsin theorganisation?
•whowilltheevaluatorreportto? when?howoften?
•howwillthefollowingpeoplebeinvolvedintheevaluation-participants?staff?volunteers?thecommunity?audiences?
•whatissuesdoIneedtoconsiderforparticipants,e.g.confidentiality;unwillingnessto
participate;language,literacyorothercommunicationissues;privacyissues?
•whatdoes theorganisationplantodowiththeevaluationreport?
Thereisscopetoundertaketheevaluationusinganumberofdifferentapproaches.However,beforeproceedingthefollowingshouldbeclear:
•thereasonforundertakingtheevaluation
•themanagementprocess
•theresourcesavailabletotheevaluation
Typically,thesedecisionswillbedeterminedbyfactorssuchas:
•theimportanceofundertakingtheevaluationtotheorganisation;
•availableresources:time;money;staff;infrastructure;
•capacitytocopewithinterruptionsduringthecommunityarts project,and
•thereactionofparticipantstoanevaluation.
Withclarityontheaboveissues,theorganisationcandeterminethescopeoftheevaluation.Itisimportanttorememberthattherearelevelsofevaluation.Theycanrangefromacomprehensive,independentevaluationtoanevaluationconductedusingin-houseresources.
Thismanualprovidesthetoolsfororganisationstoconducttheirownevaluations,andtoensurethatthecostofundertakinganevaluationisnotprohibitive.
Followingisanexampleoftherationaleforevaluatingapuppetryproject,forasmallcommunitybasedorganisationandtheprocesstobeused.
•thereasonforundertakingtheevaluation
Forexample:
Weareevaluatingthepuppetryproject,tofindoutifwearemeetingouraimofengagingyoungpeoplewithdisabilitiesintheoutersuburbs.Weneedtoidentifyifweareusingthemost
effectivemethodsofrecruitment,andifthepartnershipwithNGO123isworkingwell.Wealsowanttoknowifthereisscopetofurtherdevelopthisworkforthestatewidepuppetryfestivaltobeheldintwoyears.
•themanagementprocess
Forexample:
TheBoardhascommissionedtheevaluation,andit takesoverallresponsibilityforthe
evaluation.ThePerformanceArtsSub-Committeewillactasthesteeringcommitteefortheproject,andwillreportregularlytotheboard.JaneSmithwilltakedaytodaymanagement oftheevaluationandwillreportregularlytothesteeringcommittee,andwillpresentthedraft
evaluationreporttotheBoard.Anyissuesordifficultiesshouldberaisedwiththesteeringcommittee.
•theresourcesallocated
Forexample:
JaneSmithwillworkanaverageoftwodaysperweekontheevaluationoverthecourseoftheproject,andfull-timeinthefinaltwoweeks,inordertopreparethereport.Janewillbeabletoundertakeinterviewsanddiscussionswithpeopleduringworktime,andothertimewillbetimeinlieu.Janeistoliaisewiththecreativestaff,inordertoplanaccesstoparticipantsetc.
TOOLS TO USE
Toassistindeterminingthescopeoftheevaluation,aworksheetisprovidedinthetools
section.Itisimportanttoremember, thataneffectiveevaluationcanbeconducted,evenwithlimitedresourcesandtheopportunitiesforreflectionandimprovementshouldnotbe
overlooked.
refer:Sampletool 2.1:Scopingtheevaluation.
Asampleworkplanisalsoprovidedforaprojectofapproximatelythreemonthsduration.Thetasksandtimelinescouldbeadjustedtosuitthescopeofspecificevaluations.
refer:Sampletool 2.2:Indicativeevaluationtimeline
CH ECK LIST:
Haveyou:
•clarifiedthereasonsforundertakingtheproject?D
•decidedhowtheevaluationwillbemanaged?D
•decidedonthelevelofresourcesthatcanbeallocatedtotheevaluation?D
STAGE3:DETERMINING EVALUATIONINDICATORS
INTRODUCTIO N
Determiningindicatorstomonitorandevaluatethecommunityartsprojectispartofthe
preparingandplanningphaseoftheevaluation.Thissectionisdedicatedtothistask,because
thisissuchacriticalpartofthewholeevaluation.Sampleindicatorsareprovidedforyoutouse;selectfrom,oradapttosuittheneedsoftheparticularprojectbeingevaluated.Theindicators
focusonthequalitativeaspectsofcommunityartsprojectswhichcanbedifficulttoidentifyandmeasure.
WHAT ARE EVALUATION INDICATO R S?
Inresearchterms,anindicatorprovides“datathat measuresorindicatessuccessorfailureinmeetingyourobjectives”1.Anevaluationindicator, therefore,providesdatathatwilltellyou,anddemonstratetoothers,thatyouareachieving,orhaveachievedtheproject’saims.Evaluationindicatorsaresimplyatoolin theevaluationprocess.
Forexample,imagineyourproject’saimis:
Toestablishachoirforsociallyisolatedolderwomeninourcommunitywhichprovidesopportunitiesforparticipantstobeinvolvedinculturalactivitiesinasocialsetting
Whatwilltellyou,anddemonstratetoothersthatyouachievedthis?Toidentifyyourindicators,breakdownyouraimsandobjectivesintotheircomponentparts.Whatwouldindicatetoyou,ortosomeoneelse,(anothercommunityorganisation;afundingbody;alocalcouncil;alocal
newspaper…)thattheprojectachieveditsaimsandobjectives.Thefollowingtableprovidesanexampleofthisprocess.
1AnEvaluationResourceforHealthyLivingCentresJayneMeyrickandPaigeSinkler,HealthEducationAuthorityp. 10
WHAT EVALUATI ON INDIC ATOR S C AN BE U SED FOR C OMMUNITY ARTSPROJE CTS?
Whendevelopingevaluationindicatorsforcommunityartsprojects,youwillneedtoreturntotheaimsandobjectivesyousetfor:
•theprocess–from theperspectiveofparticipants;theprojectandthecommunity;
•theimpact–from theperspectiveofparticipants;theprojectandthecommunityand
•theoutcome–fromtheperspectiveofparticipants;theprojectandthecommunity.
Whatwereyouraimsforeachoftheabove?Then,youneedtoconsiderWhatwilltellyou(andothers)thattheprojectachievedtheseaims?
Mostcommunityartsprojectswouldhope tobeable toshowthatthefollowingoccurred,asaminimum.
participants / project/organisation / communityprocess / participantsenjoyedbeinginvolvedintheproject
therewerepositive/constructivegroupdynamics / •therationaleandneedfortheprojectwereclearly
recorded
•aprojectplan was developedandimplemented / •theprojectmet its aimsaboutcommunity
involvementintheproject
•theprojectmet its aimsaboutcommunity
managementoftheproject
•thecommunity wasconsulted abouttheproject
impact / participantsfeltgood aboutwhatthey achievedthroughtheproject
participantsexperiencedpositiveemotions about
themselves,asaresultofbeinginvolvedintheproject(felt
good;feltproud;felthopeful…) / •theaimsand objectivesoftheprojectweremonitoredandevaluated
•theprojectproduceda
reasonablequality creativeproduct / •theprojectwaspositivelyreceived inthecommunity
•theprojectprovidedapositivefocusforthecommunity
outcome / participantsconfidenceandselfesteemhas beenraisedasaresultoftheproject
participantslearntnewthings
aboutthemselvesasaresult oftheproject / •theprojectwas consistentwith theorganisation’slongtermstrategicaims
•theprojectcontributedpositivelytothe
organisation’sreputation
•theprojectcontributed
positivelytopartnerships / •theprojectcontributedtolongtermbenefitsforthecommunity
participants / project/organisation / community
AND WH E N THE EX TRAORDINARY HAPPE NS ?
Communityartspractitionersemphasisethatattimes,projectscanachieveextraordinaryresults
–eitherinthecreativeandartisticfield,and/orforparticipants,and/orinthelocalcommunity. Asampleofevaluationindicatorsisincludedinthetoolssectionwhichyoucanusetoevaluate
whetherminimumlevelsofsuccesshavebeenexceeded.Itisimportantthatyoucanidentifyandarticulatetheexceptionalsuccessesthatoccur. Ontheotherhand,itisalsoimportantforthesakeofparticipants(andstaff)thatrealisticaimsandobjectivesaresetfortheproject.
PRACTICAL ISS UE S
Yourtasknowistodeterminewhatevaluationindicatorsyouwillapplytogatherdatathat
measuresorindicatessuccessorfailureinmeetingyourobjectivesfortheparticularproject.Tosetappropriateindicators,youneedtobeclearabout:
•thepurposeandscopeoftheevaluation,and
•theaimsandobjectivesoftheproject.
TO O LS TO USE:
Toassistyouindeterminingtheevaluationindicators,refertoSampletool3.1:Determiningevaluationindicators.
Toassistindeterminingifminimumlevelsofachievementhavebeenexceeded,refertoSampletools3.2–3.4:Sampleevaluationindicators.Thistoolprovidesacomprehensiverangeofsampleevaluationindicators,for:
•theprocess–from theperspectiveofparticipants;theprojectandthecommunity;
•theimpact–from theperspectiveofparticipants;theprojectandthecommunityand
•theoutcome–fromtheperspectiveofparticipants;theprojectandthecommunity.Thesampleevaluationindicatorshavebeendevelopedinthefollowingthreelevels:
•minimumlevelofsuccessthis(orappropriateequivalent)indicateswhatshouldhappen,
asaminimum-alarmbellsshouldringifthis(orequivalent)isnotoccurring/didnotoccur
•reasonablysuccessfulindicatesthatmorethantheminimumhasoccurred
•ideal/extremelypositiveindicatesthatsomethingoutstandinghasoccurred.
Youcaneitherusetheseindicatorsdirectly,orusethemasaspringboardtodevelopindicatorsforyourownproject.
CH ECK LIST:
Haveyou:
•decidedonthemostappropriateevaluationindicators fortheproject?D
STAGE4:COLLECTINGTHEDAT A
INTRODUCTIO N
Thissectionisaboutcollectingthedataforyourevaluation.Itprovidesadviceonplanninghow
tocollect datafortheevaluation;discussesissuesassociatedwithcollectinginformationfortheevaluationanddescribessomeofthemethodsyoucanusetocollectinformation.
PLANN ING D ATA COLLEC TION
Beforecollectinganydatafortheevaluation,itisstronglyadvisedthatyouinvestsometimeandeffortplanningthisprocess.Whileyouwillanalysesomerecordsandwrittenmaterial,mostofthedatayoucollectwillcomefrompeople–participants;volunteers;committeemembers;
staff;audiencesandsoon.Youwillbeaskingpeopletoallocatesometimetothisresearch,forexample,bycompletingsurveysorparticipatinginafocusgroup.Mostpeoplearewillingtodothis,butonlyiftheycanseethattheiropinions,timeandcontributionisvalued.
Thefundamentalpurposeofanevaluationistofindoutiftheoriginalaimsandobjectivesoftheprojectarebeingachieved.Evaluationscanbebroaderthanthis,andlongterm,ortheycanbemuchmorefocussedandshortterm.Onceagain,itisvitaltoemphasisethatyouneedtobeclearaboutwhyyouaredoingtheevaluationbeforeyoustartcollectingdata.Iftheaudienceisinternal,andthepurposeistoidentifyimprovementsinartspractice,youwilldesignyourdatacollectioninaparticularway.However,ifyouareconductingtheevaluationforanexternal
audience–perhapsyouthinktheorganisationisworkinginparticularlyinnovativewayandyouwanttodemonstratethattotheoutsideworld,oryouaretryingtointerestotherfundingsources,youwilldesigntheresearchwithdifferentemphases.
Whenyouareclearaboutthepurpose,scopeandaudiencefortheevaluation,youcanidentifythedatayouwillneedandplanhowyouwillcollectit.
PRACTICAL ISS UE S
Tostartwith,youneedtoconsiderthefollowing:
WhatexistinginformationdoIneedtocollect?
Don’toverlookexistingreports,recordsanddatainyourevaluation.Thisinformationcanoftenprovideausefulcontexttotheresearch,andyoushouldhaveeasyaccesstoitwithinyour
organisationornetworks.
Existinginformationofusetotheevaluationcouldinclude:
•previousevaluations(yourorganisation,orothersnearby)
•localgovernmentreportsprofilingthelocalcommunity
•ifyouare workingwithaspecifictargetgroup,reports fromappropriatecommunityoradvocacygroups
•internalrecords–numbersofpeopleinvolvedin previousprojects;recordsofwaitinglists;audiencenumbers;numbersofvolunteers,staffetcinvolved
•projectrecords–e.g.projectcommitteeminutes;rehearsal/attendancerecords
•Whatotherrecordswillbeusefulfortheevaluation?
WhatnewinformationdoIneedcollect?
Toanswer theabove,youneedtobeclearaboutwho(orwhat)cantellyouthattheprojectisachieving,orachieveditsaims?
Irrespectiveofthepurpose,scopeoraudience–allevaluationsshouldincludeproject
participants.Participantsarethekeygroupabletoletyouknowiftheprojectisworking,
whetheritwasappropriatelydesigned,whetheritwasenjoyable/interesting/challengingtobeinvolved;whethertheywouldevercontemplategettinginvolvedagain;howbeinginvolved
affectedtheirlives;whattheylearnt;whattheydidn’tlike,suggestionsforimprovementsandsoon.
Dependingonthepurpose,scopeoraudiencefortheevaluation,youwillalsoneedtocollectinformationfromsomeorallofthefollowing:
•staff–project,organisation
•volunteers
•partner organisations
•audiences
Ifyouwishtoconductacomprehensiveevaluation,youmayalsowanttocollectinformationfrom:
•localcommunityorganisations,communityleaders
•other artsorganisations
•relevantestablishedartists,practitioners
•otherpeopleconnectedtotheparticipantswiththepermissionoftheparticipants–seeresearchethicsbelow.
•Isthereanyoneelsewhocanlegitimatelytellyouwhethertheprojectisachievingitsaimsandobjectives?
WhatquestionsdoIneedtoask?
Developingasetofappropriateresearchquestionsisacriticalaspectofthedatacollection.Throughtheresearchyouwillwanttoaddressbroadquestions,suchasthefollowing:
•Weretheprojectaimsandobjectivesachieved?
•Howweretheyachieved?
•Weretheyachievedforallinvolved?
•Whatweretheoutstandingachievementsoftheproject?
•Howwellwastheproject,andtheprocess,managed?
•Whatlessons werelearnt?
•What wastheexperiencelikeforparticipants,theorganisationandthecommunity?
•Whatweretheimpactsforparticipants,theorganisationandthecommunity?
•Howdidtheprojectcontributetoindividuals,partnershiporganisations,thecommunity?
•Whatunanticipatedoutcomesoccurred(positiveornegative)?
Asthismanualisspecificallydesignedtohelpcommunityartspractitionersevaluateand
articulatethepositiveimpacttheartscanhaveonindividualandcommunitywell-being,itisimportantthattheevaluationseeksthisinformation.
•Arethereanyotherkeyresearchquestionswhichshouldbeincluded?
Themosteffectiveapproachistodevelopasetofkeyresearchquestions,suchastheabove.Thesebroadquestionscanthenbetailoredtosuitthevariouspeopleyouspeaktothroughtheresearch.
Whenyou’reaskingayoungparticipantquestions,youwillneedtophraseyourquestions
differentlyanduseadifferentstyletothetypeandstyleofquestioningyouusewithstafffrompartnerorganisations.
Tofindoutwhatlessonswerelearntthroughtheprocess,youmightaskayoungpersonquestionslike:
Ifyourbestfriendsaidtheywantedtogetinvolvedinasimilarproject,whatadvicewouldyougivethem?Iftheprojectofficeraskedforyouradviceonplanninganotherproject,whatwouldyousay?
Whereasyoumightaskapartnerorganisationquestionslike:Howwelldoyouthinkweworkedtogether?Whatcouldwedodifferentlyandbetter?
Whataboutresearch ethics?
Itisimperativethatyouareclearabouttheethicsofyourresearchbeforeyoustartcollectinginformation.
Forexample,
•Youmusttellparticipantsthatyouareconductinganevaluationandyoushouldexplainthepurpose,scopeandaudienceforthereport.
•Youmustexplaintheimplicationsoftheresearch–forexample,Willphotosbetaken?Whowillthefinalreportbesent to?Willtheevaluationbeapublicdocument?Canpeopleprovidecommentsanonymously?Canpeopledropoutoftheproject?theevaluation?
•Ifparticipantsareuncomfortable,youmustprovideopportunitiestodiscuss thisbeforetheresearch,andyoumustgivethemchoicesaboutcontinuing,and/ordiscuss waystheresearchcouldbestbeconducted.
•Youmustaskpermissiontoquote,photographorotherwiseidentifyaparticipantinareport.
•Youmustaskpermissionoftheparticipantifyouwishtotalktoanother personaboutthem–forexample,youmaywishtoseekacommentfromschool,family,etcabouttheimpactoftheprojectandifrefusedyoumustnotseekcommentsfromthosesources.
•Youshouldprovidegenuine,meaningful,appropriateandnon-threateningopportunitiestoparticipate.Forexample,itisnotethicaltoexpectagroupofparticipantsfrom anotherlanguagebackgroundtocompleteawrittensurveyinEnglish,andthentoclaimparticipants
wereconsulted.Norisitethicaltoonlyprovideopportunitiesforpeopletoprovideinformationinagroupdiscussionifpeopleare shyorhaveanydifficultiescommunicatingin thatsetting.
•Youshoulddecidewhathappenstothedata,aftertheproject.Youmayhavealotof
informationincludingconfidentialinformationandparticipantsareentitledtoknowwhatwillhappentorecordings;photographsandotherrecords.Optionsinclude:theorganisation
lookingaftertherecords,asthey wouldotherconfidentialrecordsforaspecifiedperiodof
time;destroyingtherecordsaftertheresearch;offeringtherecordstoparticipants.Themostimportantconsiderationisthattheprivacyandconfidentialityofparticipantsisrespectedandmaintained.
HowwillIcollecttheinformation?
Whenyouareclearabouttheethicsoftheresearch,youneedtodeterminethemostappropriatemethodsforconsultingthepeopleinvolved.
Standardinformationcollectiontechniquesinclude:
•writtensurveys–whichcanincludeopenandclosedquestions;useofratingscales;useofpictogramsetc
•focusgroupdiscussions
•structured,orsemi-structuredinterviews–tapedornottaped
•exitpolls
Mostevaluationsincludesomelevelofsurveyingandinterviewing.Somesamplesurveysandinterviewsareincludedinthesampletoolssection.Thesehavebeenstructuredaroundthekeyresearchquestions,usingbothclosedandopenquestions.Ifyouhaveadditionalkeyresearchquestions,youwillneedtoadaptthetoolsaccordingly.
Inordertodecideifthesestandardmethodsareappropriateforthepeopleparticipatingintheevaluation,orwhetheryouneedtosupplementthesemethods,thinkaboutissuessuchas:
•Arethereliteracyissues?IsEnglishtheparticipant’sfirstlanguage?Areparticipants
comfortablereadingandwriting?Howlongshouldawrittensurveybe?Willpeopleneedhelptocompleteit?Willparticipantsthinkitistoodaunting,likeworkorhomework?Remember–literacyproblemsarequitecommon,youcan’tassumethatpeoplecanreadandwrite
confidently.
•Areparticipantscomfortablespeakinginagroup?Whatare thegroupdynamicslike?Doyouneedaninterpreter?Whatsortofenvironmentwillencouragediscussion?Howwillyoudealwithpeopledominatingthediscussionornotparticipating?
•Willparticipantsfeeluncomfortableinaone-to-oneinterview?Can someoneelsebepresent?
•Howcanparticipantstimebestbeused?
Ifitisnotappropriateforpeopletoparticipateusingconventionalresearchtools,arangeof
broader,morecreativemethodsarealsoappropriate,particularlywithnewtechnologies,suchasscanningimagesintocomputersorvideorecording.
•Youcanuseaprojectdiary orscrapbook–withclearguidelinesonwhocanmakeentries,howitistiedtothecreativeprocess;whentouseetc.
•Youcanusephotographs–withparticipants’permission.Photoscanbeofthecreativeworkand/orofparticipantsduringtheprojectandcanbeusedasevidencetosupportkeyfindings
•Youcanusedrawings,orothervisualrecords.
•Youcan audioorvideorecordings–withparticipant’spermission.
Whatisthedifferencebetweenquantitativeandqualitativedata?
Quantitativedatausuallyanswers‘howmany/muchquestions?’Itprovidesyouwiththenumbersabouttheproject–forexample:
PROJECTHowmanyparticipants?
Howmuchmoney–wasreceived?wasspent?
PROCESSHowmany–rehearsals;workshops;sessionswereheld?Howmanyevents–festivaldays;productions;exhibitions?
COMMUNITYHowmanyaudiencemembers?
Howmanycommunitymembersonthemanagementcommittee?
QualitativedataanswerstheWhy?How?Inwhatway…?Towhatextent…?questionsofa
project.Itisparticularlygoodforunderstandingwhycertainthingsdidordidnothappenwithin
theproject.Thisinformationwhichcannotbequantifiedlikequantitativedata,suchaswrittenorspokencomments.Themescanbedrawnoutandthiscanalsoprovideexcellentsourceofcasestudies,whichcanbeverypowerful.
Bothtypesofdatahaveadvantagesanddisadvantages.Quantitativedatacanbequitedry–youmaystillwanttoknowwhypeopleansweredinaparticularway.Ontheotherhand,itcanberelativelyeasytocollect;youcangatheralotofinformationfromalotofpeople;youcanmakecomparisonsandyoucanidentifystatisticallysignificantvariations,ifyouwishtogotothatlevelofanalysis.Qualitativedataprovidesyouwithlotsofquotes,ideasandstories–butyoucan endupwithalotofinformationandyouwillstillneedtoextractthekeythemes.Andyouwillstillneedthenumbers!
Bothqualitativeandquantitativedataareneededtoundertakeabalancedandeffectiveevaluation.2
•Askyourself,arethereanyotherappropriateresearchmethodsneededfortheevaluation?
Howoften willIcollectinformation?
Aswithallaspectsoftheresearchdesign,thisquestioncanonlybeansweredinthecontextofyourspecificevaluation.Asaminimum,youwillneedtocollectinformationfromparticipantsatleastonce,atthecompletionoftheproject.Atthatpoint,participantscanmakesomecommentabouttheprojectandtheirexperienceinbeinginvolved.Thiswouldproduceafairlylimited
evaluationreport.However,itmaybeallthatresourcesallow,andallthatyouneedtodo.Ifthisisthecase,makesureyouasktherightquestions,becauseyou’veonlygotonechance!
2ref: Hawe,Penelope;Degeling,DandHall,J(1990)EvaluatingHealthPromotion:Ahealthworkers’guideSydney:MaclennonandPettyPublishing(ref:Chapter6ImpactandOutcomeEvaluation)p.105
Ifyouareabletoaskquestionsofparticipantsatthebeginningandtheendoftheprocess,youareimmediatelyinastrongerpositionbecauseyoucancompareparticipants’perceptionsandviewsfromthebeginningtotheend.Similarly,withstaff,mentors,artists,volunteers,itisusefultoaskbeforeandafterquestions.Thisispreferabletoaskingpeopletorecallthetimebefore
theprojectstarted. Youwillhavesomedatathatclearlyindicatesthedifferencemade(ornotmade)by theproject.
If therewassupportforalongertermevaluation,youcouldarrangetofollowupparticipantsaftersometimehadelapsed.Thisallowsyoutofindoutfromparticipantstheinfluenceofthe
projectontheirlivesoveralongertimeframe. Thisiseasierifparticipantsarelikelytoreturntoyourorganisation–maybeit’sayouthcentreofsomesort,andpeoplekeepingcomingback.
Trackingparticipantsprovidesreallyusefulandinterestinginformationabouttheinfluenceoftheproject,howeverthiscanbecostlyandtimeconsumingthereforeyouneedtoconsiderthecostbenefitsofthistoo.
HowdoIletpeopleknowabouttheevaluationandencouragethemtoparticipate?
Ideally,youwillhavedecidedonyourresearchapproach,atleastinbroadtermsevenbefore
theprojectstarts.Thisway,youcantalktoparticipantsandothersatthecommencementoftheresearch.Youcandiscussandevennegotiatethebestresearchstrategies,andyoucan
encouragepeopletobeinvolvedfromtheoutset.Whenyouareclearaboutthepurposeoftheevaluation,youcan‘sell’thebenefitstoparticipants–
•Youcanhavearecordofthereportfor…school,youremployer,application totrainingcourseetc
•We’rehopingtousetheresultsinournextfundingapplication
•We’rehopingtoencouragethelocalyouthcentretocomeonboard
•We’rehopingtoencouragethedisabilityprogramtocontributesome funds
•We’rehopingtousetheresultstoconvincetheboardweneedafull-time worker
•We’rehopingtousetheresultsforanarticlein thelocalpapertoencouragemoreVietnamesewomentobecomeinvolvedinourgroup
•We’rehopingtousetheresultstoapplyformentoring funds
•We’rehopingtousetheresultstoseeifwecouldbecomeatrainingprovider….
Therearepotentially significantbenefits forparticipants aswellas organisations.Ifparticipantsare:
•informedaboutthepurposeoftheevaluation;
•consultedabouttheresearchmethods;
•givenadequatenoticeofactivitiesinvolvingthem–forthcomingsurveys,interviewsetc;
•involvedintheresearchdesign;
•providedwithchoicesabouthowtocontribute;
•consultedinnon-threateningandrespectfulways,and
•informedaboutprogress alongthewayandoncompletion
youwillfindthatmostpeoplearemorethanwillingtobeinvolved.
Remember–youneedtoconsiderwhoneedstoknowabouttheevaluationandthebestwaystokeeptheminformed.Otherorganisationsinyournetworksmaybeinterested;othersectionsofyourownorganisation;newcommunityartsgroups;establishedcommunityartsgroups.Itisusefulatthestartoftheevaluationtoprepareaninformationsheetwithashortoutlineoftheevaluation,whichyoucandistributeas required.
CO LLECTING THE I NFO RMATIO N
Whenyouhavecompletedtheaboveplanningandpreparation,youneedtofinaliseyourresearchtools.Thereisarangeofsampleresearchtoolsincludedwiththemanualforyouruse.Thesesurveys,focusgroupoutlines,interviewsanddiaryguidelinescanbeuseddirectlywithsomeminoridentifyinginformationfromyourorganisation,ortheycanbeadaptedtosuitthe
specificsoftheproject.
Otherissuesyouneedtoconsiderare:
•Howmuchinformationisenough?
•Howmanypeoplewillyousurvey?–everyone?randomselection?
•Howwillyousurvey them?–fillitinatrehearsal?mailit totheirhomeaddress?leavecopiesaround?
•Howoftenwillyousurvey them?beginning?middle?end?
•Whowillbeinvitedtofocusgroupdiscussions?
•When?
•Whowillfacilitate?
•Whowilltakenotes?
•Theroleoftheevaluator–whoevaluates?howaretheirobservationsrecordedandcollected?
TOOLS TO USE
Sampletools4.1:Participantevaluationsurvey
4.2:Genericinterviewstructure/focusgroupoutline4.3:Projectjournalguidelines
CH ECKLIST:
Haveyou:
•consideredtheethicsoftheresearch?D
•identifiedtheexistingrecordsandmaterialtobecollected?D
•identifiedthenewinformationtobecollected?D
•decidedonyourkeyresearchquestions?D
•decidedonthemostappropriateresearchmethodsfortheevaluation?D
•plannedhowoftenpeoplewillbesurveyed/interviewedetc?D
•developed themostappropriatecommunicationstrategies fortheproject?D
STAGE5:ANALYSINGTHEDAT A
INTRODUCTIO N
Stage5providessomeguidelinestohelpyoureducetheamountofinformationtoamanageableamountandthen,tomakesomesenseofit.
REDUC ING THE D ATA
Oncetheinformationfortheevaluationhasbeencollected,youwillneedtostartgroupingandcompressingtheinformation.Forexample,youwillneedto:
•collatesurvey responses (manuallyorbydataprocessingforlargenumbers)
•analysethewrittenresponses toopen-endedquestions
•identifykeythemesfromwrittensurveys;focusgroupsand/orinterviews
•collatetheinformationgatheredfrom existingmaterial
Inordertomakesenseofthedata,itneedstobeorganisedintoamanageablesizeand
structure.Useagridofthekeyfactorsbeingevaluatedandrecordtheresearchresultsthere.SeeSampleTool5.1AnalysingEvaluationDatatemplate.
participants / project/organisation / communityprocess
impact
outcome
participants / project/organisation / community
ANALYSING THE DATA
Onceyouhaveorderedthedata,youneedtoanalyseit:whatdoesthisinformationmean?Forexample:
•Whatdoesittellusabouttheproject?
•Whatdoesittellusaboutwhathappenedthroughtheproject–toparticipants?theorganisation?thecommunity?
Toanalysethedata,youneedtoapplyyourevaluationindicatorsandtheninterprettheresults.Forexample,thefollowingareprovidedasminimumlevelevaluationindicatorsforparticipants(seeStage3).Toanalyseyourinformation,youneedtoturntheindicatorsaroundandask
questionssuchasDidwefindanyevidenceofthishappening?
outcome
Workingthroughthisprocess,youwillidentifythekeythemesandfindingsoftheresearch.Itisusefultodothisasagroup,orwithsomeoneelse. Youwillfindthatyouneedtodiscussand
thinkthroughsomeoftheissues,toreallygettothecoreinformation.Itisworthtalkingaboutyourinterpretationstogether,andtestingeachother’sthinking.Bewareofissuessuchas:
•jumpingtoconclusionswithoutsupportingevidence
•notwantingtoincludeoridentifynegativefindings
•notwantingtoincludeoridentifymediocrefindings
Onceyouhaveworkedthroughandanalysedthedata,itisworthsummarisingthistoasetofkeyfindings.Developthisasadrafttousefordiscussions–forexample,withtheevaluationcommitteeifthereisone;managers;participants;partnershiporganisationsandsoon.Buildinginastepintheprocessaboutdiscussionofkey findingsprovidesavaluableopportunityfor
furtherrefinementofyourthinkingandpresentationofthedata.
TOOLS TO USE
Toassistinanalysingthedata,returntotheevaluationindicators.Workthroughthequestionsintheexampleabove,toaskyourselfwhatevidenceyoucollected.Remember,youwillonlyneedtousetheevaluationindicatorsappropriatetotheparticularevaluation.
Sampletools3.2–3.4:Sampleevaluationindicators
5.1:Analysingevaluationdatatemplate
CH ECK LIST:
Haveyou:
•collatedthedatayoucollectedintomanageableandlogicalchunks?D
•checkedthedataagainsttheevaluationindicators?D
•discussedthedatawithsomerelevantpeople?D
•developeddraftkeyfindings?D
•presentedordiscussedthekey findingswithappropriatepeople?D
STAGE6:REPORTINGONTHEDATAANDIMPROVING ON CURRENT PRACT ICE
INTRODUCTIO N
Thissectionprovidesguidelinesonreportingontheevaluation.
HOW DO YOU P REP ARE AN EV ALUATIO N REPORT?
Essentially,theevaluationreportwillwanttoansweryourkeyresearchquestions.Theexamplesprovidedare:
•Weretheprojectaimsandobjectivesachieved?
•Howweretheyachieved?
•Weretheyachievedforallinvolved?
•Whatweretheoutstandingachievementsoftheproject?
•Howwellwastheproject,andtheprocess,managed?
•Whatlessons werelearnt?
•What wastheexperiencelikeforparticipants,theorganisationandthecommunity?
•Whatweretheimpactsforparticipants,theorganisationandthecommunity?
•Howdidtheprojectcontributetoindividuals,partnershiporganisations,thecommunity?
•Whatunanticipatedoutcomesoccurred(positiveornegative)?
Howyoustructure,writeandpresentyourreportwilldependontheevaluationaudience.Ifthereportistobedistributedoutsideyourorganisation,itshouldbeformalandwell-presented.Ifitisforaninternalaudience,itcanbemuchshorter–youwon’tneedtoprovideasmuchcontextandbackground–andmorerelaxed.
Aswiththedraftfindings,itisusefultoprepareanddistributethereportindraftformfirst.Thisgivesopportunitiesforpeopletocontributetotheprocess;tocorrectinaccuracies;toimproveexpressionandpresentationandtorefinetheanalysis.
WHAT C AN WE LEAR N FROM AN EVALUATION?
Arigorouslyconductedevaluationwillprovidetheorganisationwithawealthofinformation.Ensurethattight,achievableandconstructiverecommendationsaredevelopedfromthereport.
Considertherangeofpeoplewhomaybeinterestedinthefindingsoftheevaluation–other
organisations;recruits;fundingbodies;philanthropicorganisations;otherprogramareasandsoon.Ifyouthinkarangeofpeoplewillbeinterested,itisusefultoprepareanexecutive
summary.Thisshouldincludesomebackgroundinformation,thekeyfindingsand
recommendations.Theexecutivesummarycanthenbedistributedtointerestedparties,andthefullreportcanbeavailableifrequested.
Ensurethataprocessisdevelopedtoconsultfurtherontherecommendations,ifnecessary. Ifthereisclearsupport,developaprocesstoimplementtherecommendations.Anyevaluation–whethershortandsnappyorcomprehensiveshouldprovideinformationtoimprovethecommunityartspracticeoftheorganisation.
TOOLS TO USE
Thereisasampletoolprovidetoassistwithstructuringthereport.Thistoolassumesthatthereportisbeingpreparedforanexternalaudience.
refertoSampletool6.1:Draftevaluationreportstructure
Thereisalsoaworksheettoassistyouidentifywhatimprovementneedtobemade.refertoSampletool6.2:ImprovingProjectAimsandOutcomes
CH ECK LIST:
Haveyou:
•developedadraftevaluationreport?D
•answeredthekeyresearchquestionsintheevaluationreport?D
•collectedfeedbackonthedraft?D
•finalisedthereportandpreparedanexecutivesummary?D
•distributedtheexecutivesummarytotheappropriateparties?D
•linkedtheevaluationresultsintoanimprovementprocess?D
FURTHERREADING
Fiske,EdwardB.(2000)ChampionsofChange:TheImpact OfTheArtsOnLearning.ArtsEducationPartnershipandPresident’sCommitteeontheArtsandtheHumanities.
Hawe,Penelope;Degeling,D.andHallJ.1990)“ImpactAndEvaluationOutcome”Chapter6, InHawe,Penelope;Degeling,D.AndHallJ.EvaluatingHealthPromotion:AHealthWorkers’
Guide.Sydney:MaclennonPettyPublishing.
Matarasso,Francois(1996)DefiningValues:EvaluatingArtsPrograms(SocialImpactoftheArts:WorkingPaper1).BournesGreen,Comedia.
Matarasso,Francois(1997)UseOrOrnament?:TheSocialImpactofParticipationintheArts.Stroud,England:Comedia.
Meyrick,JaneandSinkler,PaigeAnEvaluationResourceforHealthLivingCentres,HealthEducationAuthority
Owen,JohnM.andRogers,PatriciaJ.(1999)ProgramEvaluation:FormsAndApproaches.StLeonards,N.S.W.:AllenUnwin.
Pope,Jeanette(2000)SocialCapitalAndSocialCapitalIndicators.
SIAP(2001)SocialImpact OfTheArtsProject.Philadelphia:UniversityOfPhiladelphia,SchoolofSocialWork.
Wadsworth,Yoland(1991)EverydayEvaluationontheRun,ActionResearchIssuesAssociationInc.,Melbourne.
Williams,Deidre(1997)HowtheArtsMeasureUp:AustralianResearchintoSocialImpact
Gloustershire:Comdedia(SocialImpactoftheArtsWorkingPaperNo.8