An Evaluatio N Gui De for Community Arts Practitioners

Prepared by EFFECTIVE CHANGE Pty Ltdfor Arts Victoria; Darebin City Council; City of Whittlesea and VicHealth

EvaluatingCommunityArts

CommunityWellBeing

AN EVALUATIO N GUI DE FOR COMMUNITY ARTS PRACTITIONERS

Thisguideisavailablefromthefollowingwebsites:

ArtsVictoria:

VicHealth:

EffectiveChange:

DarebinCityCouncil: :

EvaluatingCommunityArtsandCommunityWell-Being

AnEvaluationGuideforCommunityArtsPractitioners

Evaluationresearch;Evaluationindicators;Communityarts;Communitywell-being;Communitydevelopment;Communitycapacitybuilding

Researchedandwrittenby:ClareKeating

EffectiveChangePtyLtd(03)93881661

©StateofVictoria2002

Thisworkiscopyright.Itmaybereproducedinwholeorpartforitsintendeduseincommunityartsprojects,forstudyortrainingpurposes,subjecttotheinclusionofanacknowledgementofthesource.Commercial

usageorsaleisnotpermitted.

Acknowledgements

Theauthor wishestoacknowledgethesupportoftheorganisationswhichcommissionedthisworkandthecommitmentofthestafffrom thoseorganisationstothisguide:

•MariaKatsonisandJudyMorton(ArtsVictoria)

•SusanBall(VicHealth)

•MarkWilkinson(DarebinCityCouncil)

•JudyTurner,KathMelbourneandSarahPoole(CityofWhittlesea)

Ialsowishtoacknowledge andthankofthebroadrangeofcommunityarts practitioners,

managersandparticipantswhowereconsultedin thedevelopmentstages;commentedondraftsoractuallypilotedtheguidewiththeirowncommunityartsprojects.Yourcontributionshavebeeninvaluable.

A MESSAGEFROM THE MINISTER FOR THEARTS

Theartsengage,provoke,amuseandexciteusbut,aboveall,theartsreflectandshapeoursenseofcommunityidentityandhelpsbuildsocialcohesion.

AsMinister fortheArts,Iampassionateaboutspreadingthewordthatparticipatingintheartsandculturalactivitiesisarewardingandcreativeexperience,onbothanindividualandcommunitylevel.

Weneedabalancedunderstandingoftheroleandworthoftheartsinoursociety–onethatembracestheiraesthetic,cultural,economicandsocialvalues.

Oneofthechallengesfacinggovernmentsandtheartssectorisfindingrigorousresearchtodemonstratethepositivesocialimpactofartsinvolvement,ratherthanrelyingonanecdotalevidence.

WiththereleaseofEvaluatingCommunityArtsCommunityWellBeing,wehaveauser-friendlytoolthatallowscommunityartspractitionerstodocumentandassessthesocialoutcomesoftheirprojectsandprogramscomprehensivelyandsystematically.

ThewidespreaduseofthisEvaluationGuidewillincreaseknowledgeandunderstandingofthebenefitsofartsinvolvement,leadingtomore,andmoreeffective,artsprogramsinVictoria’s

diversecommunities.

Istronglycommendtheguide.

TheHonMaryDelahuntyMPMinisterfortheArts

FOREWORD

Iampleasedtocommendthisevaluationguidetoyou. Evaluationisimportant,notonlyforprovingwhatyoudoiseffectivebutalsoforimprovingprogramsandcontributingtobetteroutcomes.

VicHealthisanactivesupporterofcommunityartsthrougharangeoffundingschemes

developedwithinourArtsforHealthProgram. Weregularlyobserveimportanthealthandcommunitywell-beingoutcomesthroughcommunityartsprojects,however,wefindthat

organisationshavelimitedcapacitytoprovideevidenceofthisbeyondtheanecdotal.Thisinturnlimitsallofourabilitytoshowcasesuccesses;tolearnfrom failures;andtoreflectontherangeofachievements.

Iunderstandthatmanypeoplefaceevaluationwithsometrepidation. Itishopedthisevaluationguidewilleasethatfearandbeawelcomeresourcewhichprovidestheframeworkandthenecessarytools. Ithasbeendevelopedincloseconsultationwiththesector,respondingtotheneedforbothashort‘Express’guideandamorecomprehensiveversionwhichhasthecapacitytobeusedasis,oradaptedtoyourspecificneeds. Wehaveensuredthattheresourceis

widelydistributedandreadilyavailable–thedocumentisonthewebsitesofthefiveprojectpartners.

Wethankourprojectpartners,ArtsVictoria,DarebinCityCouncilandtheCityofWhittleseafortheircommitmenttotheproject.We thankClareKeatingofEffectiveChangeforproducingsuchahighqualitydocument.Wehopeithelpsyoutoshowthedifferencecommunityartscanmaketohealth.

DrRobMoodie

ChiefExecutiveOfficer

CONTENT S

AMessage FromTheMinisterForTheArts...... ii

Foreword...... i

Preface...... v

Introduction...... v

EvaluationOverview...... x

Stage1:PrepareForTheEvaluation...... 1

Stage2:PlanTheEvaluation...... 2

Stage3:DetermineEvaluationIndicators...... 3

Stage4:CollectData...... 4

Stage5:AnalyseData...... 5

Stage6:ReportAndImprove...... 6

Stage1:Preparing ForAnEvaluation...... 7

Stage2:PlanningTheEvaluation...... 10

Stage3:DeterminingEvaluationIndicators...... 12

Stage4:CollectingThe Data...... 15

Stage5:AnalysingTheData...... 22

Stage6:ReportingOnThe DataAnd Improving On CurrentPractice...... 24

FurtherReading...... 26

TOOLS:

SampleTool1.1:Setting ProjectAims...... 29

SampleTool2.1:PlanningThe Evaluation...... 30

SampleTool2.2:IndicativeEvaluationTimeline...... 31

SampleTool3.1:DeterminingEvaluationIndicators...... 32

SampleTool 3.2–3.4:SampleEvaluationIndicators...... 33

SampleTool4.1:ParticipantEvaluationSurvey...... 36

SampleTool4.2:GenericInterviewStructure/FocusGroupOutline...... 37

SampleTool4.3:ProjectJournal Guidelines...... 38

SampleTool5.1:Analysing EvaluationDataTemplate...... 39

SampleTool6.1:DraftEvaluationReportStructure...... 40

SampleTool6.2:ImprovingProjectAimsAndOutcomes...... 42

PREF ACE

COMMUNI T Y ARTS PRACT ITIO NERS…

Wouldyouliketohaveyournextfundingapplicationreadybeforeyou’vefinishedyourcurrentproject?

•Wouldyouliketobeabletoplanfutureprojects?

•Wouldyouliketodevelopstrongerpartnershipsinthecommunity?Evaluationisaprocesswhichcanhelpinalloftheabove.

•Haveyoueveraskedparticipantswhethertheyenjoyedacommunityartsproject?

•Haveyoueveraskedapartnerinacommunityartsprojectwhattheythoughtoftheexperience?

•Haveyoueveraskedthepublicwhattheythoughtofaperformanceorpiece?

Ifyouansweredyes,thenyouhaveundertakenatleastabasiclevelofevaluation.Let’sfaceit,mostcommunityartsworkersdothisallthetime–it’sanaturalpracticeformostpeopleinvolvedinthe

communityarts.It’seasy,it’sfun,it’sinterestinganditkeepsyouinvolvedandstrivingfornewgoalsandnew projects.

•But,doyouthensitdownandconsiderthefeedback?

•Doyouthinkaboutwhatyouneedtodobeforecommencinganotherproject?

Yes?Then,youareundertakingsomelevelofcontinuousimprovement,throughevaluation.Thismightbeabitharderandhavesomepitfalls.Often,thereisnoincentivetodothisandnotimetodoitsystematically.Youprobablyhavealreadyexperiencedsomedirectbenefitsfromthisapproachthough,andyou’rekeentokeepgoing.But…

•Doyouhaveaplanreadynowtoevaluateyournextproject?

•Haveyouthoughtaboutwhotospeakto?When?Whatwillyouaskthem?How?

•Howwouldyourecogniseandpreventaloomingdisaster?

•Ifyoudocumentedtheprojectoutcomes,howwouldyoureport,andtowhom?

Ifyoudon’thaveaplanreadyNOWforevaluatingyournextcommunityartsproject,thisguideisforyou.Recognisingtimeconstraintsinthecommunityarts,theguideissimple,straightforwardandreadytouse.

Theguideisorganisedaroundthestagesofevaluation.Atthebackoftheguidearetoolsforeachstageoftheevaluationwhichcanbeusedastheyare,oradapted.Theevaluationmodelrecognisestheimportanceoftheprocessaswellastheoutcomeincommunityartsandhelpsyoutoidentifyachievementsineach

area.Themodelisdesignedasaresource,withallthetoolsprovidedtohelpyouevaluate,reportand

improve.Hopefully,withtheevaluationplannedoutforyou,communityartspractitionerswillhavesupporttoevaluate,reflectandimprove.Theguideisintwoformats–asummaryversioncalledtheExpressGuide

andacomprehensiveversion,whichincludestheExpressGuide.

Bothversionsoftheguideareavailablefromanyofthefollowingwebsites:ArtsVictoria

VicHealth

EffectiveChange

DarebinCityCouncil ebin.vic.gov.au/arts/publications .html

CityofWhittlesea

Itisrecommendedthatyouaccessbothversions.Goodluck!

INT RODUCT ION

PURPO SE OF THE EVALUATION GUIDE

Thepurposeofthisguideistoprovidearesourceforcommunityartspractitionerstousewhentheyevaluatecommunityartsprojects.Itisdesignedtobeasflexibleaspossible,sothatitcanbeusedbythewidevarietyoforganisationsinvolvedinthecommunityartsandsothatitcancovertherangeofcommunityartsprojects.Theguideprovidesacomprehensivebasefor

undertakingevaluations.Theprocessoutlinedintheguidecanbefollowedthroughsequentially.

Alternatively,theprocess,toolsandsuggestionscanbeamendedandrefinedtosuitthe

specificneedsofvariousprojectsandorganisations.Theguidealsoaimstohelpbuildabodyofevidenceabouttheworkofthecommunityartssector,byusingaconsistentandstructured

researchframework.

B ACK GR OU ND

Aconsortiumoforganisationshasbeeninvolvedinthedevelopmentoftheguide.TheprojectwasfundedbyArtsVictoria;DarebinCityCouncilandtheCityofWhittlesea.ArtsVictoriaandVicHealthhavefundedthefinalproductionoftheguideandmakingthemavailableon-line.

Togethertheprojectpartnersrecognisedthat:

•withinthecommunity,increasingvalueisbeingplacedoncommunityactivitieswhichhavepositiveimpactsonthecommunity;

•thereisagrowingrecognitionofthesocialandcommunitywell-beingoutcomesofcommunity

artsprojects,yet

•thereisagapintheresourcesavailabletocommunityartspractitionerstoassistthemin

identifying;collatingandreportingonthepositiveoutcomesachievedthroughcommunityartsprojects.

Thepurposeofthisguideisthereforetoaddressthatgapandtoserveasaresourceforcommunityartspractitioners.

TheguidehasbeenpreparedanddevelopedbyEffectiveChangePtyLtd,aMelbourne-basedconsultancycompany.Inresearchingtheguide,relevantliteraturewasconsultedandarangeofkeyinformantswereinterviewed.

WHO I S THE G UIDE FO R?

Theguideisforcommunityartspractitioners,ratherthanspecificorganisations,ascommunityartsprojectscanbelocatedacrossavarietyoforganisations.

HOW SHOULD I T BE USED?

Theguideessentially‘talks’toagenericcommunityartsprojectworkerandassumesthatthispersonwilldrivetheevaluation.However,itcanbeusedinarangeofways.Forexample,allparticipantsinacommunityartsprojectcouldsharerolesinundertakinganevaluation.Oritcouldbeusedanexternalstudentorconsultanttoundertakeanindependentevaluation.

COMMUN ITY ARTS – N OT JUST COMMUN ITY, NOT JUST ARTS

Thisguiderecognisestheuniqueplaceofcommunityartswithincommunityandculturalactivity.

Casestoriesaboundofcommunityartsprojectsproducingstunningandlastingcreative

achievements.Therearealsonumerousexampleswhereextraordinarylifechangesoccurredforparticipantswhilethecreativeoutcomeswerelessmemorable.Othertimes,communityarts

projectshavecontributedenormouslytotheirlocalcommunity,througharangeofstrategies,suchasengagingdisenfranchisedmembersofthecommunity,orproducingvaluableresourcesforaparticularcommunity.Sometimesexceptionalcommunityartsprojectsproduce

extraordinaryresultsacrossallareas.Othertimes,peoplejustfeelgoodbybeinginvolvedandtakingrisks.Thisguidehasbeendevelopedtoworkacrossallareas–fromthecreativeoutput,tothecreativejourney,totheindividualsandcommunitiesinvolved.Thismodelofevaluation

willallowyoutoidentify theareasofachievementandsuccess.It worksevenwhen,inconventionalterms,theprojectmaynotappear tobethatsuccessful.Itwillalsoallowyoutorecogniseextraordinaryachievementsacrossthisrangeof factorswhentheydooccur.

Thediagrambelowshowsthekeyelementstobeevaluated,usingthismodelofevaluation.

Diagram1:KeyElementsofCommunityArtsProjectstobeevaluated

Rememberthatyouprobablywon’tbeabletoconsideralltheabovefactors,allofthetime.Thisguide providesacomprehensiveapproach–itisuptoyoutoselectthepartsthatarerelevant

toyourproject,yourorganisationandyourresources. Youmayonlyhavetheresourcestofocusononekeyaspectoftheproject.Mostcommonly,youwillfocusontheimpactoftheproject-Whathappenedasaresultofthisproject?Positiveanswerstothisquestioninparticularyieldsusefulinformationforpromotingtheproject;promotingtheorganisation;thattheparticipants canuse;thatcanbeusedinfundingapplicationsorprogramexpansionapplicationsandsoon.Andwhenyouwanttoevaluateaprojectingreaterdepth–whenyou’reclearthatyouhavetheresources;there’sagoodcaseforevaluatingtheprojectandtheorganisationsupports the

evaluation-theguide’ssimpleframeworkandreadyresourceswillenableyoutocollectthe

informationandhelpyouarticulatetheelementsthatgototheheartofcommunityartsprojects,butaretoooftenleftunsaid.

STRUCTURE O F THE GUI DE

Theguideisstructuredchronologically.Itguidestheevaluatorthroughthestagesofevaluationfrompreparingthegroundthroughtoreportingontheevaluation.Thesixstagesare:

•Stage1:Prepareforanevaluation

•Stage2:Plantheevaluation

•Stage3:Determineevaluationindicators

•Stage4:Collectthedata

•Stage5:Analysethedata

•Stage6:Preparetheevaluationreportandimproveoncurrentpractice

Eachstageispresentedasaworksheetforyoutocompleteand:

•takesyouthroughthedecisionsyouneed tomake

•askstriggerquestionstofocusyourthoughts

•highlightstheissuesyouarelikelytoencounter

•providesachecklist

•refersyoutotoolsyoucanuseandadaptifyouwish

•refersyoutothefulltextforfurtherdetail.

TheOverviewdiagramonthenextpageisasummaryofthecompleteprocess.Inthecomprehensiveguide,eachStagecontains:

•anintroduction;

•discussionontherationalebehindtheparticularstepoftheprocess;

•discussiononresearchmethodsandissues;

•practicalissues–outliningthetaskstobedone,and

•refersyoutothetoolstouse.

Therearesampletoolsattheendoftheguide,rangingfromsamplesurveystoadraft

evaluationreportoutlinewhichcanbeadaptedforyourownproject.Thetoolsarenumberedtocorrespondwiththerelevantstageoftheprocess.

EVALUAT IONOVERVIEW

STAG E 1

PreparefortheEvaluation

STAG E 2

PlantheEvaluation

STAG E 3

DetermineEvaluationIndicators

STAG E 4

CollectData

STAG E 5

AnalyseData

STAG E 6

ReportandImprove

STAGE1:PREPAREFORTHEEVALUAT ION

SETProjectaimsandobjectives

Whyarewe doingthisproject? Whatareourshort,mediumandlongtermaims?

Whataretheaimsforparticipants?

Whatareourorganisation’saims?

Defineyour‘community’targetforthisproject–thelocalcommunity?aparticulargroupinthecommunity–youngpeople?men?women?

Whataretheaimsforourcommunitytargetgroup?

Howwillwedotheproject?

I SSU E SLookatyouranswers–

Whatdotheymeanfortheevaluation?Haveyouplannedtheprojectwell?

Doyouhaveenoughresources(peopleincluded!) tomanagetheproject?Howimportantwilltheevaluationbe?

Investingabout10%ofaprojectbudgetonanevaluationisacommonpractice.

Whatsortofinvestmentcanyourorganisationrealisticallymake?

STAG E 1

CH ECK LI ST

MORE

INFORMATION?

Haveyou:definedaimsandobjectives fortheproject?D

consideredaimsandobjectives forparticipants?D

theproject/organisation?D

andthelocalcommunity?D

consideredtheprocess(howyouwilldotheproject);

theimpact(whatyou wouldliketohappeninthemediumterm)D

andtheoutcome(whatyouwouldliketohappeninthelongterm)?

Refer to:Stage1:Preparingforanevaluationin thefulltextversion

SampleTool1.1:Settingproject aims

[

STAGE2:PLANTHEEVALUATION

DEFI N Ethepurposeoftheevaluation

Whyarewe evaluatingthis project?Doweusuallyevaluateourprojects?Willthisevaluationbeanydifferent?How?Why?

DEFI N Etheaudiencefor theevaluation

Whowillwereportto?(afundingbody?projectpartners?thecommunity?ourboardofmanagement?)Dowehaveaninternalorexternalfocus?orboth?

Howcanweusetheevaluationresults?(Consider:futurefunding

applications;improvingpractice;keepingthisprojectontrack;feedbacktoparticipants…)

DETERMI NEtheprocessforevaluation

Howwillwemanagetheevaluation?Whatsuitablestructuresexist?Doweneedacommittee/group–oranindividual?

Willweneedaninternalorexternalevaluator? Whoisavailable;ableandcapableofmanagingtheevaluation?Are thereanyconflicts?

DETERMI NEtheresourcesavailablefortheevaluation

Whatresources(time;money;materials;equipment;records)can we

allocatetotheevaluation?Canweafford/justifyanexternalevaluator?

I SSU E SLook atyouranswers–

Areyouclearaboutwhyyouwantto dotheevaluation?

Areyouclearabouthowmuchtimeandeffortcanbeputintotheevaluation?Remember,ifyoureflectonyourpracticenow,youalreadyevaluate!

Whatarethepotentialbenefitsofanevaluation?e.g.anewfundingapplication,leading tomoremoney,moretime,morepeople?

Apartfrom money,whatarethepotential coststo consider?e.g.politicalissues?Canyouwriteoneevaluationreportandmodifyitfor variousaudiences?

Howcanyoubecreativewithresources?e.g.Aretherelocaluniversitystudentswilling/appropriate/able totakeontheevaluationfortheircoursework?

Howcanyouintegratestandardpractices?e.g.projectphotos?projectjournals?

STAG E 2:

CH ECK LI ST

NEED MORE

INFORMATION?

Haveyou:clarifiedthereasonsforundertakingtheproject?D

decidedhowtheevaluationwillbemanaged?D

decidedonthelevelofresourcestobeallocatedtotheevaluation?D

Refer toStage2:Planninganevaluationin thefulltextversion

SampleTools 2.1:Planningtheevaluation

2.2:Indicativeevaluationtimeline

STAGE 3: DETERMINE EVALUATION INDICATORS

DETERMI NEKeyevaluationindicators

Broadly,whatwilltellyouthattheaimsandobjectivesoftheprojecthavebeenachieved?(e.g.participantswererecruited fromthetargetgroup;

participantswereinvolvedinthecreativeprocess;weaimedto developaplayandwereachedplayreadingstage…)

Usingthemodelofevaluationinthis guide, itisrecommendedthatyoubreakdowntheprojectintokeyparts.So,firstlylistthestakeholders

involved–e.g.:

Then,breaktheprojectdownintoits partsandhowfaryouare‘thinking’

intothefutureandlistbelow.Forexample,processisparticularlyimportantforcommunityartsprojects.Doyoualsohavemediumorlongtermaims?

Then,thinkaboutwhatyouhopetoachieveforeachgroupofstakeholders

–bothintermsofdoingtheproject,andas aresultoftheprojectandputthetwopartstogether inagrid.Keepthissimple–forexample:

I SSU E SLookatyouranswers–

Areyouclearaboutwhatwilltellyouyouhaveachievedyouraims /doneagoodjob? Theseareyourevaluationindicators.

Evaluationindicatorsareatool–notanaim!Youraimistodothecommunityartsprojectwell–challenging;mentoring;supporting;skillingparticipantsin acreativeprocessthattheyenjoyandthattheycanbeproudof!

Effectiveevaluationindicatorsareatooltokeepyouontrack.Usethemtohelpyouarticulatetheachievementsoftheproject.

STAG E 3:

CH ECK LI STHave you:decidedonevaluationindicatorsfortheproject?D

NEED MO R EINFORMATION?

Referto Stage3:Determiningevaluationindicatorsinthefulltextversion(recommended)

SampleTools3.1:Determiningevaluationindicators

3.2–3.4:Sampleevaluationindicators

STAGE4:COLLECTDAT A

DESIG Ndatacollection methods

Whatarethekeyquestionstobeansweredthroughtheevaluation?Tip:turnyourevaluationindicatorsintoquestions.

Whatdata collection methodswillyouuse? (e.g.focus groupdiscussions;written surveys;structured interviews).Matchmethodstokey researchquestions;availabletime;resourcesandtothestakeholders involved.Thinkcreatively e.g.projectjournals;surveyswithpictograms;videos;photos

Whenwillyou collectdata?throughoutthe project?attheend?monthsafter

theprojectfinished? Listbelowhowoftenyou willcollectdata,fromwhom.

DESIG Ndata collectiontools

Whatdatacollectiontools doyouneedtodesign?e.g.surveyusing

pictograms;focusgroupoutline.Tip:designeacharoundyourkeyresearchquestions.

COLLECTevaluation data

Whatexistingdatacanbeusedintheevaluation?e.g.originalfundingapplication;waiting lists;projectarchives…

Whatissues doyouneedtoconsider incollectingthedata?e.g.research

ethics;keepingpeopleinformed;planningbestuseofeverybody’stime;storageofdata

I SSU E SLook atyouranswers–

The keytoefficientdatacollection isfocussed keyresearchquestions–limityourselfto aboutten.Whenthisbasicframeworkisright,designingsurveysorfocusgroupsto suitdifferentgroupsofpeopleisstraightforward.Don’ttrytocollecttoomuchinformation–makeitmanageable.

Considertheethicsoftheresearchbeforeaskingquestions.

Thinkaboutcollectinginformationefficientlyasyougo-itmayonlytake15

minsextraonrehearsalormeetingsfora discussionorto completeasurvey.

STAG E 4:

CH ECK LI ST

NEED MO R EINFORMATION?

Haveyou:consideredtheethicsoftheresearch?Didentifiedtheexistingrecordsandmaterialtobecollected? Didentifiedthenewinformationtobecollected? D

decidedonyourkeyresearchquestions?Ddecidedonthemostappropriateresearch methodsfortheevaluation? Dplannedhowoftenpeoplewillbesurveyed/interviewedetc? D

developedappropriatecommunicationstrategiesfortheproject?D

RefertoStage4: Collectingthedatain thefulltextversion

SampleTools4.1:Participantevaluationsurvey

4.2:Genericinterviewstructure/focusgroupoutline

4.3:Projectjournalguidelines

STAGE5:ANALYSEDAT A

ANALYSEevaluation data

Howcanyoureducethedataintomanageablechunks?Readthroughthe

dataandidentifythekeythemescomingthrough;usetablesandgraphsforquantitativeresponses

HowdoIcheckthedataagainsttheevaluationindicators?Testthedataagainsttheevaluationindicatorse.g.Iftheevaluationindicatoris:

Participantswererecruitedfromthetargetgroup–analysethedatatoseeifthisisthecase.

Whoshouldthedatabediscussedwith?Considerconfidentialityorothersensitivities.

Whatarethekeyfindingsatthisstage?

Whoshouldwereportdraftfindingsto?

I SSU E SLook atyouranswers–

Are youclearabouthowto reducethe dataintomeaningfulchunks?

Are youclearaboutwhoisinvolved incollating;reducing andanalysingdata?Considerconfidentialityissueswhenanalysingdata.

Considerwhetheryouwill beproducinganinternal orpublicdocument.

Ifappropriate,itisoften usefulto discussdraftfindingswitharangeofpeople,preferablyin agroup.Youcantesttheclarityofthereportwiththesepeople.Whatwillyoudoifthefindingsareeithernotverygoodormediocre?Howwillthisimpactonthe organisationoronindividuals?

STAG E 5:

CH ECK LI ST

NEED MO R EINFORMATION?

Haveyou:

collatedthedatayoucollectedintomanageableandlogicalchunks?D

checkedthedataagainsttheevaluationindicators?D

discussedthedatawithsomerelevantpeople?D

developeddraftkeyfindings?Dpresentedordiscussedthekey findingswithappropriatepeople? DRefer toStage5:Analysingthedatain thefulltextversion

SampleTool3.2–3.4:Sampleevaluationindicators

5.1Analysingevaluationdatatemplate

STAGE6:REPORTANDIMPROVE

R EPO RTevaluationfindings

Whatdidyoufindinrelationtotheprocess?(forparticipants;fortheproject,ortheorganisation;forthecommunity)

Whatdidyoufindinrelationtotheimpact oftheproject?(forparticipants;fortheproject,ortheorganisation;forthecommunity)

Whatdidyoufindinrelationtotheoutcomesoftheproject?(for

participants;fortheproject,ortheorganisation;forthecommunity)

I DENTI F Yprojectoutcomes

Whatdidweachieveinrelationtothisproject?Thinkaboutachievementsfromtherangeofstakeholderperspectives.

Whodoweneedtotellabouttheprojectoutcomesandachievements?

Thinkaboutparticipants;fundingbodies;thecommunity;partnershiporganisations

potentialimprovements

Whatdowe needto improve?

I SSU E SLook atyouranswers–

Afteranevaluation,youmayneed totake care thattheinformationisinterpretedappropriatelybyall–somestaffmayfeeltheirworkhasbeenhighlightednegatively.Ensurethatyou usethe evaluationas anopportunitytoturnthesituation aroundto ensureeverybodylearnsfromtheexperienceandimprovementsareimplemented.

Iftheevaluationhashighlightedareasofseriousconcern,e.g.underperformance;inappropriateperformanceetc–ensurethattheissuesarediscussedattheappropriatemanagementlevel oftheorganisationand yourdecisionsandstrategies are carefullythoughtthroughanddocumented.

Youalsoneed toplanhowyouimproveareasthatwereidentifiedasrequiring

improvement.andevento startthinkingaboutyournextevaluation!

STAG E 6:

CH ECK LI ST

NEED MO R EINFORMATION?

Haveyou:developedadraftevaluationreport?D

answeredthekeyresearchquestionsintheevaluationreport?D

collectedfeedbackonthedraft?Dfinalisedthereportandpreparedanexecutivesummary? Ddistributedtheexecutivesummaryto theappropriateparties? Dlinkedtheevaluationresultsintoanimprovementprocess? D

RefertoStage6:Reportingonthedataandimprovingcurrentpractice

inthefull textversion

SampleTools6.1:Sampleevaluationreport

6.2:Improvingcurrentpractice

STAGE1:PREPARINGFORANEVALUAT ION

INTRODUCTIO N

Thissectionisaboutpreparingthegroundforanevaluation.Itconcentratesonhowtosetaimsandobjectivesforthecommunityartsproject,andwhyyouneedtodothisbeforeattemptingtoevaluatetheproject.

WHY EV ALUATE I N THE FIRST PLACE?

Thereisenormousvaluetobegainedbycommunityartspractitionersthroughevaluatingcommunityartsprojects.Anevaluationprovidestheopportunityforallparticipantstomeasureandcomparewheretheywerebeforetheprojectstarted,andwheretheyreachedbythe

project’send.Evaluationresultsalsoprovidevaluableinformationforprojectstaff,includingcreativestaff,volunteers,mentorsandsoon.

Evaluationinformationcanbeusefulacrossarangeofareas.Forexample,aprojectevaluationcanhelpfeedintotheorganisation’sownplanningandevaluationprocesses.Itcanprovide

usefulinformationforfuturefundingapplications.Itcanprovideausefulrecordformeasuringprogresswithcommunityartsprojects,andhelpingtodevelopandextendtheworkofthe

organisation.Theinformationcanhelpshapefuturepromotionandrecruitmentstrategiesfornewprojects.

Onabroaderscale,whencommunityartsprojectsevaluateanddocumenttheirachievements,thecommunityartssectorbeginstoaccumulateabodyofevidencethatprovidesmeaningful

informationaboutthecontributionofthesectortothewidercommunity.Ontheotherhand,iftheoutcomesofcommunityartsprojectsarenotdocumentedwithrigourandconsistency,the

sectorisleftwithoutarecordofitshistoryorachievements.Inaddition,projectevaluationscontributetothecollectivepracticewisdomofthesector.Evaluationresultscanprovide

valuablelessonsandinsightsforpractitionersinotherplaces,neworganisationsandsoon.Buildingsector-wideinformationalsosupportsopportunitiesforcross-sectorwork.Themorethatisknownaboutcommunityartsanditswork, thegreater theopportunitiestoform

partnershipsandlinkswith,forexample,theyouthsector;thehealthsector;agedcaresectorandsoon.

Oneofthekeyaimsofthismanualistothereforesupporttheundertakingofevaluationsinthecommunityartssector,usingaconsistentandstructuredresearchframework.Thisinitselfwillcontributetothesectordevelopingabodyofevidencewhichcanhelpwiththesector-wideandcross-sectordevelopments.Thetablebelowprovidesasummaryofthebenefitsofcommunityartsprojectsevaluations.

SOME BENE F ITS O F COMMUNI T Y ART S PROJECT EVAL UATIO NS

PRACTICAL ISSUE S:

If you don’t know where you’re going, how will you know when you get there?

Evaluationisnotsomethingtothinkaboutattheendoftheproject. Itneedstobeconsideredatthesametimeasplanningtheproject.Havingprojectaimsandobjectivesisfundamentalto

beinginapositiontoevaluatetheproject.As theheadingsays,ifyoudon’tknowwhereyou’regoing,howwillyouknowwhenyougetthere?Settingaimsfortheprojectprovidesagoalto

worktowards,anditprovidesabenchmarkthatyoucanuseatthecompletionoftheprojecttomeasurewhetheryouachievedwhatyousetouttoachieve.Ifyoudidn’t,youhavethechancetoreflectonthat–

•whydidn’tyouachieveyourgoal?

•didyouachievesomethingunanticipated?

•didyouhaveunrealisticexpectations?

•didsomethingoutsideyourcontrolpreventyou achievingit?

•didyoufailormismanagetheprojectinsomeway?

•andifso,howcanthisbeavoidedinthefuture?

Insettingprojectaimsandobjectives,itisusefultoconsideralltheelementsoftheproject,andfromtheperspectiveofthemainstakeholders.Thetablebelowsummarisesthese

considerations.

TOOLS TO USE

Toassistyouinsettingprojectaimsfor:

•theprocess–from theperspectiveofparticipants;theprojectandthecommunity;

•theimpact–from theperspectiveofparticipants;theprojectandthecommunityand

•theoutcome–fromtheperspectiveofparticipants;theprojectandthecommunity.refertoSampletool1.1:Settingprojectaims.

CH ECK LIST:

Haveyou:

•definedaims andobjectives fortheproject?D

•consideredaimsandobjectives forparticipants;theproject/organisationandtheD

localcommunity?

•consideredtheprocess;theimpactandtheoutcomes?D

STAGE2:PL ANNINGTHEEVALUATION

INTRODUCTIO N

Nowyou’vedecidedtoevaluateyourcommunityartsproject,thissectionhelpyouplanthesteps.Thisisprobablythemostimportantphaseofyourwholeevaluation,anditisworthspendingthetimeplanningtheprocessproperly.

I SSUES TO CO NSIDER

Therearenumerousissuestoconsiderintheplanningphase.Yourorganisationwillneedtobeclearonissuessuchasthefollowing:

•whyare weundertakingtheevaluation?

•whowillmanagetheevaluation?

•whatresourcescanbeallocated?

•howlongwilltheevaluationtake?

•howwilltheevaluationworkin withtheactualproject?

•howwilltheevaluationintegratewith othersystemsin theorganisation?

•whowilltheevaluatorreportto? when?howoften?

•howwillthefollowingpeoplebeinvolvedintheevaluation-participants?staff?volunteers?thecommunity?audiences?

•whatissuesdoIneedtoconsiderforparticipants,e.g.confidentiality;unwillingnessto

participate;language,literacyorothercommunicationissues;privacyissues?

•whatdoes theorganisationplantodowiththeevaluationreport?

Thereisscopetoundertaketheevaluationusinganumberofdifferentapproaches.However,beforeproceedingthefollowingshouldbeclear:

•thereasonforundertakingtheevaluation

•themanagementprocess

•theresourcesavailabletotheevaluation

Typically,thesedecisionswillbedeterminedbyfactorssuchas:

•theimportanceofundertakingtheevaluationtotheorganisation;

•availableresources:time;money;staff;infrastructure;

•capacitytocopewithinterruptionsduringthecommunityarts project,and

•thereactionofparticipantstoanevaluation.

Withclarityontheaboveissues,theorganisationcandeterminethescopeoftheevaluation.Itisimportanttorememberthattherearelevelsofevaluation.Theycanrangefromacomprehensive,independentevaluationtoanevaluationconductedusingin-houseresources.

Thismanualprovidesthetoolsfororganisationstoconducttheirownevaluations,andtoensurethatthecostofundertakinganevaluationisnotprohibitive.

Followingisanexampleoftherationaleforevaluatingapuppetryproject,forasmallcommunitybasedorganisationandtheprocesstobeused.

•thereasonforundertakingtheevaluation

Forexample:

Weareevaluatingthepuppetryproject,tofindoutifwearemeetingouraimofengagingyoungpeoplewithdisabilitiesintheoutersuburbs.Weneedtoidentifyifweareusingthemost

effectivemethodsofrecruitment,andifthepartnershipwithNGO123isworkingwell.Wealsowanttoknowifthereisscopetofurtherdevelopthisworkforthestatewidepuppetryfestivaltobeheldintwoyears.

•themanagementprocess

Forexample:

TheBoardhascommissionedtheevaluation,andit takesoverallresponsibilityforthe

evaluation.ThePerformanceArtsSub-Committeewillactasthesteeringcommitteefortheproject,andwillreportregularlytotheboard.JaneSmithwilltakedaytodaymanagement oftheevaluationandwillreportregularlytothesteeringcommittee,andwillpresentthedraft

evaluationreporttotheBoard.Anyissuesordifficultiesshouldberaisedwiththesteeringcommittee.

•theresourcesallocated

Forexample:

JaneSmithwillworkanaverageoftwodaysperweekontheevaluationoverthecourseoftheproject,andfull-timeinthefinaltwoweeks,inordertopreparethereport.Janewillbeabletoundertakeinterviewsanddiscussionswithpeopleduringworktime,andothertimewillbetimeinlieu.Janeistoliaisewiththecreativestaff,inordertoplanaccesstoparticipantsetc.

TOOLS TO USE

Toassistindeterminingthescopeoftheevaluation,aworksheetisprovidedinthetools

section.Itisimportanttoremember, thataneffectiveevaluationcanbeconducted,evenwithlimitedresourcesandtheopportunitiesforreflectionandimprovementshouldnotbe

overlooked.

refer:Sampletool 2.1:Scopingtheevaluation.

Asampleworkplanisalsoprovidedforaprojectofapproximatelythreemonthsduration.Thetasksandtimelinescouldbeadjustedtosuitthescopeofspecificevaluations.

refer:Sampletool 2.2:Indicativeevaluationtimeline

CH ECK LIST:

Haveyou:

•clarifiedthereasonsforundertakingtheproject?D

•decidedhowtheevaluationwillbemanaged?D

•decidedonthelevelofresourcesthatcanbeallocatedtotheevaluation?D

STAGE3:DETERMINING EVALUATIONINDICATORS

INTRODUCTIO N

Determiningindicatorstomonitorandevaluatethecommunityartsprojectispartofthe

preparingandplanningphaseoftheevaluation.Thissectionisdedicatedtothistask,because

thisissuchacriticalpartofthewholeevaluation.Sampleindicatorsareprovidedforyoutouse;selectfrom,oradapttosuittheneedsoftheparticularprojectbeingevaluated.Theindicators

focusonthequalitativeaspectsofcommunityartsprojectswhichcanbedifficulttoidentifyandmeasure.

WHAT ARE EVALUATION INDICATO R S?

Inresearchterms,anindicatorprovides“datathat measuresorindicatessuccessorfailureinmeetingyourobjectives”1.Anevaluationindicator, therefore,providesdatathatwilltellyou,anddemonstratetoothers,thatyouareachieving,orhaveachievedtheproject’saims.Evaluationindicatorsaresimplyatoolin theevaluationprocess.

Forexample,imagineyourproject’saimis:

Toestablishachoirforsociallyisolatedolderwomeninourcommunitywhichprovidesopportunitiesforparticipantstobeinvolvedinculturalactivitiesinasocialsetting

Whatwilltellyou,anddemonstratetoothersthatyouachievedthis?Toidentifyyourindicators,breakdownyouraimsandobjectivesintotheircomponentparts.Whatwouldindicatetoyou,ortosomeoneelse,(anothercommunityorganisation;afundingbody;alocalcouncil;alocal

newspaper…)thattheprojectachieveditsaimsandobjectives.Thefollowingtableprovidesanexampleofthisprocess.

1AnEvaluationResourceforHealthyLivingCentresJayneMeyrickandPaigeSinkler,HealthEducationAuthorityp. 10

WHAT EVALUATI ON INDIC ATOR S C AN BE U SED FOR C OMMUNITY ARTSPROJE CTS?

Whendevelopingevaluationindicatorsforcommunityartsprojects,youwillneedtoreturntotheaimsandobjectivesyousetfor:

•theprocess–from theperspectiveofparticipants;theprojectandthecommunity;

•theimpact–from theperspectiveofparticipants;theprojectandthecommunityand

•theoutcome–fromtheperspectiveofparticipants;theprojectandthecommunity.

Whatwereyouraimsforeachoftheabove?Then,youneedtoconsiderWhatwilltellyou(andothers)thattheprojectachievedtheseaims?

Mostcommunityartsprojectswouldhope tobeable toshowthatthefollowingoccurred,asaminimum.

AND WH E N THE EX TRAORDINARY HAPPE NS ?

Communityartspractitionersemphasisethatattimes,projectscanachieveextraordinaryresults

–eitherinthecreativeandartisticfield,and/orforparticipants,and/orinthelocalcommunity. Asampleofevaluationindicatorsisincludedinthetoolssectionwhichyoucanusetoevaluate

whetherminimumlevelsofsuccesshavebeenexceeded.Itisimportantthatyoucanidentifyandarticulatetheexceptionalsuccessesthatoccur. Ontheotherhand,itisalsoimportantforthesakeofparticipants(andstaff)thatrealisticaimsandobjectivesaresetfortheproject.

PRACTICAL ISS UE S

Yourtasknowistodeterminewhatevaluationindicatorsyouwillapplytogatherdatathat

measuresorindicatessuccessorfailureinmeetingyourobjectivesfortheparticularproject.Tosetappropriateindicators,youneedtobeclearabout:

•thepurposeandscopeoftheevaluation,and

•theaimsandobjectivesoftheproject.

TO O LS TO USE:

Toassistyouindeterminingtheevaluationindicators,refertoSampletool3.1:Determiningevaluationindicators.

Toassistindeterminingifminimumlevelsofachievementhavebeenexceeded,refertoSampletools3.2–3.4:Sampleevaluationindicators.Thistoolprovidesacomprehensiverangeofsampleevaluationindicators,for:

•theprocess–from theperspectiveofparticipants;theprojectandthecommunity;

•theimpact–from theperspectiveofparticipants;theprojectandthecommunityand

•theoutcome–fromtheperspectiveofparticipants;theprojectandthecommunity.Thesampleevaluationindicatorshavebeendevelopedinthefollowingthreelevels:

•minimumlevelofsuccessthis(orappropriateequivalent)indicateswhatshouldhappen,

asaminimum-alarmbellsshouldringifthis(orequivalent)isnotoccurring/didnotoccur

•reasonablysuccessfulindicatesthatmorethantheminimumhasoccurred

•ideal/extremelypositiveindicatesthatsomethingoutstandinghasoccurred.

Youcaneitherusetheseindicatorsdirectly,orusethemasaspringboardtodevelopindicatorsforyourownproject.

CH ECK LIST:

Haveyou:

•decidedonthemostappropriateevaluationindicators fortheproject?D

STAGE4:COLLECTINGTHEDAT A

INTRODUCTIO N

Thissectionisaboutcollectingthedataforyourevaluation.Itprovidesadviceonplanninghow

tocollect datafortheevaluation;discussesissuesassociatedwithcollectinginformationfortheevaluationanddescribessomeofthemethodsyoucanusetocollectinformation.

PLANN ING D ATA COLLEC TION

Beforecollectinganydatafortheevaluation,itisstronglyadvisedthatyouinvestsometimeandeffortplanningthisprocess.Whileyouwillanalysesomerecordsandwrittenmaterial,mostofthedatayoucollectwillcomefrompeople–participants;volunteers;committeemembers;

staff;audiencesandsoon.Youwillbeaskingpeopletoallocatesometimetothisresearch,forexample,bycompletingsurveysorparticipatinginafocusgroup.Mostpeoplearewillingtodothis,butonlyiftheycanseethattheiropinions,timeandcontributionisvalued.

Thefundamentalpurposeofanevaluationistofindoutiftheoriginalaimsandobjectivesoftheprojectarebeingachieved.Evaluationscanbebroaderthanthis,andlongterm,ortheycanbemuchmorefocussedandshortterm.Onceagain,itisvitaltoemphasisethatyouneedtobeclearaboutwhyyouaredoingtheevaluationbeforeyoustartcollectingdata.Iftheaudienceisinternal,andthepurposeistoidentifyimprovementsinartspractice,youwilldesignyourdatacollectioninaparticularway.However,ifyouareconductingtheevaluationforanexternal

audience–perhapsyouthinktheorganisationisworkinginparticularlyinnovativewayandyouwanttodemonstratethattotheoutsideworld,oryouaretryingtointerestotherfundingsources,youwilldesigntheresearchwithdifferentemphases.

Whenyouareclearaboutthepurpose,scopeandaudiencefortheevaluation,youcanidentifythedatayouwillneedandplanhowyouwillcollectit.

PRACTICAL ISS UE S

Tostartwith,youneedtoconsiderthefollowing:

WhatexistinginformationdoIneedtocollect?

Don’toverlookexistingreports,recordsanddatainyourevaluation.Thisinformationcanoftenprovideausefulcontexttotheresearch,andyoushouldhaveeasyaccesstoitwithinyour

organisationornetworks.

Existinginformationofusetotheevaluationcouldinclude:

•previousevaluations(yourorganisation,orothersnearby)

•localgovernmentreportsprofilingthelocalcommunity

•ifyouare workingwithaspecifictargetgroup,reports fromappropriatecommunityoradvocacygroups

•internalrecords–numbersofpeopleinvolvedin previousprojects;recordsofwaitinglists;audiencenumbers;numbersofvolunteers,staffetcinvolved

•projectrecords–e.g.projectcommitteeminutes;rehearsal/attendancerecords

•Whatotherrecordswillbeusefulfortheevaluation?

WhatnewinformationdoIneedcollect?

Toanswer theabove,youneedtobeclearaboutwho(orwhat)cantellyouthattheprojectisachieving,orachieveditsaims?

Irrespectiveofthepurpose,scopeoraudience–allevaluationsshouldincludeproject

participants.Participantsarethekeygroupabletoletyouknowiftheprojectisworking,

whetheritwasappropriatelydesigned,whetheritwasenjoyable/interesting/challengingtobeinvolved;whethertheywouldevercontemplategettinginvolvedagain;howbeinginvolved

affectedtheirlives;whattheylearnt;whattheydidn’tlike,suggestionsforimprovementsandsoon.

Dependingonthepurpose,scopeoraudiencefortheevaluation,youwillalsoneedtocollectinformationfromsomeorallofthefollowing:

•staff–project,organisation

•volunteers

•partner organisations

•audiences

Ifyouwishtoconductacomprehensiveevaluation,youmayalsowanttocollectinformationfrom:

•localcommunityorganisations,communityleaders

•other artsorganisations

•relevantestablishedartists,practitioners

•otherpeopleconnectedtotheparticipantswiththepermissionoftheparticipants–seeresearchethicsbelow.

•Isthereanyoneelsewhocanlegitimatelytellyouwhethertheprojectisachievingitsaimsandobjectives?

WhatquestionsdoIneedtoask?

Developingasetofappropriateresearchquestionsisacriticalaspectofthedatacollection.Throughtheresearchyouwillwanttoaddressbroadquestions,suchasthefollowing:

•Weretheprojectaimsandobjectivesachieved?

•Howweretheyachieved?

•Weretheyachievedforallinvolved?

•Whatweretheoutstandingachievementsoftheproject?

•Howwellwastheproject,andtheprocess,managed?

•Whatlessons werelearnt?

•What wastheexperiencelikeforparticipants,theorganisationandthecommunity?

•Whatweretheimpactsforparticipants,theorganisationandthecommunity?

•Howdidtheprojectcontributetoindividuals,partnershiporganisations,thecommunity?

•Whatunanticipatedoutcomesoccurred(positiveornegative)?

Asthismanualisspecificallydesignedtohelpcommunityartspractitionersevaluateand

articulatethepositiveimpacttheartscanhaveonindividualandcommunitywell-being,itisimportantthattheevaluationseeksthisinformation.

•Arethereanyotherkeyresearchquestionswhichshouldbeincluded?

Themosteffectiveapproachistodevelopasetofkeyresearchquestions,suchastheabove.Thesebroadquestionscanthenbetailoredtosuitthevariouspeopleyouspeaktothroughtheresearch.

Whenyou’reaskingayoungparticipantquestions,youwillneedtophraseyourquestions

differentlyanduseadifferentstyletothetypeandstyleofquestioningyouusewithstafffrompartnerorganisations.

Tofindoutwhatlessonswerelearntthroughtheprocess,youmightaskayoungpersonquestionslike:

Ifyourbestfriendsaidtheywantedtogetinvolvedinasimilarproject,whatadvicewouldyougivethem?Iftheprojectofficeraskedforyouradviceonplanninganotherproject,whatwouldyousay?

Whereasyoumightaskapartnerorganisationquestionslike:Howwelldoyouthinkweworkedtogether?Whatcouldwedodifferentlyandbetter?

Whataboutresearch ethics?

Itisimperativethatyouareclearabouttheethicsofyourresearchbeforeyoustartcollectinginformation.

Forexample,

•Youmusttellparticipantsthatyouareconductinganevaluationandyoushouldexplainthepurpose,scopeandaudienceforthereport.

•Youmustexplaintheimplicationsoftheresearch–forexample,Willphotosbetaken?Whowillthefinalreportbesent to?Willtheevaluationbeapublicdocument?Canpeopleprovidecommentsanonymously?Canpeopledropoutoftheproject?theevaluation?

•Ifparticipantsareuncomfortable,youmustprovideopportunitiestodiscuss thisbeforetheresearch,andyoumustgivethemchoicesaboutcontinuing,and/ordiscuss waystheresearchcouldbestbeconducted.

•Youmustaskpermissiontoquote,photographorotherwiseidentifyaparticipantinareport.

•Youmustaskpermissionoftheparticipantifyouwishtotalktoanother personaboutthem–forexample,youmaywishtoseekacommentfromschool,family,etcabouttheimpactoftheprojectandifrefusedyoumustnotseekcommentsfromthosesources.

•Youshouldprovidegenuine,meaningful,appropriateandnon-threateningopportunitiestoparticipate.Forexample,itisnotethicaltoexpectagroupofparticipantsfrom anotherlanguagebackgroundtocompleteawrittensurveyinEnglish,andthentoclaimparticipants

wereconsulted.Norisitethicaltoonlyprovideopportunitiesforpeopletoprovideinformationinagroupdiscussionifpeopleare shyorhaveanydifficultiescommunicatingin thatsetting.

•Youshoulddecidewhathappenstothedata,aftertheproject.Youmayhavealotof

informationincludingconfidentialinformationandparticipantsareentitledtoknowwhatwillhappentorecordings;photographsandotherrecords.Optionsinclude:theorganisation

lookingaftertherecords,asthey wouldotherconfidentialrecordsforaspecifiedperiodof

time;destroyingtherecordsaftertheresearch;offeringtherecordstoparticipants.Themostimportantconsiderationisthattheprivacyandconfidentialityofparticipantsisrespectedandmaintained.

HowwillIcollecttheinformation?

Whenyouareclearabouttheethicsoftheresearch,youneedtodeterminethemostappropriatemethodsforconsultingthepeopleinvolved.

Standardinformationcollectiontechniquesinclude:

•writtensurveys–whichcanincludeopenandclosedquestions;useofratingscales;useofpictogramsetc

•focusgroupdiscussions

•structured,orsemi-structuredinterviews–tapedornottaped

•exitpolls

Mostevaluationsincludesomelevelofsurveyingandinterviewing.Somesamplesurveysandinterviewsareincludedinthesampletoolssection.Thesehavebeenstructuredaroundthekeyresearchquestions,usingbothclosedandopenquestions.Ifyouhaveadditionalkeyresearchquestions,youwillneedtoadaptthetoolsaccordingly.

Inordertodecideifthesestandardmethodsareappropriateforthepeopleparticipatingintheevaluation,orwhetheryouneedtosupplementthesemethods,thinkaboutissuessuchas:

•Arethereliteracyissues?IsEnglishtheparticipant’sfirstlanguage?Areparticipants

comfortablereadingandwriting?Howlongshouldawrittensurveybe?Willpeopleneedhelptocompleteit?Willparticipantsthinkitistoodaunting,likeworkorhomework?Remember–literacyproblemsarequitecommon,youcan’tassumethatpeoplecanreadandwrite

confidently.

•Areparticipantscomfortablespeakinginagroup?Whatare thegroupdynamicslike?Doyouneedaninterpreter?Whatsortofenvironmentwillencouragediscussion?Howwillyoudealwithpeopledominatingthediscussionornotparticipating?

•Willparticipantsfeeluncomfortableinaone-to-oneinterview?Can someoneelsebepresent?

•Howcanparticipantstimebestbeused?

Ifitisnotappropriateforpeopletoparticipateusingconventionalresearchtools,arangeof

broader,morecreativemethodsarealsoappropriate,particularlywithnewtechnologies,suchasscanningimagesintocomputersorvideorecording.

•Youcanuseaprojectdiary orscrapbook–withclearguidelinesonwhocanmakeentries,howitistiedtothecreativeprocess;whentouseetc.

•Youcanusephotographs–withparticipants’permission.Photoscanbeofthecreativeworkand/orofparticipantsduringtheprojectandcanbeusedasevidencetosupportkeyfindings

•Youcanusedrawings,orothervisualrecords.

•Youcan audioorvideorecordings–withparticipant’spermission.

Whatisthedifferencebetweenquantitativeandqualitativedata?

Quantitativedatausuallyanswers‘howmany/muchquestions?’Itprovidesyouwiththenumbersabouttheproject–forexample:

PROJECTHowmanyparticipants?

Howmuchmoney–wasreceived?wasspent?

PROCESSHowmany–rehearsals;workshops;sessionswereheld?Howmanyevents–festivaldays;productions;exhibitions?

COMMUNITYHowmanyaudiencemembers?

Howmanycommunitymembersonthemanagementcommittee?

QualitativedataanswerstheWhy?How?Inwhatway…?Towhatextent…?questionsofa

project.Itisparticularlygoodforunderstandingwhycertainthingsdidordidnothappenwithin

theproject.Thisinformationwhichcannotbequantifiedlikequantitativedata,suchaswrittenorspokencomments.Themescanbedrawnoutandthiscanalsoprovideexcellentsourceofcasestudies,whichcanbeverypowerful.

Bothtypesofdatahaveadvantagesanddisadvantages.Quantitativedatacanbequitedry–youmaystillwanttoknowwhypeopleansweredinaparticularway.Ontheotherhand,itcanberelativelyeasytocollect;youcangatheralotofinformationfromalotofpeople;youcanmakecomparisonsandyoucanidentifystatisticallysignificantvariations,ifyouwishtogotothatlevelofanalysis.Qualitativedataprovidesyouwithlotsofquotes,ideasandstories–butyoucan endupwithalotofinformationandyouwillstillneedtoextractthekeythemes.Andyouwillstillneedthenumbers!

Bothqualitativeandquantitativedataareneededtoundertakeabalancedandeffectiveevaluation.2

•Askyourself,arethereanyotherappropriateresearchmethodsneededfortheevaluation?

Howoften willIcollectinformation?

Aswithallaspectsoftheresearchdesign,thisquestioncanonlybeansweredinthecontextofyourspecificevaluation.Asaminimum,youwillneedtocollectinformationfromparticipantsatleastonce,atthecompletionoftheproject.Atthatpoint,participantscanmakesomecommentabouttheprojectandtheirexperienceinbeinginvolved.Thiswouldproduceafairlylimited

evaluationreport.However,itmaybeallthatresourcesallow,andallthatyouneedtodo.Ifthisisthecase,makesureyouasktherightquestions,becauseyou’veonlygotonechance!

2ref: Hawe,Penelope;Degeling,DandHall,J(1990)EvaluatingHealthPromotion:Ahealthworkers’guideSydney:MaclennonandPettyPublishing(ref:Chapter6ImpactandOutcomeEvaluation)p.105

Ifyouareabletoaskquestionsofparticipantsatthebeginningandtheendoftheprocess,youareimmediatelyinastrongerpositionbecauseyoucancompareparticipants’perceptionsandviewsfromthebeginningtotheend.Similarly,withstaff,mentors,artists,volunteers,itisusefultoaskbeforeandafterquestions.Thisispreferabletoaskingpeopletorecallthetimebefore

theprojectstarted. Youwillhavesomedatathatclearlyindicatesthedifferencemade(ornotmade)by theproject.

If therewassupportforalongertermevaluation,youcouldarrangetofollowupparticipantsaftersometimehadelapsed.Thisallowsyoutofindoutfromparticipantstheinfluenceofthe

projectontheirlivesoveralongertimeframe. Thisiseasierifparticipantsarelikelytoreturntoyourorganisation–maybeit’sayouthcentreofsomesort,andpeoplekeepingcomingback.

Trackingparticipantsprovidesreallyusefulandinterestinginformationabouttheinfluenceoftheproject,howeverthiscanbecostlyandtimeconsumingthereforeyouneedtoconsiderthecostbenefitsofthistoo.

HowdoIletpeopleknowabouttheevaluationandencouragethemtoparticipate?

Ideally,youwillhavedecidedonyourresearchapproach,atleastinbroadtermsevenbefore

theprojectstarts.Thisway,youcantalktoparticipantsandothersatthecommencementoftheresearch.Youcandiscussandevennegotiatethebestresearchstrategies,andyoucan

encouragepeopletobeinvolvedfromtheoutset.Whenyouareclearaboutthepurposeoftheevaluation,youcan‘sell’thebenefitstoparticipants–

•Youcanhavearecordofthereportfor…school,youremployer,application totrainingcourseetc

•We’rehopingtousetheresultsinournextfundingapplication

•We’rehopingtoencouragethelocalyouthcentretocomeonboard

•We’rehopingtoencouragethedisabilityprogramtocontributesome funds

•We’rehopingtousetheresultstoconvincetheboardweneedafull-time worker

•We’rehopingtousetheresultsforanarticlein thelocalpapertoencouragemoreVietnamesewomentobecomeinvolvedinourgroup

•We’rehopingtousetheresultstoapplyformentoring funds

•We’rehopingtousetheresultstoseeifwecouldbecomeatrainingprovider….

Therearepotentially significantbenefits forparticipants aswellas organisations.Ifparticipantsare:

•informedaboutthepurposeoftheevaluation;

•consultedabouttheresearchmethods;

•givenadequatenoticeofactivitiesinvolvingthem–forthcomingsurveys,interviewsetc;

•involvedintheresearchdesign;

•providedwithchoicesabouthowtocontribute;

•consultedinnon-threateningandrespectfulways,and

•informedaboutprogress alongthewayandoncompletion

youwillfindthatmostpeoplearemorethanwillingtobeinvolved.

Remember–youneedtoconsiderwhoneedstoknowabouttheevaluationandthebestwaystokeeptheminformed.Otherorganisationsinyournetworksmaybeinterested;othersectionsofyourownorganisation;newcommunityartsgroups;establishedcommunityartsgroups.Itisusefulatthestartoftheevaluationtoprepareaninformationsheetwithashortoutlineoftheevaluation,whichyoucandistributeas required.

CO LLECTING THE I NFO RMATIO N

Whenyouhavecompletedtheaboveplanningandpreparation,youneedtofinaliseyourresearchtools.Thereisarangeofsampleresearchtoolsincludedwiththemanualforyouruse.Thesesurveys,focusgroupoutlines,interviewsanddiaryguidelinescanbeuseddirectlywithsomeminoridentifyinginformationfromyourorganisation,ortheycanbeadaptedtosuitthe

specificsoftheproject.

Otherissuesyouneedtoconsiderare:

•Howmuchinformationisenough?

•Howmanypeoplewillyousurvey?–everyone?randomselection?

•Howwillyousurvey them?–fillitinatrehearsal?mailit totheirhomeaddress?leavecopiesaround?

•Howoftenwillyousurvey them?beginning?middle?end?

•Whowillbeinvitedtofocusgroupdiscussions?

•When?

•Whowillfacilitate?

•Whowilltakenotes?

•Theroleoftheevaluator–whoevaluates?howaretheirobservationsrecordedandcollected?

TOOLS TO USE

Sampletools4.1:Participantevaluationsurvey

4.2:Genericinterviewstructure/focusgroupoutline4.3:Projectjournalguidelines

CH ECKLIST:

Haveyou:

•consideredtheethicsoftheresearch?D

•identifiedtheexistingrecordsandmaterialtobecollected?D

•identifiedthenewinformationtobecollected?D

•decidedonyourkeyresearchquestions?D

•decidedonthemostappropriateresearchmethodsfortheevaluation?D

•plannedhowoftenpeoplewillbesurveyed/interviewedetc?D

•developed themostappropriatecommunicationstrategies fortheproject?D

STAGE5:ANALYSINGTHEDAT A

INTRODUCTIO N

Stage5providessomeguidelinestohelpyoureducetheamountofinformationtoamanageableamountandthen,tomakesomesenseofit.

REDUC ING THE D ATA

Oncetheinformationfortheevaluationhasbeencollected,youwillneedtostartgroupingandcompressingtheinformation.Forexample,youwillneedto:

•collatesurvey responses (manuallyorbydataprocessingforlargenumbers)

•analysethewrittenresponses toopen-endedquestions

•identifykeythemesfromwrittensurveys;focusgroupsand/orinterviews

•collatetheinformationgatheredfrom existingmaterial

Inordertomakesenseofthedata,itneedstobeorganisedintoamanageablesizeand

structure.Useagridofthekeyfactorsbeingevaluatedandrecordtheresearchresultsthere.SeeSampleTool5.1AnalysingEvaluationDatatemplate.

ANALYSING THE DATA

Onceyouhaveorderedthedata,youneedtoanalyseit:whatdoesthisinformationmean?Forexample:

•Whatdoesittellusabouttheproject?

•Whatdoesittellusaboutwhathappenedthroughtheproject–toparticipants?theorganisation?thecommunity?

Toanalysethedata,youneedtoapplyyourevaluationindicatorsandtheninterprettheresults.Forexample,thefollowingareprovidedasminimumlevelevaluationindicatorsforparticipants(seeStage3).Toanalyseyourinformation,youneedtoturntheindicatorsaroundandask

questionssuchasDidwefindanyevidenceofthishappening?

outcome

Workingthroughthisprocess,youwillidentifythekeythemesandfindingsoftheresearch.Itisusefultodothisasagroup,orwithsomeoneelse. Youwillfindthatyouneedtodiscussand

thinkthroughsomeoftheissues,toreallygettothecoreinformation.Itisworthtalkingaboutyourinterpretationstogether,andtestingeachother’sthinking.Bewareofissuessuchas:

•jumpingtoconclusionswithoutsupportingevidence

•notwantingtoincludeoridentifynegativefindings

•notwantingtoincludeoridentifymediocrefindings

Onceyouhaveworkedthroughandanalysedthedata,itisworthsummarisingthistoasetofkeyfindings.Developthisasadrafttousefordiscussions–forexample,withtheevaluationcommitteeifthereisone;managers;participants;partnershiporganisationsandsoon.Buildinginastepintheprocessaboutdiscussionofkey findingsprovidesavaluableopportunityfor

furtherrefinementofyourthinkingandpresentationofthedata.

TOOLS TO USE

Toassistinanalysingthedata,returntotheevaluationindicators.Workthroughthequestionsintheexampleabove,toaskyourselfwhatevidenceyoucollected.Remember,youwillonlyneedtousetheevaluationindicatorsappropriatetotheparticularevaluation.

Sampletools3.2–3.4:Sampleevaluationindicators

5.1:Analysingevaluationdatatemplate

CH ECK LIST:

Haveyou:

•collatedthedatayoucollectedintomanageableandlogicalchunks?D

•checkedthedataagainsttheevaluationindicators?D

•discussedthedatawithsomerelevantpeople?D

•developeddraftkeyfindings?D

•presentedordiscussedthekey findingswithappropriatepeople?D

STAGE6:REPORTINGONTHEDATAANDIMPROVING ON CURRENT PRACT ICE

INTRODUCTIO N

Thissectionprovidesguidelinesonreportingontheevaluation.

HOW DO YOU P REP ARE AN EV ALUATIO N REPORT?

Essentially,theevaluationreportwillwanttoansweryourkeyresearchquestions.Theexamplesprovidedare:

•Weretheprojectaimsandobjectivesachieved?

•Howweretheyachieved?

•Weretheyachievedforallinvolved?

•Whatweretheoutstandingachievementsoftheproject?

•Howwellwastheproject,andtheprocess,managed?

•Whatlessons werelearnt?

•What wastheexperiencelikeforparticipants,theorganisationandthecommunity?

•Whatweretheimpactsforparticipants,theorganisationandthecommunity?

•Howdidtheprojectcontributetoindividuals,partnershiporganisations,thecommunity?

•Whatunanticipatedoutcomesoccurred(positiveornegative)?

Howyoustructure,writeandpresentyourreportwilldependontheevaluationaudience.Ifthereportistobedistributedoutsideyourorganisation,itshouldbeformalandwell-presented.Ifitisforaninternalaudience,itcanbemuchshorter–youwon’tneedtoprovideasmuchcontextandbackground–andmorerelaxed.

Aswiththedraftfindings,itisusefultoprepareanddistributethereportindraftformfirst.Thisgivesopportunitiesforpeopletocontributetotheprocess;tocorrectinaccuracies;toimproveexpressionandpresentationandtorefinetheanalysis.

WHAT C AN WE LEAR N FROM AN EVALUATION?

Arigorouslyconductedevaluationwillprovidetheorganisationwithawealthofinformation.Ensurethattight,achievableandconstructiverecommendationsaredevelopedfromthereport.

Considertherangeofpeoplewhomaybeinterestedinthefindingsoftheevaluation–other

organisations;recruits;fundingbodies;philanthropicorganisations;otherprogramareasandsoon.Ifyouthinkarangeofpeoplewillbeinterested,itisusefultoprepareanexecutive

summary.Thisshouldincludesomebackgroundinformation,thekeyfindingsand

recommendations.Theexecutivesummarycanthenbedistributedtointerestedparties,andthefullreportcanbeavailableifrequested.

Ensurethataprocessisdevelopedtoconsultfurtherontherecommendations,ifnecessary. Ifthereisclearsupport,developaprocesstoimplementtherecommendations.Anyevaluation–whethershortandsnappyorcomprehensiveshouldprovideinformationtoimprovethecommunityartspracticeoftheorganisation.

TOOLS TO USE

Thereisasampletoolprovidetoassistwithstructuringthereport.Thistoolassumesthatthereportisbeingpreparedforanexternalaudience.

refertoSampletool6.1:Draftevaluationreportstructure

Thereisalsoaworksheettoassistyouidentifywhatimprovementneedtobemade.refertoSampletool6.2:ImprovingProjectAimsandOutcomes

CH ECK LIST:

Haveyou:

•developedadraftevaluationreport?D

•answeredthekeyresearchquestionsintheevaluationreport?D

•collectedfeedbackonthedraft?D

•finalisedthereportandpreparedanexecutivesummary?D

•distributedtheexecutivesummarytotheappropriateparties?D

•linkedtheevaluationresultsintoanimprovementprocess?D

FURTHERREADING

Fiske,EdwardB.(2000)ChampionsofChange:TheImpact OfTheArtsOnLearning.ArtsEducationPartnershipandPresident’sCommitteeontheArtsandtheHumanities.

Hawe,Penelope;Degeling,D.andHallJ.1990)“ImpactAndEvaluationOutcome”Chapter6, InHawe,Penelope;Degeling,D.AndHallJ.EvaluatingHealthPromotion:AHealthWorkers’

Guide.Sydney:MaclennonPettyPublishing.

Matarasso,Francois(1996)DefiningValues:EvaluatingArtsPrograms(SocialImpactoftheArts:WorkingPaper1).BournesGreen,Comedia.

Matarasso,Francois(1997)UseOrOrnament?:TheSocialImpactofParticipationintheArts.Stroud,England:Comedia.

Meyrick,JaneandSinkler,PaigeAnEvaluationResourceforHealthLivingCentres,HealthEducationAuthority

Owen,JohnM.andRogers,PatriciaJ.(1999)ProgramEvaluation:FormsAndApproaches.StLeonards,N.S.W.:AllenUnwin.

Pope,Jeanette(2000)SocialCapitalAndSocialCapitalIndicators.

SIAP(2001)SocialImpact OfTheArtsProject.Philadelphia:UniversityOfPhiladelphia,SchoolofSocialWork.

Wadsworth,Yoland(1991)EverydayEvaluationontheRun,ActionResearchIssuesAssociationInc.,Melbourne.

Williams,Deidre(1997)HowtheArtsMeasureUp:AustralianResearchintoSocialImpact

Gloustershire:Comdedia(SocialImpactoftheArtsWorkingPaperNo.8