Research and Evaluation Capacity-Building Initiative at the AIDS Committee of Toronto - Final Report

Acknowledgements

This research capacity-building initiative was supported by a grant from the Ontario HIV Treatment Network.

The AIDS Committee of Toronto (ACT) acknowledges the assistance of Amy Andrews, the project consultant.

Special thanks are also due to ACT’s staff for their cooperation and keen participation in the research and evaluation capacity-building initiative.

Executive Summary

The AIDS Committee of Toronto (ACT) commenced this research and evaluation capacity-building initiative in October 2000 by striking a staff advisory group to develop the project. In the first phase of the project, the advisory group planned a consultation with staff to assess staff members’ interest in research, the function of research at ACT, their concerns about an increased profile for research at ACT, and their assessments of their current capacity to undertake research. The consultation consisted of focus groups and personal interviews.

Staff identified a number of concerns related to increasing the profile of research in the organization. However, staff also identified the importance of community-based research (CBR) for the work of the organization, and the need to improve their own ability to conduct, assess or use research.

Based on the issues raised in the consultation, the advisory group developed a capacity-building workshop plan, consisting of a series of seven workshops on research skills and research literacy. The workshops were delivered from July to November 2001 by a consultant, and included over 20 hours of instruction and discussion. The workshops represented the second phase of the capacity-building initiative.

The workshop evaluations show that ACT’s staff members have been able to improve their basic research skills, their ability to critically assess research, and their appreciation of the usefulness of research.

This report also includes a number of recommendations to further enhance staff’s capacity to undertake and/or use research for the benefit of ACT’s programs and services. ACT will address a number of these recommendations in the coming year (2002-2003) with a follow-up series of four workshops and research seminars. In addition, ACT’s research advisory committee is now assisting the organization to develop its research priorities and agenda consistent with the current strategic plan (2001-2004).

PART I:

BACKGROUND:
“Research” in the broadest sense, can be defined as the systematic gathering and production of knowledge. The AIDS Committee of Toronto (ACT) has already been involved in many “research-like” activities since its inception in the form of outreach work and community development - these activities are akin to research in that knowledge is systematically gathered and acted upon vis-a-vis program development and implementation. ACT has also shown leadership as an AIDS service organization in more formal types of research as well - survey development and administration, needs assessments, focus groups, and program evaluations. Because ACT possesses an intimate knowledge of the HIV/AIDS community, it is in an ideal position to continue to contribute to more formalized research.

In March, 2000, Health Canada acknowledged the specialized knowledge that ACT and other AIDS service organizations possess by announcing a four-year commitment to community-based research (herein referred to as CBR). Health Canada supports the notion that the academic community must work collaboratively with community organizations to provide research that is relevant, timely, and useful for HIV/AIDS prevention and support programming. The Ontario HIV Treatment Network (OHTN) followed suit by offering a capacity-building fund so that ASOs might develop the requisite knowledge and infrastructure to conduct or participate in CBR.

ACT’s Research and Evaluation Capacity-Building Initiative began in October, 2000. With a grant from the OHTN, ACT embarked on the development of an infrastructure to conduct community-based research and to make program evaluation an integrated activity of the agency.

Community workers often critique traditional forms of research as being irrelevant to their daily work. Research that is driven by community needs, however, can greatly enhance the design and evaluation of programs within ACT. In building a research infrastructure at ACT that is founded on the principles of CBR, ACT’s capacity to serve its communities will be enhanced. Additionally, ACT will be in a position to produce knowledge that is regarded as credible by funders, providing the community with an avenue for influencing future funding directions.

Definition of Capacity-Building:

“Capacity building it is not just about training people… it’s about taking what they have - their strengths and building on them.” (ACT staff)

The term “capacity-building” is widely used and its meaning can vary with context. In the HIV/AIDS sector, it is a term that is increasingly heard yet largely undefined for those closest to frontline work.

For the purposes of this report, capacity-building is defined as it is in the quote above in addition to:

  • the process by which community and community organizations are empowered to initiate research projects according to changing/emerging needs;
  • the provision of support and practical skills that will allow staff to develop their own research initiatives;
  • exposure to different forms of research.


PART II:

PROJECT OBJECTIVES:
The purpose of ACT’s Research and Evaluation Capacity-Building Initiative is to increase the research capacity of ACT staff. Enhancing ACT’s research capacity not only involves providing necessary training for staff to work critically and collaboratively with each other and with academic researchers; it also entails identifying the strengths of the organization and building on these assets so that ACT can initiate CBR projects independently.

ACT’s Research and Capacity-Building Initiative then has the following objectives:

1. to increase awareness of the principles of community-based research among management, staff, and volunteers;
2. to improve research knowledge and skills among management, staff, and volunteers;
3. to ensure that research capacity-building continues on a long- term basis.

Capacity-Building Staff Advisory Group:
A Staff Advisory Group comprised of individuals representing various ACT Program Staff teams, was struck for the capacity-building initiative. This committee included the following individuals who would work closely with the research team (Robert Lorway, Kay Ray, Robb Travers) in implementing various stages of the capacity-building initiative:

      Capacity-Building Staff Advisory Group:
      Betty-Anne Rutledge – Program Volunteer Coordinator
      Janet Rowe - Women’s Community Development Coordinator
      James Murray - Gay Men’s Community Development Coordinator
      Kathy Dickson - Employment Action Consultant
      Donna Braybrook – Practical Assistance Coordinator

The overall goal established for the Capacity-Building Staff Advisory Group was to ensure adequate staff input into the capacity building initiative at ACT.

The specific objectives (terms of reference) for this advisory group were to:
  • provide input on the scope and conceptual framework of the project;
  • define appropriate methodological strategies;
  • provide input and direction on the development of research instruments;
  • identify potential stakeholders staff, management, volunteers and other community partners;
  • provide input into the scope and content of a training program for ACT staff;
  • define dissemination and feedback strategy.

PART III:

CAPACITY-BUILDING CONSULTATIONS:
Initial consultation:
In November of 2000, a meeting was held between Program Management and staff (23 attended in total) to begin discussions about research and evaluation capacity-building at ACT. There were three objectives to this initial meeting:
  • to introduce staff to ACT’s Research and Evaluation Capacity-Building Initiative;
  • to begin discussion about CBR - focusing on its core principles and how it differs from more traditional forms of research;
  • to discuss barriers to implementing CBR.

Focus group consultations:
In conjunction with the Capacity-Building Staff Advisory Group, the research team agreed that focus groups were the most appropriate method for gathering the information required to develop a capacity-building training program for ACT. We also recognized, however, from Part 1 of the consultations that the nature of the topic would bring about fears and anxieties for some staff. Individual face-to-face interviews were therefore offered to those who preferred this option. ACT Program Staff and managers were advised that capacity-building focus groups would be conducted by team and were provided with a choice of time slots. Twenty-three ACT staff and management took part in the focus groups or interviews (two individuals were interviewed separately) - see Appendix A - Interview Guide - staff groups.

Pilot focus group:
The initial focus group was regarded as a “pilot” where a set of questions were tested for their fit with the overall project objectives. The objectives for this initial focus group were to:
  • assess fears, concerns, comfort level, organizational dynamics (structural barriers) in relation to the CBR program;
  • assess strengths, capacity assets, future-building and, learning needs in relation to CBR and program evaluation;
  • aid the research team in the assessment process and to determine future needs;
  • determine the scope and content areas for developing a training program;
  • assess how to effectively involve partner agencies.

These initial questions were also used for the focus groups with other staff teams.

Focus group between ACT management and the Staff Advisory Group:
Concerns about the Research and Evaluation Capacity-Building Initiative emerged in an early Staff Advisory Group meeting. These concerns related to broader structural concerns about the agency (e.g., decision-making processes, trust, etc.). Based on these concerns, the research team suggested a joint focus group between Management and the Staff Advisory Group. The goal for this focus group was to begin to discuss a shared vision of CBR at ACT. The objectives were the same as for other focus groups (see Appendix B - Interview Guide - Staff Advisory Group/Management Focus Group).

PART IV:

SUMMARY OF THE CONSULTATIONS
Several themes, issues and concerns emerged from the consultations with staff and management and are outlined in the following table:

        Table 1 - Summary of themes emerging from consultations
        1. Concerns regarding the implementation of a CBR infrastructure at ACT including: ?
      • trust issues between frontline staff and management;
      • suspicions about research and researchers;
      • concerns about time and resources being taken away from frontline activities for research;
      • confusion about CBR and its relevance to frontline prevention and support work;
      • fears about program evaluation and its “relationship” to staff performance evaluations.
        2. Recognition that the core principles of CBR are similar to those of many community development initiatives already familiar to staff;
        3. A common vision of what CBR initiatives at ACT should look like;
        4. Recognition among staff that many already possess research skills (capacities) and do research-like activities in their work;
        5. Recommendations for enhancing the research capacity of ACT staff.

1) Concerns regarding the implementation of a CBR infrastructure at ACT:
Significant concerns about CBR and the capacity-building initiative emerged in the staff consultations.

Trust: Principal among these was the issue of trust. Some participants voiced concerns that management (or the funders) “has a hidden agenda” in relation to CBR; most notably, staff mistrusted management and worried about the “sudden” appearance of four research team members. Some even argued that if “capacity to do research already exists at ACT,” there was no need to bring in additional personnel.

Suspicions about researchers: Others doubted the capacity of researchers to understand the nature of community work - they were suspicious about the credentials and experience of the research team and worried that their own work was incompatible with that of researchers.

Concerns related to program evaluation: Staff were concerned that research initiatives would be used as a means to evaluate programs. Still others worried that program evaluation could expose program weaknesses, and would therefore be used as a justification to eliminate programs.

Staff were also concerned that an agency-wide move toward research and program evaluation signaled the “credentialing” of AIDS in general, and the ongoing “professionalization” of ACT specifically. Some worried that people would have to have degrees to be able to work at ACT in the future, and some wondered about how research in general relates to their work.

Effect on clients: Others raised concerns about the effect on programming for clients - i.e., would research dollars take away from program resources? Still others worried about the effects on clients themselves - anxious that PHAs (who have been the subjects of many research studies) would begin to feel exploited - as if they were being “studied,” and that it would change the fundamental nature of their relationship to ACT.

Effect on ACT’s reputation: Finally, some raised the external profile of the agency as a concern. ACT’s relationship to other community organizations in Toronto was specifically mentioned. Most significantly, some worried that ACT may come to be viewed by community people as “academics doing research on the community.”


2) A desire for CBR core principles in ACT’s research activities:
Staff and management were quick to realize that the principles which define community-development work are also shared by advocates of CBR. The following guidelines or desired principles for CBR at ACT emerged out of the consultations:
  • research priorities and questions evolve out of community concerns;
  • communities “drive” research processes, are involved in all aspects of the research process, and own data;
  • research capacity at ACT becomes part of the organizational culture - it does not diminish when an individual leaves the agency.
  • collaboration between individuals in the research process is actively encouraged;
  • diverse contributions that various partners/stakeholders bring to the research process are valued;
  • capacity-building in communities to conduct further research is of utmost importance;
  • transferring knowledge back to communities is crucial;
  • CBR does not take the stance of “objectivity” in relation to who is being researched - is more concerned with gathering knowledge from the inside - primarily concerned with the “subjective” experiences of people;
  • social change and action components are central goals of CBR - research reports should not simply sit on shelves.


3) ACT’s vision for CBR:
There were also divergent definitions of CBR among ACT staff - most differences in opinion related to the “role” of community people in relation to academics. Staff members were able to agree, however, that community-based research at ACT should:

q be directed by ACT- ACT is not simply a place where researchers collect data;
q clearly define who “community” is and who appropriate and relevant partners should be;
q inform programming - i.e., be practical;
q improve the lives of the vulnerable people ACT works with - i.e., have a social change agenda driving them;
q assist ACT in advocacy work;
q foster partnerships - it was recognized that academics and community people bring a rich array of experiences and skills to research partnerships. Moreover, it was acknowledged that some academics are also community members and therefore particularly suited to be research partners.

4) “More than just tables and graphs:” How ACT staff use research in their daily work:
A realization that everyday work involves significant research-oriented activities began to emerge during the consultations. Numerous examples of current staff and management research activities were cited and are listed in the table below:

      Table 2 - Inventory of staff/management research activities
      Internet searches
      Surveys
      Needs assessments
      Focus groups
      Evaluation forms
      Telephone logs
      Talking to others Observing and listening
      Through meetings
      Interviewing clients
      Keeping track of who is coming and coming
      Observing trends
      Library searches Representing act on research advisory committees Representing act on investigators’ teams

Many began to realize that what separates these from more formalized research was the “systematization” of these activities coupled with the formal training of researchers. One staff person aptly stated, “if you take out the word research and said consultation, everyone would say, “Oh yes. I do that.” Another defined what he does everyday in his job as akin to research - “collecting data.” Still another, said that her presumptions about research projects all having to be large-scale precludes her from viewing any of the activities in her daily work as research-like.

ACT staff and management were generally able to articulate how research benefits their work. Many said that being on top of current information in the field enhanced program development and ultimately benefited clients. This group also tended to be most involved in more formalized types of research activities in their work - focus groups, needs assessments, surveys, etc. Most interesting about this group, is that few possess the more “formalized” research training typically learned through academia. The majority started doing research at ACT by default - quickly learning that much of it “wasn’t rocket science.” For some, their predecessors had built evaluation and research into their daily work, while for others, they learned “hands on” (often with little formal guidance or time to reflect).

For those staff members who had already been doing research in their jobs, an appreciation of the small-scale nature of their studies was evident. Some mentioned the expedience with which data can be gathered through smaller studies - and how efficiently the data can be transferred into programs.

5) Recommendations for enhancing the research capacity of ACT staff:
ACT staff and management put forth numerous requests for skills training in relation to research and evaluation. Most ACT management and staff viewed the opportunity to learn about research with enthusiasm. They recognized the potential to improve programs, be stronger advocates for clients, and to enhance the overall quality of life in the communities they serve.

Many recounted situations where they had worked in “partnership” with academics or attended research presentations, or research conferences (e.g., CAHR). Many said they had felt alienated and out of place in these circumstances. One fittingly stated that learning about research would be a source of power for him. It would help in future situations - “I need to have credibility. It is about credibility - I guess I feel confident that I know we have lots of knowledge but there are pieces that I don’t know that I need to learn.” Another fittingly stated that learning about research would be invaluable in her work - “it’s like knowing which fork to use.”

Research training needs identified by management and staff fall into those best described as “research skills” and those most appropriately identified as “structural questions” in relation CBR at ACT. They are summarized in the tables below:

      Table 3 - Training needs
      Research skills
    • How to identify a research need
    • How to identify a research question
    • How to write a research proposal
      Qualitative research methods
    • How to conduct research interviews
    • How to develop interview guides
    • Building trust in research interviews
    • How to do conduct focus groups
    • How to develop discussion guides
      Quantitative research methods
    • How to develop survey tools
    • How to develop needs assessment tools
      Issues relate to research ethics
      Learning research lingo (e.g., “cohort study”)
The initial process of beginning to discuss community-based research and how it differs from more traditional forms of research was invaluable to most of the participants in these capacity-building consultations. Given the barriers to implementing CBR at ACT identified in this report, this discussion should be revisited in an initial training session. Staff may require reassurance that the core principles of CBR and the vision for research at ACT that arises out of this report is shared by management an the Board.

        Table 4 - Training needs
        Structural questions in relation to CBR at ACT
        Understanding CBR and its principles
        Understanding action research vs. more traditional research
        What other communities are doing in relation to research
        Understanding dissemination strategies (i.e. in relation to CBR principles)
        Building capacity among academics to understand community work
        ACT’s research role vis-a-vis other ASOs

PART V:

THE CAPACITY-BUILDING WORKSHOP PLAN:

According to earlier sections of this report, ACT staff members were enthusiastic about the opportunity to develop research and evaluation skills. In spite of the concerns present within the agency (as outlined in Parts II-IV in this report), there was little doubt that research skills building workshops would add greatly to ACT staff members’ ability to evaluate and conduct research for the purposes of improved program planning and service provision. As staff members become more comfortable with research ‘lingo,’ methods and processes, they will add to their already impressive abilities as community leaders and be able to use their own research and the research of others as a basis for effective client and client-community advocacy.

The workshops outlined below were designed to orient staff members who have little research experience to the basic theoretical and practical aspects of research. Simultaneously, these workshops challenged more knowledgeable staff to push a little further into conceptual areas with which they were not familiar. For example, Workshop #1 covered the basic differences between qualitative and quantitative research while also challenging experienced staff members to explore and ‘unpack’ ways in which their personal attitudes towards client service provision might influence the way they conduct program and service research. Every attempt was made to offer the workshops (outlined below) in such a way as to be non-intimidating for the novice and, yet, challenging for the more experienced participant.

Workshop Timeline:
The workshops commenced in July 2001 and continued until mid-November 2001.

Workshop sequence:
As will be detailed below, the first two workshops provided the theoretical framework for the workshop series. As a basis for workshops #4 through #7, the first two workshops explored the ways in which one’s attitudes towards and assumptions about HIV/AIDS service work are reflected in the research methods one uses and conducts. The first two workshops, mandatory for all ACT staff, also provided participants with the skills to critique and discuss the quality, reliability and validity of research results. These skills were used and further developed in the following four workshops that focused on specific methods of data collection and basic analysis.

The third workshop provided an opportunity for ACT staff to collectively articulate a vision for CBR at ACT. This workshop was mandatory for all ACT staff. Parts II through IV of this report identified several legitimate concerns that were discussed and addressed as part of the process of incorporating CBR more visibly and formally into the work of ACT staff. During this workshop, staff members were invited to voice their hopes for and concerns about the implementation of a CBR framework at ACT. Staff participants were also encouraged through small-and-large-group work to identify resolutions to concerns or problems identified throughout the workshop. Following this workshop, several volunteer staff participants made themselves available to work with ACT’s Director of Research and Program Development, Winston Husbands, to construct a ‘values statement’ for CBR at ACT to serve as a guiding principle for all future community-based capacity-building work and research. This statement was developed using transcripts of the Workshop # 3 discussion.

Workshops four, five, six and seven addressed specific skills-building needs identified by ACT staff throughout this report. These workshops were provided in the middle of the day in a “lunch n’ learn” format. They were optional for ACT staff.

Community Partnership Plans:
During the writing of the April 2001 interim version of this report, ACT indicated that it hoped to invite members of the community to participate in workshops # 1, 2, 4, 5, 6 and 7, providing that funds and enrollment opportunities allowed. However, due to the overwhelming advance demand for the workshop by ACT staff members, these plans have been put on hold until another time.


PART VI:

OUTLINE OF CAPACITY-BUILDING WORKSHOPS:

Workshop #1: Evaluating the Evidence - Approaches
N.B. One session of this workshop was mandatory for all ACT staff members.

Date: July 19, 2001 from 1:00 PM - 5:00 PM and August 16, 2001 from 1:00 PM - 5:00 PM

Duration: 4 hours

Rationale for Workshop: Part IV of this report indicates that ACT staff members wanted to explore issues of research ethics and that they would like to increase their understanding of research question definition and research ‘lingo.’ This workshop offered a basic introduction to qualitative and quantitative research and research vocabulary, and introduced critical reading and evidence evaluation skills that were further developed in the workshop series.

Objectives:
1. To explore the notions of ‘objective’ research and research as ‘truth’
2. To become aware of the wide spectrum of approaches to human service provision
3. To identify one’s own approach(es) to service provision
4. To learn how to identify approaches used by other service providers and/or published authors
5. To familiarize oneself with ‘evidence jargon’ used in publications, at conferences, etc.
6. To raise awareness of the difference between qualitative and quantitative research - highlighting the strengths and weaknesses of each style of inquiry
7. To increase awareness as to how the two styles might be integrated within a single project
8. To identify the differences between action research and more traditional research
9. To increase staff research question definition skills

Method: Interactive presentation style interspersed with individual and small group exercises. Participants were encouraged to contribute at any time during the workshop.

Workshop #2: Evaluating the Evidence - The Details
N.B. One session of this workshop was mandatory for all ACT staff members.

Date: July 26, 2001 from 1:00 PM - 5:00 PM and August 23, 2001 from 1:00 PM - 5:00 PM

Duration: 4 hours

Rationale for Workshop: This workshop was a continuation of workshop #1 and was an introduction to the various study designs commonly experienced when performing a literature search on an HIV/AIDS topic. Participants continued to develop their understanding of the research process and became more familiar with research ‘lingo.’ Participants also learned how to recognize a well done and a poorly done quantitative and/or qualitative study. This workshop offered a basic introduction to quantitative research reporting techniques (statistical indicators, tables and graphs) and research vocabulary, and continued to introduce critical reading and evidence evaluation skills.

Objectives:
1. To review the content of the previous workshop and to ensure that all participants are comfortable with the previous material
2. To familiarize oneself with ‘evidence jargon’ used in publications, at conferences, etc.
3. To become familiar with several types of quantitative study carried out in health care/social science research (e.g. randomized control trials, cohort studies, case-control studies) and to review the strengths and weaknesses of each
4. To become familiar with ways to critique and assess the quality of a quantitative study
5. To become familiar with several basic statistical indicators and concepts used in quantitative study reports (e.g. p-values, r2, RR and OR)
6. To become aware of the types of qualitative inquiry used in health care/social science research (e.g. ethnography, mapping, etc.)
7. To become familiar with ways to critique and assess the quality of a qualitative study

Method: Interactive presentation style interspersed with individual and small group exercises. Participants were encouraged to contribute at any time during the workshop.

Workshop #3: CBR: The Vision
N.B. This workshop was mandatory for all ACT staff members.

Date: September 27, 2001: 1:00 PM - 5:00 PM

Duration: 4 hours

Rationale for Workshop: Part IV of this report illustrated ACT staff members’ desire to engage in an agency-wide dialogue regarding the nature of CBR and its principles. This workshop provided a forum for ACT staff to come together and to focus on their vision(s), goal(s) and objective(s) for CBR work at ACT. This was an opportunity for all ACT staff to freely voice their hopes for and concerns regarding CBR at ACT. It was also be a venue for staff members to explore possible options for further discussion.

Objectives:
1. To explore the differences between traditional research and community-based research paradigms
2. To engage in an open dialogue regarding the vision that ACT staff have for the agency’s role in community-based research
3. To begin to identify goals and a ‘values statement’ for CBR at ACT
4. To begin to identify departmental and agency-wide objectives for CBR at ACT (e.g. what we want to do/to avoid?)

Method: Interactive presentation style interspersed with individual and small group exercises. Participants were encouraged to contribute at any time during the workshop.

The main ideas and issues on CBR enunciated by ACT staff in the workshop are reproduced in Appendix C (Research with a Purpose).

Workshop #4: Sampling and Study Group Maintenance
N.B. This workshop was optional for ACT staff members

Date: October 4, 2001: Noon - 2:30 PM

Duration: 2.5 hours - Light lunch provided

Rationale for Workshop: In part IV of this report, ACT staff members expressed an interest in learning concrete skills in research project management and data collection. This workshop addressed ways in which research project coordinators can advertise for research project participants and retain study participants.

Objectives:
1. To collectively identify strategies for obtaining a sample group for a quantitative or qualitative (or both) study
2. To identify barriers to study group participation maintenance
3. To strategize ways, using available theoretical models and activities, to maintain study groups in the face of limited funding, pressures external to the participant and lack of staff time

Method: Interactive presentation style interspersed with individual and small group exercises. Participants were encouraged to contribute at any time during the workshop.

Workshop #5: Interviewing
N.B. This workshop was optional for ACT staff members

Date: October 18, 2001: Noon - 2: 30 PM

Duration: 2.5 hours - Light lunch provided

Rationale for Workshop: In part IV of this report, ACT staff members expressed an interest in learning concrete skills in research data collection. This workshop addressed techniques for conducting research interviews. Participants were encouraged to discuss ways of creating a comfortable and safe interview environment and learned of interview guide preparation methods.

Objectives:
1. To become aware of a variety of ‘best practice’ interview methods in health care and social science research (e.g. structured, semi-structured, in-depth, open-ended, etc.)
2. To become aware of the importance of environment in an interview setting
3. To develop personal skills in interview question/conversation guide construction
4. To discuss possible challenges and limitations of interviews as a research method
5. To briefly survey methods of interview data capture, translation and analysis

Method: Interactive presentation style interspersed with individual and small group exercises. Participants were encouraged to contribute at any time during the workshop.

Workshop #6: Focus Group Facilitation
N.B. This workshop was optional for ACT staff

Date: November 1, 2001: Noon - 2:30 PM

Duration: 2.5 hours - Light lunch provided

Rationale for Workshop: In part IV of this report, ACT staff members expressed an interest in learning concrete skills in research project management and data collection. This workshop addressed ways in which research project coordinators can develop focus group discussion guides. The participating group was also encouraged to discuss techniques for small group management and group discussion facilitation.

Objectives:
1. To become aware of a variety of ‘best practice’ focus group facilitation methods in health care and social science research
2. To become aware of the importance of environment, group construction and group dynamics in a focus group setting
3. To develop personal skills in focus group question/discussion guide construction
4. To discuss possible challenges and limitations of focus group discussion as a research method
5. To briefly survey methods of focus group data capture, translation and analysis

Method: Interactive presentation style interspersed with individual and small group exercises. Participants were encouraged to contribute at any time during the workshop.

Workshop #7: Survey Development
N.B. This workshop was optional for ACT staff members

Date: November 15, 2001: Noon - 2:30 PM

Duration: 2.5 hours

Rationale for Workshop: In part IV of this report, ACT staff members expressed an interest in learning concrete skills in research project management and data collection. This workshop shared ‘best practices’ techniques for survey and questionnaire development. The participating group was also encouraged to discuss techniques for survey layout, clear language use and dissemination.

Objectives:
1. To become aware of a variety of ‘best practice’ methods for questionnaire/survey development in health care and social science research
2. To become aware of the importance and use of chronology, word use, and open-and-closed questions in a survey
3. To develop personal skills in survey question construction
4. To discuss possible challenges and limitations of survey use as a research method
5. To briefly survey methods of survey coding, data capture and analysis

Method: Interactive presentation style interspersed with individual and small group exercises. Participants were encouraged to contribute at any time during the workshop.


PART VII:

WORKSHOP EVALUATION DATA COLLECTION, ANALYSIS AND RESULTS

Pre-Workshop Self-Assessment Form:
All workshop participants were asked to conduct a pre-workshop assessment form at the beginning of the workshop. This form asked participants to indicate, on a scale of 1 (not at all confident) to 7 (very confident), their level of confidence in their current ability to employ the material scheduled for that session. Participants were also asked to communicate any questions or concerns that they had about the session so that the facilitator could address those during the course of the workshop, as appropriate.

Post-Workshop Evaluation Form:
After the workshop, participants were then asked to complete a facilitator and workshop evaluation. (Please see Appendix D). Participants were asked to re-indicate, on a scale of 1 (not at all confident) to 7 (very confident), their post-workshop level of confidence in their ability to employ the workshop material. They were also asked to indicate, on a scale of 1 (not at all comfortable) to 7 (very comfortable), their level of comfort with the pace and atmosphere of the workshop. Participants were also asked to rate, on a scale of 1 (I achieved none of my goals) to 7 (I achieved all of my goals), the extent to which they reached their learning goals for the session. Finally, participants were invited, via open-ended questions, to share how the facilitator did nor did not work with them to achieve their goals. Participants were also invited to indicate, in an open-ended way, what they liked and did not like about their workshop experience.

In order to track and compare workshop participants’ pre-and-post-workshop comments, participants were asked to develop a “top-secret code” - consisting of a series of words, symbols or numbers that the participant could easily remember – that would act as an anonymous identifier for the purposes of tracking workshop comments. Participants were asked to use this “code” identifier throughout their involvement in the workshop series.

WORKSHOP EVALUATION DATA ANALYSIS

The data from the workshop evaluations were summarily analyzed throughout the workshop process in order to remain responsive to the needs of participants. A final analysis of the workshop data was conducted, using SPSS 10.0 analysis software, in December 2001.

Using the evaluation tool scale data, the Research Consultant calculated the group means for each evaluation indicator and compared the pre/post-workshop means.

The Research Consultant also transcribed all of the open-ended participant responses and grouped them according to response themes.

WORKSHOP EVALUATION RESULTS - Scale Responses

Workshop #1: Evaluating the Evidence - Approaches

Q. "How confident are you in your ability to assess evidence applicable to your work?"

At the beginning of the workshop, participants reported that they were "mildly confident" in their ability to assess evidence applicable to their work (M=5.00, SD=0.97). Their reported confidence increased significantly by the end of the workshop (M=5.37, SD=0.97), p<0.05.

Learning Goals:

Q. "Do you feel that you were able to achieve your learning goals for this session?"
By the end of workshop # 1, participants reported that they had achieved several of their learning goals for the session (M=5.28, SD=0.98).

Comfort Level with Material and Facilitation Style:

Q. " Did you feel comfortable with the way the material was presented?"

By the end of workshop # 1, participants reported that they were comfortable with the way the workshop material was presented (M=5.92, SD=0.83).

Previous Participation in the Capacity-Building Consultation Process:

Thirteen individuals indicated that they had participated in the process leading up to the workshops.

Workshop #2: Evaluating the Evidence - The Details

Q. " How confident are you in your ability to assess evidence applicable to your work?"

At the beginning of the workshop, participants reported that they were "mildly confident" in their ability to assess evidence applicable to their work (M=5.12, SD=0.82). Their reported confidence increased, but not significantly, by the end of the workshop (M=5.36, SD=1.08), ns.

Learning Goals:

Q. "Do you feel that you were able to achieve your learning goals for this session?"

By the end of workshop # 2, participants reported that they had achieved several of their learning goals for the session (M=5.25, SD=1.13).

Comfort Level with Material and Facilitation Style:

Q. " Did you feel comfortable with the way the material was presented?"

By the end of workshop # 2, participants reported that they were comfortable with the way the workshop material was presented (M=5.88, SD=0.95).

Previous Participation in the Capacity-Building Consultation Process:

Twelve individuals indicated that they had participated in the process leading up to the workshops.
Workshop #3: CBR: The Vision

Q. "How confident are you in your ability to envision how community-based research (CBR) fits into your work at ACT?"

At the beginning of the workshop, participants reported that they were "mildly not confident" in their ability to envision how CBR fit into their work at ACT (M=3.24, SD=0.73). Their reported confidence increased significantly by the end of the workshop (M=5.40, SD=0.84), p<0.05.

Learning Goals:

Q. "Do you feel that you were able to achieve your learning goals for this session?"

By the end of workshop # 3, participants reported that they had achieved several of their learning goals for the session (M=5.14, SD=1.02).

Comfort Level with Material and Facilitation Style:

Q. " Did you feel comfortable with the way the workshop discussion had been facilitated and conducted?"

By the end of workshop # 3, participants reported that they were comfortable with the way the workshop discussion had been facilitated and conducted (M=5.32, SD=0.85).

Previous Participation in the Capacity-Building Consultation Process:

Twenty-seven individuals indicated that they had participated in the consultation process leading up to the workshops.

Workshop #4: Sampling and Study Group Maintenance

Q. "How confident are you in your ability to develop a sampling strategy for a research project?"

At the beginning of the workshop, participants reported that they were generally "undecided" in their ability to develop a sampling strategy for a research project (M=4.76, SD=0.87). Their reported confidence increased significantly by the end of the workshop (M=5.36, SD=0.93), p<0.05.

Q. "How confident are you in your ability to develop a respondent retention strategy for a research project?"

At the beginning of the workshop, participants reported that they were "mildly confident" in their ability to develop a respondent retention strategy (M=5.03, SD=0.71). Their reported confidence increased significantly by the end of the workshop (M=5.64, SD=0.81), p<0.05.

Learning Goals:

Q. "Do you feel that you were able to achieve your learning goals for this session?"

By the end of workshop # 4, participants reported that they had achieved several of their learning goals for the session (M=5.89, SD=0.78).

Comfort Level with Material and Facilitation Style:

Q. " Did you feel comfortable with the way the material was presented?"

By the end of workshop # 4, participants reported that they were comfortable with the way the workshop material was presented (M=5.73, SD=0.81).

Previous Participation in the Capacity-Building Consultation Process:

Five individuals indicated that they had participated in the consultation process leading up to the workshops.

Workshop #5: Interviewing

Q. "How confident are you in your ability to develop an interview strategy (i.e., a plan to help you get at the issues you want to address) for a research project?"

At the beginning of the workshop, participants reported that they were "mildly confident" in their ability to develop and interview strategy (M=5.21, SD=1.04). Their reported confidence increased significantly by the end of the workshop (M=5.93, SD=0.88), p<0.05.

Q. "How confident are you in your ability to develop an interview guide (i.e., a question guide to help you ask the right questions at the right time) for a research project?"

At the beginning of the workshop, participants reported that they were "mildly confident" in their ability to develop and interview strategy (M=5.07, SD=0.83). Their reported confidence increased significantly by the end of the workshop (M=5.74, SD=0.98), p<0.05.

Learning Goals:

Q. "Do you feel that you were able to achieve your learning goals for this session?"

By the end of workshop # 5, participants reported that they had achieved several of their learning goals for the session (M=5.67, SD=0.89).
Comfort Level with Material and Facilitation Style:

Q. " Did you feel comfortable with the way the material was presented?"

By the end of workshop # 5, participants reported that they were comfortable with the way the workshop material was presented (M=5.94, SD=0.91).

Previous Participation in the Capacity-Building Consultation Process:

Four individuals indicated that they had participated in the consultation process leading up to the workshops.

Workshop #6: Focus Group Facilitation

Q. "How confident are you in your ability to develop a focus group strategy (i.e., a plan to help you get at the issues you want to address) for a research project?"

At the beginning of the workshop, participants reported that they were "mildly not confident" in their ability to develop a focus group strategy (M=3.89, SD=0.87). Their reported confidence increased significantly by the end of the workshop (M=5.26, SD=1.02), p<0.05.

Q. "How confident are you in your ability to develop a focus group guide (i.e., a question guide to help you ask the right questions at the right time) for a research project?"

At the beginning of the workshop, participants reported that they were generally "undecided" in their ability to develop a focus group guide (M=4.54, SD=0.92). Their reported confidence increased significantly by the end of the workshop (M=5.29, SD=0.97), p<0.05.

Learning Goals:

Q. "Do you feel that you were able to achieve your learning goals for this session?"

By the end of workshop # 6, participants reported that they had achieved several of their learning goals for the session (M=5.92, SD=0.81).

Comfort Level with Material and Facilitation Style:

Q. " Did you feel comfortable with the way the material was presented?"

By the end of workshop # 6, participants reported that they were comfortable with the way the workshop material was presented (M=5.88, SD=0.95).

Previous Participation in the Capacity-Building Consultation Process:

Four individuals indicated that they had participated in the consultation process leading up to the workshops.

Workshop #7: Questionnaire Development

Q. "How confident are you in your ability to develop a questionnaire package (i.e., a package that presents your questionnaire to potential participants)?"

At the beginning of the workshop, participants reported that they were "mildly not confident" in their ability to develop a questionnaire package (M=3.89, SD=1.01). Their reported confidence increased significantly by the end of the workshop (M=5.23, SD=0.97), p<0.05.
Q. "How confident are you in your ability to develop a questionnaire?"

At the beginning of the workshop, participants reported that they were "mildly not confident" in their ability to develop a questionnaire (M=3.96, SD=0.88). Their reported confidence increased significantly by the end of the workshop (M=5.34, SD=0.82), p<0.05.

Learning Goals:

Q. "Do you feel that you were able to achieve your learning goals for this session?"

By the end of workshop # 7, participants reported that they had achieved several of their learning goals for the session (M=5.76, SD=1.03).

Comfort Level with Material and Facilitation Style:

Q. " Did you feel comfortable with the way the material was presented?"

By the end of workshop # 7, participants reported that they were comfortable with the way the workshop material was presented (M=5.94, SD=0.98).

Previous Participation in the Capacity-Building Consultation Process:

Six individuals indicated that they had participated in the consultation process leading up to the workshops.

WORKSHOP EVALUATION RESULTS - Open-Ended Responses
Comments Regarding Confidence with Material, Learning Goals and Comfort Level with Material:

When asked to reflect on how the workshop facilitator and Capacity Building Staff Advisory Group worked with the individual participant to help develop their confidence regarding the workshop material, to achieve their learning goals and to feel comfortable with the workshop material, participants responded in one or more of four general ways:

Response Type # 1: Workshop Handouts

Several participants indicated that the presence of workshop handouts enhanced their learning experience. Many indicated that they would take the handouts for further reference and others stated that they liked to write notes during the presentation. Many participants also indicated that they enjoyed the mix of presentation handouts and overheads.

Response Type # 2: Facilitator Presentation Style

Most participants noted that they enjoyed their interaction with the workshop facilitator. Many felt that the personality of the facilitator made the workshops enjoyable and the learning less "dry". Several participants also indicated that the facilitator was able to effectively manage the group discussion.

Response Type # 3: Large-and-Small-Group Discussion and Activities

Most participants indicated that they enjoyed the combination of large-and-small-group discussion and activity that was present in each of the workshops. Several participants indicated that this facilitated their learning by allowing them to voice questions and opinions. The mix in discussion and activity format also allowed several participants to connect with co-workers that they seldom see.
Response Type # 4: Group Dynamics

Some workshop participants in the smaller, non-mandatory workshops (#4 through #7) mentioned that they felt the group dynamics were instrumental in their skills development. These respondents indicated that the small group format was very comfortable for them and that they benefited from a more intimate environment.

When asked to reflect on how the workshop facilitator and Capacity Building Staff Advisory Group could have worked more effectively with the individual participant to help develop their confidence regarding the workshop material, to achieve their learning goals and to feel comfortable with the workshop material, participants responded in one or more of five general ways:

Response Type # 1: Group Dynamics

Some workshop participants in the larger, mandatory workshops (#1 through #3) indicated that they were not satisfied with the way in which the large group was working together. These respondents indicated that certain participants spoke more than others and that this was unacceptable. Of these respondents, some indicated that the facilitator spent too much time answering the questions of certain staff members.

Response Type # 2: Not Enough Detail

Some respondents in Workshops #5, 6 and 7 indicated that they would have liked the workshops to include more details regarding the particular session topic.

Response Type # 3: Too Much Information

Some respondents in Workshops #1 through #7 indicated that the facilitator tried to include too much information for the time allotted to the session.

Response Type # 4: Not Enough Breaks

Some workshop participants felt that more breaks should have been included in the workshop format. Some of these respondents indicated that more frequent breaks would enhance their ability to engage with the workshop material.

Response Type # 5: Too much/Too Little Workshop Time

Many workshop participants felt that the time allotted for the workshops was problematic. These respondents felt that a four-hour workshop was too long and/or a two-and-a-half-hour workshop was too short for effective workshop learning.


PART VIII:

STUDY LIMITATIONS

Due to the fact that the post-workshop evaluation took place immediately following the workshop session, it is likely that participants were inclined to report a higher confidence level with the workshop material than they had two-to-four hours previously. Thus, it is possible that the post-workshop participant reports are inflated and do not accurately reflect the true confidence level of the participant.

PART IX:

RECOMMENDATIONS FOR FUTURE RESEARCH SKILL CAPACITY BUILDING

Continue Research Skill Building for Individual Staff
Although staff workshop participants consistently indicated that they were pleased with the information and training that they received throughout the workshop series, there remains a vast amount of research information to which the AIDS Committee of Toronto (ACT) staff members have little access. It is highly recommended that ACT encourage interested staff members to seek additional research training within the larger community. It is recommended that the ACT consider funding further staff research education as part of the agency’s commitment to on-going professional development.

Continue to Evaluate Workshop Participant Learning
In order to assess the long-term value of the aforementioned workshop series, it will be necessary to follow-up with the ACT staff members who participated. Thus, it is recommended that members of the Capacity Building Staff Advisory Group make plans to conduct a long-term evaluation of workshop participant learning.

Continue to Provide Opportunities for Staff to Collaborate as Community Researchers
ACT is currently involved in a number of exciting community and academic research projects. It is recommended that the ACT continue with its practice of involving staff members in the development and carrying-out of collaborative research efforts. Given that research collaboration and involvement are an excellent way to develop new research skills, it is recommended that the ACT make every effort to ensure that such opportunities are open to all interested staff members.

Continue Staff Member Education RE: Research Proposal Development
Many ACT staff members have developed basic research skills throughout the course of the workshop series. It is recommended that the ACT provide additional training and support to those staff members who wish to develop research proposals for either internal or external funding and/or approval.

Develop Procedures and Policies regarding Staff Work-plan and Research Duties
As ACT staff members become more active in agency-and-community-based research projects and processes, it will be necessary to clarify the process by which staff members will undertake new research projects while maintaining their other work-plan duties. It is recommended that the ACT initiate a policy development process to address future issues regarding staff time for research activity and other professional research skills development.

Develop Policies and Procedures regarding Ethical Review Procedures for ACT-Proposed Research
As the ACT begins to initiate its own research projects, it will be necessary for the organization to devise a sound ethical review procedure that is capable of providing a thorough review of research protocols. It is recommended that the ACT investigate ‘best practice’ models of institutional and/or agency-based ethical review processes and that they devise their own policies and procedures in this area according to nationally accepted standards.

ACT’s Role As an Agent for Community Research Skill-Building
Due to an overwhelming response from ACT staff members, the skills-building workshops planned during 2001 were fully booked well in advance of the commencement of the workshop series. Thus, interested community members were not accommodated as the ACT had originally hoped.
Through its involvement in the aforementioned workshop series, the ACT has had access to more research skills training than most other AIDS Service Organizations in the Toronto area. Thus, it is recommended that the ACT explore ways to share its developing skill in the area of research processes with the larger community. It is also recommended that the ACT make every effort to assist in the development of a local network of AIDS Service Organizations that are interested in research for the purposes of program planning, development and evaluation.


Appendix A - Interview Guide - Staff Focus Groups

Introduction and background:
The AIDS Committee of Toronto is well positioned to develop and conduct community-based research that would have a significant impact on AIDS support, prevention and advocacy in Ontario. This project, funded by the Ontario HIV Treatment Network (OHTN), grew out of a proposal written by John Maxwell in the fall of 1999. The ultimate goal of the project is to build on the existing skills and specialized knowledge of staff and volunteers at ACT thereby increasing their capacity to engage in research in the community setting.


Where is the information that we collect here going to be used?

1. Will be used in further focus groups to facilitate the collection of ideas and opinion.
2. Workshops - that staff will determine as the outcome of these focus groups.
3. Ongoing capacity building and a written report.
4. What we hope to accomplish out of these workshops it to begin a discussion that will allow a common understanding of CBR to emerge.
General comments:
  • We are taping for purposes of transcribing common themes - the research staff will transcribe the tapes and then erase them. We will store the tapes in a locked cabinet.
  • We will not use any names in the finished research - unless permission is give.
  • If we do use quotes, we will obtain permission first from the participant and all quotes will be anonymous and we will remove any identifying features.
  • Anyone who wished to may arrange a meeting with Robert or Kay to discuss and concerns, uses or to make additional comments.
  • You may also contact us by email if there are issues/concerns that you didn’t feel comfortable raising here.


Discussion Questions:
Theme # 1: Research - expectations - fears
“When you first heard that ACT was moving in the direction of Community Based Research what came to mind? Negatives/Positives? What do you think now? What do you think about what we are doing?
  • Sources of research - ie. government reports, news reports, newspapers etc.
  • Words that describe research?
  • What do these words mean to you?
  • Negative/Positive connotations of these words
  • How do you define the term Community based research? Different definitions.
  • Do you know of any instances where CBR has been used successfully? Negatively?
  • (Do you think research produced in a hospital or university differs from research produced at ACT or other community based agency if so how?)


Theme # 2: Your work experience and your specialized knowledge is unique? How did you learn when you started here?
  • Your work experiences: how has it changed since coming here? When you started here how did you learn? Misconceptions? Learning experiences?
  • Your specialized knowledge - how do you share this with others? ie. Team meetings, reports, check-ins, informal conversations with other community workers etc.
  • How do you use the knowledge that you have accumulated? Where does it go? Who uses it?
  • Documenting your work - what would be helpful to you to build an information base of your work - for your own personal use - time constraints

Theme # 3: Your work and research at ACT - how can you contribute in the future how have you contributed so far?
  • Do you see yourself as contributing to a research project? What do you think are your strengths and weakness.
  • What do you feel are your barriers to actively shaping research initiatives at ACT?
  • Funding came from the Ontario HIV Treatment Network (OHTN).
  • Future wants/needs in terms of workshops: Women’s kitchen, Project proposal
  • Do you see research as relating to your work?
  • Do you think that this can be useful for you in your job?

Appendix B - Interview Guide - Staff Advisory Group/Management Focus Group

Introduction and background:
The AIDS Committee of Toronto is well positioned to develop and conduct community-based research that would have a significant impact on AIDS support, prevention and advocacy in Ontario. This project, funded by the Ontario HIV Treatment Network (OHTN), grew out of a proposal written by John Maxwell in the fall of 1999. The ultimate goal of the project is to build on the existing skills and specialized knowledge of staff and volunteers at ACT thereby increasing their capacity to engage in research in the community setting.


Where is the information that we collect here going to be used?

1. Will be used in further focus groups to facilitate the collection of ideas and opinion.
2. Workshops - that staff will determine as the outcome of these focus groups.
3. Ongoing capacity building and a written report.
4. What we hope to accomplish out of these workshops it to begin a discussion that will allow a common understanding of CBR to emerge.

General comments:
  • We are taping for purposes of transcribing common themes - the research staff will transcribe the tapes and then erase them. We will store the tapes in a locked cabinet.
  • We will not use any names in the finished research - unless permission is give.
  • If we do use quotes, we will obtain permission first from the participant and all quotes will be anonymous and we will remove any identifying features.
  • Anyone who wished to may arrange a meeting with Robert or Kay to discuss and concerns, uses or to make additional comments.
  • You may also contact us by email if there are issues/concerns that you didn’t feel comfortable raising here.

Discussion questions:

1) Definitions of Capacity Building:

a) What does capacity building mean to you?
b) What does community based research mean to you?

2) How do you know what you know (about your job, HIV/AIDS etc)

a) How do you use research in your work?
b) What barriers do you face to using research at work?

3) Barriers to CBR at ACT: Here are some common themes we have identified so far:

a) Discontent with past decisions
b) Concerns about performance evaluation and job descriptions
c) Suspicion: Towards both management and research

Concerns that we have identified so far:
  • Trust - Decisions making process
  • Evaluation
  • Hidden agenda
  • Strategic planning how does this fit in
  • Us - the research team - we feel as if we are not trusted
  • Feel like we are selling something
  • Not authentic community workers)

4) What could we offer in terms of training to allow for a common vision of CBR to emerge?

What to do with this barrier stuff - should probably be a workshop topic (with a review of the principles of CBR given staff turnover at ACT)

Concerns with implementing community-based research:
  • it can be difficult to secure funds for community-based research if funders regard it as less “scientifically rigorous,” etc.
  • some individuals may believe that research is not important - “what we really need is programs and not numbers.”
  • inequitable distribution of power between players involved in research partnerships (academic vs. community knowledge, issues related to race, class, gender, etc.).
  • it can be hard to define “community” - who “represents/speaks for the community” must be clearly articulated.
  • leadership and decision-making should be democratic to ensure people feel valued and respected.
  • issues of mistrust among community members - feeling “burned” because researchers have not “given back.”
  • community-based research is time consuming - it can involve complicated partnerships between individuals and as such, requires patience.
  • there may be differences of opinion about what are considered acceptable research methods (if academics are involved, they may want more “objective” and “distancing” quantitative methods such as surveys - community people may prefer qualitative methods such as interviews.

Appendix C


Research with a Purpose
Enhancing Community-based research
at the AIDS Committee of Toronto


AIDS Committee of Toronto
January 2002


Commitment to Community-based research

The AIDS Committee of Toronto recognizes that community-based research (CBR) complements and enhances the organization’s values. In particular, CBR is integral to the development of services that are client-centred and responsive, timely and relevant to clients’ needs, and caring and non-judgemental. In addition, CBR advances ACT’s role of “working co-operatively with other agencies, sharing experiences and resources and ensuring that needs are addressed in appropriate, sensitive ways.” AIDS Committee of Toronto (2001). Strategic Plan 2001-2004.

ACT’s approach to CBR is based on a set of principles identified and discussed by ACT staff in September 2001. These principles (outlined below) complement the “Guiding Principles” previously developed by the International Network for Community-Based Research on HIV/AIDS (INCBR). International Network for Community-Based Research on HIV/AIDS (2000). Communities Creating Knowledge: A Consensus Statement on Community-Based Research. Vancouver: Community-Based Research Centre

In expressing their commitment to CBR, ACT’s staff also identified the following:

· Supports needed to improve their ability to undertake or participate in research within their workplan activities;
· How workplan activities would benefit from research; and
· Research priorities.

These perspectives and commitments follow the Principles under separate headings.

Principles of community-based research

Community need and practical benefit

· CBR projects should be explicit about how the community will benefit from the research, the audience for the research output, and the social context of the research.

· Research should address community needs or issues that affect the community (i.e., research should fulfill practical needs)

Community involvement

· Researchers must provide opportunity for the community to influence the research question(s)

· Research must involve people who work in the community, but who are not necessarily trained researchers

· Relationships between the researchers and community members should be equitable. Researchers and community members may be knowledgeable about different aspects of an issue, or exhibit different ways of understanding an issue. Therefore, projects should be designed to encourage the contributions of researchers and community members according to their strengths and knowledge.

Partnerships and collaboration

· CBR is a tool for developing and sustaining partnerships. CBR should also contribute to building community capacity to identify research opportunities, develop and execute research projects, and use research results to develop or improve HIV/AIDS education and prevention.

· There should be a role for service users as partners or collaborators

· Potential research partners include: service users, ASOs and other community-based or nonprofit organizations, community members, community researchers, and university-based researchers.

Research methods

· CBR can be qualitative or quantitative. Projects should be designed to ensure that participants are treated with respect.

Dissemination and follow-up

· Dissemination is a way of sharing knowledge

· Research proposals must always include a plan of how the research will be disseminated and used

· Research must include a combination of dissemination avenues and formats to make the research results accessible to a broad spectrum of interests. These avenues and formats include:
  • articles in community newspapers and magazines
  • web sites and electronic bulletin boards
  • briefs and other reports targeted to a public policy audience
  • brochures
  • annual reports
  • community forums
  • workshops, seminars or discussion groups for ACT’s staff, volunteers, community members or other service providers
  • books and book chapters
  • articles in peer-reviewed journals


Accountability, honesty and respect

· Researchers are accountable to the community. This means that (a) communities have a say in how they expect to benefit from the research, (b) communities and researchers have a common understanding of the potential benefits, and (c) where possible, researchers assist communities in realizing the benefits.

· Researchers should acknowledge their assumptions about the community and the issues under investigation, and inconsistencies or shortcomings in the research. This gives communities a better sense of how to interpret or use research results

· Researchers must be honest about their aims and intentions in negotiating with the community and potential informants

· Researchers must respect participants’ confidentiality and privacy, and must design their recruitment and data collection methods to ensure participants’ safety

Language

· Researchers should use language that is non-judgemental and accessible

· Where necessary or possible, researchers should provide opportunities for participants from minority language communities to participate in their projects, and should prepare dissemination materials in languages other than English

Supports needed by staff to undertake research

To more efficiently incorporate research into their workplans and on-going tasks, ACT staff needs access to:

· Time to plan and execute research (including administrative and reporting requirements)
· Assistance and advice from a trained researcher
· Community-based ethics review process
· Skills building workshops
· Training on software generally used by researchers to manage and analyze data
· Opportunities to network with researchers from other organizations
· Appropriate tools for program evaluation, focus groups and other research practices
· Tools and resources for literature searches

How research can be of value to staff

The workplan activities of ACT staff would benefit from research that:

· Supports program development and provides justification for decisions affecting the design of programs and services
· Identifies service gaps or populations that are not served
· Improves accountability to funders
· Improves fundraising activities and understanding of fundraising trends
· Facilitates long range planning
· Gives credibility to services and programs and to ACT as a whole
· Generates up-to-date and informed perspectives on advocacy

Research priorities

ACT’s staff identified a number of research priorities to address the services and programs that are (or could be) available to various audiences, or in which various external stakeholders participate. Staff members may incorporate these priorities into their specific workplans, or may design them as special projects. The priorities are outlined below.

Three of the priorities (i.e., services, under-served or marginalized populations, and volunteers) address important aspects of ACT’s strategic plan for 2001-2004. AIDS Committee of Toronto (2001). Strategic Plan 2001-2004.

Strategic Direction 1 (Access and Equity) states that “ACT will continue to identify, implement and coordinate strategies … to reflect and serve the diverse communities of people infected and affected by HIV/AIDS.”

Strategic Direction 2 (Who we serve. How we serve) categorizes the programs and services (education, prevention, support and advocacy) and identifies ACT’s principle target groups (gay, bisexual men and other MSM; PHAs; women; and youth).

In addition, the Strategic Plan highlights a commitment to “involving volunteers in all aspects of ACT direction setting, decision-making and service delivery,” and recognizes that ACT would be unable “to maintain either the range or quality of our services” without the skill, generosity and commitment of [its] volunteers.”

Research priorities are as follows:

Research to improve services and access to services:
  • Assessing the barriers limiting access to services by potential clients and communities
  • Evaluating strategies to improve the accessibility of services to potential clients and communities
  • Client satisfaction with services and programs
  • Program evaluation
  • Identifying and understanding innovative outreach strategies


Research to improve ACT’s profile, role and quality of service among under-served or marginalized populations in Toronto (e.g., youth, Portuguese-speaking communities):
  • Sexual attitudes and behaviours among youth
  • Mechanisms and strategies to target women in minority ethnocultural communities
  • Access to HIV/AIDS services in ethnocultural minority communities

Research to evaluate and improve the role of volunteering and volunteers’ well-being:
  • Evaluating volunteers’ contributions to ACT
  • The role and impact of volunteering on volunteers’ sexual health and well-being.

Research to improve fundraising:
  • Evaluating the methods, feasibility and benefits related to diversifying revenue streams
  • The cost-effectiveness of fundraising
  • Evaluating the benefits of fundraising events for sponsors and stakeholders

In addition to the priorities outlined above, staff members also identified CBR opportunities related to the internal functioning of ACT (i.e., procurement of physical resources, processing of gifts and donations, etc.).

Staff expressed an interest in longitudinal evaluation of on-going programs and services (i.e., identifying critical elements of a program, and examining them every six months or annually, depending on the nature of the program). For example, issues related to whether the types or needs of clients change during the life of a program workplan may be addressed in this type of evaluation.

ACT staff recognizes that every research project may not incorporate all the Principles with the same high degree of attention. However, on balance, ACT’s research should observe the Principles as time, purpose and budgets permit.

Follow-up

ACT’s Research Committee discussed a draft of this report in December 2001. The Committee acknowledged staff’s interest in and commitment to community-based research, and noted that the priorities and topics were specific enough to be incorporated into staff members’ workplans. However, the Committee decided to embark on a discussion of strategic priorities and direction to identify and outline “big picture” issues that may guide the development of ACT’s research program to 2004. The Committee will base its discussion on “Research with a Purpose”, the current Strategic Plan, current programs and services offered by ACT, and the external environment for research (e.g., directions in research funding).


Appendix D – Sample Post-Workshop Evaluation Tool
AIDS Committee of Toronto – Workshop # 4
Workshop Evaluation (October 4, 2001)

Top-Secret Code: _______________

1a). How confident are you now in your ability to develop a sampling strategy for a research project? (Please Circle One.)

1
2
3
4
5
6
7
Not Confident At All
Not Confident
Mildly Not Confident
Neutral or Undecided
Mildly Confident
Confident
Very Confident

1b). If yes, how were we able to facilitate this?
______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

1c). If not, how could we have worked with you in a better way?
______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________


2a). How confident are you now in your ability to develop a respondent retention strategy for a research project? (Please Circle One.)

1
2
3
4
5
6
7
Not Confident At All
Not Confident
Mildly Not Confident
Neutral or Undecided
Mildly Confident
Confident
Very Confident

2b). If yes, how were we able to facilitate this?
______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________


2c). If not, how could we have worked with you in a better way?
________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

3a). Do you feel that you were able to achieve your learning goals for this session?

1
2
3
4
5
6
7
No, I achieved none of my goals
No
No, I achieved very few of my goals
Neutral or Undecided
Yes, I achieved several of my goals
Yes
Yes, I achieved all of my goals

3b). If yes, how were we able to facilitate this?
______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

3c). If not, how could we have worked with you in a better way?
______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

4a). Did you feel comfortable with the way the material was presented?

1
2
3
4
5
6
7
No.
I was very uncomfortable
No
No. I was
mildly
uncomfortable
Neutral or Undecided
Yes. I was mildly comfortable
Yes
Yes. I was very comfortable

4b). Please explain your answer. ______________________________________________________________________
________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________



5. One of the things I liked most about this workshop was . . .
______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

6. One of the things I liked least about this workshop was . . .
______________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________________

7. Are you a full time or a part time staff member? (Please Circle One.)

Full time Part time Other (Please describe: _____________________)


8. How would you describe your work within the agency? Are you currently working as a:
(Please Circle One.)

Community Development Staff Member
Support Service Staff Member
Finance, Administration or Library (Research and Program Devel.) Staff Member
Development Staff Member
Program Management, Human Resources Coordinator
Thank you for your thoughts!
We will carefully consider this information throughout the course of the workshop series.

Capacity Building Staff Advisory Group