The Evaluation Report of The Living Guide: Services for people in the Toronto area living with HIV or AIDS

prepared by eeb consulting & associates (Canada) for the
AIDS Committee of Toronto and
The HIV/AIDS Cultural Network of Metropolitan Toronto & Surrounding Area
April 1998

Note: The (paper-based) report is held as part of the circulating and reference collection of the Library



    Acknowledgement
      eeb consulting & associates (Canada) would like to thank all the individuals who took time out of their busy schedule to participate in this evaluation. Thank you. Without your contribution, we would not have been able to do the work.
      .
    Summary
      As part of the partnership agreement between the HIV/AIDS Cultural Network of Metropolitan Toronto & Surrounding Area and the AIDS Committee of Toronto (HACN & ACT), a third party evaluation of the design and impact of The Living Guide was necessary before a new edition could be produced.

      The evaluation was designed to find out who was using the Guide, how it was being used, and how it could be improved.

      During the months of February - April 1998 evaluations were administered through the mail, over the phone, one to one, and in round table discussions with Service Providers and Service Users.

      Out of 216 surveys mailed out, 9.7% of the Service Providers responded. 60 responses were received from Service users of which 96% was a result of distributing surveys at the Toronto Persons With AIDS Foundation.

      In addition, advertisements and notices were placed strategically throughout the city. A confidential e-mail and phone-in line was provided. There were no responses from the e-mail and only three calls on the telephone line.

      It is not clear what the factors are affecting this rate of response.

      Those in the "AIDS business" (the Evaluation Advisory Group) suggested that the community is "surveyed out"; that given the timing with megacity - changing of Toronto politically and the funding implications - people were pre-occupied with keeping their services/agency alive.

      Whatever the reason, of those organizations/individuals who did respond, both Service Users and Service Providers, indicate that the Guide is a very useful tool. For Service Providers, it is a part of their professional kit for working with a range of clients. This includes clients who may or may not be affected by HIV/AIDS but are looking for services in their language of comfort.

      A number of suggestions were made regarding how to improve the Guide. These include making language improvements, designing the cover of the Guide to reflect the richness of the diversity of the targeted groups and finding the ways and means of updating the Guide on a regular basis. Updating is used to include the management of all aspects of the Guide including the production and promotion of the Guide on an ongoing basis.

      The Guide is promoted as being a product of the joint partnership between the HIV/AIDS Cultural Network and the AIDS Committee of Toronto. The impression received through this evaluation process is that the Guide is taken for granted. In producing a new edition of the Guide, a number of actions need to take place.

      .
    Recommendations
      The recommendations based on the data collected are identified below.

      1. The HACN and ACT need to revisit the nature of their partnership, identifying clearly their expectations, roles and responsibilities of each other especially with regard to keeping The Living Guide alive.

      2. HACN/ACT should determine the best way to resolve improving the content of the Guide by involving a cross section of the user population in consultations along with representatives of HACN.

      3. The HACN and ACT should rely on the expertise within the group to determine the best layout and design keeping in mind the importance of colours to the majority of the group.

      4. HACN/ACT need to redesign the distribution strategy of the Guide to incorporate maintaining regular ongoing contact with agencies in order to monitor the use of the Guide

      5. HACN/ACT need to determine a funding strategy which takes into account the adequate allocation of human, financial, physical resources in order to manage all aspects of producing, maintaining and managing The Living Guide on an ongoing basis.

      .
    Background
      What is The Living Guide?

      "The Living Guide is a directory of services available in Metropolitan Toronto for people living with HIV or AIDS. The Living Guide is targeted to service users and service providers and strives to make its information accessible to people from diverse communities."

      (Note: at the time The Living Guide was produced, the city of Toronto as it is today - megacity - was not in effect. This terminology refers to the former understanding of Metro Toronto, Metropolitan Toronto, Greater Toronto, Toronto and Surrounding Areas. Hence some of the listings were not previously considered to be a part of Toronto. They are now considered a part of Toronto)

      "The introduction to The Living Guide states "The variety of services in Metro Toronto has grown a great deal from the original printing of The Living Guide in April 1992. However, as in 1992, it's still difficult for people living with HIV or AIDS to find out what and where their service options are."

      "The Living Guide is an attempt to provide a self-help roadmap to services that are relevant and appropriate to people living with HIV or AIDS. This edition also includes new features that begin to address the difficulties that people from diverse ethnocultural communities' experience finding and making use of linguistically and culturally appropriate services."

      "Although education activities are vitally related to support and advocacy services (and in many cases are provided by the same organizations), The Living Guide does not attempt to provide an exhaustive listing of such programs."

      The press release written for release during AIDS Awareness Week, September 29th to October 5th, 1996 states,

      "Over 5000 copies of the 1996 edition of The Living Guide: services for people in the Toronto area living with HIV or AIDS have been distributed to community-based service organizations, members of the Toronto HIV primary care physicians group and public health units across Metropolitan Toronto.

      "This 192 page directory lists the services of over 375 groups, agencies and institutions in Metro Toronto that provide the care and support services that people living with HIV or AIDS can access to as part of their ongoing health maintenance program.

      "The directory's improved format makes it easier for users to find the services they need, as well as to alert them to some of the issues related to accessing those services.

      "Special attention was paid to the needs of people living with HIV or AIDS throughout Metro's diverse ethnocultural communities. An expanded languages index lists agencies which can provide services in one or more of 89 different spoken and signed languages. Translated passages on the cover and within the directory make it easy for people to find the information they need in the language of their choice. And a revised table of contents and services index refers users quickly to the agencies that provide targeted services to ethnocultural or other populations.

      "This is the second time that the AIDS Committee of Toronto and the HIV/AIDS Cultural Network of Metropolitan Toronto & Surrounding Area ( a partnership between more than 30 community-based service organizations and departments of public health) have collaborated to produce The Living Guide".

      .
    Introduction
      As part of the partnership agreement between the HIV/AIDS Cultural Network of Metropolitan Toronto & Surrounding Area and the AIDS Committee of Toronto, a third party evaluation of the design and impact of The Living Guide was necessary before a new edition could be produced.

      This document is the report of the evaluation of The Living Guide, 1996 edition. The original rasion d'etre behind the production of The Living Guide continues to hold true.

      "Self help can be an important part of coping with the personal discovery of AIDS and HIV-related health problems. People living with HIV and AIDS have a right to access the wide variety of support services that exists in Metropolitan Toronto. Accurate information and an improved understanding of the "system" empowers people with HIV and AIDS to help themselves."

      This is the statement of principle behind the objective to develop the Guide as presented in the original proposal developed by Community Information Centre in Toronto in its proposal to Health and Welfare Canada in November 1990.

      The evaluation was designed to find out who was using The Living Guide, how was it being used and how could it be improved. With this in mind the following approach and methodology was used.

      In particular, the evaluation looked at:

      .
    Effectiveness/
    Usefulness:
    • Does the Guide make it easy for people to find the services they need?
    • What are people's experiences contacting agencies listed in the Guide?
    • What effect (if any) has the Guide had on the lives of people with HIV or AIDS?
    • Is the Guide useful to both service users and service providers?
    • What types of information are especially important to service users? To service providers?
    • Does the information become out-of-date too quickly?
    • What are the expectations that service users and service providers have of the publishers?
      .
    Content:
    • Is the information readable and accessible (what are the cultural barriers to the document)?
    • Are there to many / too few agencies listed?
    • Is there a concern about who gets listed in the Guide?
    • Are the section narratives useful / comprehensive enough?
    • Is a sufficient range of services covered?
    • Is there enough information given in each agency listing? Is it pertinent?
      .
    Design / Layout:
    • Does the layout make it easy to get to the information needed?
    • Is it easy to read?
    • Is the document broken up into logical sections?
      .
    Distribution:
    • How did people hear about the Guide?
    • Where did people get a copy of the Guide?
    • Where do people expect to be able to get a copy of the Guide?
    • Where and how should the Guide be advertised?
      .
    Production
    • How often should the Guide be produced?
    • Should the information in The Living Guide be broken down into smaller, specialized Guides that are published separately and at different times of the year?
    • What is the best way to keep track of the information contained in The Living Guide?
    • What are public and private sources of funding that could support The Living Guide on an ongoing basis?
      .
    Approach and Methodology
      The Living Guide is targeted to service users and service providers (two groups who don't often speak the same language), and strives to make its information accessible to people from diverse communities. As such, innovative approaches to gathering user feedback needed to be used:

      Most Living Guides are distributed anonymously, so finding people who have used it posed quite a challenge. In order to make contact with people who had ongoing relationships with service providers, our strategy focused on tapping into the existing client and community networks of the member organizations of the HIV/AIDS Cultural Network of Metropolitan Toronto & Surrounding Area. More traditional methods of advertising were then used in an attempt to make contact with harder-to-reach people.

      In order to obtain as much information as possible, the following approaches and methodologies were used:

      1) through existing networks:

      ACT's AIDS Biz list includes AIDS service organizations and AIDS programmes of community-based and social service agencies, Hospital HIV clinics, HIV primary care physicians, anonymous HIV testing sites, and community health centres.

      2) through other means:

      Flyers were posted at key sites across Toronto (including bars and bathhouses where gay men meet, the offices of HIV primary care physicians, the waiting rooms of hospital-based HIV clinics, and community events such as league nights of the Toronto Historical Bowling Society) indicating that The Living Guide was being evaluated (see Appendix I)

      .
    What worked?
      Approaching service users at the Toronto People With AIDS Foundation. Most persons approached in this setting did not mind completing the questionnaire. A few indicated that they would prefer not to do so stating their reasons as having too much to deal with in their lives at that time, not feeling comfortable in English or not being familiar with the Guide.

      Telephone surveys. While the actual numbers of those who responded by telephone is not significantly more than the completed questionnaires returned by mail, the response was more encouraging. Over two hundred surveys were mailed out, and over 90 telephone calls were made with 21 responses recorded, a response rate of 23%.

      .
    What did not work?
      Round Table discussions. Although arranged at different locations around the city and at different times of the day and evening, the response was limited. Five sessions were held. Three sessions were attended by one person each and the other two by two persons at each session.

      Confidential telephone line and e-mail. Three persons called and left their name and number for us to call and interview them. There was no response by e-mail.

      Drop -off boxes at AIDS Service Agencies. 5% of the service users responses were received in this manner. No Service Providers responses were received in this manner.

      .
    Details of the Methodology Used
      Through Service Providers
    • 216 Service Provider questionnaires were distributed to ACT's AIDS Biz. List. Each Service Provider also received a copy of the Service User questionnaire to be copied and distributed to their clients.
    • A confidential phone line and e-mail address was set up through ACT to allow individuals to make contact with us and/or to send in their feedback.
    • 5 focus groups were held across Toronto the week of March 9th through to March 13th. (see Appendix for list)
    • Individuals were approached at the Toronto People with AIDS' food bank on two consecutive Thursdays in an effort to obtain Service User data.
    • Survey Drop-off boxes were placed at a variety of AIDS service agencies.
    • Telephone interviews were done with key organizations in order to obtain Service Provider data.
    • Similar organizations were approached to identify best practices.
    • Possible funders were identified and initial contact made regarding their level of interest re funding The Living Guide.
    • Regular meetings were held with the Evaluation Advisory Committee.
    • The draft final report was discussed with the Evaluation Advisory Committee prior to the final report being submitted.

      Other Methods

    • A confidential voice mailbox was set up through ACT to allow individuals to call in their feedback. Only the consultants had access to the box to retrieve messages.
    • Flyers were posted at key sites across Toronto indicating the ways in which organizations and/or individuals could participate in the evaluation process.
    • Advertisements were placed in community papers, such as in XTRA, and in HACN member's newsletters, such as ACT's volunteer bulletin (mailed to over 1000 people) and PSM.
      .
    Evaluation Results
      Service Providers

      The results of the Service Provider questionnaires are tabulated in the chart on the next page. A summary of the comments follows the chart. Eighteen service providers responded by mail. Twenty-one service providers where interviewed by telephone **(see Appendix 1 for interview questions). The totals in question # 1 indicate the total number of those who responded. Organizations were able to self identify in more than one category listed. Therefore, the total numbers do not necessarily reflect total numbers surveyed.

.
        Questions
        Results
              .
        Please tell us whom you are by circling as many responses as apply.
        HIV/AIDS Service Provider (18)
        Medical Office (1)
        Hospital (2)
        Ethnic Specific Organization (2)
        Community based organization with multiple services (7)
        Community Health Clinic (2)
        Public Health Unit (9)
        Legal Clinic (0)
        Anonymous HIV Testing Site (0)
        Hostel/Shelter (0)
        Social Services Office (0)
        Other (3)
              .
        3. How did you get a copy of the Guide?
      • 8 organizations received the guide from ACT
      • 2 organizations received the guide through the mail
      • 2 organizations were not sure how they obtained the guide
      • 1 organization received the guide from PWA
      • 1 organization indicated the guide was shipped to them
              .
        4. Do you have more than one copy of the Guide?
        Yes (15)
        No (3)
              .
        5. How do you use the Guide?
      • Service directory/like a phone book
      • Information and referral
      • As a resource
      • As a research tool
      • For phone numbers
      • Counseling tool with clients
      • Use as part of job
      • Outreach
      • Conference / workshop work
      • Educational tool
              .
        5c) Do you promote the use of the Guide?
        Yes (14)
        No (2)
              .
        unedited responses
      • To anyone interested in AIDS/HIV, for personal and agency use
      • Make available to all clients, especially those with limited knowledge of Toronto HIV/AIDS services/organizations
      • By providing it to individuals who are HIV positive
      • Share it with clients, volunteers and offer it as a resource tool
      • As a tool to newly diagnosed, new to Toronto, also for new works in HIV/AIDS and beyond
      • Refer clients/colleagues to it
      • We provide 2-day workshops to others who do AIDS education. We have a display copy and advise participants of its usefulness & how to obtain a copy
              .
        5d) Are there any barriers to promoting the Guide?
        Yes (3)
        No (11)
              .
        unedited responses
      • No - excellent resource
      • Some clients who have not disclosed their HIV status to family or friends hesitate to take the guide home. Some cultures are reluctant to be referred to community groups who represent their own culture because they are concerned that they might meet someone who knows them.
      • Language barriers. Need for language translation under the description of ethnic ASOs.
              .
        6. Is the information in the Guide useful to you,
        as a Service provider?
        Yes (15)
        No (1)
              .
        unedited responses
      • Assists in making me aware of service providers in the Metro area
      • Sometimes use it to double check geographic boundaries
      • Summary of agency services very useful
      • Gives good overview of variety of services involved in AIDS education.
      • Clear; concise; specific
      • Quick and easy to locate services
      • Outdated
      • Clients can be referred to services and agencies closer to where they live - easily found in the guide
      • Well organized; user friendly
              .
        What information is most useful to you?
      • Support groups; description of content
      • Phone # and addresses of agencies
      • All info is useful
      • Information services offered before approaching individual listed services
      • Details regarding programs and services
      • Like the organization of contents of this edition
      • Fax #s use to send faxes to other ASOs
              .
        What information is least useful?
      • Hard to keep so much information up-to-date
      • Contact person they change frequently
      • Service directory - use different coloured paper as markers
      • I need to screen services listed to see if they are proactive for my clients
      • Some of the ethnocultural stuff we have little use for it - but many would
              .
        9. Is the information on agencies/services enough?
        Yes (13)
        No (1)
              .
        unedited responses
      • Yes for Toronto - we have our own small list of physicians & dentists that we also use
      • The guide is very helpful; lots of resources listed
      • Need to go outside of ASO & HIV. Clients have multiple problems need to list e.g. battered women services, shelters, and other ethnic agencies doing immigrant settlement work.
      • Yes, people should call if they need more info - otherwise guide gets too wordy
      • Public health is not listed as an agency involved in home visits
      • We do not use the guide enough to comment on this
      • Perhaps now could include email contacts or websites
      • I believe it is_each summary is detailed
      • Basically the information there is enough to get an idea of services provided - call each are for further info.
      • It needs to be updated yearly to keep up with program changes and new services
      • Need indication of how much HIV knowledge an agency has
              .
        9a) If you use the guide for referrals, does the information make it easy to decide who to refer Clients to?
        Yes (13)
        No (0)
              .
        10. Are you aware of any other health
        issue specific guide?
        Yes (4)
        No (11)
              .
        unedited responses
      • Four organizations indicated the Blue Book
      • One organization indicated the Cancer society and Heart and Stroke foundation
              .
        11. Does the layout of the Guide make it easy for you to use?
        Yes (12)
        No (3)
              .
        unedited responses
      • Well organized
      • Good layout
      • Agency Directory  yes - Service Directory  no
      • Indexes and list of services section too long and complicated
      • Different colours for the sections
      • Incorporating Mega city and services outside of Toronto
      • Language translation for ethnic agencies description
              .
        12. Can you think of any way in which we can improve the layout of the Guide?
        Yes (7)
        No (7)
              .
        How often should the guide be produced?
      • Yearly - 9 organizations said yearly
      • Every 2 years - 6 organizations said every 2 years
      • 3-5 years- 2 organizations said every 3-5 years
              .
        How can you help to keep the information
        in the Guide updated and relevant?
      • Fax (overwhelming response for this method)
      • Mail
      • Telephone messages
      • email
      • printed document in Living Guide
              .
        15. Would you be interested in advertising
        in the Guide?
        Yes (6)
        No (8)
              .
        unedited responses
      • Yes - however would need widespread distribution
      • No - most organizations which said no because no dollars to do so
              .
        What would you be willing to pay for a ½ page to advertise your service
      • Responses ranged from free to $250. With the average being $100.00
              .
        17. Have you received feedback from
        other Service Providers about the Guide?
        Yes (6)
        No (9)
              .
        "Overall it is a great tool to have and very
        helpful to our agency in serving our clients"
      • very positive, helpful
      • the book is a great guide to use and helpful
      • most find it useful/worthwhile
              .
        18. Have you received feedback from your
        clients about the Guide?
        Yes (8)
        No (6)
              .
        "an indispensable resource for a
        great many people"
      • Yes - people like it and use it. It is a valuable resource. The only other one equally as valuable is the blue book
      • very positive
      • clients find the guide very helpful and comment on the detail of each agency
      • comments on the lack of out of town services
              .
        What else would you like to tell us
        about the Guide?



        "The guide is useful for service providers,
        but many clients find it too big to take home, perhaps a smaller pocket guide could be
        provided for clients e.g. something similar
        to what the health food stores put out."
      • Keep up the good work
      • Keep up the good work  it helps us all
      • Keep it going
      • It provides a basic and useful resource to people
      • An excellent resource
      • The information on page 32 on HIV testing is misleading. It is rare that others request medical records. The information here makes it sound as though lots of people have access. It also does not stress that one has to give consent for this information to be shared.
      • More need for language and cultural sensitivity; need for widespread distribution, need for ongoing update.
      • Need wider advertising - clients often surprised guide exists
      • Indexing not to the point enough needs improvement
      • Division of services needs spelling out more
      • Subject headings could be more specific - e.g. no subject heading for services related to women (pg. 53)
.
          Summary of Service Providers' Survey Responses

          The table above details the responses of the Service Providers. The comments identified after each question are direct quotes from responses. The purpose of presenting the information in this manner is to give as much details as possible.

          Generally speaking, Service Providers use The Living Guide as a resource and reference guide in identifying a range of services for their clients. (Note 'client' is used interchangeable with 'service users') Often the Guide is used for far more than responding to an HIV/AIDS specific related need. For example, more than one Service Provider indicated that they used it to identify services for clients in their mother tongue - as a springboard to other services.

          A significant amount of Service Providers mentioned using the housing section extensively.

          Some Providers promoted the Guide by giving a copy (in person or by mail) to their client; by showing/talking about how the Guide can be used and by confirming specific services listed in the Guide prior to recommending a service to their client.

          While there were some suggestions for improving the layout of the Guide, there was general agreement that it worked well, that the information on each Agency was enough and should anyone need more details a call could be made to the Agency.

          The suggestions for improvement included: finding a way to keep the information up to date, as it becomes outdated quickly. Currently, Service Providers "in the know" know which Agency has closed, change of service and so forth through the informal network. There needs to be a way to capture this "knowing" into keeping the Guide current. This is demonstrated by someone indicating that the "contact person" at agencies change often. This is true, but this person was drawing on their informal knowledge of the AIDS business, as there are no contact names listed in the Guide!

          Others thought that related services should be listed. For example, services for "battered women", settlement services and shelters.

          There was also a suggestion that it would be useful to have some indication of how much, to what extent, the Agency listed does provide services targeted for HIV/AIDS persons.

          It was suggested that the Guide needs to be updated regularly - perhaps on a yearly basis. And, that the best way to update the Guide is by having the agencies send in regular updates by fax. While the electronic form of communication was mentioned, the discussions usually turned to seeing this as a barrier for some agencies.

          Discussions regarding placing paying advertisement in the Guide suggested that while it was an idea to explore, it should not become a source of funding the production of the Guide as this could change the purpose of the Guide. Most agencies could not advertise in the Guide as their funds are limited. But, if it were an option to consider, at most, an average of $100 - $150 per page was suggested.

          Taking all of these responses into consideration, this group of Service Providers use the Guide on a regular basis. They consider it a part of their "tool kit" in working with their service users and would notice its absence. It was felt that a process and procedure in which agencies fax in their updated information on a regular basis would be the best way of keeping the Guide current.

              .
      "Some of the agencies
      that indicate that they
      provide services in
      Spanish, actually do not.
      Only one or two of these agencies actually provide AIDS related service- we need to check with each organization that they
      actually do what they
      say they do."

      "Keep the format, keep
      the layout. It's an easy
      way to do it. No need to spend $$$$ to do
      something new.
      Based on information
      in print, it is critical to
      check on its accuracy."

      "The information is very comprehensive - the information is easily accessible to us but
      not to our clients based
      on their language skills- some of the more
      general service should
      be removed
      e.g. no need to list a
      dental clinic, can they
      do HIV counseling?"
          Summary of Roundtable Discussions

          As a part of the methodology, round table discussions were planned for the week of March 9 -13. Five different locations at five different times of the day were organized and the information widely distributed several times to the appropriate agencies. A total of seven persons participated in these discussions, all seven were service providers.

          A significant amount of what these service providers had to say is already reflected in some of the comments received over the telephone and on the surveys. But, given the nature of the interaction - face to face, there is additional information.

          Six persons had copies of the Guide and indicated that they had received them during the initial distribution of the Guide. One person was relatively new to the AIDS business and had received his copy from CATIE fairly recently.

          They all use the Guide as a resource for referrals over the phone and during in-person consultation. Face to face consultations usually result in the client receiving a copy of the Guide with some tips on how to use the Guide - depending on their comfort level in English.

          More than one person mentioned that the Guide was too gray. It was suggested that more lively graphics was needed. Pictures of people, a collage perhaps - but not the current cover.

          Some of the comments received are recorded below.

        • Is it possible to have doctors who will take HIV/AIDS patients listed by languages which s/he speaks ?
        • Can we ask each agency what % of their service is in whatever language they are claiming to offer service in?
        • Get more details on the type of counseling offered, and keep general counseling to a minimum. Also at minimum list the agencies with the latest information on the research - drugs.
        • The HACN needs to maintain contact with the services/voices in the Directory. We need to find a way to continue the relationship." For example, should we be thinking of providing a service of educational activities to share what we do, to find what we need from the community and what the community needs from us.
        • We need a universal code for the services to accommodate low literacy for our service users .
        • The Living Guide is misleading, we need to give tips on living. Perhaps we need to partner with "Managing your Health" and promote them both together.
        • There needs to be more room to make notes.
        • The cover looks like an ACT document, it needs to be richer in order to represent the diversity of the people it is reaching out to.
        • The section on Drug Use needs to be broken down some more. E.g. Substance Abuse, Relapse Prevention.
        • PHA Rights/Legal Advocacy section needed  to empower PHAs.
        • Need to include South Asian Family Support services.
        • Need to make sure that mainstream agency has copies of the Guide.
        • Heads and Sub-heads are not clear, it looks like a solid block. Need to see it at a glance.
        • Some fax numbers are missing. Phone numbers need to say phone. E-mail addresses are needed.
        • Bold headings on left side.
        • Do not mix English and another language.
        • What does community-counseling mean?

          The point was made that "the South Asian community" is not a term with which the persons referred to use to describe themselves. There needs to be a decision made about what terminology to use that is acceptable to the persons being described. "Ethnocultural specific, clients do not know what these words mean, use plain language for them. These are social workers terminology."

          Discussions with the ACT staff person responsible for The Living Guide indicates that these types of discussions were a part of making The Living Guide a reality. As a result, all the terminology used in the Guide was agreed to by the HACN. Also, the Guide was reviewed by East End Literacy for plain language. (Assisting agencies and organizations with producing documents in plain language is a service which is provided by this agency.)

          In reviewing the draft report with members of the Evaluation Advisory Group, discussions reveal that the Guide was reviewed by Service Providers. Service Users were not involved in the process. Consequently, the tone, feel and language in the Guide is that of a group of social service professionals. If in fact, there is a real expectation that Service Users will use this more than the results of this evaluation suggests, then consideration will have to be given to involving them in the pulling together of the next Living Guide.

          In summary, while only seven Service Providers participated in the round table discussions, the process encouraged more detailed exploration of the areas under review. In addition to repeating some of the observations made by those who participated in the written and telephone surveys some insightful comments were made. Attention needs to be given to these comments and ways and means found to respond to keeping not only the Guide alive but also the HACN/ACT partnership - re its role and responsibility towards The Living Guide.

          Guide Users

          In designing the questions for Guide Users, the questions were kept short and to the point. The rationale was, filling out another form would be the last thing that a Service User would want to do in addition to all the other forms one usually completes in using services related to being HIV/AIDS. The hope was that users would see the notices placed strategically and call in to respond. This did not materialize as already mentioned under Approach and Methodology.

          The results of the Guide Users questionnaire are tabulated in the chart below. Sixty individuals responded to the Guide Users questionnaire. Ninety-five percent of the questionnaires received were distributed at PWA's offices during food bank distribution.

          A significant number of those who responded to the questions designed to see to what extent the Guide was being used by Service Users indicate that the Guide was useful as the information in the Guide contributed to them deciding which services to use.

          Over 50% of the Users have a personal copy of the Guide. Of this group more than half of them lend their copies to friends and clients. When the Guide is used, it is used for a range of information. This information includes: seeking appropriate social services, housing information, emergency services, group therapy and so forth.

          There was overwhelming positive response to the question, "does the layout of the Guide help you to decide which agencies to approach?"

          In summary, based on the responses received, it appears that Service Users use the Guide and find it helpful in meeting their needs. One of the suggestions for improvement suggested by this group not mentioned by Service Providers is the need to have larger print. This is a consideration which the next reprinting of the Guide should incorporate in recognizing the effects of AIDS on one's eye sight. A chart summarizing the responses is presented on the following page.

.
        Questions
        Results
              .
        1. How did you hear about The Living Guide?
        PWA Foundation (19)
        Doctor's office (8)
        ACT (17)
        Word of mouth (2)
        Other (13)
              .
        2. Do you have a personal copy of the Living Guide?
        Yes (38 or 68%)
        No (3 or 5%)
              .
        3. If you have a personal copy of The Living Guide, do you lend it out?
        Yes (22 or 36.6%)
        No (8 or 13%)
              .
        4. Who did you lend the Guide to?
        Friends
        clients
              .
        5. When you use The Living Guide, what information are you seeking?
        AIDS NETWORK
        SOCIAL SERVICES
        HOUSING INFO
        HEALTH SUGGESTIONS
        REFERENCE
        GROUP THERAPY
        EMERGENCY SERVICES
              .
        6. Did the information in The Living Guide help
        you to decide which agencies to approach?
        Yes (46 or 76.6%)
        No (4 or 6%)
              .
        7. Does the layout of The Living Guide help
        you to use it easily?
        Yes (56 or 93%)
        No (4 or 6%)
              .
        8. Did the information help you to identify
        which agency would help you?
        Yes (46 or 76.6%)
        No (4 or 6%)
              .
        9. What suggestions can you make
        to help us improve the Guide?
      • GOOD WORK
      • CHANGE THE COVER
      • LARGER PRINT
      • UPDATE MORE FREQUENTLY
      • INDEX TO INCLUDE LIST NOT ONLY BY GROUP BUT BY MAJOR INTEREST
      • CLARITY
      • FRENCH
.
      Best Practices

      "The ACN needs to
      maintain contact with
      the services/voices in
      the Directory
      ...e.g. do we invite
      them to events without charging them?"
          The Living Guide. How does it compare to similar publications, i.e. directories for persons living with HIV/AIDS or directory of services designed to assist persons seeking specific health related services? We wanted to know if other publications had developed and implemented strategies for evaluating and producing their directory. And if so, what could be learnt and applied to the next production of The Living Guide. (See Appendix IV for the guidelines used)

          With the information received from the Evaluation Advisory Group, six organizations were contacted. These were the Gay Men's Health Crisis, (New York), Hamilton AIDS Network, AIDS Vancouver, Canadian HIV/AIDS Care Giving Professional Network, Ontario Hospices Association and Community Information Center of Toronto.

          Based on the discussions with these organizations, The Living Guide appears to be a trendsetter. None have conducted formal evaluations of their directories. Also, with the exception of the Blue Book, the identification of a department or a person with the responsibility for managing all aspects of their directory was not possible.

          AIDS related directories appear to be driven by an individual's commitment to providing such a service. Once this initiative is done, reprinting is done based on updating the information, not necessarily on a combination of updating and evaluating the effective use of the directory.

          The results of discussion are presented below. The names of the individuals are provided under separate cover.

              .
      Funding Search
          In keeping with the some of the areas to be explored in this evaluation, possible funding was identified. Specifically, the question "What are public and private sources of funding that could support The Living Guide on an ongoing basis?" guided the data gathered in this area.

          A random selection of 13 potential funders were identified and initial discussions held with them. It was made clear that these were "feeler" discussions and that the appropriate Living Guide person would follow-up at an appropriate time.

          All 13 stated their level of interests, where and how their sponsorship dollars are allocated, and in some cases what the initial processes were for them to consider The Living Guide as a fit.

          All 13 responses are detailed under Appendix IV.

          Paid advertisement was explored with Service Providers. The response did not suggest that enough funds could be raised in this manner to contribute to producing the Guide. Most agencies do not have a budget, which would accommodate this item.

              .
      What Does It
      All Mean?
          As indicated earlier, the evaluation was designed to get answers, suggestions, which would improve the next edition of The Living Guide. The areas to be examined were very specific. They were effectiveness/usefulness; content; design/layout; distribution; and production.

          What follows is the analysis of the information received from Service Users, Service Providers, the Best Practices Review and Potential Funding sources. The specific questions from which this information is drawn are identified in the Introduction on page 5 and detailed in Appendix I. The questions are not repeated here. What is presented here is the analysis to the responses under the areas probed.

          Summary and Recommendations

          Generally speaking, The Living Guide is a useful resource for those who have used it. It provides information on services which meet the specific needs of clients who may or may not be living with HIV or AIDS.

          The content of the Guide is, in general, accessible by those who participated in the evaluation indicating that the information about each agency is the right amount. Some felt that there ought to be a way to state to what extent an agency in the Guide provided HIV/AIDS related service without having to call on the phone.

          The design/layout is acceptable to most. Some felt that the design was "grey" and needed to be made more colourful. And some felt that while the attempt to speak to non-English speaking clients was a remarkable improvement over the last edition, this needed to be continued.

          The distribution of the Guide - Service Providers received the Guide mostly from the initial distribution while Service Users received it from Service Providers or by picking a copy up at significant places such as ACT, PWA.

          The production of the Guide should be done on a yearly basis with agencies faxing in their updates. The size should remain more or less the same. In order for this to become a reality, ongoing funding for the Guide needs to be established. Whether or not some of the funding sources identified can provide this assistance will have to be explored.

          The Details

          The results presented in the above text is expanded on in the following section. The headings correspond with the areas probed in the evaluation.

          Effectiveness/Usefulness

          The questions asked were designed to identify the effectiveness and usefulness of the Guide to both Service Providers and Service Users.

          In general, both Service Providers and Service Users found the Guide to be an invaluable resource. Not only for HIV/AIDS related services but also including services for housing, ESL services and other supportive services.

          Given the rate of response it is difficult to state to what extent the Guide has had (or not had) an effect on the lives of people with HIV or AIDS. Based on the information received, those who have used the Guide find it useful. No one suggested that they could do without the Guide. It is implied that the Guide is now a recognized and reliable source of information for a range of services. It would be sorely missed if it were not reprinted.

          The description of services appears to be among the most useful information in the Guide. It allows the reader to determine which service might best serve his or her current need. This is true whether or not it is a Service Provider working with a client or a client using the Guide on his/her own.

          The main issue identified with the usefulness of the Guide appears to be how quickly the printed information becomes out dated. For Service Providers, while the informal sources of keeping updated work, it is still identified as a source of frustration. Especially, when a client calls in to report that the service recommended is no longer in existence. Or, their interpretation of the service offered, based on how it its presented in the Guide is different in reality.

          The most common suggestion in resolving this issue was that agencies should keep The Living Guide up to date by faxing in changes on a regular basis.

          This is an excellent suggestion, one which is barrier free. That is, most agencies have access to fax machines. They do not necessarily have access to electronic mail. What is assumed throughout all of the discussions is that there is a staff person, somewhere who is responsible for The Living Guide. The expectation is that the publishers of the Guide will take all the suggestions made during the course of the evaluation and incorporate them into producing an update Guide. At no point did any one (with the exception of the ACT staff person ) suggest the "how" to making this a reality. Discussions about this and other issues at the Evaluation Advisory Group suggested that a clear recommendation should come forth which would allow staffing to be a reality.

          Clearly, HACN and ACT need to determine how best to update, produce, promote and keep The Living Guide alive and well. The identification of resources - human, financial and physical is critical for the survival of the Guide.

          The evaluation process did not leave the Evaluators with a sense of the Guide belonging to the HACN. The most caring and helpful person was the ACT staff person. There was limited participation by some members of the HACN Evaluation Advisory Group and very little participation by the majority of the HACN. The level of response to the data collection process speaks for itself.

          Recommendation: The HACN and ACT need to revisit the nature of their partnership, identifying clearly their expectations, roles and responsibilities of each other especially with regard to keeping The Living Guide alive.

          Content

          With regard to content, the intent was to find out if the information in the Guide was readable and accessible to all given the diverse group of persons catered to in the Guide.

          The results indicate that generally speaking the information was readable and accessible. This was especially true in the case of Service Providers who are familiar with working their way through resources like the "Blue Book" and who have an appreciation of the logic of the Guide. At the same time, Service Users did not have any major concerns with accessibility.

          Most Service Providers indicated that the number of pages was appropriate and that they would not want to see the Guide any thicker. Nor did they want to see it broken down into smaller specialized guides. (Although one person did suggest doing this.)

          The issues regarding content, from the Service Providers point of view, were: the swiftness with which information becomes dated; a cost effective way of keeping the Guide up to date; the need to have a better sense of the extent to which an agency has the expertise/specialization to work with an HIV/AIDS client; and the need to provide a language index with all the information written in the appropriate language, with no English used.

          The content of the Guide is acceptable. While requiring more language specific information, most do not want to see the Guide increase in size. Also, some services not currently included, for example, Public Health, should be included in the new edition,.

          Perhaps one of the ways to better meet as many people's needs as possible is to have a "help" telephone line as a part of The Living Guide Service. The appropriate information could be accessed in a wide range of languages and in so doing assisting in responding to users of the Guide who are challenged by their level of literacy in English.

          Where this service would reside and who would maintain it is a question for HACN and ACT to resolve. Again, based on the level of participation of the HACN in the evaluation, it is difficult to suggest any one Agency.

          Recommendation: HACN/ACT should determine the best way to resolve improving the content of the Guide by involving a cross section of the user population in consultations along with representatives of HACN.

          Design/layout

          The questions were designed to find out if the layout of the Guide made it easy to get to the information needed in a logical manner, the responses to the questions were mixed.

          While indicating that the design and layout contributed to finding information easily, some Service Providers said that this was true only because they knew what they were looking for. It was felt that if English was not one's first language or if one was not familiar with the Agencies it could be challenging finding information.

          On the other hand, Service Users had no problem using the Guide as is.

          The recurring issue, in one form or the other, was the cover of the Guide, the look of the Guide. It was suggested that the cover of the Guide needed to be more reflective of the diversity

          of the people it was trying to reach. While appreciating the languages on the cover, the grayness of it needed to be

          improved. This could be done, for example, by having brighter colours, a collage of people, pictures of people and so forth. While this is an excellent suggestion, it needs to be tempered with an eye to being cost-effective. The more colours, the more the cost.

          Recommendation: The HACN and ACT should rely on the expertise within the group to determine the best layout and design keeping in mind the importance of colours to the majority of the group.

          Distribution

          Most Service Providers heard about the Guide through the initial distribution of the Guide in 1996. Most Service Users heard about/received the Guide from their Service Providers. There was not much thought given to how the Guide should be distributed by Service providers. One gets the sense that they assumed the Guide would arrive.

          Based on the response to the evaluation, it is suggested that the initial distribution strategy should be changed. Instead of delivering boxes of The Living Guide to agencies, it is suggested that one copy of the new Guide be mailed indicating that should the Agency request more copies that they indicate so. This would contribute to building an ongoing process around the Guide and encourages regular interaction/feedback/update on an agency.

          A staff person responsible for maintaining the Guide could be pro-active and maintain ongoing outreach to agencies. This is based on the expectation that funding will be found.

          Recommendation: HACN/ACT need to redesign the distribution strategy of the Guide to incorporate maintaining regular ongoing contact with agencies in order to monitor the use of the Guide.

          Production

          Questions were designed to find out how often the Guide should be produced. While a range of responses were received,

          the majority want the Guide to be produced annually. This makes sense. But presents some challenges.

          In order for the Guide to be produced annually a number of things have to be in place. First, there needs to be confirmed ongoing funding. Funding which takes into account staffing the production of the Guide. In the past, funds have been secured for pulling together, producing and the initial distribution of the Guide. What has become clear through this evaluation is that this is not enough.

          Producing the Guide needs to be a part of managing all aspects of the Guide. A staff person needs to be fully responsible for the production, distribution, and proactive update of the Guide - all aspects of managing the Guide.

          The Best Practices Review did not yield substantial feedback. What was learnt is that unless the Guide is seen as a product of HACN/ACT housed in an Agency that demonstrates its understanding and caring about the importance of the Guide by allocating the needed resources, the Guide will eventually disappear or lose its intent and responsiveness to the diverse communities. It may or may not be updated regularly depending on the availability of funds and/or someone's interest.

          The Blue Book is updated annually. It does have a form for agencies to fax in updates. One completed form was received last year. Another one was received this year. Most of the responses for updates were received by someone sitting on the phone and calling (up to four times in some cases) the agencies. Based on the information received, the Blue Book has the human and financial resources to do this. How The Living Guide will be updated, is the question.

          The potential funders identified in Appendix IV need to be re-contacted and pursued in order to obtain ongoing funding for the production - including all aspects of managing and maintaining The Living Guide.

          Recommendation: HACN/ACT need to determine a funding strategy which takes into account the adequate allocation of human, financial, physical resources in order to manage all aspects of producing, maintaining and managing The Living Guide on an ongoing basis.

          In conclusion, The Living Guide is a superior publication. It is an excellent resource, which is worthy of a new publication guided by a group of committed stakeholders.

          The Living Guide has become a part of the reliable reference material used by professionals in the AIDS business. There is an expectation that this source of information will continue to be available.

              .
      APPENDICES
          Note: Most of the information contained in the Appendices is not available in an electronic format. However, the information can be viewed in the (paper-based) full report held in ACT's Library.

          I DATA COLLECTION INSTRUMENTS

          II DOCUMENTS / PUBLICATIONS REVIEWED

          III LIST OF BEST PRACTICE CONTACTS/ORGANIZATIONS

          IV LIST OF POTENTIAL FUNDERS
              .
      APPENDIX I
      Data Collection Instruments
          Note: These instruments are not available in electronic format. However, they can be viewed in the (paper-based) full report held in ACT's Library:

          (1) Questionnaire for Service Providers and Material which was sent with the questionnaires
          (2) Questionnaire for Guide Users
          (3) Telephone Interviews
          (4) Do You Know the Living Guide - Flyer
          (5) General Questions for Roundtable Discussion
              .
      APPENDIX II
      Documents and
      Publications
      Reviewed
          1) AIDS Committee of Toronto: Strategic Plan 1996/97

          2) AIDS Cultural Network: HIV/AIDS: Sharing Our Cultural Experiences and Strategies.

          3) AIDS Committee of Toronto and the HIV/AIDS Cultural Network of Metropolitan Toronto & Surrounding Areas: The Living Guide. 1992

          4) AIDS Committee of Toronto and the HIV/AIDS Cultural Network of Metropolitan Toronto & Surrounding Areas: The Living Guide. May 1996

          5) Community AIDS Treatment Information Exchange: A Practical Guide for People Living with HIV Disease. 1997.

          6) CHAP Directory - Canadian HIV/AIDS Care Giving Professionals Network. 1997.

          7) Gay Men's Health Crisis: Living with AIDS - A Guide to Resources in New York City. 3rd edition & 4th edition.

          8) Hooking Up to Social Services: A Report on the Barriers People Living with HIV/AIDS Face accessing Social Services. March 1994.

          9) The Positive Living Guide. Resources for people in Hamilton/Wentworth and surrounding areas living with HIV/AIDS.
              .
      APPENDIX III
      Best Practices
          In summary, the evaluation of The Living Guide will contribute to building best practices for similar directories. Based on the data collected, the current process for reprinting similar directories seems to be to update the profiles of the initial agencies and include new ones without any critical analysis of fit and feedback from users of the guide/directory.

          Gay Men's Health Crisis, New York.
        • Limited information available.
        • No evaluation of the directory has been done.
        • The people who worked on the Guide are no longer with the agency.
        • Suppose to be reprinted twice a year but due to finances this has not been possible.
        • It is distributed to area hospitals, social service agencies, nursing home etc.
        • Charge of $5.00 per copy after initial distribution
        • It is a very popular resource with AIDS workers and is used mostly by professionals.

          Hamilton AIDS Network, North Hamilton CHC
        • No evaluation done
        • The person who co-ordinated the first publication is no longer there.
        • Updated twice by having two-summer students call each listed service and update the information.
        • To be listed in the Guide, had to sign a consent form.
        • Funded by the Province of Ontario.

          AIDS Vancouver
        • No directory of AIDS specific services.
        • They use the Red Book (= to Ontario's Blue Book) and several other directory of services. E.g. the Survival Manual, the Pacific AIDS Network Directory for B.B. & Yukon - a directory of members.

          Canadian HIV/AIDS Care Giving Professional Network (CHAP)
        • Indicated they did not have a directory
        • (NB they do, trying to get more information)

          Community Information Centre of Toronto
        • No evaluation done like what is being done with The Living Guide.
        • Evaluation is done: 1) form at the back of the Blue Book - 1 completed form was received last year and this year. 2) informal feedback during the course of doing other business with service users. 3) feedback from internal colleagues based on what users are saying to them over the phone.
        • Rethink every year if layout/ structure/content of book is the best possible way of doing it.
        • Updated annually, every agency is sent its current profile and asked to update the information. It usually takes 3 to 4 phone calls to get a response from at least 50% of the agencies.
        • 3 full time research editors spend half of their time on the Blue Book.
        • Temporary help is used through federally funded programs during peak times (June through to January).
        • Subscription of $120/yr - a hard copy and an electronic copy.
        • There are about 12 pages of paid advertisement, which allows agencies that do not really fit the norm of the Blue Book to list their services.
        • Advertisement cost approximately $100 per page.
        • Funding is split three ways between the United Way, government and commercial enterprises.
        • The Blue Book list 1270 organizations and cost over $150,000 to produce.

          Ontario Hospices Association
        • Documentation received after initial phone call - not applicable to this process.
              .
      APPENDIX IV
      Funding Search
          Thirteen organizations were approached by telephone to determine the possibility of funding for The Living Guide. The individuals we talked to were told that this was just an initial contact call and that follow-up would be done directly by ACT on behalf of The Living Guide.

          The information contained in this Appendix is not available in an electronic format. However, the information can be viewed in the (paper-based) full report held in ACT's Library.