Inclusive Communities Advertising Campaign Evaluation Tool (ACET)

Final report

Prepared for: Immigration, Refugees and Citizenship Canada

Supplier Name: Leger Marketing Inc
Contract Number: CW2333278
Contract Value: $43,542.29 (including HST)
Award Date: 2023-10-06
Delivery Date: 2024-03-26

Registration Number: POR 064-23

For more information on this report, please contact IRCC at:
IRCC.COMMPOR-ROPCOMM.IRCC@cic.gc.ca

Ce rapport est aussi disponible en français.

Copyright

This public opinion research report presents the results of a quantitative study conducted by Léger Marketing Inc. on behalf of Immigration, Refugees and Citizenship Canada (IRCC).

The research involved two online pre-wave and post-wave panel surveys with 2,500 adults for each wave aged 18 to 75 from the Canadian general public (excluding Quebec) using the Advertising Campaign Evaluation tool (ACET) questionnaires.

1. A baseline pre-campaign survey to be completed before the beginning of the advertising campaign, and
2. A post-campaign online survey to measure awareness, recall and effectiveness of the Inclusive Communities campaign after it is completed.

Cette publication est aussi disponible en français sous le titre : 2023-24 Outil d’évaluation de campagnes publicitaires (OECP) pour la campagne Communités inclusives - Rapport final.

This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Immigration, Refugees and Citizenship Canada. For more information on this report, please contact Immigration, Refugees and Citizenship Canada at IRCC.COMMPOR-ROPCOMM.IRCC@cic.gc.ca or at:

Communications Branch
Immigration, Refugees and Citizenship Canada
Jean Edmonds Tower South
365 Laurier Ave W
Ottawa ON K1A 1L1

Catalogue Number: Ci4-248/2024E-PDF
International Standard Book Number (ISBN): 978-0-660-70997-0

Related publications (registration number: POR 064-23):
Catalogue Number: Ci4-248/2024F-PDF (Final Report, French)
ISBN: 978-0-660-70998-7

Copyright: His Majesty the King in Right of Canada, as represented by the Minister of Immigration, Refugees and Citizenship Canada, 2024

Executive Summary

Léger is pleased to present IRCC with the detailed methodology used on the quantitative survey designed to evaluate IRCC’s Inclusive Communities campaign using the Advertising Campaign Evaluation Tool (ACET). This report was prepared by Léger Marketing Inc. who was contracted by IRCC (contract number B8815-24-0112 awarded October 6, 2023).

Background and Objectives

The Inclusive Communities campaign first launched in 2020-21. This year’s advertising campaign will continue to encourage Canadians to support, welcome and engage with newcomers in an effort to help their community be inclusive. The tactics included digital, SEM, and audio ads (e.g Spotify) print ads, video ads and out-of-home ads.

The campaign will help Canadians understand the benefits of immigration by encouraging contact between newcomers and Canadians. Demonstrating the benefits of immigration at the local, community level, dispelling common myths about immigration and refugees, and promoting positive engagement between Canadians and newcomers is vital to ensuring that newcomers are welcomed and communities are inclusive.

The purpose of this ACET study is to ensure that the Inclusive Communities campaign was effective and appropriate, and that it accomplishes the goal of encouraging Canadians to get involved in welcoming newcomers.

The main objective of this quantitative study was to measure awareness, recall, and impact of the ads among the target audience. The surveys used the Government of Canada’s Advertising Campaign Evaluation Tool and consisted of two waves: a baseline wave conducted before the campaign launches and a wave at the end of the campaign.

The ACET assessed:

• Unaided and aided recall of the ads
• Awareness and use of information channel (i.e. website) promoted in the ads
• The perceived messages of the ads, as well as the effectiveness of the ads in increasing intention or action to welcome newcomers
• Receptivity to government advertising and ratings of government performance and priorities

Methodology

Advertising Campaign Evaluation Tool (ACET)

This research for both waves was conducted through online surveys, using Computer Aided Web Interviewing (CAWI) technology. Since a sample drawn from an Internet panel is not probabilistic in nature, the margin of error cannot be calculated for this survey. The questionnaires used for both phases of the study are the Government of Canada ACET questionnaires. Leger Marketing Inc. used these questionnaires without making any modifications to their format nor to their content, except for the modifications necessary to adapt the questions to this present study on the Inclusive Communities ad campaign requested by Immigration, Refugees and Citizenship Canada.

The targeted audience of this study included a national sample reflecting the 18- to 75-year-old Canadian population outside the province of Quebec. Both the baseline and post-campaign surveys were conducted with a sample of around 2,500 Canadian adults. The respondents were recruited via Leger’s web panel using a random selection stratified by region. The general adult population sample was distributed as follows:

Table A.1 Regional Distribution of Respondents

1.2.1 Baseline Study

Fieldwork for the survey was carried out from October 26 to November 05, 2023. A total of 2,506 Canadians aged between 18 and 75 living outside the province of Quebec were surveyed. The sample was drawn randomly from the Leo panel and the overall response rate for the survey was 12.3% (see Table A.3 in the Appendix for the calculation details).

Using data from the most recent Canadian census, the weighting was done within each region by gender, age, language, presence of children in the household and education to ensure the best possible representativeness of the sample within each region and overall. The weight of each region was adjusted to be equivalent to its actual weight in relation to the distribution of the Canadian population. The weighting factors are presented in detail in the appendix of this report.

A pre-test of 52 interviews was completed before launching data collection to validate the programming of the questionnaire in both English and French. An average of four minutes and forty-six seconds was required for the respondents to complete the survey.

Leger adheres to the most stringent guidelines for quantitative research. The survey instrument was compliant with the Standards of Conduct of Government of Canada Public Opinion Research.

A complete methodological description is provided in the Appendices section of this document (please see Appendix A).

1.2.2 Post-Campaign Study

Fieldwork for the survey was carried out from February 8^th to 19^th, 2024. A total of 2,503 Canadians aged between 18 and 75 living outside the province of Quebec were surveyed. The sample was drawn randomly from the Leo panel and the overall response rate for the survey was 10.32% (see Table 11 in the Appendix for the calculation details).

Using data from the most recent Canadian census, the weighting was done within each region by gender, age, language, presence of children in the household and education to ensure the best possible representativeness of the sample within each region and overall. The weight of each region was adjusted to be equivalent to its actual weight in relation to the distribution of the Canadian population. The weighting factors are presented in detail in the appendix of this report.

A pre-test of 48 interviews was completed before launching data collection to validate the programming of the questionnaire in both English and French. An average of six minutes and thirty-three seconds was required for the respondents to complete the survey.

Leger adheres to the most stringent guidelines for quantitative research. The survey instrument was compliant with the Standards of Conduct of Government of Canada Public Opinion Research.

A complete methodological description is provided in the Appendices section of this document (please see Appendix A).

Contract value

The contract value was $43,542.29 (HST included).

Political Neutrality Statement and Contact Information

Leger certifies that the final deliverables fully comply with the Government of Canada’s political neutrality requirements outlined in the Policy on Communications and Federal Identity and the Directive on the Management of Communications.

Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leaders.

Signed:

Christian Bourque, Senior Researcher
Léger

Appendix 1: Quantitative Methodology

Quantitative research was conducted through online surveys, using Computer Aided Web Interviewing (CAWI) technology.

As a CRIC Member, Leger adheres to the most stringent guidelines for quantitative research. The survey was conducted in accordance with Government of Canada requirements for quantitative research, including the Standards of the Conduct of Government of Canada Public Opinion Research - Online Surveys.

Respondents were assured of the voluntary, confidential and anonymous nature of this research. As with all research conducted by Leger, all information that could allow for the identification of participants was removed from the data, in accordance with the Privacy Act.

The questionnaire is available in Appendix A2.

A.1 Sampling Procedure

Leger conducted a panel-based Internet survey with a sample of adult Canadians. The same sampling procedure was used for both the baseline and the post-campaign phases of the study. The exact distribution for each of the two phases of the study is presented in the following section. Participant selection was done randomly from Leo’s online panel.

Leger owns and operates an Internet panel of more than 400,000 Canadians from coast to coast. An Internet panel is made up of Web users profiled on different sociodemographic variables. The majority of Leger’s panel members (61%) have been recruited randomly over the phone over the past decade, making it highly similar to the actual Canadian population on many demographic characteristics.

A.2 Methodology for the Baseline survey

A.2.1 Data Collection

Fieldwork for the survey was conducted from October 26 to November 05, 2023. The participation rate for the survey was 12.3%. A pre-test of 52 interviews was completed on October 25, 2023.

To achieve data reliability in all subgroups, a total sample of 2,506 Canadians aged 18 to 75 living outside the province of Quebec were surveyed.

Since a sample drawn from an Internet panel is not probabilistic in nature, the margin of error cannot be calculated for this survey. Respondents for this survey were selected from among those who have volunteered to participate/registered to participate in online surveys. The results of such surveys cannot be described as statistically projectable to the target population. The data have been weighted to reflect the demographic composition of the target population. Because the sample is based on those who initially self-selected for participation, no estimates of sampling error can be calculated.

Based on data from Statistics Canada’s 2021 national census, Leger weighted the results of this survey by age, gender, language, education and presence of children in the household within each region of the country.

The following table details the regional distribution of respondents. The baseline sample attempted to replicate as closely as possible the actual distribution of the Canadian population.

Table A.2 Regional Distribution of Respondents

A.2.2 Participation Rate

The overall participation rate for this study is 12.3%.

Below is the calculation of the Web survey’s participation rate. The participation rate is calculated using the following formula: Participation rate / response rate = R ÷ (U + IS + R). The table below provides details of the calculation.

Table A.3 Participation Rate Calculation

Typical participation rates for web surveys are between 20% and 30%. A response rate of 12.3% may seem a bit low, but due to time restraints, we had to spread the invitations more widely in the panel to achieve our objectives, which has an impact on the participation rate.

A.2.3 Unweighted and Weighted Samples

A basic comparison of the unweighted and weighted sample sizes was conducted to identify any potential non-response bias that could be introduced by lower response rates among specific demographic subgroups (see tables below).

The table below presents the geographic distribution of respondents, before and after weighting. The weighting adjusted for some minor discrepancies so that each province has a representative importance in the results.

Table A.4 Unweighted and Weighted Sample Distribution by Region

The following tables present the demographic distribution of respondents, according to gender and age.

First, regarding gender, we can see that weighting has adjusted slightly the proportion of male and female. The adjustments made by weighting are minor, and in no way can we believe that the small differences observed in the effective samples could have introduced a non-response bias for either of these two sample subgroups.

Table A.5 Unweighted and Weighted Sample Distribution by Gender

Note: Gender-diverse people and answer refusals make up the rest of the sample.

Regarding age distribution, the weighting process has corrected some minor discrepancies. The actual distribution of the sample generally follows the distribution of age groups in the actual population. In this case, it is unlikely that the observed distributions introduce a non-response bias for a particular age group. Because the differences were so small, weighting allowed the weights to be corrected without further manipulation.

Table A.6 Unweighted and Weighted Sample Distribution by Age Group

There is no evidence from the data that having achieved a different age or gender distribution prior to weighting would have significantly changed the results for this study.

The following tables present the demographic distribution of respondents according to native language, the presence of children in the household, and education.

Table A.7 Unweighted and Weighted Sample Distribution by Native Language

Note: As multiple answers were possible, total may exceed the total sample size.

Table A.8 Unweighted and Weighted Sample Distribution by Presence of Children in the Household

Table A.9 Unweighted and Weighted Sample Distribution by Education level

Again, the corrections were minor and there is no evidence that they would have had an impact on the results.

The relatively small weight factors and differences in responses between various subgroups suggest that data quality was not affected. The weight that was applied corrected the initial imbalance for data analysis purposes and no further manipulations were necessary.

A.3 Methodology for the Post-Campaign Survey

A.3.1 Data Collection

Fieldwork for the survey was conducted from February 8^th to 19^th, 2024. The participation rate for the survey was 10.28%. A pre-test of 48 interviews was completed on February 8^th, 2024.

To achieve data reliability in all subgroups, a total sample of 2,503 Canadians aged 18-75 living outside the province of Quebec were surveyed.

Since a sample drawn from an Internet panel is not probabilistic in nature, the margin of error cannot be calculated for this survey. Respondents for this survey were selected from among those who have volunteered to participate/registered to participate in online surveys. The results of such surveys cannot be described as statistically projectable to the target population. The data have been weighted to reflect the demographic composition of the target population. Because the sample is based on those who initially self-selected for participation, no estimates of sampling error can be calculated.

Based on data from Statistics Canada’s 2021 national census, Leger weighted the results of this survey by age, gender, language, education and presence of children in the household within each region of the country.

The following table details the regional distribution of respondents. The baseline sample attempted to replicate as closely as possible the actual distribution of the Canadian population.

Table A.10 Regional Distribution of Respondents

A.3.2 Participation Rate

The overall participation rate for this study is 10.32%.

Below is the calculation of the Web survey’s participation rate. The participation rate is calculated using the following formula: Participation rate / response rate = R ÷ (U + IS + R). The table below provides details of the calculation.

Table A.11 Participation Rate Calculation

Typical participation rates for web surveys are between 20% and 30%. A response rate of 10.32% may seem a bit low, but due to time restraints, we had to spread the invitations more widely in the panel to achieve our objectives, which has an impact on the participation rate.

A.3.3 Unweighted and Weighted Samples

A basic comparison of the unweighted and weighted sample sizes was conducted to identify any potential non-response bias that could be introduced by lower response rates among specific demographic subgroups (see tables below).

The table below presents the geographic distribution of respondents, before and after weighting. The weighting adjusted for some minor discrepancies so that each province has a representative importance in the results.

Table A.12 Unweighted and Weighted Sample Distribution by Region

The following tables present the demographic distribution of respondents, according to gender and age.

First, regarding gender, we can see that weighting has adjusted slightly the proportion of male and female. The adjustments made by weighting are minor, and in no way can we believe that the small differences observed in the effective samples could have introduced a non-response bias for either of these two sample subgroups.

Table A.13 Unweighted and Weighted Sample Distribution by Gender

Note: Gender-diverse people and answer refusals make up the rest of the sample.

Regarding age distribution, the weighting process has corrected some minor discrepancies. The actual distribution of the sample generally follows the distribution of age groups in the actual population. In this case, it is unlikely that the observed distributions introduce a non-response bias for a particular age group. Because the differences were so small, weighting allowed the weights to be corrected without further manipulation.

Table A.14 Unweighted and Weighted Sample Distribution by Age Group

There is no evidence from the data that having achieved a different age or gender distribution prior to weighting would have significantly changed the results for this study.

The following tables present the demographic distribution of respondents according to native language, the presence of children in the household, and education.

Table A.15 Unweighted and Weighted Sample Distribution by Native Language

Note: As multiple answers were possible, total may exceed the total sample size.

Table A.16 Unweighted and Weighted Sample Distribution by Presence of Children in the Household

Table A.17 Unweighted and Weighted Sample Distribution by Education level

Again, the corrections were minor and there is no evidence that they would have had an impact on the results.

The relatively small weight factors and differences in responses between various subgroups suggest that data quality was not affected. The weight that was applied corrected the initial imbalance for data analysis purposes and no further manipulations were necessary.

Appendix 2: Quantitative Instruments

English and French quantitative instruments are provided under separate cover.

Appendix 3: Full set of tabulated data

A full set of tabulated data for the baseline and post-campaign waves are provided under a separate cover.