2023-24 Newcomer Services Campaign Advertising Campaign Evaluation Tool (ACET)

Methodological Report

Prepared for Immigration, Refugees and Citizenship Canada

Supplier name: Advanis Inc.

Contract number: CW2333794

Contract value: $54,420.80 (including HST)

Award date: October 12, 2023

Delivery date: March 21, 2024

Registration number: POR 065-23

For more information on this report, please contact IRCC at:
IRCC.COMMPOR-ROPCOMM.IRCC@cic.gc.ca

Ce rapport est aussi disponible en français

Copyright

This public opinion research report presents the methodological details for the Study on Newcomer Services Campaign (2023-2024) conducted by Advanis Inc. on behalf of Immigration, Refugees and Citizenship Canada (“IRCC”). The survey for the baseline (pre-advertising campaign) wave was administered with 511 newcomers to Canada between October 31 and November 13, 2023, while the post-advertising campaign wave was conducted with 515 newcomers between February 22 to March 15, 2024.

Cette publication est aussi disponible en français sous le titre : Campagne publicitaire sur les services à l’intention des nouveaux arrivants 2023-24 outil d’évaluation de campagnes publicitaires (OECP)

This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Immigration, Refugees and Citizenship Canada. For more information on this report, please contact Immigration, Refugees and Citizenship Canada at IRCC.COMMPOR-ROPCOMM.IRCC@cic.gc.ca or at:

Communications Branch
Immigration, Refugees and Citizenship Canada
Jean Edmonds Tower South
365 Laurier Ave W
Ottawa ON K1A 1L1

Catalogue Number: Ci4-226/2024E-PDF
International Standard Book Number (ISBN): 978-0-660-70999-4

Related publications (registration number: POR 065-23):
Catalogue Number: Ci4-226/2024F-PDF (Final Report, French)
ISBN: 978-0-660-71000-6

Copyright: His Majesty the King in Right of Canada, as represented by the Minister of Immigration, Refugees and Citizenship Canada, 2024

Political neutrality certification

I hereby certify as Senior Officer of Advanis that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Policy on Communications and Federal Identity and the Directive on the Management of Communications.

Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leaders.

Signed: Nicolas Toutant, Vice-President, Research and Evaluation

Executive Summary

Background

The Government of Canada wishes to make it easier for immigrants to build successful lives in Canada. In order to raise awareness among newcomers about services permanent residents can access, the Government of Canada launched the Newcomer Services advertising campaign in 2009, to inform newcomers to Canada about the settlement services available to help them succeed in their new country.

The Government of Canada requires a campaign evaluation using the Advertising Campaign Evaluation Tool (ACET) for all advertising campaigns with a media buy exceeding $1,000,000^[1].

Objectives

The purpose of this quantitative study is to ensure that advertising aimed at newcomers to Canada was effective and appropriate and that it accomplished its objective to build awareness and understanding of the settlement services available during the pandemic.

The focus of the survey was newcomers to Canada across the country (outside Quebec). This study measured awareness, recall and impact of the ads among the target population. It consisted of two survey waves: a baseline wave conducted before the campaign launch and a wave at the end of the campaign. More precisely the study aimed to assess:

• Unaided and aided recall of the ads
• Awareness and use of information channel (i.e., website) promoted in the ads
• The perceived messages of the ads, as well as the effectiveness of the ads in increasing awareness of IRCC and newcomer services
• Receptivity to government advertising and ratings of government performance and priorities

The results of this research provided feedback on the effectiveness of the ads in raising awareness among the target audience about the services available to them. The results will also be used to guide and inform the design and implementation of future public awareness and outreach activities aimed at newcomers.

Methodology

The baseline wave of the survey was conducted online with 511 newcomers to Canada (since 2018, aged 24 to 60) outside of Quebec from October 31 to November 13, 2023. The post-campaign wave of the survey was conducted with 515 newcomers to Canada outside of Quebec from February 22 to March 15, 2024. Respondents from a pre-profiled database recruited randomly via telephone were sent SMS or email invites to participate in the study. The average survey length of the baseline survey was 6 minutes, and was 10 minutes for the post-campaign survey. The margin of error for the baseline survey was +/-4.3%, at a 95% confidence interval and +/-4.3%, at a 95% confidence interval for the post-campaign survey. The data reported was weighed to replicate, as closely as possible, actual newcomer population distribution by age and gender within Canada according to IRCC data. Respondents were offered the survey in the official language of their choice.

A full quantitative methodology report, including all information about the execution of the fieldwork that is needed to replicate the research initiative, can be found in Appendix A. The quantitative survey research instruments are provided in appendix B. A set of tabulated results from the quantitative surveys are provided in English and French under separate covers.

The value of this contract, including HST, is $54,420.80.

Appendix A: Methodology

Sampling and Survey Design

Advanis used its General Population Random Sample (GPRS), which is a pre-profiled database to find potential respondents. Our GPRS recruits people living in Canada using random digit telephone dialing (including both landline and cell phone numbers) to collect a random sample of potential respondents. Advanis identified newcomers from all Canadian provinces (except Quebec) who have been in Canada for less than 5 years to populate its sample. Potential respondents were selected to attain the targeted number of completed surveys for each province/region of Canada.

Table 1: Sample Frame by Province/Region of Canada

The baseline and post-campaign questionnaires were designed by IRCC and supplied to Advanis. Questions within the survey further filtered out responses from anyone under the age of 24 or above 60, anyone who mentioned having been born in Canada, anyone who mentioned having arrived in Canada prior to 2018 for the baseline wave and prior to 2019 for the post-campaign wave and anyone working for a market research firm, a magazine or a newspaper, an advertising agency, a graphic design firm, a political party, a radio or television station, a public relations company or the federal/provincial government.

Most questions were the same across both surveys (baseline and post-campaign) but additional questions were added for the post-campaign survey. Respondents were shown advertisement banners and videos from IRCC and were asked follow-up questions about their exposure to the ads, their reaction to the ads and their assessment of the ads.

CATI Recruit

For both waves of the survey, Advanis used a CATI (Computer Assisted Telephone Interviewing) survey to recruit potential participants to participate in the web survey.Potential participants pulled from our GPRS sample were randomly called and were asked if they agreed to participate in an online study. Those who agreed were sent an invitation via SMS or via email. Overall, 1,730 people agreed to receive an invitation to participate to the web survey for the baseline wave. For the post-campaign wave 2,826 agreed.

Email and SMS invites/reminders

Invitations were mainly sent by SMS. Emails were sent only if it was demanded specifically by the potential participant. After sending the initial invitation, a reminder message was sent every three days to applicants who did not complete a survey or were not screened out of the survey. A maximum of three reminder messages were sent.

A test pre-launch was conducted on October 31 for the baseline survey and on February 22 for the post-campaign survey. Neither pretest highlighted any difficulties with the survey itself or data collection.

A support email address and phone number were provided in the survey itself for any technical issues related to the survey. Surveys had a unique number embedded in the hyperlink to eliminate the possibility of duplicate responses from one participant.

Results

The table below shows the details of initial invitations sent, unopened surveys, screened out applicants, as well as complete and incomplete surveys. In total, 511 web questionnaires were completed during the baseline wave and 515 during the post-campaign wave. The margin of error for the baseline survey was +/-4.3%, at a 95% confidence interval and +/-4.3%, at a 95% confidence interval for the post-campaign survey.

The response rate, calculated as the number of in-scope – responding units divided by the sum of unresolved units, in-scope – non-responding units, and in-scope – responding units, was 39.2% for the baseline pre-campaign wave and 26.6% for the post-campaign wave. This response rate is within normal ranges for a survey of this type.

Table 2: Invitations Sent and Response Rate

Inclusive participation is a requirement of public opinion research contracted by the Government of Canada. Online surveys are programmed for example to adapt to mobile devices, limit large tables of scaled questions, support screen readers, and support use of tab-key functions to navigate screens. Alternative formats (e.g., interviews) are also available if requested. The measures Advanis puts in place addressed the needs of participants who may have had challenges, as no one requested additional support.

Completes and Targets

Overall, 1,730 were invited to participate to the web survey during the first wave and 2,826 during the second wave. The average survey length of the baseline survey was 6 minutes and 10 minutes for the post-campaign survey.

For both waves, the number of completes per province matched approximately the number of completes targeted by IRCC.

Table 3: Number of Completes and Targeted Number of Completes per Province

Table 4: Number of Screened out participants

Weighting and Data Cleaning

Overall, 1,026 web interviews were conducted during data collection. The final data was weighted within each survey wave by age and gender. The weighting scheme was provided by IRCC based on internal estimates of the demographic characteristics of the newcomer population within Canada (encompassing both permanent and temporary residents) and Advanis applied the weights to the database.

Table 5: The Weights

All demographic and derived variables used in the analysis and/or weighting were included in the dataset. The open-ended responses were coded to general themes. Coded and original verbatim responses were maintained in the dataset. A dataset was provided for the baseline wave and the post-campaign wave. A merged dataset for both waves was also provided to IRCC.

Questions were crossed with themed categories to provide insight into the results. Banner tables were provided for the baseline and the post-campaign wave. Merged banner tables comparing the results for both waves were provided as well. Final banner tables included unweighted (total) and weighted frequencies and percent.

Non-Response Bias Analysis

Non-response bias occurs when non-responders differ in a meaningful way from respondents and this difference impacts the information gathered. It is difficult to assess the presence of non-response bias since information about why non-responders did not participate is usually unavailable. That said, one way to gauge the potential impacts of non-response bias is to evaluate if the sample is representative by comparing the respondents' characteristics and gauge if they reflect known population characteristics. Where possible, we can check the distribution of respondents across various demographics (e.g., age and gender) and geographic categories and compare those distributions against known population characteristics. If the variation is fairly small and we have no reason to believe there are other factors impacting respondents’ willingness to participate, we can conclude that the likelihood of non-response bias impacting the information gathered in the study is minimal. This is the case with the current study.

Several strategies were employed to increase response rates and reduce the effects of non-response bias. This includes:

• Recruiting respondents by telephone, which achieves a higher response rate compared to email invitations.
• Outpulsing a local phone number (rather than a toll-free number) which increases pick-up rates (reducing call screening).
• Sending an SMS text message to recruits, which assures a seamless transition from the telephone survey to the online survey, as receipt can be confirmed in real-time and encourages respondents to complete the survey as soon as the call ends.
• Offering the survey in both official languages to maximize ease of completion.

Appendix B: Quantitative Instruments

The quantitative survey instruments in French and English are provided under a separate cover.

Appendix C: Full set of tabulated data

A full set of tabulated data for the baseline and post-campaign waves are provided under a separate cover.

1. See the Directive on the Management of Communications: https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=30682#appB