Prepared for Immigration, Refugees and Citizenship Canada
Supplier name: Advanis Inc.
Contract number: CW2239189
Contract value: $54,420.80 (including HST)
Award date: September 1, 2022
Delivery date: March 15, 2023
Registration number: POR 034-22
For more information on this report, please contact IRCC at:
IRCC.COMMPOR-ROPCOMM.IRCC@cic.gc.ca
Ce rapport est aussi disponible en français
This public opinion research report presents the methodological details for the Study on Newcomer Services Campaign (2022-2023) conducted by Advanis Inc. on behalf of Immigration, Refugees and Citizenship Canada (“IRCC”). The survey for the baseline (pre-advertising campaign) wave was administered with 506 newcomers to Canada between October 17 and November 1, 2022, while the post-advertising campaign wave was conducted with 544 newcomers between February 6 to February 27, 2023.
Cette publication est aussi disponible en français sous le titre : Campagne publicitaire sur les services à l’intention des nouveaux arrivants 2022-23 outil d’évaluation de campagnes publicitaires (OECP) rapport méthodologique
This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Immigration, Refugees and Citizenship Canada. For more information on this report, please contact Immigration, Refugees and Citizenship Canada at IRCC.COMMPOR-ROPCOMM.IRCC@cic.gc.ca or at:
Communications Branch
Immigration, Refugees and Citizenship Canada
Jean Edmonds Tower South
365 Laurier Ave W
Ottawa ON K1A 1L1
Catalogue Number: Ci4-226/2023E-PDF
International Standard Book Number (ISBN): 978-0-660-49925-3
Related publications (registration number: POR 034-22):
Catalogue Number: Ci4-226/2023F-PDF (Methodological Report, French)
ISBN: 978-0-660-49926-0
Copyright: His Majesty the King in Right of Canada, as represented by the Minister of Immigration, Refugees and Citizenship Canada, 2023
Political neutrality certification
I hereby certify as Senior Officer of Advanis that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Policy on Communications and Federal Identity and the Directive on the Management of Communications.
Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate, or ratings of the performance of a political party or its leaders.
Signed:
Nicolas Toutant, Vice-President, Research and Evaluation
The Government of Canada wishes to make it easier for immigrants to build successful lives in Canada. In order to raise awareness among newcomers about services permanent residents can access, the Government of Canada launched the Newcomer Services advertising campaign in 2009, to inform newcomers to Canada about the settlement services available to help them succeed in their new country.
The Government of Canada requires a campaign evaluation using the Advertising Campaign Evaluation Tool (ACET) for all advertising campaigns with a media buy exceeding $1,000,000 (See the Directive on the Management of Communications: https://www.tbs-sct.gc.ca/pol/doc-eng.aspx?id=30682#appB).
The purpose of this quantitative study is to ensure that advertising aimed at newcomers to Canada was effective and appropriate and that it accomplished its objective to build awareness and understanding of the settlement services available during the pandemic.
The focus of the survey was newcomers to Canada across the country (outside Quebec). This study measured awareness, recall and impact of the ads among the target population. It consisted of two survey waves: a baseline wave conducted before the campaign launch and a wave at the end of the campaign. More precisely the study aimed to assess:
The results of this research provided feedback on the effectiveness of the ads in raising awareness among the target audience about the services available to them. The results will also be used to guide and inform the design and implementation of future public awareness and outreach activities aimed at newcomers.
The baseline wave of the survey was conducted online with 506 newcomers to Canada (since 2017, aged 24 to 60) outside of Quebec from October 17 to November 1, 2022. The post-campaign wave of the survey was conducted with 544 newcomers to Canada outside of Quebec from February 6 to February 27, 2023. Respondents from a pre-profiled database recruited randomly via telephone were sent SMS or email invites to participate in the study. The average survey length of the baseline survey was 5.8 minutes, and was 8.2 minutes for the post-campaign survey. The margin of error for the baseline survey was +/-4.36%, at a 95% confidence interval and +/-4.20%, at a 95% confidence interval for the post-campaign survey. The data reported was weighed to replicate, as closely as possible, actual newcomer population distribution by age and gender within Canada according to IRCC data. Respondents were offered the survey in the official language of their choice.
A full quantitative methodology report, including all information about the execution of the fieldwork that is needed to replicate the research initiative, can be found in Appendix A. The quantitative survey research instruments are provided in appendix B. A set of tabulated results from the quantitative surveys are provided in English and French under separate covers.
The value of this contract, including HST, is $54,420.80.
Advanis used its General Population Random Sample, which is a pre-profiled database to find potential respondents. Our GPRS recruits people living in Canada using random digit telephone dialing (including both landline and cell phone numbers) to collect a random sample of potential respondents. Advanis identified newcomers from all Canadian provinces (except Quebec) who have been in Canada for less than 5 years to populate its sample. Potential respondents were selected to attain the targeted number of completed surveys for each province/region of Canada.
| Province/Region | Baseline Survey | Post-Campaign Survey |
|---|---|---|
| Atlantic Canada | 206 | 208 |
| Ontario | 2,080 | 1,695 |
| Manitoba | 152 | 128 |
| Saskatchewan | 114 | 89 |
| Alberta | 401 | 235 |
| British Columbia | 837 | 556 |
| Territories | 8 | 8 |
| Total | 3,798 | 2,919 |
The baseline and post-campaign questionnaires were designed by IRCC and supplied to Advanis. Questions within the survey further filtered out responses from anyone under the age of 24 or above 60, anyone who mentioned having been born in Canada, anyone who mentioned having arrived in Canada prior to 2017 for the baseline wave and prior to 2018 for the post-campaign wave and anyone working for a market research firm, a magazine or a newspaper, an advertising agency, a graphic design firm, a political party, a radio or television station, a public relations company or the federal/provincial government.
Most questions were the same across both surveys (baseline and post-campaign) but additional questions were added for the post-campaign survey. Respondents were shown advertisement banners and videos from IRCC and were asked follow-up questions about their exposure to the ads, their reaction to the ads and their assessment of the ads.
For both waves of the survey, Advanis used a CATI (Computer Assisted Telephone Interviewing) survey to recruit potential participants to participate in the web survey. Potential participants pulled from our GPRS sample were randomly called and were asked if they agreed to participate in an online study. Those who agreed were sent an invitation via SMS or via email. Overall, 3,069 people agreed to receive an invitation to participate to the web survey for the baseline wave. For the post-campaign wave 2,230 agreed.
Invitations were mainly sent by SMS. Emails were sent only if it was demanded specifically by the potential participant. After sending the initial invitation, a reminder message was sent every three days to applicants who did not complete a survey or were not screened out of the survey. A maximum of three reminder messages were sent.
A test pre-launch was conducted on October 17 for the baseline survey and on February 6-7 for the post-campaign survey. Neither pretest highlighted any difficulties with the survey itself or data collection.
A support email address and phone number were provided in the survey itself for any technical issues related to the survey. Surveys had a unique number embedded in the hyperlink to eliminate the possibility of duplicate responses from one participant.
The baseline wave of the survey was conducted online with 506 newcomers to Canada (since 2017, aged 24 to 60) outside of Quebec from October 17 to November 1, 2022. The post-campaign wave of the survey was conducted with 544 newcomers to Canada outside of Quebec from February 6 to February 27, 2023.
The table below shows the details of initial invitations sent, unopened surveys, screened out applicants, as well as complete and incomplete surveys. In total, 506 web questionnaires were completed during the baseline wave and 544 during the post-campaign wave. The margin of error for the baseline survey was +/-4.36%, at a 95% confidence interval and +/-4.20%, at a 95% confidence interval for the post-campaign survey.
The response rate, calculated as the number of in-scope – responding units divided by the sum of unresolved units, in-scope – non-responding units, and in-scope – responding units, was 25.5% for the baseline pre-campaign wave and 33.7% for the post-campaign wave. This response rate is within normal ranges for a survey of this type.
| Disposition | Baseline Wave | Post-Campaign |
|---|---|---|
| Total Invitations Issued (SMS/Email) | 3,069 | 2,230 |
| Invalid (incomplete/incorrect email address, email invitation bounce backs) | 1 | 3 |
| Total unresolved units (no response at all) | 2,261 | 1,443 |
| Total in-scope - non-responding units | 24 | 34 |
| Qualified respondent break-off (incomplete) | 24 | 34 |
| Total in-scope - responding units | 783 | 750 |
| Other disqualified | 277 | 206 |
| Completed questionnaires | 506 | 544 |
| Response Rate (in-scope – responding / (unresolved + in-scope – non-responding) + in-scope – responding) | 25.5% | 33.7% |
Inclusive participation is a requirement of public opinion research contracted by the Government of Canada. Online surveys are programmed for example to adapt to mobile devices, limit large tables of scaled questions, support screen readers, and support use of tab-key functions to navigate screens. Alternative formats (e.g., interviews) are also available if requested. The measures Advanis puts in place addressed the needs of participants who may have had challenges, as no one requested additional support.
Overall, 3,069 were invited to participate to the web survey during the first wave and 2,230 during the second wave. The average survey length of the baseline survey was 5.8 minutes and 8.2 minutes for the post-campaign survey.
For both waves, the number of completes per province matched approximately the number of completes targeted by IRCC.
| Province/Region | Baseline Wave: Completed the web survey | Baseline Wave: Target | Post-Campaign Wave: Completed the web survey | Post-Campaign Wave: Target |
|---|---|---|---|---|
| Atlantic Canada | 32 | 27 | 43 | 27 |
| Ontario | 288 | 270 | 312 | 270 |
| Manitoba | 24 | 20 | 24 | 20 |
| Saskatchewan | 15 | 17 | 15 | 17 |
| Alberta | 49 | 54 | 44 | 54 |
| British Columbia | 94 | 111 | 105 | 111 |
| Territories | 2 | 1 | 0 | 1 |
| Did not confirm province | 2 | N/A | 1 | N/A |
| Total | 506 | 500 | 544 | 500 |
| Survey wave | Working in ineligible sector | Born in Canada | Moved to Canada before 2017 or 2018 | Not between 24 and 60 | Living in Quebec | Total |
|---|---|---|---|---|---|---|
| Baseline | 66 | 11 | 197 | 0 | 3 | 277 |
| Post-Campaign | 57 | 9 | 134 | 4 | 2 | 206 |
| Total | 123 | 20 | 331 | 4 | 5 | 483 |
Overall, 1,050 web interviews were conducted during data collection. The final data was weighted within each survey wave by age and gender. The weighting scheme was provided by IRCC based on internal estimates of the demographic characteristics of the newcomer population within Canada (encompassing both permanent and temporary residents) and Advanis applied the weights to the database. Those whose gender was recorded as ‘other’ (n=2 in the baseline survey and n=1 in the post-campaign survey) received a weight of 1.0.
| Survey wave | Age | Gender | Unweighted N | Weighted N | Weight |
|---|---|---|---|---|---|
| Baseline Survey | 24-34 | Male | 81 | 165 | 2.035 |
| Baseline Survey | 24-34 | Female | 60 | 150 | 2.497 |
| Baseline Survey | 35 + | Male | 242 | 107 | 0.440 |
| Baseline Survey | 35 + | Female | 121 | 83 | 0.684 |
| Baseline Survey | 35 + | Other | 2 | 2 | 1.000 |
| Post-Campaign Survey | 24-34 | Male | 112 | 178 | 1.586 |
| Post-Campaign Survey | 24-34 | Female | 62 | 161 | 2.603 |
| Post-Campaign Survey | 24-34 | Other | 1 | 1 | 1.000 |
| Post-Campaign Survey | 35 + | Male | 244 | 115 | 0.471 |
| Post-Campaign Survey | 35+ | Female | 125 | 89 | 0.713 |
All demographic and derived variables used in the analysis and/or weighting were included in the dataset. The open-ended responses were coded to general themes. Coded and original verbatim responses were maintained in the dataset. A dataset was provided for the baseline wave and the post-campaign wave. A merged dataset for both waves was also provided to IRCC.
Questions were crossed with themed categories to provide insight into the results. Banner tables were provided for the baseline and the post-campaign wave. Merged banner tables comparing the results for both waves were provided as well. Final banner tables included unweighted (total) and weighted frequencies and percent.
Non-response bias occurs when non-responders differ in a meaningful way from respondents and this difference impacts the information gathered. It is difficult to assess the presence of non-response bias since information about why non-responders did not participate is usually unavailable. That said, one way to gauge the potential impacts of non-response bias is to evaluate if the sample is representative by comparing the respondents' characteristics and gauge if they reflect known population characteristics. Where possible, we can check the distribution of respondents across various demographics (e.g., age and gender) and geographic categories and compare those distributions against known population characteristics. If the variation is fairly small and we have no reason to believe there are other factors impacting respondents’ willingness to participate, we can conclude that the likelihood of non-response bias impacting the information gathered in the study is minimal. This is the case with the current study.
Several strategies were employed to increase response rates and reduce the effects of non-response bias. This includes:
English and French quantitative survey instruments for the Baseline and Post-Campaign waves are provided under a separate cover.
A full set of tabulated data for the Baseline and Post-Campaign waves are provided under a separate cover.