Government of Canada

2023 Remembrance and Recognition Ad Campaign ACET

December 2023

Methodological Report

Contract Information

  • Supplier Name: Advanis Inc.
  • Contract number: CW2334787
  • Contract Value: $57,695.50 (tax included)
  • Award date: October 24, 2023
  • Delivery date: December 19, 2023
Prepared For:
Public Service Commission of Canada
Registration number:
POR 075-23

For more information on this executive summary, please contact Veterans Affairs Canada at:commsresearch-commsrecherche@veterans.gc.ca

Ce rapport est aussi disponible en français.

Prepared for Veterans Affairs Canada
Supplier Name: Advanis Inc.
December 2023

This report presents the methodological details for the 2023 Remembrance and Recognition Ad Campaign ACET conducted by Advanis Inc. on behalf of Veterans Affairs Canada. This study was completed from October 23 to October 30, 2023, in the form of a pre-campaign survey and from November 16 to December 5, 2023, in the form of a post-campaign survey. Both waves were conducted using Advanis’ General Population Representative Sample (GPRS) sample. Advanis sought a probability-based sample of 2,000 Canadian adults aged 18 or older drawn from the general population.

This publication may be reproduced for non-commercial purposes only. Prior written permission must be obtained from Veterans Affairs Canada. For more information on this report, please contact Veterans Affairs Canada at: commsresearch-commsrecherche@veterans.gc.ca

Veterans Affairs Canada
161 Grafton Street, PO Box 7700
Charlottetown, PE, C1A 8M9

Ce rapport est aussi disponible en français sous le titre: OECP pour la campagne publicitaire de commémoration et de reconnaissance 2023 Rapport méthodologique.

© His Majesty the King in right of Canada, as represented by the Minister of the Environment and Climate Change and the Minister responsible for Veterans Affairs Canada, 2023

1. Executive Summary

Veterans Affairs Canada (VAC) ran the Remembrance and Recognition campaign in the fall of 2023 to honour Canadians who served our country during times of war, military conflict and peace. This national advertising campaign seeks to encourage Canadians to engage in acts of remembrance.

As per Government of Canada requirements, all advertising campaigns valued at least $1M are required to undertake an evaluation using the Advertising and Communication Evaluation Tool (ACET) via a pre (or baseline) and post evaluation survey.

The purpose of this quantitative research was to evaluate the 2023 Remembrance and Recognition Advertising Campaign using the Government of Canada’s (GC) ACET, and as required, is conducted before and after the campaign’s paid media placement.

The results of this study will assess whether the campaign met its objectives and to inform future Veterans Affairs Canada advertising campaigns.

This first phase of the study was completed from October 23 to October 30, 2023, in the form of a pre-campaign survey through the use of Advanis’ General Population Representative Sample (GPRS) sample. The post-campaign survey was conducted from November 16 to December 5, 2023. A consistent set of questions is used in both surveys to compare across the timeframe in which the advertising campaign enters and leaves the public domain.

For both waves, Advanis sought a probability-based sample of 2,000 Canadian adults aged 18 or older drawn from the general population. Participants were invited via email or SMS to take part in a Web survey. The study was drawn from a random sample and can be extrapolated to the broader population only on a national level.

Questions within the survey further filtered out responses from anyone under the age of 18, those who did not live in Canada, and anyone working for organizations that would not qualify to participate (e.g., a market research firm). A total of 4,064 adult Canadians participated in the study. For the aided recall evaluation, a 16 second video ad was shown to respondents as well as ad visuals.

The contract value for this study was $57,695.50 (tax included).

1.1 Political Neutrality Certification

I hereby certify as Senior Officer of Advanis that the deliverables fully comply with the Government of Canada political neutrality requirements outlined in the Policy on Communications and Federal Identity and the Directive on the Management of Communications.

Specifically, the deliverables do not include information on electoral voting intentions, political party preferences, standings with the electorate or ratings of the performance of a political party or its leaders.

Signed:
Signature
Nicolas Toutant, Vice-President, Research and Evaluation

2. Introduction

Veterans Affairs Canada (VAC) ran the Remembrance and Recognition campaign in the fall of 2023 to honour Canadians who served our country during times of war, military conflict and peace. This annual, national advertising campaign seeks to encourage Canadians to engage in acts of remembrance.

As per Government of Canada requirements, all advertising campaigns valued at least $1M are required to undertake an evaluation using the Advertising and Communication Evaluation Tool (ACET) via a pre (or baseline) and post evaluation survey.

The purpose of this quantitative research was to evaluate the 2023 Remembrance and Recognition Advertising Campaign using the Government of Canada’s (GC) ACET, and as required, is conducted before and after the campaign’s paid media placement.

3. Survey design

Survey content and the French translations were provided by Veterans Affairs Canada and were programmed using SurveyBuilder, a software program that is proprietary to Advanis. The surveys were available to be completed online and were compatible with both desktop computers and mobile devices (tablets and smartphones). It was also designed to be accessible according to the Government of Canada’s Standards for Web Accessibility. The surveys were housed on a website hosted by Advanis.

The surveys were designed to include multiple-choice questions, including open-ended and demographic questions. Skip logic was applied throughout, including three thank you messages used to screen out ineligible participants (having an occupation that makes them ineligible to participate, not living in Canada, or being under the age of 18). The surveys were thoroughly pre-tested to ensure that skip patterns and survey questions were correctly programmed.

Participants were asked the same questions in each of the pre and post waves for comparative purposes. Additional questions were asked in the post survey specific to the actual campaign, as participants are shown the ads that aired on the main media platforms. For the aided recall evaluation, a 16 second video ad was shown to respondents as well as ad visuals.

A pre-test was conducted October 23, 2023, for the pre-campaign survey (150 completed surveys) and again November 16, 2023, for the post-campaign survey (58 completed surveys). Respondents were asked to provide feedback on the survey. Neither language pre-tests highlighted any difficulties with the survey itself or with data collection.

A minor change was made to the pre-campaign questionnaire after the pre-test. At question D1A, the questionnaire terminated people working for the “Federal or provincial government” (among others). We removed “provincial” from this level and only kept “Federal” since there was no reason to terminate respondents working for a provincial government. The data collected during the pre-test was retained for the analysis.

4. Sampling

4.1 Project Characteristics

This study was completed from October 23 to October 30, 2023, in the form of a pre-campaign survey through the use of Advanis’ General Population Representative Sample (GPRS) sample. The post-campaign survey was conducted from November 16 to December 5, 2023.

4.2 Sampling Plan

The target audience for this project was Canadians who are 18 years and older. The targeted number of completed surveys for each of the post and pre-campaign survey was 2,000 Canadian adults. Questions within the survey further filtered out responses from anyone under the age of 18, who did not live in Canada, and anyone working for a market research firm or other disqualified organizations. Each survey had a unique number embedded in the hyperlink to eliminate the possibility of duplicate responses from one participant.

Table 1: Sample plan by province/region of Canada
Region Percentage of population Target population
Atlantic Canada 6.7% 135
Quebec 23.1% 462
Ontario 38.7% 774
Manitoba/Saskatchewan 6.4% 129
Alberta 11.1% 222
British Columbia 13.9% 278
Total 100% 2000

5. Invitations

Invitations were sent by SMS and email and grouped by province, to ensure that they were sent out during appropriate hours within each time zone. After sending the initial invitation, a reminder message was sent three (3) days later to applicants who did not complete a survey or were not screened out of the survey. A maximum of two (2) reminder messages were sent. Overall, 33,655 SMS and emails were sent during data collection for these surveys.

Table 2: Details of survey invitations and reminders
Message ID Purpose Pre-campaign Post-campaign Total Sent
1 Invitation (EN) 6,849 6,611 13,460
2 Invitation (FR) 1,333 1,203 2,536
3 Reminder 1 (EN) 4,192 4,441 8,633
4 Reminder 1 (FR) 921 881 1,802
5 Reminder 2 (EN) 2,947 2,972 5,919
6 Reminder 2 (FR) 655 650 1,305
Total 16,897 16,758 33,655

6. Completes and Targets

Overall, 15,996 Canadians were invited to participate in the pre and post-campaign surveys, 6,849 in the pre survey and 6,611 in the post survey. The average survey length of the pre-campaign survey was eight (8) minutes and 11 minutes for the post-campaign survey. In total, 4,064 surveys were completed during the data-collection period – 2,011 completes in the pre-campaign survey and 2,053 in the post-campaign survey.

For the pre-campaign survey, 165 responded to the survey but were screened out because they were not eligible. It was the case for 161 respondents in the post-campaign survey. When considering those who were not eligible, the response rate for the pre-campaign was 31.8% and the margin of error was +/-2.19%, at a 95% confidence interval. The response rate for the post-campaign was 33.5% and the margin of error was +/- 2.16%.

Table 3: Targeted number of completes per province, per wave
Province/Region Target Pre-campaign completes Post-campaign completes
Atlantic provinces 135 138 144
Quebec 462 412 481
Ontario 774 812 772
Manitoba/ Saskatchewan 129 141 128
Alberta 222 229 226
British Columbia 278 279 302
Total 2000 2,011 2,053
Table 4: Number of screened-out participants per wave
Wave Working in ineligible sector Does not live in Canada Younger than 18 Total
Pre-campaign 159 3 3 165
Post-campaign 152 9 0 161
Total 311 12 3 326

7. Non-response Bias

Non-response bias occurs when non-responders differ in a meaningful way from respondents and this difference impacts the information gathered. It is difficult to assess the presence of non-response bias since information about why non-responders did not participate is usually not available. That said, one way to gauge the potential impacts of non-response bias is to evaluate if the sample is representative by comparing the respondents' characteristics and gauge if they reflect known population characteristics. Where possible, we can check the distribution of respondents across various demographics (e.g., age and gender) and geographic categories and compare those distributions against known population characteristics. If the variation is fairly small and we have no reason to believe there are other factors impacting respondents’ willingness to participate, we can conclude that the likelihood of non-response bias impacting the information gathered in the study is minimal. This is the case with the current study.

Several strategies were employed to increase response rates and reduce the effects of non-response bias. This includes:

8. Weighting, Database and Banners

Overall, 4,064 surveys were completed. The weighting of each dataset (pre / post-campaign) was based on 3 variables: age, gender and region. The six regions used for weighting were Atlantic Canada, Quebec, Ontario, Manitoba/Saskatchewan, Alberta and British Columbia. North was excluded. A weight value of 1 was attributed to respondents who had missing information on a weighting variable. The weights are based on the most recent Statistics Canada census information that is publicly available, the 2021 census.

Each of the pre-campaign and post-campaign survey databases was cleaned to remove any errors, as well as all unique identifiers in the client profiles used by Advanis for delivery and management of the research. All survey answers have been matched and compiled into banner tables.

New variables were created to include in the banner tables. Using the responses to the survey questions, Advanis created variables for the following subgroups to facilitate further analysis:

9. Appendices: Surveys

Pre-campaign Veterans Affairs ACET

Login page

Content
wcag

Si vous préférez répondre à l'étude en français, veuillez cliquer sur français

Thank you for taking a few minutes to complete this survey on current issues that matter to Canadians.

Your participation is voluntary and your responses will be kept entirely confidential.

The survey takes about 10 minutes to complete. If you get interrupted while doing the survey, you can select the same link to pick up right where you left off.

To view our privacy policy, click here (opens in a new window).

If you require technical assistance, please contact survey+vacpre@tellcityhall.ca .

This survey uses Advanis' TellCityHall (opens in a new window) data collection methodology. Your participation is voluntary. Your personal information is treated with the highest standards of confidentiality by Advanis. Survey answers will be grouped with other participant responses, and your personal information will only be used for the purposes stated in the survey. We will not use this information for any purposes other than those consistent with market research requirements and guidelines.

This survey is registered with the Canadian Research Insights Council's (CRIC) Research Verification Service. The project verification number is: 20231019-AD652. Click here (opens in a new window) to verify the legitimacy of this survey.

Section Survey Questions

Page Screeners
D1A

Does anyone in your household work for any of the following organizations?

Select all that apply
TermWork

We regret that your responses have shown that you are ineligible to participate in this survey. Thank you for your time.

(Status code 501)

D1Bb

What is your gender?

D1Bc

In what year were you born?

____________________

TermAge1

We regret that your responses have shown that you are ineligible to participate in this survey. Thank you for your time.

(Status code 502)

D1C

In which of the following age categories do you belong?

TermAge2

We regret that your responses have shown that you are ineligible to participate in this survey. Thank you for your time.

(Status code 503)

D1E

In which province or territory do you live?

TermProv

We regret that your responses have shown that you are ineligible to participate in this survey. Thank you for your time.

(Status code 507)

Page Ad Awareness
Q1

Over the past three weeks, have you seen, read or heard any advertising from the Government of Canada?

Q3

Think about the most recent ad from the Government of Canada that comes to mind. What do you remember about this ad?

Please do not enter personally identifying information (e.g., name, email address, phone number, mailing address), as anything you enter may be shared with the sponsor of this research.

____________________

T1A

Over the past three weeks, have you seen, read or heard any Government of Canada advertising about Canadian Veterans?

T1B

Where have you seen, read or heard this ad about Canadian Veterans?

Select all that apply
T1C

What do you remember about this ad?

Please do not enter personally identifying information (e.g., name, email address, phone number, mailing address), as anything you enter may be shared with the sponsor of this research.

____________________

T1D

Which sources would you rely on when looking for information about remembrance and the contributions of Canada's Veterans?

Select all that apply
T1E

Over the past three weeks, have you seen or read content related to Canadian Veterans on social media sites such as Facebook, YouTube, Twitter, Instagram or LinkedIn?

T1F

In the lead-up to and on Remembrance Day, do you or any of your immediate family members plan to do any of the following acts of remembrance?

Select all that apply
T1G

Please rate the importance of the following acts of Remembrance that Veterans Affairs Canada is involved with.

T1N

How significant is Remembrance Day (November 11) to you?

Page Veterans
T1P

Please describe in your own words who you consider a Veteran.

Please do not enter personally identifying information (e.g., name, email address, phone number, mailing address), as anything you enter may be shared with the sponsor of this research.

____________________

T1R

What do you think is the biggest issue facing Veterans today?

Please do not enter personally identifying information (e.g., name, email address, phone number, mailing address), as anything you enter may be shared with the sponsor of this research.

____________________

Page Demographics
D1

And in closing, a few questions that will help us to analyze the survey results.

Are you a Canadian Veteran, or do you personally know a Canadian Veteran? If so ... what is your relation to the Canadian Veteran?

D2

Which of the following categories best describes your current employment status? Are you...?

D3

What is the highest level of formal education that you have completed?

D4

Are there any children under the age of 18 currently living in your household?

D5

Which of the following categories best describes your total annual household income, including income from all household members, before taxes are deducted?

D6

Where were you born?

D6B

What country were you born in?

D7

In what year did you first move to Canada?

____________________

D8

What is the language you first learned at home as a child and still understand?

Select up to two
D9

Do you identify as any of the following?

Select all that apply
QEND

That concludes the survey. This survey was conducted on behalf of Veterans Affairs Canada. In the coming months the report will be available from Library and Archives Canada. We thank you very much for taking the time to answer this survey. Your help is greatly appreciated.

(Status code -1)

Post-campaign Veterans Affairs ACET

Login page

Content
wcag

Si vous préférez répondre à l'étude en français, veuillez cliquer sur français

Thank you for taking a few minutes to complete this survey on current issues that matter to Canadians.

Your participation is voluntary and your responses will be kept entirely confidential.

The survey takes about 10 minutes to complete. If you get interrupted while doing the survey, you can select the same link to pick up right where you left off.

To view our privacy policy, click here (opens in a new window).

If you require technical assistance, please contact survey+vacpre@tellcityhall.ca .

This survey uses Advanis' TellCityHall (opens in a new window) data collection methodology. Your participation is voluntary. Your personal information is treated with the highest standards of confidentiality by Advanis. Survey answers will be grouped with other participant responses, and your personal information will only be used for the purposes stated in the survey. We will not use this information for any purposes other than those consistent with market research requirements and guidelines.

This survey is registered with the Canadian Research Insights Council's (CRIC) Research Verification Service. The project verification number is: 20231019-AD652. Click here (opens in a new window) to verify the legitimacy of this survey.

Section Survey Questions

Page Screeners
D1A

Does anyone in your household work for any of the following organizations?

Select all that apply
TermWork

We regret that your responses have shown that you are ineligible to participate in this survey. Thank you for your time.

(Status code 501)

D1Bb

What is your gender?

D1Bc

In what year were you born?

____________________

TermAge1

We regret that your responses have shown that you are ineligible to participate in this survey. Thank you for your time.

(Status code 502)

D1C

In which of the following age categories do you belong?

TermAge2

We regret that your responses have shown that you are ineligible to participate in this survey. Thank you for your time.

(Status code 503)

D1E

In which province or territory do you live?

TermProv

We regret that your responses have shown that you are ineligible to participate in this survey. Thank you for your time.

(Status code 507)

Page Ad Awareness
Q1

Over the past three weeks, have you seen, read or heard any advertising from the Government of Canada?

Q3

Think about the most recent ad from the Government of Canada that comes to mind. What do you remember about this ad?

Please do not enter personally identifying information (e.g., name, email address, phone number, mailing address), as anything you enter may be shared with the sponsor of this research.

____________________

T1A

Over the past three weeks, have you seen, read or heard any Government of Canada advertising about Canadian Veterans?

T1B

Where have you seen, read or heard this ad about Canadian Veterans?

Select all that apply
T1C

What do you remember about this ad?

Please do not enter personally identifying information (e.g., name, email address, phone number, mailing address), as anything you enter may be shared with the sponsor of this research.

____________________

T1D

Which sources would you rely on when looking for information about remembrance and the contributions of Canada's Veterans?

Select all that apply
T1E

Over the past three weeks, have you seen or read content related to Canadian Veterans on social media sites such as Facebook, YouTube, X (formerly known as Twitter), Instagram or LinkedIn?

T1F

In the lead-up to and on Remembrance Day, did you or any of your immediate family members do any of the following acts of remembrance?

Select all that apply
T1G

Please rate the importance of the following acts of Remembrance that Veterans Affairs Canada is involved with.

T1N

How significant is Remembrance Day (November 11) to you?

Page Veterans
T1P

Please describe in your own words who you consider a Veteran.

Please do not enter personally identifying information (e.g., name, email address, phone number, mailing address), as anything you enter may be shared with the sponsor of this research.

____________________

T1R

What do you think is the biggest issue facing Veterans today?

Please do not enter personally identifying information (e.g., name, email address, phone number, mailing address), as anything you enter may be shared with the sponsor of this research.

____________________

Page Ad Recall
T1Hmedia1

Here are some ads that have recently been broadcast on various media. Click to watch.

T1Hmedia2

Here are some ads that have recently been shown on various media.

Advertisement 1

Advertisement 2

Advertisement 3

Click on the right arrow to go to the next page.

T1Ha

Over the past three weeks, have you seen, read or heard these ads?

T1I

Where have you seen, read or heard these ads?

Select all that apply
T1J

What do you think is the main point these ads are trying to get across?

Please do not enter personally identifying information (e.g., name, email address, phone number, mailing address), as anything you enter may be shared with the sponsor of this research.

____________________

T1K

Please indicate your level of agreement with the following statements about these ads.

Page Demographics
D1

And in closing, a few questions that will help us to analyze the survey results.

Are you a Canadian Veteran, or do you personally know a Canadian Veteran? If so ... what is your relation to the Canadian Veteran?

D2

Which of the following categories best describes your current employment status? Are you...?

D3

What is the highest level of formal education that you have completed?

D4

Are there any children under the age of 18 currently living in your household?

D5

Which of the following categories best describes your total annual household income, including income from all household members, before taxes are deducted?

D6

Where were you born?

D6B

What country were you born in?

D7

In what year did you first move to Canada?

____________________

D8

What is the language you first learned at home as a child and still understand?

Select up to two
D9

Do you identify as any of the following?

Select all that apply
QEND

That concludes the survey. This survey was conducted on behalf of Veterans Affairs Canada. In the coming months the report will be available from Library and Archives Canada. We thank you very much for taking the time to answer this survey. Your help is greatly appreciated.

(Status code -1)