November 7, 1997

Prepared for:

Industry Canada

PREFACE

This document is an interim working document. It provides a context for discussion of the Student Connection Program. It is organized according to the Evaluation Framework and presents results from the initial findings of the first work term of the Program. By discussing the components of the Evaluation Framework at the conclusion of each work term, we can present feedback to identify opportunities for Program review and redirection. The report will also assist in the refinement of the final report.

This document draws on, and refers to, the four interim reports that constitute the data gathered for the evaluation of the first term of the Program. These reports include:

Summary Results of Interviews with Administrative Centre Coordinators

Summary Results of Interviews with Program Managers

Summary Report of the Survey of Student Business Advisors

Summary Report of the Survey of Business Clients.

Naturally, the Student Connection Framework Review (1996), is also quoted and referred to.

For verification, and details on the results cited in this working document, please refer to these reports.

TABLE OF CONTENTS

1.0	Introduction	1

2.0	Program rationale and continued relevance	1

2.1	Has the Program adapted to the changing needs of small and medium sized enterprise?	2

2.2	Are the purposes and objectives of the SCP consistent with IC and HRD's missions and objectives?	3

2.3	Are SCP's objectives clearly stated? Are they relevant, appropriate, and

consistent with its purpose?	3

2.4	Is the Program adapting to the emerging information technology?	5

2.5	Are there other federal or provincial programs similar in purpose?	5

2.6	Is SCP complementary to related programs?	5

2.7	Does the SCP have the appropriate structure to address the perceived needs?	5

2.8	Are the SCP activities and outputs consistent with its mandate?	5

3.0	Skills and competitive development among client firms	6

3.1	Does business regard the student business advisors as a valuable resource?	6

3.2	Are as many businesses as expected hiring the students?	7

3.3	Are business managers learning practical and useful skills?	7

3.4	Is the length of the sessions long enough for effective instruction and knowledge dissemination?	7

3.5	Are the student business advisors capable and appropriate instructors in this situation?	8

3.6	Are client businesses ready for this instruction? Do some industry participants show more potential and interest than others?	8

3.7	Does the instruction increase the competitiveness of the client businesses?	9

3.8	Are the new information skills used?	9

3.9	Have the firms invested in IT education and equipment after their SCP session?	9

3.10	What are the attributes of the SMEs that have been assisted?	10

4.0	Skills development among student business advisors	10

4.1	Has the SCP increased SBA's ability to appreciate business problems?	10

4.2	Has the SCP increased their business skills, presentation skills, and other competencies?	11

4.3	Has the SCP enhanced the Internet and computer skills of the SBAs?	11

4.4	Have participating SBAs gained work experience?	11

4.5	Has the Program increased the SBAs knowledge of information needs

of business?	12

4.6	Have SBAs had sufficient opportunity to demonstrate their skills to each employer?	12

4.7	Are SBAs working with clients that provide high future employment potential?	12

4.8	Do SBAs receive adequate payment for their services? Does the SCP provide them with income comparable to other sources?	12

5.0	Informal impact on the Canadian Technology Sector	13

5.1	Has the Program strengthened the position of the Canadian Information Technology Industry?	13

5.2	Have client firms made use of Strategis and other Internet resources?	13

5.3	Have academic institutions developed partnerships or closer relations with client firms as a result of the SCP?	14

6.0	Education/Instruction standards and materials	14

6.1	Are SBAs effective instructors for business managers?	14

7.0	Unintended consequences	14

7.1	Has the SCP had an adverse impact on private Internet suppliers or consultants? Has it had a positive impact?	15

7.2	Has the uptake of the Program reflected expected variation in regional

economic activity?	15

7.3	Have the SBAs ever provided poor advice? Has this ever resulted

in complaints?	16

8.0	Cost effectiveness	16

8.1	Is the Program cost effective? What is the total cost of each project?

What proportion is funded by the client business?	16

8.3	Does the Program require the continued financial support of

Industry Canada?	17

9.0	Management	17

9.1	Have the post-secondary institutions been successful in their roles?	17

9.2	Have Team Leaders provided appropriate support to student

business advisors?	17

9.3	Has SCP been properly represented by Team Leaders to the participating businesses?	17

9.4	Does the AUCC/ACCC effectively coordinate the Program?	18

9.5	Has Industry Canada provided sufficient administrative support and advice

to the other management groups?	18

9.6	Have all program participants supported necessary evolution in program delivery to meet evolving business needs?	18

APPENDIX A	Evaluation issues, questions and indicators and their references in this report

1.0	Introduction

PRA prepared the Evaluation Framework in consultation with AUCC, ACCC, and Industry Canada. The evaluation framework translates the general objectives into discrete issues.

The evaluation framework presents the following issues as concrete expectations of the Program's goals and objectives.

a.	Program Rationale and Continued Relevance

b.	Skills and Competitive Development Among Client Firms

c.	Skills Development Among Student Business Advisors (SBAs)

d.	Indirect Impact on Canadian Information Technology Sector

e.	Educational/Instructional Standards

The remaining three issues derive from a number of management concerns over the delivery of this program as well as the requirements of the Program Review process. These include:

f.	Unintended Consequences

g.	Cost Effectiveness

h.	Management

2.0	Program rationale and continued relevance

On-line business information (OLBI) is growing exponentially with the economy. Any program designed to assist firms in its utilization must be able to readily adapt to rapidly evolving technologies. It should also equip clients to participate in the OLBI rapidly. The OLBI should be legitimately seen by business as a desirable, and even necessary asset to their operations.

The basic requirement for the SCP is that the training provided to businesses remains on the 'cutting edge.' Specifically, the Program must deliver training that is both relevant to business and presents the state-of-the-art in business on-line resources. The pace of innovation in information technology is swift and SBAs must be able to provide valuable and timely information to middle and senior managers.

This issue includes a range of program review questions. These include, but are not restricted to, the clarity of goals, the need for business services, the consistency of the Program goals with that of the sponsoring departments (Industry Canada and Human Resources Development Canada), potential duplication with other programs, complementarity with other programs (i.e., is the spending levered by other public sector activity?) and the consistency of outputs with the mandate of the SCP.

Finally, it is particularly important to include a question on program structure given that the SCP is delivered by Non-
Governmental Agencies and post-secondary educational institutions. Whether this decentralized delivery is effective and efficient will be determined by management judgement over the level of outcomes and costs.

2.1	Has the Program adapted to the changing needs of small and medium sized enterprise?

The program's focus on the requirements of small and medium sized enterprises (SMEs) has redefined the Program's view of clients and training. Program management expressed the view that this definition of a client, and client training was restrictive. The obligation of the Program, as expressed by the management is "to introduce small business managers to the potential that the Internet has to meet their daily business needs." Initially, the Program provided 15 hours of Internet training to SME's in exchange for $150 in fees.

2.1.1	Definition of a client

Interviews with Administration Centre management, Head Office, and feedback from Client Surveys indicated that the 15 hour sessions are either not always feasible or desirable. Respondents suggested that a client could be defined as anyone who attends a SCP presentation of an hour or more, or who accesses 50 or more pages of the SCP web site, as this introduces them to the potential of the Internet.

Respondents argued that by changing the definition of a client, the Program may redirect its efforts to the needs and requirements of Canadian SMEs as they emerge. It allows the Program to serve a broader range of existing and emerging business.

2.2	Are the purposes and objectives of the SCP consistent with IC and HRD's missions and objectives?

Industry Canada's mission and role is defined as follows.

Fostering a growing, competitive, knowledge-based Canadian economy that:

provides more and better-paying jobs for Canadians

supports stronger, sustainable business growth

gives consumers, businesses, and investors confidence that the marketplace is fair and efficient.

One of Industry Canada's four key strategic objectives aimed at improving the climate for economic growth in Canada is improving Canada's innovation performance and the transition to the knowledge-based economy.

The SCP is designed to help Canadian small and medium sized businesses become more competitive by connecting to, and learning how to use business services on the Information Highway.

Canadian businesses need to be shown how to:

connect to the Internet and to use it effectively

access and use on-line business services to support their day-to-day decision making processes.

Clearly, these purposes are consistent with the overall mandate of Industry Canada.

2.3	Are SCP's objectives clearly stated? Are they relevant, appropriate, and consistent with its purpose?

The program originally had three objectives established to realize its purpose. The program

offers basic training in the use of Canadian on-line services to address common business applications, with an aim to increasing the global competitiveness of Canadian SMEs. A specific goal was training at least 50,000 managers from small and medium sized businesses in Canada over 3 years.

develops an environment where students and recent graduates can demonstrate business related skills to potential future employers, learn additional valuable skills, and earn a salary to finance their continuing education. The program was expected to involve 2000 students over a three-year period.

improves the strength of the Canadian information technology industry by demonstrating their valuable on-line business services to a market that is difficult for the information industry to penetrate - Canadian small and medium sized business.

These objectives were originally set out early in 1996. To be successful, the Program objectives must also be able to reflect the changes that occur in the information technology industry.

Based on the rate of change in information technology, it is inappropriate to expect the full set of program goals set early in 1996 to apply for the entire length of the Program (until 1999). Any program dealing with the information technology industry that adheres rigidly to objectives set out four years earlier will either end up providing an inappropriate service at the end of its term, and/or fail in its primary purpose.

Recently, the SCP has begun to move away from the goal of training businesses in the use of information technology, to providing an introduction to how the Internet can meet business needs, and a resource that links SMEs to the on-line sources of information that will improve their competitiveness. This goal is consistent with the Program purpose of strengthening the Canadian information technology industry. The original targets for training now appear very high and unrealistic. Line managers are clearly adapting to needs as expressed by the "market."

2.4	Is the Program adapting to the emerging information technology?

The SCP is adapting to the emerging information technology. This process is evident through the development of utilities such as the on-line resource. However, as an introductory training, the real objective may be more extensive than intensive. By extensive service we mean a general introduction of the possibilities of the OBLI to a wider audience. An intensive introduction of advanced OBLI procedures to a smaller group is not appropriate and duplicates existing private and public sector offerings.

2.5	Are there other federal or provincial programs similar in purpose?

(To be integrated into future reports.)

2.6	Is SCP complementary to related programs?

(To be integrated into future reports.)

2.7	Does the SCP have the appropriate structure to address the perceived needs?

Based on the results of the Survey of Businesses, clients would like to see more flexible training. The original design of a 15 hour course - an introduction to the Internet - is frequently too long, too short, too simple, or too advanced.

The Program must adapt to meet the needs of business clients and to increase the use of IT in SMEs.

2.8	Are the SCP activities and outputs consistent with its mandate?

Based on the results of the Survey of Student Business Advisors, and Business clients, the activities and outputs of the SCP have been consistent with its mandate.

3.0	Skills and competitive development among client firms

A basic question for the evaluation is whether the acquisition of information technology skills leads to increased competitiveness for SMEs. If businesses do not see value in the training, it may be because the on-line information systems are less valuable than assumed.

Another related question is the extent to which businesses gain - be it monetarily or strategically - through using new information technologies. It is important not to measure incremental profitably in the immediate post-instruction period. Rather, the initial outcome is changed attitudes by business and increased investigation on their part of on-line information. Student Business Advisors are expected to apply their information technology skills to a range of business problems including marketing, new product development, promotion, and employee relations, thereby increasing the potential gain to business.

Several questions on the survey of business managers probe this area:

where business clients heard of the Program

had they been advised of the value by other managers.

3.1	Does business regard the student business advisors as a valuable resource?

Based on survey responses from participants in the 1997 Winter term, the majority of business clients view the Program as a very good value.

Based on questions about the student's understanding of the technology, the clarity of their instruction, and their ability to understand their client's business needs, most respondents reported that the student's provided valuable information on the new technology. Overall, the clients felt that students were a good source of information on computer technology and the Internet. However, several clients noted students were not well equipped to teach. Although the students knew the information, they were not always able to communicate it clearly according to several respondents.

3.2	Are as many businesses as expected hiring the students?

There are clearly fewer businesses than expected hiring the students.

When looking at why fewer businesses are hiring the students, it is important to consider why 50,000 clients were expected, and the "true" definition of a client, to explain the lower usage.

3.3	Are business managers learning practical and useful skills?

More than 80% of Term 1 clients reported that the skills they learned from the Program were relevant to their business and more than 90% said the skills were up-to-date.

Although the overwhelming majority of clients said that the skills they are learning are practical and useful, a few offered comments indicating that the skills were too simplistic for their needs. This mismatch is low is not be a concern. Over time, however, as on-
line skills diffuse through the business economy, the demand for more advanced training will grow.

3.4	Is the length of the sessions long enough for effective instruction and knowledge dissemination?

The length of the sessions needs to be more flexible to accommodate the needs of business clients.

Based on interviews with Administrative Centre coordinators, and feedback from the Survey of Business Clients, the 15 hour session length is restrictive, and dissuades some potential clients from using the Program.

Some clients expressed the desire for longer sessions, some for shorter more focused sessions, and others suggested that a follow-
up several months after the initial training would be appropriate.

3.5	Are the student business advisors capable and appropriate instructors in this situation?

Business clients regard students as computer and Internet authorities. However, as mentioned above, some clients feel they are not adequately skilled at teaching.

Almost 90% of clients said that their SBA had a good understanding of the technology, while 80% of the clients surveyed said their SBA could communicate and present information clearly. However, there were several comments made by clients indicating that although their SBAs clearly understood the technology, they could not communicate or teach the information.

Increasing SBA training in teaching and presentation skills will increase the value of the Program to clients.

3.6	Are client businesses ready for this instruction? Do some industry participants show more potential and interest than others?

The appropriateness of the instruction depends on the computer knowledge of the client, and their expectations for the Program.

The Survey of Business Clients revealed that client's expectations varied. Some wanted HTML instruction and web page design, others wanted to learn about using electronic commerce in their business, and some simply wanted to know about the Internet.

In many cases, the SBA adapted the instruction to fit the requirements and requests of their client. When the instruction was successfully adapted to the client's needs, the interest was high. When the SBA did not adapt the training as requested, the clients expected to do little with their training.

This raises a very difficult problem for SCP. If it tailors courses, then it becomes a consultant and not a training provider. An evolution in this direction could encroach on other training providers.

3.7	Does the instruction increase the competitiveness of the client businesses?

More than half of the clients surveyed said the training they received will make them more competitive.

While the majority of survey respondents indicated that they would be more competitive as a result of the training, about a third could not say.

3.8	Are the new information skills used?

On average, client's use of Internet skills increased between 113% for e-mail, and more than 1200% for the Strategis web site. Overall, total use of the Internet resources increased by over 600% or a factor of 6.

Using the data collected from the Survey of Business Clients, we can calculate that the average weekly use of the new information skills increased by the following amounts.

Web browsers and search engines were used 5.4 times more per week after the training.

E-mail was used 5.1 times more a week after training.

Business related web sites were accessed 5 times more per week.

Databases were accessed 2.5 times more per week.

Strategis was accessed 1.2 times more per week.

Electronic Commerce was used 0.7 times more per week.

These quantitative measures are concrete measures of program impact within the client community.

3.9	Have the firms invested in IT education and equipment after their SCP session?

Many business clients reported investing in IT education and equipment after their SCP session.

Within six months of their SCP training, almost half of the clients had a new or additional subscription to an Internet service.

Approximately one quarter purchased modems, software, and computers.

7% of respondents invested in further IT training.

The high rate of new subscriptions to Internet services can be partly linked to the "deals" that SBAs could offer their clients. By developing new partnerships with companies involved in hardware, and software, further investment in IT can be encouraged.

3.10	What are the attributes of the SMEs that have been assisted?

Due to the limited data gathered by the SBAs and Administrative Centres, it is not possible to classify clients by industry type, or SIC codes. The original evaluation methodology recommended that this information be gathered by the SBAs, but this has not been done.

Based on the results of the Survey of Business Clients, the average client comes from a business with less than 10 full-time employees, and under $500,000 in annual sales.

4.0	Skills development among student business advisors

Many of the Student Business Advisors will be in mid academic career. Exposure to practical business needs and requirements can allow them to develop the ability to impart information technologies in a business environment. Also, while the Program is intended to benefit business, students are also expected to derive significant benefit from their activity. More precisely, they are expected to increase their insight into the problems of businesses, and in addition to intellectual skills, to gain self-confidence dealing with businesses -- a trait which is expected to increase their 'employability.'

4.1	Has the SCP increased SBA's ability to appreciate business problems?

On the Survey of Student Business Advisors (SBAs), we asked students to rank how much their abilities had improved to:

conduct business needs analysis

identify and solve problems faced by businesses.

Most respondents said that their ability to do these tasks increased.

4.2	Has the SCP increased their business skills, presentation skills, and other competencies?

Respondents to the Summer Survey of SBAs said that their marketing and sales skills had been increased. SBAs also indicated that their business communication skills had been increased significantly as a result of their work term. Only 13% stated that their skills had increased slightly or not at all. Over 60% stated that their skills had increased significantly.

4.3	Has the SCP enhanced the Internet and computer skills of the SBAs?

Students participating in the SCP generally already possessed some core computer competencies. About one quarter said that they did not learn much new information about e-mail, connecting to the Internet, or computers and computer systems. However, 50% of the students said they learned 'much more' information in these areas.

Almost 80% of the students said they learned 'much more' information about commercial Internet sites, an area outside of the conventional students' computer knowledge.

4.4	Have participating SBAs gained work experience?

When we asked students if they had gained work experience that would be an asset when looking for work in the future, every respondent said that they had. Over 60% strongly agreed with the statement, "I gained work experience that will be an asset when I look for work in the future."

4.5	Has the Program increased the SBAs knowledge of information needs of business?

The SCP has moderately increased the SBAs' knowledge of the information needs of business.

Based on the Survey of SBAs, just over 50% of respondents said the Program had significantly increased their knowledge and ability to solve problems faced by business. Approximately the same number also agreed that their ability to conduct needs analysis had significantly increased.

4.6	Have SBAs had sufficient opportunity to demonstrate their skills to each employer?

Approximately 90% of all students responding to our Survey of SBAs agreed that the Program had provided an opportunity to showcase their skills to their clients, and to network with potential employers.

4.7	Are SBAs working with clients that provide high future employment potential?

The Survey of SBAs was administered to students at the conclusion of their work term. At this time, 9% of respondents said that they had received actual employment from their clients, while an additional 17% said they had offers of employment. Overall, just over one-quarter of SBAs received offers of employment within four months of training their clients.

4.8	Do SBAs receive adequate payment for their services? Does the SCP provide them with income comparable to other sources?

Approximately 40% of students said that they were not adequately compensated for the work and skill levels required for their position as a SBA. On average, students said that they could have made $10 an hour had they chosen to accept a different job offer.

5.0	Informal impact on the Canadian Technology Sector

The effect of the Program on changes in the Canada IT sector is hard to discern. To the extent that participating businesses increase and retain their use of on-line systems and other aspects of on-line technology (e-mail, electronic data interchange, web sites, etc.), this program will have had a beneficial effect on this sector. Increasing use of Strategis and increased partnerships between business and academic sponsors involved in the Program are also indicators of increased development in the Canadian IT sector.

5.1	Has the Program strengthened the position of the Canadian Information Technology Industry?

As a result of the SCP, the position of the Canadian Information Technology Industry has been strengthened by increasing the demand for its services and by augmenting the supply of knowledgeable employees.

Clients' use of Information Technology (IT) increased significantly after their training session. Based on the responses to our Survey of Business Clients, we estimate that clients increased usage by approximately 600% over the levels before the training. Student Business Advisors also reported a moderate increase in their IT skills and knowledge as a result of the Program.

5.2	Have client firms made use of Strategis and other Internet resources?

On average, client use of Strategis jumped from 0.1 times a week to 1.3 times a week.

Use of other business related web sites increased from an average of once per week to 5 times a week - a 5 fold increase.

5.3	Have academic institutions developed partnerships or closer relations with client firms as a result of the SCP?

(To be integrated into future reports.)

6.0	Education/Instruction standards and materials

The quality of the instruction and supporting materials is critical to this program, especially the longer term goal of self-sufficiency. The preparation of students, more importantly, the selection and training of students with the basic appreciation of the business context, will be very important to determining success of the Program. Businesses often use word-of-mouth and referral before using consulting services; good experiences with SBAs will certainly increase utilization, just as poor experiences will limit program uptake.

6.1	Are SBAs effective instructors for business managers?

In our Survey of Business Advisors, almost 90% of respondents said that the SBA had a good understanding of the technology.

Close to 80% agreed that their SBA understood their business needs, and more than 80% said their SBA tailored their session to fit their business requirements.

Although approximately 80% said the SBAs could communicate clearly and answer questions, we received several written statements from clients stating that their SBA's instructional skills were insufficient.

7.0	Unintended consequences

It is impossible to identify all unintended consequences prior to evaluation. The program could create new opportunities for client business (an intended consequence) or SBAs might also be competitors to other business services. Although the intent is not to define the scope narrowly to the creation of web pages or Internet instruction, it is possible that some SBAs could subsidize services and inadvertently compete with private firms. Even the process of offering broader services and applying information technologies to general business problems has some potential for harming existing private firms.

In the surveys and interviews we did not discover any unintended positive or negative outcomes.

7.1	Has the SCP had an adverse impact on private Internet suppliers or consultants? Has it had a positive impact?

We do not have sufficient information yet to comment on adverse impacts.

As a result of the Program, private web page programmers have benefitted. Based on our survey results of business clients, there were 27 clients who invested money in developing a web page after undertaking the SCP training. Each of these clients spent an average of $2,500 on developing their web site.

The program therefore has had positive impacts on web page designers and programmers.

7.2	Has the uptake of the Program reflected expected variation in regional economic activity?

For the Winter 1997 work term, we have records of 539 clients trained.

The clients are distributed across the following areas:

Atlantic Provinces: 15%

Quebec: 25%

Ontario: 37%

Prairie Provinces: 8%

British Columbia: 14%

This represents the expected variation in uptake, although the prairie provinces appear to be somewhat under-represented. Certainly in terms of population, they should equal the activity in the Atlantic provinces and British Columbia.

7.3	Have the SBAs ever provided poor advice? Has this ever resulted in complaints?

Based on the Survey of Business Clients, there was only one incident where a client complained about the SBA's advice.

The client indicated that their SBA's made an error and crashed their new computer system. Although they 'made great efforts to get it running again,' it has never worked as it should have. The client was dissatisfied, as they felt the students should have had it checked by a qualified individual, and completely restored the system. This represents an isolated incident, but also shows the risks inherent in the Program. Here the liability falls on the client.

8.0	Cost effectiveness

Cost effectiveness questions are largely self-explanatory and draw on administrative data and project reports.

8.1	Is the Program cost effective? What is the total cost of each project? What proportion is funded by the client business?

Based on an estimated funding level of $2 million, and an estimated 539 clients trained in the Winter 1997 work term, and an average client fee of $150, the average cost per client served was approximately $3,850.

This figure is expected to drop as Admin Centres become more familiar with the Program, consumer knowledge of the Program increases, and as more clients are trained by the SBAs in future work terms.

For the first work term, the client businesses funded approximately 4% of the average cost per client, while Industry Canada covered 96%. Clearly, volume must increase if cost recovery is to be feasible.

8.2	What are the costs of training per SBA? Does SCP meet the training goals/standards of the federal government?

In the 1997 Winter Term, SBAs averaged seven days of training before working. As most of the training was self directed, the cost of the training largely depended on the wage the students were receiving over this time. Assuming an average wage of $8/hr and a seven hour day, training costs would be almost $400 per SBA.

8.3	Does the Program require the continued financial support of Industry Canada?

Based on the cost breakdown estimated for this work term, the Program clearly requires the continued financial support of Industry Canada in the short and medium term. Long-term program viability cannot be currently estimated and requires much higher volumes. Analysis of the summer work term will be important to the determination of cost recovery.

9.0	Management

9.1	Have the post-secondary institutions been successful in their roles?

(To be discussed in Work Term 2 report)

9.2	Have Team Leaders provided appropriate support to student business advisors?

(To be discussed in Work Term 2 report)

9.3	Has SCP been properly represented by Team Leaders to the participating businesses?

(To be discussed in Work Term 2 report)

9.4	Does the AUCC/ACCC effectively coordinate the Program?

(To be discussed in Work Term 2 report)

9.5	Has Industry Canada provided sufficient administrative support and advice to the other management groups?

(To be discussed in Work Term 2 report)

9.6	Have all program participants supported necessary evolution in program delivery to meet evolving business needs?

(To be discussed in Work Term 2 report)

APPENDIX A

Evaluation issues, questions and indicators

and their references in this report

Evaluation Issues, Questions and Indicators
Issues	Questions	Data Sources	Summary
Section 2

Program Rationale and Continued Relevance	2.1	Has the Program adapted to the changing needs of small /medium sized enterprises?	Program documentation

Business surveys	The program has begun to adapt by changing the definition of a client and by initiating the on-line resource.
2.2	Are the purposes and objectives of SCP consistent with IC's and HRD's missions and objectives? Does the SCP contribute significantly to helping IC fulfill its mission and objectives?	Program documentation	Yes.
2.3	Are SCP's objectives clearly stated? Are they relevant, appropriate, and consistent with its purpose?	Interviews/surveys of program managers	SCP objectives need to be flexible to allow the Program to achieve its purpose.
2.4	Is the Program adapting to the emerging information technology?	Technical reports (in journals) on Information technologies.	Yes.
2.5	Are there other federal or provincial government programs similar in purpose that may have changed the nature or the need for the SCP since it was implemented?	Program documentation	Incomplete
2.6	Is SCP complementary to related programs or does it create conflict and/or duplication, substantively or administratively? If so, in what ways?	Program documentation	Incomplete
2.7	*Does SCP have the appropriate program structure and concepts to realistically and effectively address perceived needs?	Program documentation	The program structure needs to encourage flexibility and innovation to allow the Program to fulfill its purpose.
2.8	*Are SCP's activities and operational outputs consistent with its mandate and conductive to achieving its objectives and producing the intended results?	Comparison of outputs with objectives	Yes.

Section 3

Skills and Competitive Development Among Client Firms	3.1	Does business regard the student business advisors as a valuable resource?

3.2	Are as many businesses as expected hiring the students? If not, why not?	Business survey	Overall, clients said the SBAs provide valuable technical information, and that the instruction is a good value.
3.3	Are business managers learning practical and useful skills?

3.4	Is the length of the sessions (up to two days) long enough for effective instruction and knowledge dissemination?

3.5	Are the student business advisors capable and appropriate instructors in this situation?	Business survey

Management interviews	Overall, clients find the Program skills to be relevant and up-to-date.

The training session length and content needs to be more flexible for effective knowledge dissemination.
3.6	Are the client businesses ready for this instruction? Do some industry participants show more potential and interest than others? Should the Program focus on firms with certain characteristics or continue to offer the service to all that apply?	Business survey

Management interviews	Clients are varied in their training requirements.

The training must be client specific to provide instruction with
3.7	Does the instruction increase the competitiveness of the client businesses?

3.8	Are the new information skills used significantly after the instruction? Can changes in sales, revenue, profits, customer satisfaction, or exposure be attributed to the instruction in information technology?

3.9	Do the client firms continue to invest in IT education and equipment after their SCP session?	Business survey	More than half the clients surveyed said they are more competitive.

Use of IT resources increased by 600%.

Half clients have a new Internet hookup, one quarter have purchased modems, software, and computers.
3.10	Does the instruction address the long and short term potential of the technology? Have participant businesses accelerated their use of Information Technology and the Internet? Are businesses accessing sourcing information? Are they using E-
mail, and electronic commerce? Did businesses construct Web Pages? How else are they using Information Technology?	Business survey	Use of e-mail increased 113%

Use of browsers increased 286%

Use of databases increased 833%

Use of Strategis increased 1200%

Electronic commerce increased 700%

Use of business related web sites increased 500%

Section 4

Skills development among student business advisors	4.1	To what extent has the SCP increased student business advisors' ability to appreciate business problems and provided skills for the modern workplace, such as self-confidence, problem solving, presentation abilities, and other competencies?	Student business advisors survey	Student's knowledge of business problems increased 'somewhat' or 'significantly.'

SBA's business and communication skills increased 'significantly.'
4.2	Has the SCP refined and enhanced the Internet and computer skills of the participating student business advisors?	Student business advisors survey	The SCP increased the SBAs knowledge of computer skills, but to a lesser degree than their business skills.
4.3	What are the attributes of SMEs that have been assisted (numbers by sector, size, region, etc.)	Project Files

Business survey	The prairie region trained proportionality fewer clients then the other areas of Canada.
4.4	Have participating student business advisors gained significant work experience as a result of the Program?	Student business advisors survey	The SCP has provided and exceptional opportunity for students to gain work experience.
4.5	Has the Program increased the student's knowledge and understanding of the information needs of business and industry?	Student business advisors survey	The SCP has 'moderately' increased the SBA's knowledge of the information needs of business.
4.6	Have student business advisors had sufficient opportunity to demonstrate their skills to each employer?

4.7	Are student business advisors provided with client businesses that provide high future employment potential?	Student business advisors survey	The SCP has provided a very good opportunity for networking with future employers.
4.8	Do student business advisors receive adequate payment for their services? Does the SCP provide them with income comparable to other sources?	Student business advisors survey	Forty percent of SBAs felt that they were underpaid.
Have participating student business advisors received employment as a result of their participation in SCP? (see Section 4.7)	Student business advisors survey	Nine percent of SBAs said they had received actual employment from their clients. 17% received employment offers.

Section 5

Informal Impact on the Canadian Technology Sector	5.1	Are participating businesses using the available resources in an effective and competitive manner? Has this program strengthened the position of the Canadian Information Technology Industry?	Business survey

Industry consultation	The SCP has strengthened the Canadian IT industry
5.2	Have client firms made use of Strategis Internet resource? Have they been properly exposed to the resource and its facilities?	Business survey	Use of Strategis has jumped by and average of 1200% among SCP clients.
5.3	Have academic institutions developed partnerships or closer relations with client firms as a result of the SCP?	Management interviews	Incomplete
Section 6

Education/Instruction Standards and Material	6.1	Are student business advisors effective instructors for business managers? Can they analyse business marketing/information needs? Can they adapt the use of information technologies such as Internet to meeting these marketing information needs?	Business survey	Overall SBAs are effective instructors.

However, improving their teaching skills and helping them to tailor their sessions to fit their client's needs would improve customer satisfaction.
Do student business advisors have enough knowledge to assist businesses? Can they identify resources that managers will find timely and useful in everyday business situations? (see Section 6.1)	Student business advisors survey

Business survey
Section 7

Unintended Consequences	7.1	Has the SCP had an adverse impact on private Internet suppliers or consultants? Has it had a positive impact?	Complaints received by management

Survey of Internet providers	The SCP has had a positive effect on demand for HTML programming and programmers.
7.2	Has the uptake of the Program reflected expected variation in regional economic and sectoral activity?	Project files	Not verifiable.
7.3	Have Student business advisors ever provided poor advice? Has this resulted in complaints? If so, has program management resolved the problems in a timely and effective manner?	Complaints received by

management (post-
secondary institutions, , etc.) Business survey	There was one complaint on the Survey of Business Clients. SBAs had crashed the client's system, and were not able to restore it.

Section 8

Cost Effectiveness	8.1	*Does the Program provide a cost effective method for meeting its goals? What changes would make the Program more efficient while maintaining program quality/impact? Are management, program and delivery mechanisms appropriate/adequate in this context?	Project files

Management interviews	Total cost of each client trained was approximately $3,850.
*Are there alternative approaches to better achieving the Program objectives? (see Section 8.2)	Management interviews	Incomplete
*What is the total cost of each project? What proportion is funded by the client business? By Industry Canada? (see Section 8.1)	Project files	In term 1, Industry Canada carried approximately 96% of the cost for each client.

The business client paid 4% of the costs of training.
*Have resource allocations been sufficient to allow the Program to meet its objectives?	Management interviews	Incomplete
*What are the costs of training per Student business advisors? How does this compare to other training programs? Does SCP meet the training goals/standards of the federal government? (see Section 8.2)	Project files

Management interviews	Based on an average wage of $8.00 per hour, costs of training a SBA are approximately $400.
*Does the Program require the continued financial support of Industry Canada? Under what conditions could SCP become self-sustaining? (see Section 8.3)	Business survey

Management interviews	The SCP will require IC funding for the short and medium term. Long term viability without IC funding is uncertain.
Section 9

Management

9.1	Have the post-secondary institutions been successful in recruiting and hiring students, marketing the Program, supporting the student business advisors, and administering the Program? Have regional offices of Industry Canada supported the local market. Has the national office provided sufficient marketing. Are program standards consistent across the country?	Student business advisor survey

Management interviews	Incomplete
9.2	Have Team Leaders provided appropriate support to student business advisors. Has a viable needs assessment tool been created? Have the Team Leaders assisted student business advisors in their dealings with clients?

9.3	Has SCP been properly represented by Team Leaders to the participating businesses? Have any problems been resolved in a timely and constructive manner?	Student business advisor survey

Management interviews	Incomplete
9.4	Does the AUCC/ACCC effectively co-ordinate the Program? Is there sufficient communication with Industry Canada and the participating post-secondary institutions? Is the Program developed to the specified parameters at all participating institutions?	Management interviews	Incomplete
9.5	*Has Industry Canada provided sufficient administrative support and advice to the AUCC/ACCC, and the post-secondary educational institutions?	Management interviews	Incomplete
9.6	Have all program participants (Industry Canada, AUCC/ACCC, Post-Secondary Institutions supported necessary evolution in program delivery to meet evolving business needs?	Management interviews	Incomplete