Chapter 5 – Detailed recommendations

Table of Contents

Introduction

This chapter summarizes our findings and makes recommendations for optimizing the credential mix to improve graduate transitions to the labour market. Drawing on the key findings of our review, we consider how to position Ontario's postsecondary education system to continue to support and improve quality and innovation that better prepares graduates to make a smooth and timely transition into the labour market.

Overview of findings

Postsecondary education in Ontario is a solid investment, but delayed labour market transitions are costing students, families, and the economy

Our analysis of Ontario's credential mix found strong evidence of large returns associated with obtaining an Ontario postsecondary credential. However, we also found evidence to suggest that it can take time before graduates reach their full potential in the labour market, especially for graduates in some fields of study such as humanities, social sciences and sciences. Most postsecondary graduates establish themselves in the labour market by the time they are aged 25-29, after experiencing weaker outcomes in the first few years after graduation. This suggests a gap between the needs of the labour market and the skills and experiences of recent graduates.

Although this delay in transitioning to full-time work may only be for a few years, it may be long enough to be associated with significant societal consequences such as later home ownership and family formation.

How do we accelerate graduate labour market transitions?

While our study suggests that it takes time for students' investment in postsecondary education to pay off, those who can demonstrate advanced abilities, such as critical thinking and strong communications abilities, grounded in technical skills – especially through work experience – have a considerable edge in the labour market.

Students in Ontario and around the world are increasingly seeing applied education as a critical part of their career pathways. Among Ontario and international PSE commentators there is a growing consensus that increasing options for applied education may improve labour market transitions (Fallis, 2013; Canadian Chamber of Commerce, 2014; Mourshed et al., 2014; Economist, 2014). Our analysis suggests that there is no 'one-size-fits-all' solution that will meet the needs of all students. A range of options is needed to improve school-to-work transitions.

Ontario, like other jurisdictions, is experimenting with innovative approaches to strengthen applied options including:


  • Expanding baccalaureate granting authority to applied institutions;
  • Encouraging pathways between applied and academic institutions; and
  • Experimenting with a range of delivery models, including collaborative programming, and competency-based delivery, and work-integrated learning.

These innovations reflect an understanding that the knowledge and skills graduates need to successfully participate in society and contribute to the economy continue to change over time.

What else is needed for continued quality and innovation?

Many jurisdictions around the world are also thinking about credentials within the broader question of how best to position the postsecondary education system to support students and the labour market through various forms of advanced technical education. Our analysis revealed that continued quality and innovation are strengthened through a unified understanding of credentials as a whole, supported by a common understanding of what students are expected to know and do at each credential level. These are essential for developing greater confidence in the quality of Ontario postsecondary credentials at all levels which is necessary for institutions to move forward and innovate together.

While other jurisdictional models are key to understanding global trends in higher education and looking towards best practices, it is important to reflect on the regulatory context in which new credentials are introduced, and to note the unique character of Ontario's system. Ontario is singular among jurisdictions scanned for this report for its high degree of sector autonomy in defining credentials, programs, and quality assurance processes. In many other jurisdictions, government plays a strong role in defining and monitoring minimum quality standards for publicly funded postsecondary education. This has contributed to a context in which new credentials can be considered and introduced with a common understanding of quality standards and how they will contribute to the credential mix as a unified whole.

Like Ontario, governments around the world have a heightened focus on coordination and synergy between institutions and sectors. Differentiation is seen as a fiscally sustainable approach to maintaining quality and innovative public postsecondary education systems. Here and elsewhere, the concept of distinct and separate postsecondary sectors that defined postsecondary education in the post-war era – vocational and academic postsecondary sectors – is being challenged. Postsecondary education systems are increasingly becoming better coordinated, integrated, and understood as one comprehensive whole. Governments are working with postsecondary institutions to refine institutional mandates to achieve greater system coordination and leverage institutional strengths.

Charting a path for Ontario

High quality and innovative postsecondary education, represented with clear and consistent credentials

At the centre of our recommendations is a focus on how Ontario can continue to deliver high quality and innovative postsecondary education, represented with clear and consistent credentials.

Like other jurisdictions around the world, Ontario is focused on strengthening the conditions for a system-wide focus on quality. Increases in higher education enrolment over the past three decades and constrained public resources are challenging postsecondary institutions to find new ways to deliver high-quality programming. Ontario must realign to be more nimble and responsive to the needs of students and the changing labour market.

Increasing emphasis on quality requires a focus on both learning and labour market outcomes. Despite considerable consensus on the importance of skills students should acquire in their postsecondary education, there is no consistent approach for Ontario postsecondary institutions to formally and systematically assess cognitive skill development. Without robust assessment tools to evaluate student skill development, educators cannot make an evidence-based determination of their impact on student learning, or identify gaps in current curricular and pedagogical approaches. Given the magnitude of Ontario's investment in postsecondary education, we should know whether Ontario postsecondary students acquire the knowledge and skills they need to succeed in life and work (Weingarten, 2014).

Students also need a better understanding of their employment and earnings prospects based on what they choose to study. While Ontario collects a substantial amount of information on its postsecondary students, the data is spread across a number of organizations and it is difficult to obtain. Publicly released data is fragmented and not easily comparable. This makes it difficult for prospective students to compare the costs and benefits of attending different programs and make informed choices about their careers.

A renewed focus on quality to support student outcomes in the labour market requires a whole-of-system approach that is driven first and foremost by student needs and a shared commitment to foster continued economic growth and make a positive impact in the lives of every Ontarian. There are encouraging jurisdictional examples that demonstrate success in balancing academic integrity with improving student transitions to the labour market. Institutional autonomy and system coordination and integration are not mutually exclusive. System partners – institutions, government and employers – work effectively together to create a high quality postsecondary experience where students learn what they need to contribute to a prosperous society and economy.

Overview of recommendations

High-quality and innovative postsecondary education

Strengthen conditions for quality

  1. Reaffirm system-wide learning outcomes for the baccalaureate degree qualification in Ontario
  2. Re-think credential mix as an integrated whole with a focus on articulation and laddering to support student mobility
  3. Establish a province-wide approach to quality, with a protocol for the introduction of new credentials

Accelerate progress on learning outcomes assessment

  1. Support projects that advance the measurement of learning outcomes

Improve accessibility, comprehensiveness, and comparability of data

  1. Link applicant, registration, and graduate data to enable comprehensive analysis
  2. Align data sharing practices with Open Government and Open Data commitments

Enable more deliberate experimentation with innovation

  1. Establish a research agenda and common evaluation framework to identify and scale what works
  2. Pilot test innovative approaches that accelerate program completion and labour market transitions

Use transparency to foster trust and public confidence

  1. Ensure that the Ontario Qualifications Framework is a complete and up-to-date information tool for the public
  2. Create an online dashboard to allow students to compare the costs, quality and labour market outcomes of attending individual programs

Strengthen conditions for system quality

Ontario's colleges and universities are responding to student and labour market needs and delivering postsecondary education that combines theoretical and technical content in their applied degree programming. Our research suggests that Ontario's postsecondary education landscape can be strengthened by the reaffirmation that the bachelor's degree constitutes a meaningful qualification applicable to both colleges and universities, consistent with pan-Canadian degree standards and the provisions of the Postsecondary Education Excellence and Choice Act, 2000. This requires the reaffirmation of a clear distinction for the honours nomenclature, as described in the Ontario Qualifications Framework (i.e., levels 10 and 11).

There is a growing recognition that the credential mix is more than the sum of its parts. The postsecondary education system must work as a unified whole to support student choice, mobility, and confidence that any chosen path leads to success. Likewise, employers and the broader public need to know that regardless of the institution or sector, graduates are entering the labour market with high quality publicly funded postsecondary credentials. In other words, general principles and a shared understanding of what degree graduates are expected to know and do are required, while maintaining the different strengths and mandates of Ontario's institutions and sectors. This does not imply convergence across colleges and universities. Instead it requires leveraging system strengths to deliver credentials that work for students and the labour market.

Accelerate progress on learning outcomes

Quality needs to be grounded in learning, in what students know and are able to demonstrate as a result of their given program of study (McKiernan, 2013). Ontario has experience in defining learning outcomes, and should establish an approach for measuring whether learning outcomes are being met at Ontario postsecondary institutions.

Despite considerable agreement on the importance of acquiring skills in postsecondary education, Ontario postsecondary institutions do not formally and systematically assess cognitive skill development. Substantial progress is being made towards this end in Ontario: HEQCO and a number of Ontario postsecondary institutions have formed the Learning Outcomes Assessment Consortium to identify robust and scalable tools to measure student skill development. This is an invaluable asset to Ontario's postsecondary system, and initiatives similar to this consortium should be encouraged and accelerated.

Without robust assessment tools to evaluate student skill development, educators cannot make an evidence-based determination of their impact on student learning, and whether there are gaps in current curricular and pedagogical approaches. When combined with strong and transparent quality assurance, learning outcomes can serve as the foundation of stronger relationships between institutions and articulation between credentials.

Enable more deliberate experimentation

Through their Strategic Mandate Agreements, Ontario institutions have proposed various options to improve student mobility and choice, including new and expanded collaborative programs, increased options in applied baccalaureate-level education, and new credentials. Underlying the majority of these proposals is the desire to build stronger connections between applied and academic education, and between the worlds of school and work.

We see the type of experimentation underway at Ontario institutions as providing significant value to the province moving forward. Core to our recommendations is to harness this experimentation with a focused research agenda, higher quality student and graduate outcome data, and an approach to effectively measure progress.

Improve accessibility, comprehensiveness, and comparability of data

While our analysis provides meaningful insights that highlight key gaps and potential opportunities for enhancing Ontario's credential mix, it cannot tell us everything we need to know to completely answer the research question of whether Ontario has an appropriate mix of labour market focused credentials.

Current data sharing practices make accessing information on the outcomes associated with Ontario postsecondary education credentials difficult to obtain for researchers, the general public and even the government. KPI data define labour force participation differently for college and university graduates, and neither is consistent with the Statistics Canada definition. It only tells us what is happening with college graduates in the six months after they graduate, and with university graduates in the six months and two years after they graduate. Looking out any further requires us to change datasets, lose precision and make assumptions.

Accessible, linked and aligned datasets enable policy-relevant research and would allow Ontario to more definitively evaluate outcomes and educational impact that considers student characteristics, as well as monitor the longer term labour market outcomes of all Ontario graduates consistently.

Use transparency to foster trust and public confidence

Increases in higher education enrolment over the past three decades and the scarcity of public resources challenge postsecondary institutions to continue to deliver high-quality programming. Postsecondary education is a costly investment, and as a result students, parents and governments are increasingly focused on the quality and labour market outcomes of individual programs and institutions.

Students are seeking a clear picture of their employment prospects based on what they choose to study. While Ontario collects a substantial amount of information on its postsecondary students, publicly released data is fragmented and not easily comparable. This makes it difficult for prospective students to compare the costs and benefits of attending different programs and to make informed choices about their careers.

Recommendation 1: Strengthen conditions for system quality

Ensure that Ontario credentials are designed and delivered within a system-wide approach that fosters high-quality, collaborative and responsive programming across the continuum of certificate, diploma, and degree-level postsecondary education. The approach is based on two broad principles:

  • System-wide learning outcomes for bachelor's degrees – regardless of the type of institution delivering a bachelor's degree in Ontario, each degree must meet a substantial and common set of learning outcomes to justify the bachelor's label.
  • The credential mix is more than the sum of its parts – while Ontario's colleges and universities were not originally designed as an integrated system, a substantial amount of progress has been made in building stronger linkages between sectors. Collaboration is not only necessary to support student mobility, but to foster a culture of lifelong learning. This involves looking at the credential mix as a unified whole to build clear articulation paths between credentials, between sectors – and strong relationships between institutions – to make learning portable, stackable, flexible, and accessible.

Whether delivered by a college or university, all bachelor's degrees should signify the same quality and academic achievement for the student. The credential of a bachelor's degree in Ontario should signify to students, employers, and the public that the graduate has achieved the learning and skills associated with that education. In addition, an honours distinction, as defined by the OQF level 11, conveys that the graduate has achieved the bachelor's degree requirements with “more conceptual sophistication, specialized knowledge and intellectual autonomy” that is reserved for this designation.

A reaffirmed qualifications framework for degree education in Ontario is consistent with the differentiation policy framework: institutions' degree offerings and aspirations should be considered through the lens of their demonstrated strengths, as well as the coordination of the postsecondary education system as a whole. In this context, it is increasingly important to look at the system's credential mix as more than the sum of its parts, where credit transfer plays a crucial role in supporting portability and flexibility between qualifications.

There needs to be a balance in terms of enabling innovation while ensuring that Ontario's postsecondary education credentials are of the highest quality and that public resources are used effectively. While the existing credential mix is performing well, we see opportunities for stronger alignment and transparency of quality assurance processes to maintain the focus on quality and to strengthen coordination for the review of new credential proposals for public funding.

1.a. Reaffirm system-wide learning outcomes for the baccalaureate degree qualification in Ontario

Ontario should maintain and reaffirm the principles agreed to in the Council of Ministers of Education, Canada 2007 Ministerial Statement on Quality Assurance of Degree Education in Canada that a single definition of a bachelor's degree can accommodate academic, professional and applied programs. A bachelor's degree conveys common education requirements, while allowing for degrees of specialization and sophistication – as defined in the learning outcomes described in levels 10 and 11 of the OQF.

Ontario should also reaffirm that there is a clear distinction for honours bachelor's degrees and develop and affirm nomenclature that reflects this distinction consistently across institutions so that students, employers and other institutions can clearly distinguish an honours bachelor's degree from the bachelor's degree.

College proposals for new degree programs in applied areas of study should be based on areas of strength, consistent with the principles of the differentiation policy framework, which is predicated on quality (i.e., institutional capacity, expertise, collaboration) and provincial need. Furthermore, new degree programs should be considered in the context of the Ontario Colleges of Applied Arts and Technology Act, 2002 which outlines the college mandate to provide career-oriented, postsecondary college education and training. This is integral to ensuring Ontario protects its commitment to serve the needs of all students across the province.

The Postsecondary Education Quality Assessment Board should continue to carry out its role in assessing the quality of all Ontario college degree proposals in applied areas of study, and its processes for granting consent should be maintained, as outlined in the Postsecondary Excellence and Choice Act, 2000.

1.b. Re-think the credential mix as an integrated whole with a focus on articulation and laddering to support student mobility

Ontario credentials should be better integrated within a single postsecondary education system that transcends the boundaries of the province's college and university sectors. Ontario should review opportunities to build clearer standards for how credentials articulate to one another to support student mobility and lifelong skills development. An objective of the review should be to identify principles for establishing common outcomes that apply across all types of postsecondary education qualifications with the view that the range of qualifications constitutes a unified whole rather than two parallel college and university ‘tracks.” The results of this review should identify procedures for facilitating pathways that genuinely support student mobility and lifelong learning.

In essence, we recommend that Ontario continue to strengthen its commitment to credit transfer, with a focus on articulating and transferring between colleges and universities. Ontario has made significant progress on credit transfer in a relatively short period of time, moving from articulation on an institution-to-institution and program-to-program basis towards system-wide student mobility. Building on this work, Ontario needs to establish broader recognition of shared principles behind program design and the beginnings of a shared way of understanding learning outcomes. This will enable systemic and systematic pathways at the credential level.

Ontario should also work towards a strategy for foundational credentials and credential modularization. In this burgeoning area of experimentation, further exploration is required around credentials awarded as shorter, stackable “chunks” of learning, versus those that are foundational credentials intended to allow the learner to ladder into another credential, and those that are designed as “early exit” credentials. This strategy should be developed in tandem with a process for the introduction of new credentials as we describe below.

1.c. Establish a province-wide approach to quality, with a protocol for the introduction of new credentials

Ontario needs a coordinated approach to quality with a focus on alignment and transparency to facilitate the introduction of new credentials for public funding. Ontario should ensure that quality assurance processes and quality measures supporting public postsecondary education are appropriate as well as transparent to students and the broader public. Ontario should consider exploring the different needs of stakeholders to enable the identification of useful, transparent and comparable information on quality and evaluate where there is room for greater commonality and consistency between the three major quality assurance bodies.

Ultimately, quality assurance reports and follow up/related reports should be published in their entirety and should be clear and accessible to the academic community, external partners (including government) and other interested individuals. Any decision or recommendation contained in reports should be easy for the reader to find and to understand. Considering the current legislative framework for Ontario's postsecondary institutions, recognition of institutions eligible for public funding as a proxy for institutional accreditation should continue to be a responsibility of the government.

Balancing government stewardship with college and university leadership, all parties should work together during the development and introduction of new credentials. We recommend that Ontario establish a system-wide protocol for a more transparent, collaborative and coordinated approach to developing new postsecondary credentials and assessing their eligibility for public funding.

The purpose of the protocol would be to ensure that new credentials align with the government's overall vision for publicly funded postsecondary education, and that the government has the appropriate information to make informed decisions.

The protocol should focus on collaboration and information sharing related to the following:

  • Alignment with Ontario's vision and priorities for the postsecondary sector
  • Alignment with institution/sector mandates and prevention of unnecessary duplication
  • Articulation between the proposed credential and existing credentials on the Ontario Qualifications Framework. (i.e., do the proposed learning outcomes facilitate student mobility?)
  • Detailed financial estimates of the cost of credential implementation and ongoing delivery
  • Robust evidence of student and employer demand for a new credential
  • Evidence of quality assurance processes and procedures, and evidence of institutional/system capacity to award the credential.

Rationale

Why reaffirm system-wide learning outcomes for the baccalaureate degree qualification in Ontario?

Consistent with the principles agreed to in the CMEC Ministerial Statement on Quality Assurance of Degree Education in Canada, a single definition of a bachelor's degree can accommodate academic, professional and applied programs. The core principles and fundamental components of a bachelor's degree must be maintained to signify the quality and learning outcomes associated with this qualification.

System-wide principles for bachelor's degrees do not stand in the way of innovations in the design and delivery of credentials, such as competency-based assessment or other approaches outlined in recommendation 3.b. Instead, they reaffirm that the bachelor's degree qualification represents an education that can be delivered by both colleges and universities – with the same quality – within the context of individual institutions' differentiated strengths and mandates.

A reaffirmation of the parameters set out for bachelor's degrees, as described in the Ontario Qualifications Framework levels 10 and 11 would further promote equity to all Ontario students completing a degree credential (either college or university). In our consultations with stakeholders we heard that there was inconsistency of nomenclature across institutions leading to confusion about the honours distinction. Reaffirming this distinction would give a clearer understanding to students, employers and institutions when they are making decisions. This should be applied to the provision of all bachelor's degree education in Ontario – regardless of whether they are delivered in applied areas of focus – as well as principles and guidelines for degree nomenclature common to both colleges and universities.

Our analysis was able to tell us a great deal about the performance of Ontario's credential mix. We found strong evidence that postsecondary education remains a solid investment in Ontario. We were able to present considerable detail on how the returns to education vary by credential and field of study. There were, however, limitations to the available data which affected the robustness of our analysis: a lack of linked application and outcomes data prevented us from analyzing credential performance, while controlling for student characteristics, and a lack of long term outcomes made it difficult to provide definitive answers about labour market transitions.

With respect to the proposal put forward by Colleges Ontario to deliver three-year labour market focused degrees, our study did not point to a conclusion that a new, three-year labour market focused degree would improve student transitions.

It is important to note that this is a complex issue for at least two reasons. First, the question of whether advanced diplomas are already meeting the learning outcomes in OQF level 10 can only be determined by PEQAB through an actual assessment of each program. Second, the effectiveness of three-year labour market focused degrees is difficult to determine for the Ontario context. There is a lack of common characteristics between Ontario and education systems in other jurisdictions (i.e., system structures, primary, secondary, and tertiary organization) that did not allow for a meaningful comparison of findings.

Through Ontario's differentiation policy framework, the Ministry and postsecondary institutions are working together to provide the best possible learning experience to all qualified learners in an affordable, financially sustainable way. At the core of the approach to differentiation is the recognition that institutions have well-established strengths and operate together as complementary parts of a whole.

Why look at the credential mix as a unified whole?

Findings from the review of credential models in the United States (Washington State, Oregon, Wisconsin), Europe (England, Ireland), and Canada (British Columbia, Alberta) noted that like Ontario, public postsecondary education systems have introduced new credentials and expanded the degree-granting authority of the non-university sector over time. In many cases, the expansion of offerings and mandates has been facilitated by clear oversight of postsecondary qualifications to ensure their consistent interpretation and relevance, and to support flexible qualifications linkages and pathways to foster trust, portability and comparability between institutions and sectors.

Currently, institutions are experimenting with a number of models for foundational components of credentials aimed at a variety of different learners. Modularization in the United States has tended to focus on enhancing the flexibility and responsiveness of adult learning. In other jurisdictions, foundational programs are designed to ‘nest” credentials to create more seamless pathways from one credential level to the next. In both Europe and North America, jurisdictions are currently experimenting with the delivery of foundational components of a bachelor's degree, such as the Foundation degrees in UK and the two-year Applied Associate degrees in B.C. There is also extensive experimentation with “early exit” credentials being awarded for partial completion of a bachelor's degree or master's degree program. There is still considerable uncertainty as to how these trends will play out in practice. It is unclear which innovations will yield the most positive results. A commitment to learning what works is critical, as well as striking the right balance between enabling innovative approaches and protecting the integrity and the credibility of Ontario's postsecondary education credentials.

Why establish a province-wide approach to quality?

All jurisdictions examined align their quality assurance policies with regional (U.S. states), national (B.C., Alberta), or international (England, Ireland) quality assurance criteria. In all jurisdictions examined, government plays a principal role in defining and monitoring minimum quality education quality standards for publicly funded postsecondary education.

In jurisdictions where ongoing assessment of program quality is the responsibility of institutions, government maintains a strong monitoring role. For example, the Quality Assurance Agency for Higher Education UK reviews institutions to ensure they are meeting expectations for setting and maintaining academic standards. In the U.S., although regional accreditation bodies have the autonomy to set quality assurance policies and standards, they must also meet minimum standards set by the Federal Department of Education.

Ontario is unique in that the government entrusts each postsecondary sector (colleges and universities) with oversight of the quality of their institutions. As a result, however, the government is relatively less involved in quality monitoring in Ontario's postsecondary sector than governments in other jurisdictions where there is an explicit link between quality and public funding.

In general, institutional accreditation tends to be more prominent in countries with a significant new or private postsecondary education sector, such as those in the Americas and Eastern Europe. In some European jurisdictions, there is a growing need for a mechanism to validate when non-university institutions (i.e., colleges, polytechnics, fachhochschule) are “upgraded” to university status. The Bologna Process has also contributed to the greater prominence of accreditation agencies, which are owned or approved by the jurisdictions in which they operate. In the United States, accreditation is performed by a decentralized network of non-governmental accrediting agencies, which evolved to respond to the large private higher education sector. As noted above, government establishes the standards that must be met and agencies themselves must be recognized by government through the processes established by government. In Ontario, in lieu of recognized bodies that undertake institution accreditation this function is the legislative role of the Minister of Training, Colleges and Universities.

Recommendation 2: Accelerate progress on learning outcomes

Our analysis of global trends and select jurisdictions indicates that substantial progress is being made around the world in defining learning outcomes associated with postsecondary education. Here, like elsewhere, the next frontier in learning outcomes is figuring out how to measure them with sufficient precision to both yield valuable information about student learning and program quality and to encourage continuous improvement.

Through its work on exploring learning outcomes assessment tools, HEQCO has identified a number of findings to strengthen the collection of learning outcomes data (HEQCO, 2013). We recommend accelerating this research along the lines proposed by HEQCO to allow for robust analysis of the potential value of scaling up the various approaches piloted.

2.a. Support projects that advance the measurement of learning outcomes

Currently, Ontario postsecondary institutions do not formally assess whether students develop the basic skills, higher order cognitive skills, or behavioural attributes that we expect postsecondary graduates to acquire over the course of their studies (Weingarten, 2014).

Research demonstrations that test innovative and promising ways of measuring student learning and skills acquisition can advance the development and measurement of learning outcomes, with specific focus on the following areas with a focus on higher-order skills:

  • Basic cognitive skills: are postsecondary graduates developing expected levels of literacy and numeracy?
  • Higher-order cognitive skills: do postsecondary graduates have the critical thinking, problem-solving and communication skills we assume upon graduation?
  • Behavioural/transferable skills: do postsecondary graduates have the non-cognitive skills employers are looking for such as resilience, initiative and working effectively with others?

Ontario is well positioned to carry out these types of projects. HEQCO has substantial experience in designing, coordinating and evaluating research projects to advance the development and measurement of learning outcomes in Ontario.

Continued research should build on lessons learned from previous experiments and be sufficiently aligned to allow for comparative evaluation. Ontario should develop a project implementation strategy and a common evaluation framework to effectively coordinate this research.

In addition, project selection criteria should give preference to the following:

  • Projects that integrate learning outcome measurement into curriculum design and in-course/program assessments and yield quality outcome metrics where progress can be tracked over time.
  • Collaborative projects that involve two or more institutions working together to develop, implement and evaluate a common measurement tool.
  • Projects that estimate the impact of postsecondary education on skill development by including pre- and post-measurements of learning outcomes.

Rationale

Learning outcomes are statements of what a student should know and be able to do after a period of learning (Lennon et al., 2014). Measuring and assessing whether students actually learn what they are supposed to learn has the potential to add value in multiple ways:

  • Students and parents can make more informed choices about their investments, and employers can see more clearly what skills are being produced in institutions.
  • Learning outcomes assessment can increase the quality of the student learning experience by improving teaching and learning and driving continuous improvement.
  • A focus on learning outcomes assessment can enhance focus in postsecondary education programs on skills that are relevant to the 21st-century labour market.
  • During their programs, students' progress can be clearly marked, and at-risk students can be identified in terms of specific areas of need.
  • Learning outcomes can make institutional accountability and quality assurance more transparent.
  • They can assist student mobility by building transparency and trust between institutions.
  • They can be a tool for curriculum planning, clarifying how different courses combine to build a program and a credential.

The idea of scaling experiments and measuring learning impact across a wide range of institutions is appealing; it would be useful to have comparable data on the learning impacts being achieved across programs and institutions.

Recommendation 3: Enable more deliberate experimentation with innovation in credential design and delivery approaches

Innovations in credential design and delivery can accelerate transitions of graduates to the labour market. Demonstration projects that are designed to inform system-wide innovation – and supported by a common evaluation framework – will ensure that experimentation is being driven with a system-wide view of quality and student outcomes.

3.a. Establish a research agenda and common evaluation framework to identify and scale what works for improving labour market transitions

We recommend that Ontario establish a research agenda to identify and scale promising innovations that accelerate student transitions to the labour market.

The research agenda would serve as a guide towards a better understanding of the relative effectiveness of current credential design and delivery and what could be done to make the system work better for students. The demonstration projects above (see 2a.) and below (see 3b.) would form crucial inputs to the research agenda.

The research agenda should:

  1. Establish a clear list of research goals and a roadmap towards a better understanding of what works for improving student transitions.
  2. Be informed by the expertise and needs of Ontario students, employers, institutions and government.
  3. Include a common monitoring and evaluation framework.
  4. Make recommendations on how to improve student transitions into the labour market.

3.b. Pilot test innovative approaches that accelerate program completion and labour market transitions

There are a number of promising approaches and innovations in credential design and delivery that could be pilot tested in the following areas:

  • Collaborative programs that provide students with a combination of advanced technical and cognitive skills and accelerate student transitions to the labour market — drawing on the strengths of individual institutions.
  • Competency-based delivery and tools to help students articulate their skill development
    • Approaches to documenting and assessing skill development that help students articulate their skills to employers, including digital badges and e-portfolios
    • Competency-based delivery where credentials are awarded based on skills acquired rather than credits completed.
  • Work-integrated learning (WIL) in various forms, including:
    • Traditional in-program WIL placements, such as service learning, internships and co-ops
    • Applied research projects
    • Employer-institution partnerships involving deep collaborative curriculum development and long-term financial and non-financial employer commitments.
  • Given colleges' mandate to support adult education, Ontario should explore new approaches to skills upgrading and training
    • To support more adults to pursue postsecondary education credentials that are required by employers, Ontario should consider the development of modular training programs with short-term, stackable credentials
    • These programs would have multiple entry and exit points and be developed to meet both the needs of employers in specific in-demand industries and the needs of adults who want to pursue postsecondary education while maintaining employment.

Pilot testing and rigorous evaluation of projects should be informed by the following principles:

  • Proposals should clearly state how the demonstration is intended to improve student transitions, either by accelerating time to program completion or accelerating the time from graduation to employment.
  • Proposals should emphasize the potential generalizability of project findings to inform system-wide innovation. This includes an analysis of whether other institutions in Ontario are testing similar innovations, and how the proposal will build on existing research. The selection process should prioritize proposals from consortia of institutions or those that can otherwise demonstrate their potential scalability and generalizability.

Rationale

In our jurisdictional scan, we noted that improving student transitions is a challenge common to many of Ontario's comparator jurisdictions. As such, other jurisdictions are exploring new credential design and delivery approaches that accelerate transitions of graduates to the labour market.

There is a substantial amount of ongoing innovation and experimentation in Ontario and around the world to better understand student transitions to the labour market. Through their Strategic Mandate Agreements, Ontario institutions have proposed a range of options to improve student mobility and choice including new and expanded collaborative programs, increased options in applied baccalaureate-level education, and new modularized credentials.

However, experimentation is often isolated and not coordinated with ongoing research in the system. There are significant opportunities to shape research in a way that maximizes system learning. Learning and scaling up what works for improving student transitions to the labour market should be a top priority for Ontario's postsecondary education sector going forward. This task is something neither sector can do on its own.

Initiatives focused on collaboration, competency-based delivery, and work-integrated learning are potentially valuable experiments; the most promising way forward is to rigorously evaluate these various design and delivery options.

Recommendation 4: Improve accessibility, comprehensiveness and comparability of data

Building on its considerable achievements in performance measurement in postsecondary education over the past several decades, Ontario should establish a process to improve the accessibility, comprehensiveness and comparability of data on postsecondary applicants, students and graduates to enable system monitoring and to better evaluate effectiveness.

4.a. Link applicant, registration, and graduate data to enable comprehensive analysis

Ontario should establish a process to link data from application, student satisfaction surveys, and graduate surveys as is done in many other jurisdictions including British Columbia and the United Kingdom.

I. Link institution administrative data to income tax data

Ontario should support efforts already underway by the Education Policy Research Initiative to link institution administrative data to income tax data held at Statistics Canada. This has already been done for the University of Ottawa and is underway for twelve additional universities. Due to the very high tax filing rates of Canadians, this methodology has potential to provide information on long-term earnings outcomes for the vast majority of postsecondary graduates.

II. Align college and university graduate surveys to enable system-wide analysis

Recognizing the evolution of college and university graduate surveys, Ontario should work to align them to allow for system-wide analysis:

  • Align college and university graduate survey questions. Questions on the college and university graduate surveys should be aligned where appropriate. Alignment should include a set of identical core questions for both college and university graduates. (“secondary questions” can allow for sector-specific questions). The survey should be administered at the same point in time post-graduation.
  • Align survey and sampling design. The surveys should use the same techniques to increase response rates and ensure representativeness, and could be modelled after the National Graduates Survey (NGS), which asks college and university graduates the same questions at the same time after graduation. The sampling technique chosen should ensure the ability to support program-level reporting as is done in other jurisdictions such as British Columbia and the United Kingdom.
  • Add an employer survey for university graduates. An employer survey for university graduates should be developed. If a university employer survey is developed, efforts should be untaken to ensure survey alignment to allow for comparability across the university and college sectors.
  • Review survey questions for relevance. Consider whether questions about health, intended occupation, nonmonetary compensation, and life satisfaction (similar to the NGS) should be added to the KPI graduates' surveys. Continue to assess whether there are opportunities to remove questions that are not perceived as useful by system stakeholders and experts.
  • Add an annual ‘theme' of survey questions to enable analysis of current issues. Incorporate an annual subset of questions in college and university student and graduate surveys that explore current issues and opportunities common to colleges and universities. The theme would be set collaboratively, and informed by current system priorities.

Changes to the existing surveys should be accompanied by comparability research assessing the impacts of changes and their implications for comparing data from the old and new surveys.

III. Use the data to regularly monitor system performance

Ontario should consider an annual report that would present trends relevant to a broad range of questions. For example, annual reports could present mean earnings, unemployment, labour-force participation, and relatedness. These could be reported by credential, field of study, region, and other important categories. If an annual theme is added to graduate surveys, analysis of this issue could also be included in the annual report.

4.b. Align data sharing practices with Open Government and Open Data commitments

Ontario should develop and enforce a data sharing protocol that would establish responsibility for the collection, storage, and access of applicant, student and graduate outcome data to make it more accessible and transparent to the general public, researchers, practitioners and the government.

The development of the data sharing protocol should be a collaboration among all relevant organizations, including postsecondary sector application centres, Colleges Ontario, the Council of Ontario universities, and relevant ministries and departments from provincial and federal governments.

The data protocol should clearly identify the individual and collective responsibilities that each party has to ensure the postsecondary data ecosystem aligns with the principles of Ontario's Open Government and Open Data initiatives. Specifically, the data protocol should include:

  • Service standards and principles for data requests and data use: acode of conduct that clearly describes the process for how the public, researchers and the government can access and use Ontario postsecondary data, including a charter of service standards, statement of service principles, and clear step-by-step instructions for individuals applying to use data.
  • One window access: minimize transaction costs for researchers seeking to analyze Ontario's postsecondary system by establishing one service window for all data requests.
  • Fair pricing: data should be made available on a cost-recovery basis only.
  • Privacy safeguards: The data sharing protocol should include clear safeguards to protect individual privacy.

Rationale

Why link system and tax datasets together?

Linked data enables policy-relevant research opportunities that are not possible with unlinked data. For example, linked data would allow researchers to conduct analysis that would account for the role that student preparedness plays in shaping differential labour market outcomes across credentials. Adjusting earnings differences across credentials for student characteristics that existed before postsecondary entry would be helpful to better understand why labour-market outcomes differ across credentials. This type of analysis is only possible with linked application, registration, and graduate outcomes data.

Evaluating the longer term labour market outcomes of all Ontario graduates is critical given that labour market transitions have emerged as a major issue for the system. Linking institution administrative data to tax data would provide invaluable information that would provide considerable insight on this issue. While the NGS provides some information, it is unsuitable for long-term monitoring as it contains too few Ontario graduates, and treats college graduates with different credentials as a single group.

Why align graduate surveys to make them more comparable?

Aligned graduate surveys would substantially improve the ability to analyze the postsecondary system as a whole. System-wide analysis is increasingly important because the college and university sectors are becoming more integrated through collaborative programs, credit-transfer agreements, and an increasing number of students pursuing college and university programs sequentially.

Owing to their separate evolution and distinct sector approaches, the College Graduate Survey and the Ontario University Graduate Survey do not lend themselves to full-scale comparability. The most important differences are:

  • Question wording: The surveys use slightly different questions and response alternatives to measure the same concept.
  • Response rate: The GOSS response rate is approximately 70%; the OUGS response rate is approximately 35%. This raises the concern that nonresponse bias differs across the surveys.
  • Time since graduation: the GOSS is conducted six months after graduation; the OUGS is conducted approximately two years after graduation.
  • Survey format: the GOSS is a telephone survey; the OUGS is overwhelmingly completed online or in pencil and paper format.

Adding an employer survey for university graduates is also critical. Although getting a good job is not the only reason students attend university, there is strong evidence that it is a primary reason (Harmon et al., 2014). If students are attending university in part to get a good job and earn a good income, it is important to understand how well universities are delivering on this objective from the perspective of both graduates and their employers.

Finally, the annual graduates' surveys can be more useful to researchers, institutions and the government if they are dynamic and responsive to system needs, challenges and priorities. Revising an annual subset of survey questions common to both surveys and based on current issues in Ontario postsecondary education would generate timely answers to relevant questions.

Why make postsecondary data more transparent and accessible?

Current data sharing practices make accessing information on the outcomes associated with Ontario postsecondary credentials difficult to obtain for researchers, the general public and even the government. Data is spread across a number of different organizations and each organization has a different protocol for accessing this data.

This type of fragmentation creates unnecessary delays and financial barriers to independent researchers conducting comprehensive analyses of the Ontario postsecondary education system.

The Government of Ontario has recently committed to open and accessible data. In 2013, the Ontario Government established the Open Government Initiative, a commitment to increase transparency, access to data, and the public's ability to interact with government (Office of the Premier, 2013).

In their 2014 report, the Open Government Engagement Team recommended that government data be published in timely, accessible formats that correspond to accepted best practices, and that the data be available free of charge in non-proprietary formats. Further, the report recommended that these practices be extended to the broader public sector (Open Government Engagement Team, 2014).

Recommendation 5: Enhance transparency to support collaboration and strengthen public confidence

Ontario postsecondary institutions collect and analyze a wide range of information on the quality, student satisfaction, and graduate outcomes associated with their programs. But some of this information is not publicly reported; and in some cases, not even reported to the government. More transparent and user-friendly information on Ontario postsecondary qualifications and graduate outcomes would serve to support collaboration between postsecondary stakeholders, and increase public confidence in Ontario's credentials.

5.a. Ensure that the Ontario Qualifications Framework is a complete and up-to-date information tool for the public

We recommend that Ontario establish guidelines for the maintenance of the Ontario Qualifications Framework (OQF) to ensure it is up-to-date and inclusive of all postsecondary credentials offered in Ontario. Specifically, Ontario should:

  • Immediately add university-delivered credentials that are currently not included in the OQF (e.g., the undergraduate certificate and the graduate diploma).

To be an effective information tool for students, other postsecondary education stakeholders, employers, and the broader public, Ontario should explore opportunities to ensure the OQF is accessible in both form and content, with accompanying information specific to its various audiences. We see an opportunity for this information to be built into an online dashboard (see 5b. below) for students so that they have an understanding of the meaningful distinctions between different types of credentials as they explore program options.

The nomenclature used in the Ontario Qualifications Framework should be consistently applied to sector processes for naming credentials, including the honours distinction we outlined in 1.a. as well as other nomenclature policies, such as the Minister's Binding Policy Directive Framework for Programs of Instruction.

5.b. Create an online dashboard to allow students to compare the costs, quality and labour market outcomes of attending individual programs

Ontario should develop a web-based tool that allows prospective students to compare individual postsecondary programs across a range of cost, quality and labour market outcome indicators.

There is an opportunity to leverage the comprehensive and high quality data already collected by Ontario colleges and universities as the foundation for the information tool. Ontario should review the sampling techniques used in the Ontario postsecondary student and graduate surveys to determine whether adjustments would be required to yield a representative sample of program satisfaction and graduate labour market outcomes for individual programs.

Close attention should be paid to how the various components of program-level information are organized and presented:

  • One-stop access to program information. Students should be able to compare all college and university information on one, user-friendly website. The Ministry, colleges and universities should work together to incorporate the best features of their respective education planning tools into a comprehensive tool.
  • Compare individual programs. To be most useful to prospective students, program quality and employment outcomes should be presented for individual programs of study, by institution.
  • Present information for men and women. Program quality and labour market indicators should be presented for men and women separately.
  • Account for regional differences when presenting labour market data. The tool should account for the local labour market context to ensure a fair comparison across institutions.
  • Allow students to customize their search criteria. Students should be able to customize their search in a way that enables them to compare programs on the indicators that matter most to them.

Rationale

Today, getting a good job or earning a decent income are the most frequently cited reasons by students and their parents for attending postsecondary education (Harmon et al., 2014; Prairie Research Associates, 2013).

While attaining a postsecondary credential clearly improves graduates' labour market prospects, the quality of individual institutions and programs is increasingly important to students when making career decisions. There is considerable evidence to suggest that the labour market outcomes of postsecondary graduates vary strongly by field of study (Tal & Enenajor, 2013; SRDC, 2014).

Based on our scan of global trends and other jurisdictions, Ontario is a leader in the quantity and quality of postsecondary data collection; however, publicly released data is fragmented and not easily comparable. This makes it difficult for students to assess the costs and benefits of pursuing different career paths. More work can be done to leverage the data Ontario already collects.