Chapter 1 - A closer look at student outcomes
Outcomes indicators
We begin by assessing graduation rates because they are typically seen as an indicator of the extent to which students have selected programs that adequately match their interests, goals and abilities. The graduation rate is also a key performance indicator, since false starts represent a considerable cost to both the individual student and the province.
We also consider three broad types of post-graduation outcomes: earnings, employment and satisfaction. In addition to the satisfaction and labour market related indicators used in college and university KPI reporting, we have added earnings, multiple indicators of employment, and two additional satisfaction indicators creating a total of nine indicators. Taken together, these indicators present a rich portrait of graduate outcomes. Table 1 provides a brief description of each indicator. Table 2 provides a more technical description of how each indicator is measured in our data sets.
| Outcome | Indicator |
|---|---|
| 1. Graduation | The percentage of students who graduated out of all students in a particular program cohort. |
| 2. Earnings | Annual earnings. |
| 3. Employment | Unemployment – The percentage of graduates in the labour force who are not working. |
| Labour force participation – The percentage of graduates who are either working, or not working and looking for work. | |
| Employment in a related field – The percentage of graduates who are in work related to their field of study – based on graduates’ own perceptions. | |
| Returned to education – The percentage of graduates who have returned to school for further full-time or part-time education. | |
| 4. Satisfaction | Achieved postsecondary goals – Graduates’ perceptions of how useful their education was in helping them to achieve their goals after graduation. Work preparation satisfaction – Satisfaction with college preparation for the type of work they are doing. Would Recommend the Program – Would graduates recommend the program to someone else. |
| Data source | ||||||
| Outcome indicator | Measure | Ministry | GOSS (6 months post-graduation) | OUGS (2 years post-graduation) | NHS | LFS |
|---|---|---|---|---|---|---|
| 1. Graduation | Among all students who enroll in a program, the percentage that graduate from it within a given time frame. Graduation rates are calculated 7 years after enrolment for 4-year programs and 200% of typical duration for other programs4 | KPI data | ||||
| 2. Earnings | Total annual earnings before tax deductions and transfers among employed graduates. Inflation-adjusted with the Consumer Price Index for Ontario using 2010 constant dollars | Self-reported gross earnings at time of survey | Self-reported gross earnings at time of survey and retrospective to 6 months post-graduation | Annual earnings derived from self-reported wage | ||
| Synthetic, cumulative earnings age 25-64 | Self-reported earnings from 2010 | |||||
| 3. Unemployment | Among graduates in the labour force, the percentage who are unemployed | Self-reported employment status at time of survey | Self-reported employment status at time of survey and retrospective to 6 months post-graduation | Self-reported employment status | ||
| 4. Labour force participation rate | Among all graduates, the percentage who are either employed or looking for employment | Self-report of being either employed or seeking employment at time of survey | Self-report of being either employed or seeking employment at time of survey and retrospective to 6 months post-graduation | |||
| 5. Returned to education | Among all graduates, the percentage that returned to education within 6 or 24 months of graduation | Self-report of being enrolled in post-secondary education at time of survey | Self-report of being enrolled in post-secondary education at time of survey and retrospective to 6 months post-graduation | |||
| 6. Employed in related field | Among employed graduates, the percentage who are working in a field related to their postsecondary education. | Self-report of employed graduates working in a “job related to program that [they] graduated from” at time of survey. We conceptualize relatedness narrowly, categorizing only graduates who selected “yes” as being in related work. (Those selecting “Yes, partially” are not categorized as being in related work.) | Self-report of employed graduates on how closely related their work was “to the skills they acquired through the program of study” at time of survey. We conceptualize relatedness narrowly, categorizing only graduates who selected “Closely related” as being in related work. (Those selecting “Partially related” are not categorized as being in related work.) | |||
| 7. Work preparation satisfaction | Among employed graduates, the percentage who are satisfied with their college preparation for the type of work they are doing | Rating of “Satisfied” or “Very satisfied” with their college preparation for the type of work they are doing | No data collected | |||
| 8. Achieved post-secondary goals | Among all graduates, the percentage who report that their college education was useful in achieving their post-graduation goals | Rating of “Satisfied” or “Very satisfied” with the usefulness of their college education in achieving their post-graduation goals | No data collected | |||
| 9. Would recommend the program | Among all graduates, the percentage who report that they would recommend their program to someone else | Answer of “Yes” to: Would you recommend the (program name) to someone else or not? | No data collected | |||
Limitations in our analysis
While our analysis provides meaningful insights that highlight key gaps and potential opportunities for enhancing Ontario’s credential mix, it cannot tell us everything we need to know to completely answer the research question of whether Ontario has an appropriate mix of labour market focused credentials. We note the following five limitations of our analysis:
- Using short-term data – KPI data only tells us what is happening with college graduates in the six months after they graduate, and with university graduates in the six months to two years after they graduate. Looking out any further requires us to change datasets, lose precision and make assumptions.
- Issues with definition of labour force status in college and university data – Employment outcomes from the KPI are usually presented as the percentage of all graduates who are employed. We think it is important to also consider unemployment, which better reflects the reality that many recent graduates are not offering their labour.
- We cannot measure unemployment properly with the GOSS because full-time students are not asked any employment questions, leaving the labour-force status unknown. Our approach is to classify all college graduates who are full-time students as not in the labour market, even though many full-time students are probably either employed or are looking for part-time work. Because they are not in the labour force, they are excluded from calculation of the unemployment rate.
- We cannot measure unemployment properly with the OUGS because respondents who are not employed are asked if they were looking for employment, but are not asked if they seek full- or part-time employment. In our unemployment calculations, we treat all university graduates who are students and who are seeking employment as though they are seeking part-time employment. Consequently, they are classified as unemployed labour force participants, even though some are actually seeking full-time employment and should not be included in the labour force.
- Due to these survey issues, it is unclear if our GOSS unemployment estimates would be higher or lower than estimates using the standard Statistics Canada approach for measuring labour force status, as the GOSS estimates likely exclude both employed students and unemployed students seeking part-time jobs. However, our OUGS unemployment estimates will be too high because we will classify some full-time students as unemployed when they are actually seeking full-time employment and should be classified as not in the labour force.
- While our labour force status measures are not consistent with Statistics Canada’s measures and our college and university measures are not consistent with one another, our measures can be used to examine trends separately for college and university graduates and compare credentials and fields of study separately for college and university graduates. Where possible, we also construct measures of labour force status that allow for direct comparisons between college and university graduates.
-
Inability to isolate credential performance – Even if we only consider outcomes six months to two years post-graduation, we lack some important measures of student characteristics, such as ability. Thus we cannot determine what proportion of graduates’ outcomes is due to the credential they obtained versus their individual characteristics. Table 3 presents the other factors that we can and cannot account for in our analysis. The role of field of study is of particular importance. We elaborate further on this in the next subsection.
Table 3: Factors other than the credential itself that may influence our outcomes of interest Factors accounted for Factors not accounted for - Field of study
- Program length
- Gender
- Geographic region
- Year of data collection
- Labour market conditions
- Prior student ability (cognitive and non-cognitive)
- Prior credentials attained
- Teaching quality
- Actual learning outcomes
- Looking through the rear view mirror – Historical data only tells us what happened in the past. We do not know with certainty how well past performance will predict future performance.
- No obvious benchmark – Even if we accept that the KPI data are useful for providing an important part of the overall picture of the performance of the credential mix, it is difficult to interpret this part of the overall picture since we have no clearly identified system benchmarks (e.g., when it comes to recent graduate unemployment, how high is too high?).
Our intent was to measure unemployment according to Statistics Canada’s approach so that our estimates could be directly comparable to results from the Labour Force Survey. This approach is widely considered the official and correct way to measure unemployment in Canada. However, the KPI graduate surveys do not permit us to calculate unemployment rates the way that Statistics Canada does. Another issue is that each of the graduate surveys is different, so we needed to devise different unemployment measures for the college and university surveys.
The differences between the KPI and the Statistics Canada unemployment definition are not trivial and neither are the differences between the unemployment measures we use for the college and university data. Thus, we discourage direct comparisons of unemployment between our results and official unemployment statistics from Statistics Canada. We proceed with our measurement of unemployment despite these problems because offered but unutilized labour is an important consideration when examining the needs of students and employers.
We cannot measure unemployment using the standard Statistics Canada approach with the GOSS or the OUGS due to the treatment of full-time students in both surveys. According to the Statistics Canada definition, full-time students are defined as labour-market participants if they are (a) employed or (b) looking for part-time employment. Like other adults, full-time students are not labour-market participants if they are not employed and not looking for work. Unlike other adults, they are also considered not in the labour force if they are not employed and are looking for full-time employment because they are assumed to be seeking work for the summer or a co-op placement (see Statistics Canada Labour Force Survey definition).
4 Graduation Rates are collected by the college at the end of the completion timeframe for all colleges. They are then reported to the Ministry one-year later (reporting year).