Learner Gains Research Project
Service Provider Questions and Answers
What is the Learner Gains Project?
The implementation of the Learner Gains performance measure is linked to the availability of an assessment tool measuring a learner’s progress pre- and post- program intervention. Over the past three years, MTCU has been exploring options for a tool to assess a learner’s literacy gains as a result of program intervention. The assessment tool, Essential Skills for Employment and Education (ESEE), was developed by College Sector Committee (CSC), one of the LBS support organizations. The ministry successfully piloted ESEE in November 2013 to test the viability of the tool, with follow up adjustments to the tool and testing completed March 2015.
The Learner Gains Research Project is designed as a small scale research project using the ESEE assessment to collect data on learner progress. This information is not currently collected. The additional data will provide the ministry with better understanding of learner progress and current LBS program outcomes.
What are the ministry’s intended goals for the project?
The Learner Gains Research Project will contribute to making informed decisions about the future directions of the LBS program by achieving the following objectives:
- providing client level data on learner profiles
- obtaining information on learner types and learner progress
- linking service type and program effectiveness
- supporting the review of the LBS Performance Management Framework (PMF).
Is the ministry specifically endorsing the ESEE tool as the tool for the Learner Gains Measure?
At this time, MTCU is evaluating whether or not the ESEE tool meets the needs of the LBS program. The ESEE tool was chosen for this research project as it has already been tested in an Ontario context and has been well received. The assessment has validated questions which are mapped to the Ontario Adult Literacy Curriculum Framework (OALCF). The assessment is available in English, French and with supports, such as American Sign Language (ASL). MTCU is familiar with other assessment tools that are used by the LBS network and continues to support the network using those tools.
When will the full implementation of the LBS Performance Management Framework occur?
The effective date for the remaining elements of the LBS PMF is yet to be determined.
Will the project impact the current LBS PMF? (That is, are there any additional performance measures for a participating service provider?)
Testing on the learner gains measure is outside of the current Phase II-A PMF and Service Quality Standard (SQS). Reporting on the learner gains performance measure will only be for the purposes of this project. The service provider will not be measured by an additional standard from the rest of the network.
What is the general scope and timelines and of the project?
A sample of 33 LBS service providers from across all regions and delivery streams are participating in the research project to target a representative sample population of 1700 learners across the province. The project was officially launched in January 2016. Any new learners entering the program beginning January 2016 will be asked to participate until the sample size is met. Each new learner participating in the project will complete an assessment within two weeks of being registered in the LBS program. Exit tests will be administered to the same learners upon program completion. The final date to administer exit tests will be at the project end, in November 2016.
Why did only certain organizations receive an invitation to participate? How were they selected?
Based on analysis of the clients served across the region and streams, it was determined the population for the project sample should be approximately 1,700 learners. Further analysis specified the numbers of learners within each region and stream. Service providers were selected based on their organizational profile, sector, size of organization and the types of learners they serve to meet the representative sample of learners in the program.
What will be expected of the service providers involved with the project?
The College Sector Committee for Adult Upgrading provided training on the ESEE assessment tool in January 2016. Service providers will be responsible for registering new learners to the project, facilitating the administration of the assessment tool, reporting client information and learner gains scores in EOIS-CaMS. Service providers will also be responsible for providing feedback through surveys or interviews on an as-needed basis.
Service providers will be provided funding to offset the administrative costs associated with participating in the project.
Are there specific performance goals the service providers are expected to meet for the project?
Each participating service provider has an agreement stating the target number of learners that will participate in the project. The organization will be expected to have all new learners that register for the project to complete an entry test until the target is met. Understandably, not all entrants will have exited by project’s end. Successful participation in the project will be evaluated based on the number of learners participating in the project.
If a learner is referred to my organization, or my organization refers a learner to another LBS service provider that is participating in this project, towards whose target number does the learner count?
Each participating service provider will have an agreement stating the target number of learners that will participate in the project. A learner is counted towards the target number of the organization that conducts either a learner intake and administers the entry test or conducts a learner exit and administers the exit test.
If a learner is referred to my organization that is not participating in this project, will this learner count towards my organization's target number?
If your organization conducts the learner intake and the learner chooses to participate in the Learner Gains Research Project, then yes, the learner will be counted towards your organization's target number.
Will my organization's performance as a participant in this research project be considered when being reviewed and assessed according to the LBS Performance Management Framework?
No. Your organization's participation in the Learner Gains Research Project is voluntary and separate from your organization's normal commitments and obligations under the LBS Service Provider Agreement. The Learner Gains Research Project is focused on assessing a learner's performance and knowledge gains as a result of LBS programming. Service Provider performance is out of scope for this research project.
Project Implementation
What are the timelines and cutoff dates for new learner registration?
Any new learners entering the program from the start of the project, January 4th, through June 30, 2016 may participate in the project and take an entry test. The final date for exit tests is November 30, 2016.
Service providers serving Deaf stream learners may include their learners who registered as of September 1, 2015. The cutoff for Deaf stream learner registration is June 30, 2016.
When should new learners complete the entry test?
New learners should complete the entry test as early as possible from the date of registration into the LBS program and within three weeks of starting the test.
The ESEE online test has been designed to enable learners to take the assessment at their own pace, meaning that the learner may start and stop the test as needed until completion.
What are the service provider reporting requirements for participation in the project?
Each service provider should report the entry test and exit test scores to the Ministry via EOIS-CaMS. In the event that a learner is exempt from completing the assessment the service provider is required to document this exemption in EOIS-CaMs and report this reason to the Ministry.
Each service provider is also required to report on eligible expenditures incurred for participation in the project. Expenditure reporting is required for the Ministry to release project funding to the service provider.
Are there any exemptions as to who should take the test?
All new learners are eligible to take the entry and exit tests; however, there may be some instances where a test administrator determines that testing a learner is not appropriate and can exempt the learner. If the service provider determines that testing is not appropriate, the service provider should record the reasons in EOIS-CaMs for the learner not taking the test and report this to the Ministry.
These exemptions impact the Ministry’s understanding of clients served by the LBS program and the ability to make informed decisions. As such these exemptions should be determined on a case-by-case basis to accurately and fairly reflect on the client base served by an organization.
What if a learner’s literacy level is not sufficient to take the test?
If there is any uncertainty about the skills of the learner being sufficient to take the entry test and testing the learner is deemed not appropriate, the service provider can exempt the learner. The goal is not to make anyone uncomfortable, but to understand why learners may struggle with the assessment. The rationale for the exemption should be recorded and reported on. Please note, a learner exempted from taking the assessment is counted toward the service provider participant target as long as the rationale for the exemption is provided.
The ESEE assessment is accompanied with a checklist form that determines if a learner’s literacy level is below that of the test and should be exempted. A test administrator can use this checklist form to determine if a learner should take the test and to record the exemption on EOIS-CaMs.
What if the learner experiences anxiety or stress that prevents successful administration of the test?
If the learner is anxious about taking a test, the service provider may offer a pre-assessment to relieve anxiety. The goal is not to make anyone uncomfortable, but to encourage learners to take the test while relieving learner anxiety and stress, in order to understand learner perceptions around this type of assessment and understand why learners may struggle with the assessment.
What happens if a learner registered for the project leaves the program or moves to another LBS service provider not participating in the project?
In the event that a registered learner leaves the program prior to completion or moves to another LBS service provider that is not participating in the project, the service provider that conducted the intake and entry testing should report the entry test score and then indicate the exit score as “not taken” and the reason for leaving the service provider.
What happens if a learner has active learning plans with 2 or more LBS service providers participating in the project?
The Ministry recognizes that, on occasion, LBS program learners may access services at multiple service delivery sites within the province. This may occur when learners attend more than one in-person service delivery site or when they utilize a blended learning approach, by complementing in-person programming with LBS e-Channel service delivery. In such circumstances, it is important that LBS service providers coordinate their services to ensure that learners receive the assessment and training they need in order to meet their goals. As such one of the following two scenarios would apply:
Scenario 1: A learner has concurrent learning plans with two service providers..
In the event a learner is accessing services at more than one LBS service delivery site around the same time, it is expected that both LBS service providers will coordinate the services they provide to ensure that the learner only experiences one entry and one exit test.
If a learner starts learning activities with a second service provider within a short timeframe of having started with the first service provider and if there are no substantial changes with the learner then only entry test is required. Both service providers can report the same score regardless of who administered the test and maintain the necessary documentation required for the coordination of services. Both the service providers must enter the appropriate scores in EOIS-CaMs and report this data to the ministry. Where a service provider deems it is necessary to conduct a second entry/exit test then they must maintain clear documentation providing rationale for doing so.
Service providers must ensure that:
- each service delivery site maintains a separate but complementary learner plan for the learner, avoiding duplication in programming while supporting the learner to achieve their goals;
- assessment results are documented within the learner’s files at both service delivery sites regardless of which site conducted the assessment with the learner; and
- protection of personal information is strictly maintained. In all cases, learners must be aware and provide documented consent for the sharing of any personal information including but not limited to learner plans, participant registration forms and assessment results
At exit, only the service provider with the longer running program will conduct the exit test and record this score on EOIS-CaMs report the exit test score to the Ministry.
The service provider with the shorter program will be required to note that the learner was exempted as they are continuing with their LBS programming with another service provider. This exemption is to be noted in EOIS-CaMs and reported to the ministry.
If the learner exits both programs within a short period of each other and if there are no substantial changes with the learner, service providers can count that learner toward their target regardless of who administered the exit test and maintain the necessary documentation required for the coordination of services. Both the service providers must enter the entry and exit scores in EOIS-CaMs and report this data to the ministry.
See also Question 9 for performance expectations of the project.
Scenario 2: A learner exits the program with one LBS service provider and moves to another LBS service provider with no overlap between the two.
In the event that a registered learner completes their programming with the first service provider and then moves to another LBS service provider, the first service provider that conducted the Exit test score should report that test score to the second service provider as the Entry test score. The first service provider is also required to report the entry and exit scores on EOIS-CaMs. The second service provider will also be expected to administer an Exit assessment. The second service provider is required to enter the entry and exit scores in EOIS-CaMs.
What happens if a learner is referred to my organization by a non-participating service provider, can I count that learner toward my target?
Yes, so long as it is a new learner entering the program with your organization during the registration period of January 4, 2016 or, September 1, 2015 for Deaf stream sites to June 30, 2016.
My organization offers LBS programming geared toward specific competencies, or, a learner may focus on a specific competency in the LBS program at our site. Does the test focus on different competencies? Can the learner test on only specific components?
There are 3 components to the ESEE assessment: Literacy, Numeracy, and Document Use. For the purposes of this project the Ministry is requesting learners take the ESEE assessment in its entirety. Data collected for the project can be readily compared if all components of the assessment are completed by test takers. MTCU recognizes that ESEE assessments focusing on one or two components of the test have been administered and counted toward a service provider’s participant target. As of February 11th, the SPs are being requested to use the test in its entirety, testing all 3 components.
Are service providers required to spend all money paid by the Ministry for participation in the project?
Funding provided to participating service providers is in the form of transfer payments. Service providers must spend their allocated amount of funding in order to comply with the terms of the agreement to participate in this project. Any unspent money must be returned to the Ministry. Service providers are also required to report on eligible expenditures incurred for participation in the project.
The total funding allocation is split over the two fiscal years of the project. Service providers will receive fifty percent of their funding in 2015/16 and the rest in 2016/17.
What if my organization doesn’t register any new learners in 2015/16 and only a small quota of my target for the project, are service providers expected to return the funding?
Funding is not directly tied to administering a test to each learner and achieving the target, but the administrative expenses to support the delivery of the project. The expectation for service providers is to demonstrate the administrative costs incurred with being able to deliver the ESEE tool. For example, these costs could include training time and materials, hiring new staff or use of existing personnel resources, efforts to coordinate reporting.
Can my organization test more learners than the requested target number?
Yes, your organization can test more learners. Project funding will remain as indicated on the agreement.