Ed Trust Comments on U.S. Department of Education FY 22 and FY 23 Pooled Data and Evaluation Funding

Overview

The Education Trust supports the recommendations from the Institute for Higher Education Policy and Results for America for the pooled evaluation authority for Higher Education Act programs authorized in the Consolidated Appropriations Acts of 2022 and 2023, including the importance of disaggregating data by race/ethnicity. We helped develop them.

We also wanted to provide additional recommendations for the Department’s use of the approximately $7 million of FY 22 funding, and the Department’s planning for FY 23 funding, for data collection and analysis and evaluation. Our high-level recommendations are as follows:

1. Don’t spend it all in one place; make down payments on multiple sound investments.
2. Overhaul the Department’s higher education program data infrastructure to better inform policymakers and the public and incentivize institutions of higher education (IHEs)/grantees to perform at their best.
3. Review the ways higher education program grantees learn about and use evidence to design and implement their programs.
4. Support evaluation(s) of student success interventions that (a) are commonly carried out by IHEs/grantees and (b) are promising but not well supported by existing evidence of effectiveness.
5. Create and pilot a framework for conducting implementation audits of proven programs.

Don’t spend it all in one place.

Although $7 million is a small fraction of what it would take to evaluate efforts to advance postsecondary success for millions of students nationwide, we hope you won’t invest it all in a single project. Consider making smaller bets on at least two projects — at least one in the realm of data collection/analysis and one in the realm of evaluation — that you can build on in future years. This approach reduces the Department’s risk of failure across projects while multiplying engagement with external stakeholders (including colleges and potential contractors) who are aligned with your data and evaluation agenda. More touch points with more projects could also strengthen the culture of and commitment to data and evaluation inside the Department.

Overhaul ED’s higher education grant data infrastructure to better inform policymakers and the public while strengthening incentives for IHEs/grantees to perform at their best.

Each year, the Department spends many billions of dollars on direct grants to colleges and on student financial aid, which students spend at colleges. And yet little is known about these investments in postsecondary student success. Ed Trust supports ending the federal ban on a student unit record data system. In the meantime, there is much the Department could do to bring greater transparency to and unlock deeper insights into federally funded postsecondary education programs to the benefit of those working to improve educational opportunity and outcomes for students, including federal and state officials, educators, researchers, and families. We encourage you to solicit a contractor to:

• [Audit] Conduct a data infrastructure audit that includes findings and recommendations about which data are collected and how they are disaggregated; data collection, management, and analysis systems and processes, including the relationships between separate systems; staff expertise and training related to data collection, management, and analysis; and data reporting, transparency, and public engagement policies, procedures, and formats, including reviews of websites, publicly available data files, and FOIA.
• [Matrix] Produce and publish a matrix identifying (a) the most-commonly proposed and conducted interventions supported by federal higher education grants, (b) the federal cost per student for each intervention, (c) outcomes associated with each intervention, and (d) the overall evidence associated with each intervention.
• [Pilot] Create a model data infrastructure environment for at least one postsecondary education program to show what’s possible and lay the groundwork for a comprehensive overhaul of the Department’s higher education grant data infrastructure.

Review the ways higher education program grantees learn about and use evidence to design and implement their programs.

If all federal higher education grantees who are focused on increasing postsecondary student success used proven strategies with strong evidence of impact, then Congress probably would not have created the new evidence-based Postsecondary Student Success program in the consolidated appropriations acts of 2022 and 2023. Surely, part of the problem is the need for more proven strategies. It may also be the case that grantees are not aware of the existing research, or that for some reason they are not translating existing findings into their program designs and implementation. To complement efforts to grow the evidence base for college access and success efforts, the Department should fund a qualitative research project to learn what federal higher education program grantees know about existing evidence-based interventions and to identify barriers and opportunities related to increasing the adoption of evidence-based interventions in federal higher education grant programs, which may include increasing technical assistance and/or incentives throughout the grantmaking cycle to help applicants and grantees identify and successfully adopt evidence-based interventions.

Support evaluation(s) of student success interventions that (a) are commonly carried out by IHEs/grantees and (b) are least well supported by existing evidence.

We encourage you to support evaluations of one or more common and promising interventions with limited evidence of effectiveness. For example, there are certain programmatic activities statutorily required of certain TRIO programs, such as tutoring, associated with limited or mixed evidence of effectiveness for increasing college access and success. On the other hand, IHEs have substantial flexibility in designing their interventions for the Strengthening Institutions program. We encourage the Department to support evaluations of specific interventions, whether required or allowable under Department programs, that represent a substantial share of federal investment and at the same time are not yet backed by substantial evidence of impact to determine whether to continue investing in and supporting the implementation of those interventions, and if so, how best to proceed.

Create and pilot a framework for conducting implementation audits of proven programs.

To protect federal investments and ensure that students reap the benefits of tried-and-true interventions for advancing postsecondary success, we encourage you to issue a solicitation for the development and piloting of a program implementation audit framework. The purpose of a program implementation audit would be to examine the implementation of proven programs and verify their adherence to the established standards and best practices most likely to yield strong impact. For example, if you structure the $45 million Postsecondary Student Success program as a tiered competition, targeting investment in interventions with the greatest evidence, how might you plan to assess to what extent grantees are executing previously evaluated interventions based on evidence and maximizing outcomes for their students?

If a community college plans to use federal resources to implement a version of Accelerated Study in Associate Programs (ASAP), which doubled graduation rates among associate degree seeking students in New York and Ohio, how might the Department monitor — and if necessary, assist — the college to ensure that the institution successfully implements this comprehensive program of student supports as prior evaluations suggest it should be implemented?

Closing

Our hope is that you will use the $7 million in FY 22 to make meaningful progress on an ambitious data and evaluation agenda that you will build on with greater resources in FY 23 and beyond. As is indicated in the IHEP/RFA letter we signed onto, we urge you to set aside the maximum allowable funds for FY 23, so you have the resources to strengthen your data infrastructure and ramp up our collective knowledge about what works in higher education in ways that are aimed at closing opportunity gaps by race and ethnicity and income for education beyond high school.

If you have any questions or would like to discuss these ideas, please don’t hesitate to contact Jinann Bitar, director of higher education research and data analytics, or Philip Martin, director of higher education policy implementation.