Washington, DC – ARC Meeting (10/25/19) — consumerfinance.gov

Good morning. Welcome to the Consumer Financial Protection
Bureau’s Academic Research Council Meeting. My name is Zixta Martinez. I serve as the Associate Director for the
External Affairs Division at the CFPB. Today’s meeting is being held at the Bureau’s
headquarters in Washington, DC, and is being live streamed at consumerfinance.gov. A recording of the meeting will be made available
on the Bureau’s website. You can also follow the Bureau on social media
via Facebook and Twitter. In a few moments I’ll turn to Andrew Duke,
who will kick off the meeting, and introduce Director Kraninger. But first I’d like to welcome our Committee
Members, and introduce the individuals who are serving on the Bureau’s Academic Research
Council, or ARC, during the Fiscal Year 2020 cycle. Members, when I call your name, please raise
your hand. The ARC Chair is Michael Baye. he is the Burt
Elwert Professor of Business Economics and Public Policy at Indiana University’s Kelley
School of Business. Karen Dynan is Professor of the Practice of
Economics, Department of Economics at Harvard University. Terri Friedline is Associate Professor of
Social Work at the University of Michigan. John Lynch, Jr. is Director of the Center
for Research on Consumer Financial Decision Making, and Senior Associate Dean for Faculty
and Research at the Leeds School of Business, University of Colorado, Boulder. Tom Miller, Jr. is Professor of Finance and
inaugural holder of the Jack R. Lee Chair of Financial and Consumer Finance at Mississippi
State University. And Joshua Wright is University Professor
at the Executive, and the Executive Director of the Global Antitrust Institute at Scalia
Law School, at George Mason University. We also have with us Brian Johnson, Deputy
Director for the Bureau, and Matt Cameron, Staff Director for the Office of Advisory
Board and Councils. I’m now pleased to introduce Andrew Duke. He serves as the Policy Associate Director
for External Affairs, and brings 27 years of experience in public policy, including
20 years on Capitol Hill, serving with three different Members of Congress. He received his BA economics from Hampton
City College in Virginia. Andrew. Thank you, Zixta. Good morning. And thank you all again for being here with
us today. As Zixta mentioned, my name is Andrew Duke. I am, serve as the Policy Associate Director
for the Division of External Affairs, as well as the Division of Consumer Education and
Engagement, here at the Consumer Financial Protection Bureau. I’d like to thank all of our ARC members for
agreeing to serve in this capacity, to help advise the leadership of the CFPB and its
Director on a broad range of consumer financial issues, emerging market trends, and the Bureau’s
strategic research planning process and research agenda. For those of you live streaming let me share
a brief overview of our schedule today. Following the Director’s remarks our Chair,
Michael Baye, will conduct the meeting. Chair Baye will introduce Bureau subject matter
experts for a discussion on the Bureau’s Start Small Save Up initiative. After that discussion Chair Baye will then
introduce Bureau subject matter experts for a discussion about the Bureau’s research agenda. At approximately 12:45 p.m. the ARC will break
for lunch. At 2:45 p.m. the round table will reconvene. Chair Baye will introduce Bureau subject matter
experts for a discussion on defining consumer harm and consumer injury. Then at 3:45 p.m. the ARC round table will
adjourn. As background, the CFPB established this Committee
to provide advice about the Bureau’s strategic research planning process and research agenda,
related to consumer financial products or services, consumer behavior, and cost benefit
analysis, among other topics. As a reminder, the views of the ARC Members
are their views. They are greatly appreciated and welcome. Yet, they do not necessarily represent the
views of the Bureau. I’m now pleased to introduce Director Kathy
Kraninger. Director Kraninger became the second confirmed
Director of the Consumer Financial Protection Bureau in December 2018. From her early days as a Peace Corps volunteer,
to her role establishing the Department of Homeland Security, to her policy work at the
Office of Management and Budget, and now to the CFPB, Director Kraninger has dedicated
her career to public service. It is my privilege to welcome her to the today’s
round table meeting. Director Kraninger, the floor is yours. Thank you, Andrew. And good morning, everyone. Thank you so much for joining us today. This is the first meeting for some of you,
so welcome to the Bureau. And I am grateful that you accepted this appointment. Thank you for being patient during the application
and selection process. And I can tell you that I am certain that
the Bureau will benefit from your membership. But I certainly expect that all of you will
as well. I’d also like to extend my appreciation to
Michael Baye, our new Chair, for taking on this important leadership role, and helping
the Bureau manage the Academic Research Council. The ARC was formed to advise the Bureau with
respect to our strategic research planning process and research agenda, as well as to
provide technical advice with respect to research methodologies. The Bureau’s research can provide a strong
empirical basis for what I see as the primary purpose and focus on the Bureau, preventing
harm to consumers. As you know, Congress clearly articulated
the purpose and objectives of the CFPB in the Dodd Frank Act. Among other things the Act states the Bureau
shall seek to implement and where applicable enforce federal consumer financial law consistently
for the purpose of ensuring that all consumers have access to markets for consumer financial
products and services. And that those markets are fair, transparent,
and competitive. Congress gave significant powers and tools
to the Bureau to carry out that mission. These include education, regulation, supervision,
and enforcement. At the Bureau we believe the best application
of these tools is to focus on prevention of harm to consumers. We believe that if we can prevent harm before
it occurs we have saved consumers from the financial headaches, setbacks, and devastation. I’m so glad that all of you have volunteered
to share your thoughts and perspectives. And I look forward to vigorous discussions
on how best to harness our research capabilities to further our mission. Our agenda today focuses on three important
topics that are before us right now. Let me briefly touch on each of them. One of the first things we will be talking
about is our Start Small Save Up initiative. Part of the Bureau’s responsibilities include
conducting financial education programs, and ensuring that consumers receive timely and
understandable information to make responsible decisions about financial transactions. With the multitude of public and private sector
entities engaged in consumer financial education, I also want to ensure that the Bureau uses
its education tool where it is uniquely positioned to add value and provide leadership. Specifically, the Bureau recognized a gap,
and conducted extensive research to develop a consumer centric definition of financial
well-being that goes beyond income and credit score. It captures a person’s sense of financial
security and financial freedom of choice in the present and in the future. Core to that security is control over day
to day and month to month personal finances, as well as the capacity to absorb financial
shock. We have also found that consumers can experience
financial well-being, or lack of it, across a wide range of incomes. It is tied to financial skills, and the confidence
in those skills. Today’s consumers need these skills even more. For example, fewer than half of Americans
set aside money for their children’s college education. More and more people reach retirement with
incomes and savings that simply won’t meet their needs. Perhaps most concerning to me was the Federal
Reserve report that found that 12 percent of Americans say they could not cover a $400
dollar emergency expense. And an additional 27 percent said they would
turn to credit or sell something to cover an emergency. The goal of Start Small Save Up is to move
the needle on the number of Americans in this country who can cover a financial shock, like
a $400 dollar emergency. I’m committed to bringing together partners
from across sectors to develop and execute a strategy to achieve this outcome. The collective investment in financial education
is substantial across the country. And I believe we can increase the level of
emergency savings by bringing these efforts together. Research has an important role to play in
helping us formulate successful strategies. And I look forward to your thoughts about
promising areas for research. For example, research can help us measure
the extent to which consumers save for emergencies. A challenging question, in that emergency
savings can be spent, and then replenished. Research can help us understand the barriers
to savings, and why some consumers save, whereas others similarly situated do not. And research can help us evaluate which strategies
are actually moving the needle on savings, and not just enrolling consumers who would
save with or without the intervention. Another important item on our agenda today,
particularly one that all of you will no doubt have opinions about, is the Bureau’s research
agenda. Research forms the backbone of our key decisions. How do consumers make decisions that affect
their financial well-being? What methods help consumers make informed
decisions? How do we know when consumers are being harmed? Setting a research agenda will help us to
answer the most important questions facing the Bureau. We have built an exceptional research office,
with highly skilled researchers. Although much of the time in the Office of
Research is necessarily devoted to analyzing the costs, benefits, and impacts of regulations,
the Bureau is developing and supporting our fair lending examinations. And the research agenda that we adopt will
shape the research output from the Bureau over the coming years. To produce that research we have a wealth
of assets to draw upon, like our Consumer Credit Panel, and the National Mortgage Database. And we have the capability to do surveys that
we can pair with our administrative data, like the Making Ends Meet survey we have recently
concluded. Plus, we can obtain new data by conducting
lab and field trials. But we need to prioritize our efforts to make
sure we’re focused on the most important areas, and rigorously researching important questions
in those areas. So, we’re asking you for your insights on
what are the most pressing research questions in consumer finance that the Bureau should
try to answer. And as skilled researchers we’d like your
thoughts on how best to use our resources to answer those questions. And finally, in the last session we’d like
to hear your thoughts on how the Bureau can best define and measure consumer harm. The concept of consumer harm is central to
much of our work. You can find it in the statute that created
the Bureau. Indeed, the Dodd Frank Act cites risks to
consumers as a reason or factor to be considered in many specific Bureau activities, including
market monitoring, supervision, and research. Furthermore, the concept of substantial injury
is central to the concept of an unfair act or practice in both the Dodd Frank Act, and
the Federal Trade Commission Act. The Federal Trade Commission has considered
consumer harm in enforcement and rulemaking for decades. And since our inception, so have we. From my perspective, the main goal of this
session is to bring academic researchers into this conversation. We want to hear about how social scientists
define consumer harm, and how that harm can be measured as rigorously as possible. We know this is a two way street. Consumer financial products and services inherently
can present risks. Credit, for example, cannot even be discussed
without considering the time and uncertainties that come with repayment. A lot can go wrong. And the law does not envision that every bad
outcome is a harm that merits a legal remedy. Ultimately the Bureau’s work involves balancing. And we know that balancing relies on good
measurement. And that good measurement relies on clear
and practical definitions. So, we want to begin by hearing form experts
on the foundational work that you have done, or believe we should do in defining and measuring
consumer harm. So again, thank you all for being here today. Thank you for serving on our Advisory Committee. And I look forward to your partnership and
thoughts on all of these issues. Thank you very much Director. My name is Mike Baye. It’s an absolute pleasure to have the honor
to chair the Academic Research Council, or ARC. And especially to serve with such an esteemed
group of social scientists that are also like myself, academics serving in this capacity. It’s absolutely fantastic that the Chairman
had a commitment to ensuring that the actions of the Bureau are rooted in the scientific
method. And very happy to be here in that regard. I’d like to also thank you Dr. Kraninger,
for actually being here for the entire day today. And I understand you were participating with
other advisory committees yesterday. So, it’s a reveal preference, you care a lot
about this. And it’s absolutely fantastic that you do. Look forward to working with you in the future. Today’s meeting, as the Director pointed out,
are going to focus on three very important initiatives here at the Bureau. We’re going to be talking about the Bureau’s
Start Small, Save Up Initiative, its research agenda, and also defining consumer harm and
consumer injury. So, without further ado, we’re going to begin
our first session. And I’d like to introduce the staff that will
be navigating us through the important work that the Bureau is doing in this dimension. First of all, across the table from me is
Jason Brown, Assistant Director for the Office of Research. Melissa Knoll, Section Chief Decision Making
and Behavioral Studies for Office of Research. Caroline Ratcliffe, very good. Senior Economist for the Office of Research. LaShaun Warren, Campaign Manager from the
Division of Consumer Education and Management. Desmond Brown, he didn’t raise his hand. Okay. Deputy Director from the Division of Consumer
Education and Management. And to my left, we have Tom Pahl, Policy Associate
Director for the Division of Research, Markets, and Regulations. And David Silberman, Associate Director from
the same Division. So, thanks for joining us. I’ll now turn it over to you fine folks to
enlighten us. Great. Thank you so much. So, the way we’re going to set this up is
I’m going to just go through some slides. And then we want to hear from you. And we’ll facilitate a conversation among
the Start Small, Save Up team up here. So, I’m new to the ARC members. Most of the ARC members. And so just a few words about myself. I have done a lot of research on savings. While I’m new to you, I’m not new to the savings
field. I’ve done a lot of work on emergency savings,
a randomized control trial of the match savings program, as well as looking at debt in a lot
of detail, and financial well-being. So, I’m new to the Bureau. I came from the Urban Institute where a lot
of that research was done. And I’ve been here at the Bureau since early
this summer. And I just want to say before I get into the
slides, so, we are representing, Melissa and I co-chair the research and evaluation work
stream under Start Small, Save Up. But we are a cross-Bureau initiative, and
cross-disciplinary. So, a couple of our colleagues are here also
Brianna Middlewood, Jack Gardner, are in the Office of Research, as well as we have folks
in Consumer Education and Engagement that are part of the research and evaluation work
stream. So Dave Siniski’s here. And this meeting is really a great time for
us to come together. Because while we have framed the work that
we’re doing, we’re in the early stages. And so as I said, I’m just going to give a
brief overview. But the biggest focus is for us to hear from
you. And we have a disclaimer. So maybe — do I have to read this? No. Okay. Great. I hadn’t seen that earlier. Here we go. Okay. So, just as a backdrop to the initiative,
the Director mentioned this, but many consumers report a lack of emergency savings. And consumers also experience, many consumers
experience financial shocks. So, the most, a few data points, the most
commonly cited is this one from the Federal Reserve Board that the Director mentioned. There’s also a study by Pew that found that
56 percent of people worried about their finances in the past year. And among that 56 percent 83 percent worried
about a lack of savings. So, along with this lack of savings, there’s
also a lot of volatility in families’ lives. And this statistic is from another Pew report
that found that 60 percent of families experienced a financial shock in the past year. And this is drops in income, but also expense
spikes. So, about 25 percent of families experience
a significant drop in income. These expense spikes, about 30 percent report
a significant cost to repair a car, 25 percent a home repair. So, we’ve got this situation where we have
a lack of savings and a lot of volatility. I want to mention that as part of Smart — Start
Small, Save Up, we are also going to be using our Making Ends Meet survey data that the
Director mentioned. And that is built off of the consumer credit
panel. But it has information on monthly savings
habits, ability to handle financial shocks, experience running out of money. So, that’s going to be part of this work. To add to this base of what we know. So we’ve got this combination of a lack of
savings as well as these shocks. And this can affect financial well-being. And again, the Director touched on this, that
the Bureau’s done a lot of work on financial well-being. That the Bureau work shows that having higher
levels of savings is associated with higher levels of financial well-being. So, on the left side we have liquid savings. This is cash on hand, dollars in checking
and savings accounts. And what this figure shows is that as you
move up to higher levels of liquid savings, we see higher levels of financial well-being. And one thing I want to point out is that
even at small dollars, you can see a few hundred dollars as you move up levels of financial
well-being increase. And this is found in other research that controls
for income in the randomized evaluation I mentioned, we found that people were saving,
and their — we looked at financial stress was reduced. So that’s a little bit of the motivation. So the overall vision of Start Small, Save
Up, and then I want to talk about the research and evaluation work group. So the vision is to increase people’s opportunities
to save. And empower them to achieve their savings
goals as a step to improve financial well-being. And we talk about financial well-being, because
emergency savings is really a flow. So people save, there’s a shock, they spend
it. And then the goal is that they save back up. So, it’s harder to measure then if we think
about retirement savings, where the goal is that people increase their savings throughout
their working lives as a, you know, retirement. We have five work streams. We’ve put research and evaluation at the top
of this circle here. But the research and evaluation piece was
built in from the beginning. And we really see this as a foundation for
the other work streams. So, the other work streams are employers’
partnerships. So here this would be partnerships with financial
institutions, FinTech companies, other government agencies, communities, outreach and engagement. So to make this a little bit more concrete,
so our employers’ work group is actively engaging with employers around strategies to increase
the savings of their employees. So the research and evaluation work group,
we’re working with them. And we’re use — to sort of build in what
we know from the literature to inform those strategies. And at the same time we’re looking for opportunities
to set up pilots and tests. So, basically this test and learn so we can
think about strategies around financial incentives, automation, messaging. The goal of the research and evaluation work
group is enhancing the evidence base of Start Small, Save Up. Collecting, analyzing, reporting on data. So, we really do see ourselves both as a foundation
for the other work, as well as providing evidence to the field. Our planned activities, so the first is before
we embark on our work, we really need to understand what’s out there, and what works and doesn’t
work. So, we’re cataloging what’s been rigorously
tested in the field. We’re also doing a scan of the field. So we expect that there’s a lot of interesting
initiatives out there that are helping people save. And they’re promising, but they’re untested. So we want to understand what they are. And look for opportunities where we could
evaluate. And we’re also looking, doing a scan of the
field for existing data. So, particularly if we think around — about
FinTech companies, they have lots of data. What are the opportunities for us to get that
data and to learn in house about people’s, consumers and consumer needs? And this third bullet here is really, we’re
trying to get at this pilot and testing strategies to increase emergency savings. So this is field studies. And then the final two, this first one is
just what I had said earlier, that we’ll be putting together a publication using our new
Making Ends Meet survey data, which just arrived at the Bureau in September, so it’s really
exciting for us. And the final point here, is we’re conducting
a series of research experiments. So these are lab experiments on the savings
and debt puzzle. So this is the phenomenon we’re looking at,
where people simultaneously hold savings and credit card debt. And that’s being led by one of our psychologists,
Brianna Middlewood. So this is an overview of where we are with
the initiative. So now it’s really for me to — for us to
hear from you. And I will stop talking. But before that, so we’ve — we’re really
looking at opportunities where we can add to the field. And we are doing this cataloging. But so our first question, and you have the
whole series of questions. And we don’t have to be super rigid. But why don’t we start with this first one. And we can chime in. People can, you know, jump around a little
bit. But, what are the under-explored areas in
the savings space where additional research would be helpful for helping us identify effective
savings strategies? And we of course know that there’s not a one
size fits all. So, as part of this, like what are the specific
populations or strategies for specific populations, and what we need to know in that space? So, I’ll just throw it open. I have a question on, just a fundamental question. I don’t know what the literature shows. But, I don’t think it’s a secret that people
should save money. I mean, in my lifetime, you know, long lifetime,
there’s a — that — it’s pretty well known. But, what’s the fundamental reason that people
don’t save money? That’s been, obviously been studied hard. But, what is the impediment? Well, what we do know from the literature
is, so this issue of low savings is not just a low income issue. It goes up the income — up the income distribution. What we do know is that when people are given
incentives to save, they will save. If we think about the incentives that higher
income people get to save through preferential treatment for retirement savings, mortgage
interest tax deduction, there’s incentives to purchase a home. So, that’s — those are the two key ways that
people build their wealth. But we do know that low income people can
and will save, and particularly when they’re given incentives. And we know that the opt in feature of, or
the opt out feature of retirement savings has really been effective in increasing people’s
participation in retirement savings. There has been some talk about doing something
like that in — for emergency savings as well. To make it automatic. That you’re — that you don’t have to think
about it. But that you’re setting it up, and it happens
automatically. So, just — to kind of follow up, I found
that very interesting. And I guess one thought that I had is, your
focus really is on kind of these short term shocks that would lead people to need a little
nest egg to draw on. And it strikes me that there is kind of this
tradeoff between, you know, saving for retirement, these long term funds versus the nest egg. And I’m just wondering to what extent there
can be tradeoffs or crowding out across those two dimensions? Um-hum. And that could particularly adversely affect
individuals, you know, that don’t have the means to save. I think that’s where Tom was coming from. That some people just don’t earn a lot of
money, and therefore they’re not going to save. Implicit I think in the question. And it strikes me that, you know, as you contemplate
incentives to induce people to save, I mean, there are competing institutions in Washington,
like the Department of Labor, Social Security Administration that are trying to provide
incentives for people to save for retirement, right? Those incentives typically provide up front
benefits, tax deductions. That’s, as you climb up the income ladder,
as you said, those. Um-hum. But, then there are these penalties associated
with drawing those funds out, right? That when confronted with, you know, losing,
paying a 10 percent tax penalty, that might induce people to take credit card debt. And I’m just wondering to what extent the
research program that you’re envisioning here, is recognizing that short term shocks might
somehow be linked to, you know, long term savings, short term savings decisions? I’m just rambling, but. I mean, I can start. So, I think you’re raising a really important
point. And that’s why we also want to look at financial
well-being. Um-hum. We do know that there is a lot of people who
aren’t taking — they don’t have access to employer provided retirement savings. And they also, or they’re not taking advantage
of what’s available. It would be nice if our other ARC member,
Brigitte Madrian was here. She’s done a lot of work on this idea of side
car savings, with AARP. Where they are really worried about this exact
point. That people are taking money out of their
retirement savings accounts. And they’re — they have a penalty, and they’re
using it for emergencies. So the idea of these accounts is that you
enroll people into an emergency savings plan. And then once that reaches a limit, it can
go into retirement. Um-hum. So that you’re — you’re trying to help people
sort of create that buffer in the short term. But as a road map for longer term financial
security, not to hinder it. But I think you’re right, we — there’s a
lot we don’t know about sort of the short term and the long term. But, we have — we are preparing that, sorry. So one thing that I would add that I think
is related is, as Caroline mentioned, and as you alluded to, we do know a lot about
retirement savings. And the things that work and what doesn’t
work. And kind of what encourages people to save
for retirement and other things like that. So, I think one of the things that we’ll be
looking at is, what are the distinctions between retirement savings and short term emergency
savings? What makes them different? There’s the time horizon. There’s maybe the concreteness of the goal. Emergency savings is kind of this like nebulous
thing that we kind of understand what emergencies are, we might not. Some people might not want to dip into their
savings for something they don’t think rises to the level of an emergency. But we do know what retirement is. There are also kind of potentially social
norms around these different kinds of things. And kind of like I said, goals in terms of
long and short term. But also we kind of understand a little bit
how much we need for retirement. We don’t necessarily know how much we need
for emergencies. There’s different standards there. So, I think one area of exploration is kind
of studying specifically what are the differences between these types of savings that we know
more and less about. And digging into some of those things as we’re
approaching this question. So, I wanted to ask, Caroline, you’ve kind
of alluded to this, about employer benefit plans. And low income folks having access to those. And I wondered the extent to which some of
the structural conditions and the environments in which people are experiencing, how those
affect abilities to save? And also just to kind of add, you know, that
some of the questions, the foundations of your research? There is a fairly long and robust literature
on, you know, some of the mechanisms like opt in and having incentives to save, and
that people do engage in those when they’re provided to them. And what’s the extent to which those are provided
to folks in their environments? So, on the structural conditions of the workplace,
that I’m not as familiar with. I mean, I think what we are, our employer’s
work group is going into employers and to try to set up mechanisms to help folks save. You know, what can the employer do? So, if some of those barriers, like here we’re
looking at split to save. There’s other, America Saves has been doing
that with a number of employers. So basically, setting it up so people can
split their paycheck into both a checking and savings account. And one of the issues is, if people don’t
have a savings account, then that’s another barrier. So, and then I think, you know, just the opt
out part, there is a lot of, as I said, there’s a lot of interest in testing that as a strategy
around the emergency savings piece. And that because if you can get it set up,
set it and forget it, or not even have to set it. If it’s just your, people are automatically,
when they come in as an employee, say some percent of their paycheck would go into an
emergency savings account. But, it’s not been something that’s been — that
has been tested yet. If I could just jump in on the work you’re
doing with employers. I think it’s terrific. I think employers are a really important part
of helping people to save. That’s been demonstrated in the research. I wonder if part of the effort could be also
talking to employers about the burden, or barriers, or costs, of doing these interventions? Because it, you know, we for example can see
that large employers, many of them offer retirement savings plans. That’s not so true for smaller employers. So, if you’re thinking about how to actually
act policy wise, kind of figuring out what the barriers are, not just on the household
side, but on the employer side, to doing these sorts of things for these employees, is important. And then just a second comment related to
this. Which is, I think it’s not only important
to think about encouraging people who have kind of traditional relationship with their
employers to save, that’s where a lot of the literature has been. But, given the evolution of the labor market,
we know that increasing numbers of workers have kind of these non-traditional relationships. They’re going contract work, or they’re in
the gig economy. And what can be done for those people, to
help them encourage — to encourage them to save? Great. Thank you. If I could chime in. So, I think another piece of it is just the
psychological factors that affect savings. So, one of the things that I think deserves
more study is alluded to already, and that’s how people view the fungibility between these
different sources of savings. So when we talked about college savings, retirement,
emergency. And to what extent do people see those as
permeable or not? And then — and then there’s the psychology
of how people frame a pool of money. And whether or not they think it’s usable
for a particular purpose. So, one of the things that we’re studying
in our shop is, retirement plan leakage that happens at job separation. And it appears that people are using it in
ways that seem — that seem sensitive to whether they frame that pool of money as — as not
something that is necessarily for retirement security, but something that’s for — that
is flexibly labeled as something for emergencies. Where people drain their entire retirement
account when they change jobs. And now they’re back and they’re 35 years
old, and they’re back at zero with what they had saved. So, that substitution between these different
domains is something that you’re in a position to study, where I think that it’s difficult
for other entities to do that. And then I — the other thing that was alluded
to already, somebody mentioned this issue of the nebulous nature of what you should
have for emergency savings. And so one of the things that keeps people
from striving for a goal is when they don’t think they’re going to achieve it. And so the issue of what does one need for
a particular purpose is critical. I have a colleague that we recruited from
the — she’s now on our finance faculty. Her name is Emily Gallagher, from the — who
studies the issue of, well how much do people actually need in their emergency accounts? Because if you tell somebody who has few resources
that you need two thousand dollars, or some large number, then they say I shouldn’t strive. And yet, whereas if you could research and
establish that a lower number was actually going to provide a lot of the benefit, then
that may make people perceive that it’s possible. And then I’d just like to have an overarching
remark over all that, is that some of this is psychology. And so, on your team, the idea of having a
team with social scientists across the board is real useful. Thank you very much for that. One of the — I’m very interested in what
you said about the retirement and people taking money out. And I was talking to someone recently who
works a lot — who works at a credit bureau with a lot of lower income clients. And the framing, she was saying how the framing
of emergency, like people what to know like what they can use. Yeah. They don’t like the word emergency. At least her client — her clients. And so I think it is important for us to think
about as we’re framing it out in the world with people who are struggling. That the terms that we use might not be the
best terms. Or the terms that they react to or respond
well to. Which is another reason why the point that
Tom made earlier, about you were having partnerships both in the academic community and in the
business practitioner community, is super useful for you to have those institutional
insights. Um-hum. Any other thoughts here? Well, let me just put up some of our other
ques — oops, I went back. Let’s see. So, the next question here is advantages,
do you see advantages and disadvantage of working with different types of partners? So here we are with our employers and our
partnerships group. Thinking about employers, financial institutions,
FinTech companies, different ones, different sizes. Any thoughts there? And also that next question is, what types
of partnerships do you think would be most valuable for our work? Any thoughts here? Well, I think in terms of picking and choosing,
I’ve got no advice there on which types of partners you want. But, I do think, you know, one of the key
advantages of partnering with institutions, like employers for example, is, you know,
the opportunity to do field experiments. To actually test out, you know, different
mechanisms to have a — to move the needle in terms of savings rates for example. In terms of, you know, the disadvantages,
you know, one of the disadvantages, you might find that the policy you think is going to
work, doesn’t work. So a lot of places in Washington don’t know
— don’t want to do the experiment because they may not like the answer. And that’s why I’m very happy that you’re
here, to care about the underlying science. You obviously have to make sure that it’s
a randomized control. And you know, that the employer that you’re
— that you get to sign up, doesn’t have characteristics that are materially different than other employers
out there. But, one partnership that I would strongly
encourage you to think about, is partnering with the academic community. I know you’ve got a symposium coming up in
December, a research symposium where you’re going to have a number of academics come in. And you know, one of the scarce resources
that you folks have, that the typical academic doesn’t have, is the deep institutional knowledge
and the access to really fantastic data sets. And so to the extent that you can leverage
that to partner with academics that might not have those skills, but might have other
skills that could complement the types of research that you want, could be incredibly
valuable, I think, for policy making. And ultimately the types of research that
guides the things that the Director wants to do. Mike, can I ask you a question just — so,
if you’re thinking about partnering with employers, and you said a randomized control would be
ideal. That raises the — so in theory what you’d
like to be able to do is say, this group of employees gets this program, and this group
of employees doesn’t get this program, to see whether the people. Employers may have resistance to offering
a benefit to some employees, not others. Do you have thoughts as to how you might be
able to get around that, the sort of challenge of the desire for randomized control, and
the employer’s potential desire to treat everybody the same? I think that’s why my caveat was, be careful. The employer that’s willing to allow you to
do the experiment, may not be representative of the employer out there. Right. So, I just give that caution. I’ve got no inside information on how you
convince a particular type of employer to do that. Or, I mean, there’s other types of randomized
experiments you can do as well. You’re talking about financial institutions,
and others. Yeah, so I was going to say related to that. It’s not unique to employers. So, we’ve worked with financial institu — not
financial institutions, other companies before, where kind of convincing to withhold a treatment,
or that there needs to be a control group or randomization is definitely a challenge,
and something that we go back and forth. So, I’m interested in the thoughts of the
group around the different types of research. So obviously the RCT is the gold standard. Sometimes there can be other types of program
evaluations or collecting data where there might not have been an RCT. So, the types of things that Caroline lined
out, lined up about collecting data from companies that are already doing things. Or overlaying potentially some study on a
company that’s already been doing something. But they may not, have not necessarily had
a treatment and control exactly in that way. What are the different types of things we
can learn from different types of research? Yeah, if I could chime in. So, I was going to — this is answering both
that question and the preceding one. So, I’m on the board of this organization,
Common Sense Lab. Which is a nonprofit that’s aimed at FinTech
to help lower/middle income people. And I was just at their board meeting. And they were having exactly this kind of
discussion about who were the relevant partners. It turns out that it’s very difficult, some
of their enthusiastic small partners don’t have the ability to do things at scale. So that winds up militating toward larger
partners. And another thing that happens is, when you
have some research informed interventions, they don’t always work. And so, I think their hit rate is like 50
percent on some of these interventions. And so — so, which is good, right? But, part of that is that some types of research
are more expensive than others. And so I know in your capability, the fact
that you’ve got the suite of all these methodologies. That if you’re not entirely sure about the
particular way to implement this, that’s a place where laboratory experimentation is
useful as a predecessor before you go in there and do the — do the big field experiment. But I guess the other insight I would say
from talking with those other folks is that they find these FinTech companies to be readier
to do the randomized experiments than to go into a particular employer. The employers have sort of this duty to their
employer — their employees that make them reluctant to have the randomized data. So, I would add also to both the questions
about partnerships and about the types of research. I’m kind of — I was chuckling to Michael’s
suggestion about partnering with academics and researchers, which you know, as a member
of the Academic Advisory Council, that wasn’t the first partnership that rose to my mind. It was, you know, it was thinking about the
groups that are, and organizations that are most closely connected to the people with
whom you are trying to reach. So that can be an employer. It can be a financial institution. And I think there are also a range of organizations,
nonprofit intermediaries that, you know, have financial coaches or counselors. Other sorts of educators that are sitting
with clients, and helping them navigate some of these decisions that are very close to
the realities that people are experiencing. And to the question about the methods. I think RCTs are important. And empirical data is important. And when we have the ability to answer questions
with empirical data, we can and should do so. And there are, you know, clearly decisions
about costs and — related to implementing those types of studies and methods. I think there’s also kind of a multiplicity
of knowledge. And it depends on the type of question that
you’re asking, and what you want to receive in return. And often, being able to triangulate data,
different types of data from many different angles and sources, is what strengthens your
position on a particular topic. So, in some of the RCTs that myself or that
my colleagues have worked on, having a treatment control, or a treatment participant marry
a control participant, is a little bit, you know, difficult for your RCT. You know, the — the rigorous testing of your
RCT that you have been so careful to implement in the field where real things happen. So, being able to demonstrate across the wide
range of methods and data points, qualitative to quantitative, I think, builds a really
strong case. It’s not any one particular study in the literature
that we point to really. It’s a body of research that has a consistent
— a consistent story to tell us. Good point. I just wanted to, I guess largely echo some
of the thoughts that Terri and others have offered. One of the reasons at least within the economics
professions that RCTs have become the gold, we had all the nice things about causal inference,
and it was fantastic. But one of their reasons, there’s been a little
bit of an overreaction about non-RCTs, is a body of work sometimes by a, you know, I’m
an economist, but a legal academic. And some of my legal academic friends, and
maybe me sometimes, will sort of over claim where there’s not sort of real causal inference. And I think we’ve seen in economics profession,
a push back against that. And for lots of good reasons. We get lots of really, really nice things
out of well done RCTs. One of the things, and this is more comment
and compliment then question. One of the things that struck me in reading
through some of the materials sort of in preparation for this meeting, was the degree of care in
articulating what inferences could be drawn from what comparisons. When there is causal inference to be had,
there is a claim that there is causation. And when there is not, there’s sort of careful
talk about what inferences can be drawn. And that — that is what makes the heterogeneity
in studies your friend, and the friends of the policy makers, and the friends of consumers
of the studies. And I think, I mean, I was really impressed
with the degree of care, attention paid to that question in the materials that I read. And I think it, you know, should be highlighted
that the heterogeneity in approaches. Whether they’re descriptive. I mean, descriptive data on some of these
questions that really, where there are none, right. I mean, the marginal benefit of descriptive
data in those areas is really, really high. We may not know the causal question to ask
yet in some of these areas. And so I just wanted to add another voice
to the idea that those things are really valuable. And especially when coupled with sort of being
careful about what inferences can be drawn from RCTs versus not. And that should never, I think, be taken as
discouragement from doing non-RCT based work. One separate thought. I will make it quickly. Is in terms of partners and partnerships. It may be one that’s less likely to — it’s
not on the slide in any event. But, you know, there are other agencies around
that have a bunch of economists who, I used to work at one. Mike used to work at one. And at least when I was there, I mean, we
talked about there were partnerships in enforcement, right. You had to share information and data on cases
on the enforcement side. But in terms of the research and reporting
function across agencies, you guys have cool data. They have different data. I don’t know if it’s cool. You guys might have asked. And it might not be cool. I don’t know. Maybe you have all the cool data, which is
great. But, you know, I think for other agencies
that are in complementary areas, and thinking about, you know, I got a bunch of economists
and social scientists over at other agencies too. And maybe there are part, you know, benefits
— gains from trade from those sorts of partnerships as well. Maybe not. But perhaps worth thinking about. I think, I mean, Josh’s point is great. I think is Jesse Leary still at — here? I mean, he just —
No. No. Well, he did some great work at the FTC with
relate — in relation to factive scoring study. Where he was, you know, obtaining data, from,
you know, other government agencies. So even on the dat — you know, not the researcher
side, but the data side, there are ways that, I get that it’s costly for people like you
to make these things happen. But, from the researcher’s standpoint, you’ve
got access to way more than the typical academic could possibly get in terms of matching up
characteristics of individuals and so forth. So, exploit those. I do want to — I mean, I kind of led with
field experiments. As you say, that is the gold standard. But, I don’t — I mean, don’t let the perfect
be the enemy of the good here. You can — you’ve got the ability to do significant
work, you know, before or after analysis to the extent that you know that a financial
institution or an employer changed a policy over time. You can — you can get at the causality questions
that Josh is alluding to with those approaches. You might be able to use difference in difference
analysis if you’ve got one state that — I mean, so don’t — I mean, I get that it’s
great to have — you know, anyway, you understand. Thank you for saying that. Because some of the things we were — we’ve
talked a little bit about staggered roll out. You know, if you have employers in different
places, is there a way you can put it out first in one place versus another, and look
at those beforehand. So, try to do some things where it’s not true
RCT. But it’s — we’re getting close. Or natural experiments, right? I mean, if you — with all the connections
that you have, you’ve got a lot of credit unions and banks and so forth on the other
advisory boards. There’s got to be ways that you can identify
events that might have naturally occurred. Yes. Um-hum. That you can use as a natural. Because you’ve got inside information on potential
natural experiments, which is a great tool. If I can just jump in. So, first of all, I totally concur about the
perfect being the enemy of the good. And I also, just to come back to David’s point,
I mean, in my experiences, there are just a lot of people who are in a position to do
an intervention. And they really just don’t want to exclude
some group because they’re worried about leaving that group behind. A couple of general thoughts on partners. And one is just, I think Horizon is important
in this literature. So, you know, finding out that an intervention
works six months later is great. And it advances our knowledge. But, really for this question, being able
to look again after a year, or after two years to see, you know, what happened to that. Or try to figure out what happened to the
money you motivated them to save, is important. So, I think that’s a — something to consider
as you’re thinking about who would make a good partner. It’s just their willingness to kind of be
in a relationship with you and how long. The other thing I wanted to say, and maybe
it’s more of a question. Which is, you have all these fantastic data
assets here. And I’m curious, if you partner with say a
FinTech firm, or an employer, is there scope from — for taking the data that you get from
that kind of intervention, and doing some sort of administrative merge with some of
your rich data sets that you have here, just because that could just kind of greatly expand
what you have to work with? So, actually Michael, I actually had a — was
thinking of people who don’t save money in employer matching plans. Okay, that’s — so you’re buying a dollar
for 50 cents. Why don’t you buy as many as you can? And that to me that’s an interesting area
to look at. And what is the participation rate? And you can then measure there if you have
some program. You can at least measure that participation
rate in employee matching programs. Among college professors, they don’t always
save with employees. Some private universities have matching programs. And so why don’t people max those out? What’s the impediment? What’s going on? And then if you do have some program, you
can then measure how people changed over — at least in that area, so. Karen, to your point, one observation or to
answer the question, just the Making Ends Meet survey that Caroline referenced, we are
planning a follow up to the same respondents. It’s — the research design is, we asked them
to anticipate what they would do if something happens. And then we go back six months later and ask
them, have these things happened? And what did you do? So we can do that. You know, I think the research design ends
at six months, the second wave. RIGHT. Six to eight months. But, then maybe we should think about extending
that. The other thing I’d note, and then ask if
people have thoughts on, one of the capabilities that we have that we have — so John mentioned
laboratory work as a preface to field work. We have — in addition, we have some traditional
labs. We’ve contracted with universities to work
in their labs to do work. We’ve also built a mobile lab. Essentially think about it as a series of
iPads. Where we can go out in the field and do anything
you could do in the lab, but without being restricted to the population of students or
employees of universities. So to the extent people have thoughts about
how that tool might be deployed in support of this research initiative, that would be
quite interesting. One quick thing I will add about the long
term look at some of these interventions is that we think a lot about how we’re uniquely
suited. And I think actually, I mean, this one is
a question, sorry. Uniquely suited in the Bureau compared to
let’s say an academic institution. And I think one of — we love publishing of
course. And we’re excited to publish our results. But I think that we are on a little bit of
a slower time table then academics. So we kind of have the ability to sit back
and wait for more data to come in before we publish that report or something of that nature. And obviously it’s a give and take with the
employer or institution as you mentioned. But I think that we kind of have that luxury
of not having to publish on a certain timeline. That gives us the ability to do exactly that
kind of thing. So, if I could chime in. The two things you can get out of that. You’re talking about having like a data truck
type, where you — or a roving lab. So, the two things that the lab piece can
do for you, is it can help you understand the mechanism by which something has an effect. So if you just did like a field experiment
that people get it, say it’s a disclosure or not. And it worked or it didn’t work to a certain
degree, you don’t know why. Whereas the kinds of stuff that these folks
do, help understand the why. You know, some of the stuff for example Dustin
has done and so on. So that’s a key piece of it. The other thing that you can get out of a
data truck is to try to exploit the heterogeneity you actually get in your sample. And then take a look at subgroup analysis
and try to understand the which people. And the which people have a certain response,
helps you understand the why. And that helps you scale up before you do
this large scale intervention. And test it scale has got some kind of a field
setting. So, it’s a really important capability for
what you’re doing. Great. Thank you. Maybe we can — one of the questions that
sort of — came up at the beginning, is measuring. And you talked about crowding out. So, this question here, how should we measure
suc — how should we measure success? And what types of measures should we prioritize? And of course it depends on the intervention. But, just broadly if we’re thinking, we’re
trying to increase savings. Obviously we want to improve people’s well
being. Love to get your thoughts. Yeah. I think that’s — I mean, that’s the ten thousand
dollar question in my mind. I think it’s very difficult, in my judgment,
to measure success in a way that — that doesn’t result in unintended consequences. So, if the goal is to induce people to save
more, we can just adopt laws that say, you know, 20 percent of your earnings have to
go into a short term savings account. Right? And then you went into that issue is okay,
if you then preclude people from taking that money out when they, you know, have a shock,
you know, then it would really increase liquidity. That’s why the crowding out question is important
in my mind. But, what I think is particularly important
for you folks, is the unintended consequences of poor savings. Does it lead to people to go to pawn shops
more often? Are they taking out more payday loans because
they’re saving more for a rainy — and then you have the behavioral issues. You know, you’ve got these mental accounting
systems potentially, where this is my savings account. I don’t want to touch it, for behavioral reasons. Does that then induce people to be more vulnerable
to having to rely on these, you know, secondary, third area markets that I think some people
have a distaste for. So, I think it’s an incredibly important question. All I would say is, make sure you don’t answer
those questions in a vacuum. If your goal is to induce more savings, make
sure there aren’t unintended consequences that are going against other issues, areas
that you might have concerns. Yeah. I think that’s a great point. And it underlies a lot of our interest in
looking at savings and debt. Simultaneously savings and — I mean, one
of those studies that we talk about is the credit card debt puzzle, things of that nature. But also, in some of our partnerships, we’re
trying to look at companies that show the flow of cash. So, simultaneously with the savings feature,
let’s say on an app, they — a lot of those companies also have the bank account information
and things of that nature. So we can see a little bit more of the bigger
picture. So, we’re trying to work on that type of partnership. To look at exactly that type of question. I think your point, your question, also is
making me think about surveys. And the extent to which we would want to do
follow up surveys. Because to get at have you used a payday loan? What’s happening beyond what we can see in
the cash flow and the accounts. So we could think about that potentially for
some of our work. Another issue that I’m thinking about as it
relates to measuring success, is if part of our goal is to help people to prepare for
emergencies or shocks, but then they spend down. You know, we can’t look at this issue as in
the way we look at retirement where someone starts at whatever number, and over time they
get to this number. And that’s success. Helping people to manage shocks is really
important to this. So, someone can start with five hundred dollars
in June. And in September there’s a shock, so they
spend down that five hundred dollars. If you look at them over six months, how do
you — how do you classify that as a success? Where for me, I think that is a very successful
thing. They might now have two hundred dollars in
October, or whatever. So, that’s part of the conversation that I
hope that we can talk a little bit about. And also, the other thing is, motivations. How do you motivate a person to take action
to have that five hundred dollars for that emergency that they might not be thinking
about? Similarly on the motivation side, how do you
motivate employers or other partners to get involved in something like this? Because for a small or a large company getting
involved in a research project like this might be, you know, it’s an opportunity cost. So, if we can help them figure out the impact
on their employees, or the impact on the bottom line, or whatever that positive impact might
be that might motivate them, I think that might also help. So, don’t want to take us off course. But those are just a couple of things that
I commented to on mine. Can I jump in on that one? So, totally agree. Actually, if anything, I think you guys are
overselling how easy the retirement literature is. Because actually having worked on the retirement
literature, actually is frustrating. Like that number that you refer to, like we
don’t know what that is. It depends on the person. It depends on what kind of retirement benefits
they’re going to have. So, an alternative approach is just, and this
is something that Jason and I used in a paper, is look directly at hardship. And whether people are experiencing hardship. So, we did this for older households. But, you know, you can imagine this in your
studies. Which is, at the end of the day what you care
about is, you know, whether people are experiencing less hardship because of — because of the
saving you have encouraged them to do. So, I think, you know, if you can directly,
you know, measure that by looking at measures. So, for example, one of the measures I’ve
used in research is, you know, did people have to cut back on their food expenditures? Did they fall into poverty? Or did some kind of broader measure of their
kind of financial assets and income put them in some danger zone? So, then kind of directly looking, if you
can measure it, directly looking at the hardship outcomes could be constructive as part of
the agenda. I think — oh. I think yes, I agree. That’s a very important point. And how we measure success also strikes me
as somewhat of a qualitative question. The extent to which a person who is saving
for emergencies, particularly someone who is from a lower income group and financially
vulnerable, what would they define that success to look like? Because at the four hundred dollar mark, I
could imagine that there are a number of emergencies. Your bus runs late for work, or your toaster
breaks for breakfast that are at price points that would vary by the context of a person’s
situation. But success in those contexts might be able
to have the financial resources at the ready to repair them. And were they able, when that emergency arose,
to afford what they had been trying to save for? Which could be captured in other ways like
hardship. So in Mike’s comment about why if you think
a certain behavior is desirable, just legislate it. The reason why people — we don’t do that
is because there’s heterogeneity. And so the same behavior that’s desirable
for one person may be undesirable for another. So my view, one important success metric for
the Bureau, if you have some particular actions intended to help consumers in a certain way,
is to see whether it improves selectivity. If it’s possible for you to say, this subgroup
probabilistically it’s more likely that it’s a good thing for them to behave in this way,
and this group it’s different. If you see that your interventions are improving
sorting in a way that you think is appropriate, that would be something I think would be desirable. So, are
there any thoughts that folks have on types of interventions that we might want to test? I mean, we were thinking about different types
of financial incentives. There’s messaging. There’s different ways we could think about
automation. Do folks have any thoughts on what might be
priorities for us? We have lots of ideas ourselves, so. Can you test Mike’s law? Mike’s suggestion we just mandate it and test
that? Yeah. That hasn’t been in our working group conversation. I would be interested, I mean, in speaking
of things that have been mandated recently, and the opportunity for difference and differences
analyses. You know, the state of Michigan recently raised
the asset limits for SNAP benefits, right. And I would imagine if you are a person who
is receiving government benefits in some way, and all of a sudden there is a new threshold
at which you can save for emergencies, that that might be an opportunity to test for behavior
changes. And just one of the — just to add in a positive
point on the savings piece. When I had done the research at Irving looking,
doing a randomized control trial, what we found — so this was low income people, and
we — they had an incentive to save. But we found that they saved at the median
about 650 dollars. They had lower material hardship, reduced
use of check cashing. But what we actually — and they weren’t taking
it out for emergencies. But what we actually think was happening,
was that people, the program was helping people stay connected to benefits. So, that’s the way that they were able to
save. And not increase their material hardship. So, I think that your bringing in the asset
limit piece is a part of this. And the program that we evaluated, as their
savings went up, it was not counted in those asset limits. And we think that that was really important. We’ve just got just a few moments left. And I thought it would be fair to give the
Director a chance to talk. Be a fire hose to a bunch of academics. Thank you Mike for that. And thank you everyone for the discussion
around this. As you can see, we’ve certainly put a lot
of thought and energy into it. And at the same time we know that we’re not
the only font of information and thoughts about how to best go about doing this. Certainly it’s helpful, I think the partnership
with other agencies with respect to the data and thinking they have. And other sources of data is something that
we’ve been very interested in. Because as valuable as that gold standard
RCT is, also takes a really long time to set that up. We found that in our work, to set that up
and then to actually see the results of it and do the follow up of it. A lot of resources and energy around that. And I’ve been really pushing on what opportunity
is there to take some of these products and services that are midstream, where companies
have the information. Maybe they don’t have the ability to research
it, or research it in the same way we would. And the same with other agencies. So again, the opportunity to take that data
and maybe get some wider perspective on it. So, thoughts you have around that obviously
were useful and helpful to hear. And I do think there is a lot of opportunity,
probably with the Department of Labor, to talk more about the way that they see retirement
and that relationship here between, you know, the way we’re thinking about this, and they
are. And I certainly know, as I said, Caroline
and Melissa have spent a lot of time on this. The terminology that we use, and to Desmond’s
point, how we measure this, because we’re trying to encourage a habit. We’re not necessarily targeting on a number. Which is maybe a little bit different again,
of the way to think about this, and again, how to measure it. So, I really appreciate your insights into
it. And this is the beginning of the conversation
with all of you today, rather than the end. So, we’re going to certainly iterate on this. And as things come to you and as opportunity
to partner on even particular research efforts, the opportunity to do that with all of you
is — would be fantastic too. Thank you. Well, thank you. We could probably spend a week on each one
of these. But unfortunately we don’t have that kind
of time. Our next discussion is going to be on the
Bureau’s Research Agenda. As if we haven’t already been talking about
that. But, Jason Brown is going to stay to lead
this discussion, so. Okay. Thank you very much. And just I would like to thank the ARC and
the members very much for your insights and wisdom, and willingness to partner with us. So, it’s a — I’m new to the Agency. And it’s just a real benefit as a researcher
to have the dedication of excellent, outside researchers to work with us. The disclaimer that Caroline did not read,
also applies here in this session as well. So, the Dodd/Frank Act does require the Bureau
to conduct research. And I’ll just go over what the Dodd/Frank
Act mandates us to do. It’s to report on developments in the markets
for — consumer financial products or services. Access to fair and affordable credit for traditionally
under-served communities. Consumer awareness, understanding, and use
of disclosures and communications regarding consumer financial products or services. And likewise, consumer awareness in understanding
of cost risks and benefits of those products and services. Consumer behavior with respect to consumer
financial products and services, including performance on mortgage loans and experiences
of traditionally under-served consumers including the un-banked and under-banked. Likewise, we work with the Office of Financial
Education to conduct research related to consumer financial education and counseling. So, we do have a research agenda that governs
a good deal of the research that we do. It was established in 2014 for a five year
cycle, so we’re nearing the end of that, or we are at the end of that original research
agenda. And we have two focal areas. The dynamics of household balance sheets. Which is a fairly broad research area. And disclosure, which of course is of great
importance to the Bureau. We had a pretty thorough governance process
for deciding on projects under the research agenda. So we would generate ideas with the working
groups within the Bureau. And talking with others outside the Bureau. It can be — research projects can be commissioned
by the Office of Research Management, and the staff would often propose ideas through
a call for proposals. We collect the proposals through an annual
call for papers. And then we’d have a pretty thorough review
process. So, we’d have panels composed of members of
the art, visiting scholars, leadership in the Bureau. You know, these panelists were selected on
the basis of subject matter expertise. And then they would review the proposals. So finally we would — and then we would decide
on the final projects. So, you know, we talked a little bit about
in the last session some of the research inputs that we used on Start Small, Save Up. But what we have generally available to us
are existing data assets which are, you know, extremely rich as we’ve talked about a consumer
credit panel being a good example of that. The Bureau developed surveys like the Making
Ends Meet survey that we pair with our existing data assets. External data, you know, we get proprietary
data, administrative data, survey data, and then data developed through field trials and
laboratory experiments. And so our output, and we have a range of
outputs. So we’ll do reports on policy relevant questions. We’ll do research reports on developments
in credit markets. Studies with outside partners, nontechnical
issue briefs, and independent research. And of course, that’s in addition to the rule
making and statutory required work that we do. So, you know, we’ve been trying to drum up
some new research agenda topics. And we’re really interested in your thoughts
on these and others. But I’ll just go through them briefly, just
to set up the discussion. We just talked about emergency savings and
Start Small, Save Up. We have also been thinking about consumer
resilience in the face of financial challenges. Empowering consumers, and disclosures. So we already have the disclosure, but we
can think more broadly about how consumers use information. How do they access information to make decisions? Consumer injury and consumer harm. Which I know we’re going to talk about in
a little bit. Access to credit. And then discrimination in credit markets. So, we’ve got some discussion questions here
to kind of guide us through the conversation. But basically we’re interested in your thoughts
on where do you think are some, you know, the gaps in knowledge that we should be focused
on? Do you have some ideas on our sort of research
project selection process? How should we be using sort of the rich assets
we have? Or our lab capabilities? Should we be collecting other data? And then finally, what role can all of you
play in this selection and development of our research projects? So, with that I’ll sort of open it up. Well, Jason, you know the coin of the realm
in the academic world is peer-reviewed academic papers, and they’re very narrow focus by design
because we can’t be, answer all questions in one single research paper. And I don’t know to the extent that you can
follow or emulate that peer review process internally, but you can knit together a series
of narrowly focused research topics to build a mosaic of results. And the other thing I just — you know, you’ll
be tired of hearing me say this probably by the end of the day, but I do think that we
can, academics can partner with the bureau if we have data. Data is the driver for the research studies
that academics do, and so you can outsource some of this research by just providing data
around and let the worldwide network of scholars run with the data. Thank you, and I should have mentioned we
actually do have ongoing partnerships with academics through our IPA program. I think we have about eight or nine academics
who are part time basically who are attracted by our data and that we do get to work with. What’s IPA besides the —
I was just going to say that. Yeah. Intergovernmental Personnel — I’m sorry? Oh, okay. Interpersonal, yeah, per agreement. I would piggyback on what you’re saying, Tom,
too, because it is something that, you know, as we formulate this research agenda, as we
make the research agenda publicly known too, I’m looking at our ability to have a much
more robust, you know, academic relationship, network if you will, to the point, and Jason,
you know, definitely supports this obviously in terms of how we go about this, and data
being a key part of it. I do get that. That’s where those IPAs come into play too
because we have to make sure some of this data is proprietary and needs to be protected,
how you protect it, and it’s not quite the same ability to just, you know, put the data
out there widely, but certainly where we can, we want to, and where we have access to data
that others don’t, figuring out how we can more easily have these relationships, because
the IPA is similar to even the setting up of an RCT and having an agreement with a particular
employer about their data. That, each time you try to start that new
relationship, there’s a lot of effort and back and forth that goes into it to try to
figure out how we replicate that, how we can speed that up, and how we can network that
so that a lot of that up front, and important, but bureaucratic steps can be sped up a little
bit to get the work done. I have a couple of suggestions on important
research gaps kind of inspired by your list, but on the question of consumer resilience
in the face of financial challenges, I have seen a couple of one-off papers that have
tried to do kind of stress testing of households like you see stress testing of financial institutions. But this is just getting at the question of
kind of are they in a sustainable position and kind of the idea of developing a standard
framework for thinking about could they withstand a shock to, you know, their employment, or
a healthcare shock, or something, you know, a long-term care shock? I feel like that could just shed a lot of
light on whether people have enough saving or access to credit, and I think you’re really
well positioned to do it just because you’re here in this community, this policy community
where people have done so much research about what a healthcare shock looks like or what
an employment shock, you know, what sorts of social insurance do they have to kind of
offset that shock, but what do they need in terms of their own resources? So, kind of some sort of stress testing could
be really interesting. The other thing I wanted to comment on was
access to credit, and access to credit and saving. We talked a lot in the last session about
saving to buffer emergencies, and we talked about saving for retirement, but no one — we’ve
only really just kind of touched on economic mobility, and I think that’s a question that’s
related to saving and also access to credit. So, I feel like a lot of the research that
goes on in the economic mobility sphere doesn’t look at financial access, and a lot of the
financial research that goes on doesn’t think about economic mobility in their scope to
bring those worlds together. But, you know, obviously, kind of having some
saving or access to credit can facilitate investing in education. Everyone understands that, but, you know,
other examples, you know, would be starting a business, being able to move to a place
where there are better jobs, being able to move to a place where there are better skills. There are all sorts of linkages there and
I just don’t feel like can we talk enough about that as kind of the benefits to saving
and benefits to access to credit? Jason, could I ask a really basic question
that maybe everyone in the room knows the answer to, but that’s never stopped me from
asking a question before? The existing — I’m sort of thinking about
new data areas, and there’s a lot of talk about and we’ve touched upon a couple of the
survey data assets that you have. But for sort of a newbie to what you have
and don’t have in terms of data assets, can you do a sort of, you know, two-minute “Here’s
what we got” in terms of existing data assets? Because I think it’s sort of useful to get
the map to start thinking about where gaps might be or where you think the gaps are. I might invite some of my colleagues who have
been working on the data more than I have, but just briefly, we have this consumer credit
panel, which is, I think, a one in 48 sample of consumer credit records. We’ve paired that with, like I said, so it’s
got credit records, but, you know, it doesn’t have things like income or, you know, a lot
of demographic information, so this is why we think about, you know, augmenting that
with survey data that can kind of fill in some of those gaps. We have the national mortgage database, which
is a partnership with FHFA. Again, it’s sort of a sample of mortgages,
and then, yeah, and then sort of the — I’m trying to think of other ones. Any of my colleagues? Credit, credit — so the third big database
we have is the credit card database. Well, actually, and also we have Y 14 data
that the Fed is sharing with us, both credit cards and mortgages, and that’s also —
So these are all microdata, low level, account level data on credit cards, in the case of
credit cards and mortgages in the case of the Y 14 data. So, this is, I think, piggybacking on Karen’s
point about resilience. One of the things I think you’re starting
to that I think is really exciting is the idea of looking at financial well-being beyond
the individual level and looking at the person in part of some social network. Because people will sometimes rely on family,
and friends, and so on as a source of resilience, and there are very interesting cross-cultural
differences in the degree to which people do that, et cetera. And so, actually, it was in some conversation
with some of your folks yesterday afternoon. I was very excited to hear the work that they’re
doing that’s taking this more sociological, looking at the entire, the person’s entire
social network as a source of resilience. And I would just, at a high level, just kind
of piggyback off of the Start Small, Save Up statement. I mean, you know, start small and build up
in terms of the research program. You’re a relatively young agency. In my experience, you’re going to receive
a lot of pressure, a lot of political pressure to do bold things before you may have the
answers necessary to boldly impact public policy in a meaningful way. And in my experience, a lot of what — a lot
of the challenges that we faced at the FTC stemmed from years and years of policies that
have been immortalized in legal proceedings and rulemaking that are difficult to change
once new information comes into play. And I think that the best example I can think
of here is the last session we’re going to have today is how do you measure consumer
injury or consumer harm? How do you measure, as Karen was talking about
earlier, how do you measure consumer well-being? I mean, those are fundamental questions, and
it’s just, it’s mind-boggling to me that as a, you know, as a consumer protection agency,
that we would contemplate serious policies that are going to impact consumers for generations
without knowing the answers to those seemingly small questions, right? So I think you’re asking exactly the right
questions. And I would also just add that, you know,
it’s understandable that a young agency is going to want to start having an impact on
public policy to protect consumers, but it’s like the PhD student that comes into my office
with the idea of a dissertation topic that is just so broad, like, “How do we save the
world economically?” I mean, you know, yeah, that’s a great question,
but let’s tackle something small that we can move the needle in a positive direction, and
I just would — I know you face a lot of pressure. I know you have laudable goals, but I just
would encourage you to not be afraid to fund or devote resources to those seemingly small
questions like what is consumer injury? How do we measure financial well-being? That’s not going to lead to an immediate payoff
in terms of some regulation, or some policy, or some enforcement decision, but it’s going
to ensure that future generations of consumer protectors base their policies on scientifically
sound answers. Just to step in to maybe give a little more
context on your question, which is, “What data do we have and then how do we use it?”
a little over a year ago, we did a comprehensive internal review at the bureau to categorize
every type of data that we collect, and then also explain our data governance policies
which include conditions on reuse within the agency, and also look at every MOU we have
in place for the sharing of data external to the bureau with our other state and federal
partners, et cetera. And we released a public report which we’re
happy to provide, you know, copies to you all and to the public as well, that was a
couple of hundred pages basically explaining every type of collection source that we have,
the type of information that we take in and retain, the conditions on its use, so privacy
restrictions or trade secrets restrictions that attach to it, how we use the data internally,
how we reuse the data internally, and then a comprehensive, a group of appendices that
was really a data inventory, so every source of data that we have warehoused in a description
of what it is and what you can get at, so I’m happy to get you all a copy of that, which
is really the comprehensive view on what we have and how we use it. I’ll add that while the agency or the bureau
may be relatively new, the work that you’re doing intersects with decades of research
often in many of these agendas. So while knowledge and information can be
refined over time, you know, the work is connected to kind of long and rich histories and evidence
bases. And I would just raise up the possible agendas
to include access to credit again, also to think of the ways that all different types
of data being used. Earlier, there was a mention of the use of
alternative data in underwriting, and I think there are possibilities for that as well as
potential harms for that, and given what sounds like partnerships with FinTech and business,
and financial institutions that use that alternative data or have the ability to scrape that alternative
data from a variety of sources. I would be very curious to know kind of the
effects of those technologies and those processes on consumers. And when I think of consumers, consumers that
are in need of protection from harm, I think of those that are kind of like at the forefront
of being kind of most detrimentally affected from those harms, so lower income groups,
groups in rural communities, black or brown communities, women. Women are increasingly heads of households
and managing finances, and taking care of kids, and also, you know, folks, fair folks
in retirement. So as you think about these agenda topics,
I would also encourage you to think about kind of the social identities of the populations
or the consumers and how that kind of intersects with the agenda, the access to credit or disclosures
that they may be confronted with. I have a suggestion or an idea that I think
comes out of some FTC experience on particular projects, and we talked a lot about RCTs. And I think Mike earlier said, you know, where
we can do those, great, but there are natural experiments out in the world if we sort of
look for them. I think Terri raised the Michigan, the state
law change as one of those. I think thinking ahead about — I mean, you’ve
got unique access to, through partnerships and through your own data assets, to all sorts
of data around consumer behavior. Oftentimes for the natural experiments, it’s
variation in state or local law that generates the identification for the study. And one would think that maybe the comparative
advantage for compiling panels of changes in state law over time would be with law professors. They don’t do it, you know, sort of hardly
ever in these areas, and so it’s one of the places where I think — I mean, I’m thinking about sort of competition
side state law variation studies the FTC has done on issues ranging from non-competes,
to state franchise laws, to there was a study they did about the effects of interstate wine
shipment before the Supreme Court had a case. Really influential stuff that got decided
in the Supreme Court impacted the change in the law. And I think an intuitive reaction would be
the investment into sort of cataloging state law, and tracking over time would be one that
maybe wouldn’t be your comparative advantage because someone else is doing it and you can
sort of pull that data off the rack when it happens or you just sort of, by looking out
your window, know when there’s a natural experiment in a state that sort of changes the law in
a way you can exploit to study something. I’m willing to bet, with some thought and
design, it may well be worth investing resources into tracking some particular dimensions of
state law that impact consumer behavior in ways that complement and touch upon the bureau’s
mission. It may well be worth investing in creating
those databases yourself in some of those areas. I don’t have specific ideas for which ones,
but, you know, that’s where the natural experiments tend to live on some of these issues is state
by state variation over time. And, you know, the RCTs are important because
they get us identification. This is another way to get identification
and, you know, I think is probably worth thinking about. One other suggestion, since we’ve been talking
a lot about interacting with the academic community, I think there would be value in
your Office of Research promulgating a list of research priorities. If we’re talking about interacting with the
academic community, right now, my impression is it happens on kind of a one off basis,
so you know somebody who knows you, and then therefore, it happens. But another model, if you’re willing, if it’s
possible to do the matchmaking process, is you promulgate a set of priorities. By the way, you can use your other advisory
groups to help you with what those priorities might be. There are things that — and then when you
promulgate those and distribute those to the academic world, then you try to do a matchmaking
thing, and we would come to you if we thought we had a differential advantage and you had
something that could help us. So, doing that, reaching out to the academic
community on a more public basis, would have value, I would think. So are you suggesting just putting, giving
out sort of almost a list of ideas or things that we’re interested in, or is that sort
of more formal like a grant process, or — So I’m a marketing professor. There’s this amazing organization in my field
called the Marketing Science Institute, which is this academic industry partnership, and
these very thoughtful people from industry come together and they say, “What are issues
that are happening in marketing that they need frameworks for?” and they’re not just
a one off thing that a lot of people have that, and then that leads to this priority
generation process, and they promulgate this in the academic community. And by the way, their thing is they give both
matchmaking to get data and small grants, but that idea, they have a much wider net
as a consequence of doing that, so it’s not just if you work at school X and you know
this person on CFPB that you’re in the set. I think that’s a brilliant idea, John, and
just to be clear, it’s not clear that this costs the bureau anything, I mean, to the
extent that the value you provide is data and individuals with institutional knowledge. And there are a lot of people, young researchers,
that have incredibly impressive skill sets. The scarce resource they don’t have is data
and institutional knowledge, right? So, it’s a win-win, I believe. It costs you nothing. We actually, we have two ways in which we
interact with academics, and I think you’re right, John. It’s somewhat ad hoc. One is, as we mentioned, the IPA is in there. We’re bringing people in as part-time employees
of the bureau, either for pay or not for pay, but in a number of instances, some of our
economists have partnered have outside academics to jointly do research papers, and that’s
another way in which we’re able to — I guess, I think probably we keep the data
here, but we’re still able to work together with somebody, and leverage their expertise,
and produce original research there that I think is quite useful. So, Josh, to your point, just, I mean, because
we think it’s a great point, one thing, one paper that two of our economists have recently
done was to exploit variations or changes in state debt collection laws, where what
we wanted to do was to understand — There, it was not looking at consumer behavior,
but actually firm behavior, to understand to what extent changes in debt collection
laws affected access and price of credit, so our credit card database has pricing data
in it on the price of credit cards, and by comparing states which had changes. And there, to a point Tom made at one point,
by relying on our markets folks, our markets folks were able — we didn’t have the database
you talked about, but our markets folks were able to say, “Look, these states made changes
in these laws that were material to the way the collection industry went about doing their
practices.” So that if you were going to find variations,
if you looked in those states and compared them to other states that are otherwise comparably
situated where there weren’t those changes — And two of our economists produced a paper
which we relied on in the debt collection rulemaking and which they presented at various
academic conferences, and I think it was submitted to some peer review journals as well. Yeah, I think pretty highly of the CFPB’s
research efforts and your abilities to partner with academics, and I appreciate John’s suggestion
about, like, systematically reaching out so that there are entry points for others maybe
without some of those connections, that they are able to join in and kind of participate
in some of these activities. But, I mean, it strikes me as kind of asking
the question back to you of how we can be helpful to you in advancing the work that
you’re already doing. I mean, I guess, part of that is, you know,
maybe this conversation, but, you know, maybe there are other specific ways. Yeah, I mean, thank you. We will take whatever you can give us in terms
of — No, I mean, we are mindful that you’re very
busy, so, but, you know, in the past, we have relied on the AARP to kind of help review
some research project. I mean, because obviously we’re here asking
you for thoughts on coming up with a research agenda. But, yes, so, but we’re very sort of interested
in any ideas you have on how you think you can be helpful to us, being mindful of your
competing responsibilities. Can I push you a little harder on the question
of increasing academic access to your data resources? I’ve already — we talked about this a little
bit at breakfast, but I understand entirely why you can’t open up access to kind of the
microdata that you have, I mean, the privacy issues, the legal issues, you know, just your
resource constraints, but, you know, as Josh was saying, a lot of the action is, in terms
of natural experiments, is kind of at the state level. And there is research that is being done in
the academic community that looks at either state level data or kind of county level data
where you have a change in state policy, but what you want to do is you want to look at
two counties that are kind of just across the line from each other so you can compare
those two because presumably they’re very similar in other ways, and I don’t know about
your ability to kind of release kind of roll-ups of the data. So for example, the New York Fed has, they
released a little bit of their consumer credit panel at the state level. It’s just kind of student loans. It’s a limited number of years. It’s only annual data. And I just don’t know —
I mean, I know you have limited resources and you’re going to have your priorities,
but I feel like the conversation could be greatly enriched if kind of someone were releasing
data along their lines because there’s just such little data to kind of research these
questions out there. And I think the better research will always
be done inside the agency with kind of, with the microdata, but kind of just being able
to kind of release it out there, and set some roll-ups out there, and set it free might
then just spark a lot of kind of creativity in the community, you know, people, you know,
thinking they’re finding patterns that then may generate kind of real research projects
for you down the road. The one data asset you might look at, Karen,
we’ve actually — the one time we have put out microdata, actually a subset of data from
the National Mortgage Database. The National Mortgage Database, which is,
it’s a nationally representative sample of all mortgages in existence. From that, we survey each quarter new borrowers,
people that have newly obtained a mortgage, and obtain data from that, and that’s a longitudinal
survey we’ve been doing for eight, 12 quarters, I’m not sure, with FHFA, and last year, we
put out a public use dataset which uses the NSMO, the National Survey Mortgage Origination
data, and includes some administrative data, some of the variables from the administrative
data along with that, and that’s available as a public use dataset for anybody, any researcher
to use. Yeah, yeah, and I should have acknowledged
that. I mean, it’s huge because just the data — these
are such important questions, but the data that’s available to just researchers who don’t
have some relationship with a financial services company or kind of are inside some agency
that has deep pockets and can buy these data, you know, it’s just, it’s huge when you do
something like that. It really makes a huge difference. The other thing you’re able to do, like the
issue of understanding financial behavior, like one sector, is always — the ability
to understand that is hindered by not knowing the entire balance sheet, and so anything
that you have that allows one to pair data about what’s going on, say, with mortgages,
with other aspects of the person’s financial life would be huge because there is this substitution
effect across areas. John, let’s put you on the spot. Suppose you had a bunch of post docs who came
to you and said, “We have money, don’t worry about the money, to do any kind of laboratory
work, field work that you think is important. “What would you like us to do that would be
most valuable to your research or to the field of understanding consumer decision making?” Do you have any sort of top candidates of
things you might, directions you might point them in? Well, one is, I think one is something that
has come up already. Again, it’s this relationship between these
different categories of savings. I think that’s a super important thing and
I think that deserves more research, so I’d let other people chime in theirs. I think, David, you described like maybe an
academic’s dream of, you know, a team of research post docs. And that’s why we have you here, your academic’s
dream for us. Dream for us. Tell us what you — we’ve got all of those
people out there who are at your service, and all of this data, and money is not an
object. What would you do with it? I think I would go back to the question about
kind of digital data, I suppose. I think there are things that are happening
with data that, given our technologically advancing society, I think, and the use of
artificial intelligence. And connecting that with finance, I think,
is maybe a decade or two beyond kind of the research agendas, at least that I am thinking
about or that many of us are, like, trained to address, and these are things that are
happening in real time and advancing in their applications pretty quickly in different markets. And I would wish for a team that would look
at some of those kind of rapidly advancing technological changes and their impacts on
consumers, but that’s more of the, maybe the kind of 30,000 foot question. I mean, there are a series of small incremental
steps that can be taken underneath that, but in terms of that trend, I think it’s important
to be at least considering, which I think we’re already kind of behind the forefront
of that, but definitely have it on our agendas. I think, you know, we’ve been, you know, talking
about what, you know, we, as outsiders, can bring to the table. You know, you’ve got a number of, you know,
economists and social scientists within the Bureau that are doing fantastic work. And I would advocate the Bureau ensuring that
you’re allowing those individuals to go to professional conferences in their fields,
to have conversations with academics, because a lot of these partnerships that we’re talking
about, they don’t need to be top down ways of happening. You can get those engagements by having your
staff have conversations. And once the hungry academic looking for tenure
at a university in a publish or perish world understands that you’ve got smart people with
access to fantastic data, you’re going to be doing more research than you can possibly
handle, so. I think this is definitely helpful. I couldn’t agree with you more in terms of
this being — you know, there is some top down and bottom up, and I think, to your point,
Terri, we have done some pretty fantastic research out of this organization. A lot of that really has been some bottom
up work. And I think to John’s point, something that
I am particularly interested in is understanding how we support both certainly. I think we have given and will continue to
give opportunities to our economists to get out there and to harness their ideas and their
passions around different topics because that’s what’s going to help, you know, get some of
these things going, but that more systematic way to do this as well so that there is some
top down, that we’re kind of putting out our research challenge to the world or the areas
where we see gaps. And I think something that I’ve heard here
today and something that I’ve heard in just — I’ve had a couple of other outreach sessions
with academics as I’ve been out and about. Jason and I did one in Chicago not too long
ago too, and I will say that the data is a huge part of this. So I also appreciate your point, Karen, about
how we can think about what we can make available. And from my involvement in general in government,
it’s what we can make available over time too, and make that commitment to continue
to make it available so that there is an opportunity for, you know, repeatable experiments or,
you know, things that look at things over the time horizon too, which I took from, you
know, the prior conversations. So, I certainly challenge the staff to be
thinking about that as you’re all sitting there listening to us, to be thinking about
what, perhaps ways that we can make some of these things available, what there is, so
that we can share that. We haven’t talked much, I mean, again, getting
back to some of the research topics that Jason started with, and Terri, you touched on it
a little bit with access, but around discrimination in the marketplace too and what kinds of things
maybe you all have explored, what approaches we could take to that, what there is or is
not. I think the other thing I would say is as
we’re looking at these topics, it’s what we think needs greater exploration or what perhaps
there is a decent body from your experience of, you know, research out there on some of
these various topics. That might be an interesting way to wrap us
up for a couple of minutes. You know, David, since you gave us a great
big credit card, I think — when I was a senior associate dean at St. Louis U, one of my jobs
was to facilitate research by the faculty and just eliminate any problems that they
would have, you know, except for, you know, they still have to teach classes and whatnot. But, you know, but I would say just organically,
these research topics will flow if people are given the opportunity for data, and for
grants, and actually, you know, a big idea would be why not start a new journal in this
field somehow? Since you’ve got, you know, we had this money
is not a problem, let’s have a new journal of current topics or something that the Bureau
can help steer, and you have the economists to serve on the Advisory Board, and you can
have other academics be, you know, an editorial board, et cetera. But I think that it’s hard to mandate top
down research. I think it just kind of swells up from — you
know, academics chase data and they have pinpoint questions, that they follow the scientific
method, but then you put all of that collection of knowledge together and then you can build
sound policy. Let me just jump in real quick to elaborate
a little more on the genesis of the research agenda and why is there a research agenda? I mean, I think there is this desire —
There was a desire to think sort of long term, like how do we prioritize the research we
do? And there were also a couple of sort of other
motivating factors with those specific research agendas, the dynamics of household balance
sheets. The Bureau was receiving this incredibly rich
data, and I think wanted to have some way of thinking about how to sort of make the
most of it and how to use it, and so there came that one agenda. And likewise, the Bureau was beginning to
think about, you know, how do we set up our lab and field work? And so the disclosure agenda was a way of
kind of harnessing that. But, you know, if I understand you correctly,
Tom, you’re sort of thinking that maybe there doesn’t need — I mean, we have an agenda
set by the Dodd-Frank statute and just sort of leave it at that. Let 1,000 flowers bloom. You know, that’s one approach. I guess the other though is that should the
Bureau have some priorities that it really wants, you know, the staff to focus on? And I should also sort of point out that,
you know, we do have independent research that takes place that is within this broader
sort of umbrella of consumer finance that is not sort of Bureau directed. I wonder if I could follow up with one of
the specific topics here, especially because it relates so closely to our regulatory agenda
where we often are faced with choices as to whether to prohibit behavior or to mandate
disclosures as an alternative. Are there cutting-edge issues related to disclosures
in consumer financial research that would be worth the Bureau spending time and effort
to study? You, please. No, please. So the issue with disclosures is people not
reading them is one piece, and then the issue of the limited effects that they have, and
so one of the, you know, one of the main behavioral areas of research related to all of this has
to do with whether people can be — can you get like information overload? And as to the case that people don’t use information
because it’s part of some larger set, and therefore — and the placement of where it
occurs is such that by the time — if it’s, you know, at the end of some long financial
perspectives, for example, have people worn out on that? And so you do have actually a pretty interesting
agenda in that arena, like one of the things you’re doing that I think is important is
to consider the effects of disclosures not just on the consumers themselves, but also
on the sellers, and how does requiring a certain kind of disclosure change the behavior of
sellers in ways that might be helpful? So I’m actually pretty impressed with the
agenda you have around that disclosure topic. I’m glad that I let you speak first because
that dovetails nicely. I was going to basically suggest what John
did, that, you know, disclosures in and of themselves can —
You know, information is not a bad thing. I’ve never heard an economist suggest, I mean,
other than asymmetric information problems, but generally, information is a good thing,
not a bad thing. The problem is that I think about the mortgage
disclosure study the FTC did and looked at the method in which those disclosures were
taking place, and the fact that government mandated disclosures weren’t communicating
information to consumers. I think it was like 80 percent of the consumers
didn’t even know what the annual percentage rate was on their mortgage, whether they were
going to have to pay a prepayment penalty if they — so, I mean, it’s not clear that
disclosures are necessarily a panacea, I think, and so you’ve got this tension as an economist. One the one hand, I’m an advocate of giving
consumers information. On the other hand, you know, like you guys,
every time I download an app, I click accept. You know, I don’t read those darn disclosures. And so what would be really interesting is
to do some type of retrospective study to try to understand, in the disclosure space,
have there been instances in which disclosures have materially impacted consumers in a positive
way, and, I mean, I know of no study that has done that, and then so that’s just — Retrospective studies generally might help
you understand whether some of these things that on the surface an economist is going
to say is a great thing to do are really generating any benefits. If I can jump in, I had one question for our
members here today which relates to the research agenda, but not directly, it doesn’t directly
get to the question of what should we study? It gets instead to how do we maximize the
utility of the research that we are doing? And what comes to mind is in April, the Office
of Information and Regulatory Affairs put out updated information quality guidance which
instructed departments and agencies to try and pay careful attention to the research
they’re putting out to improve its quality, but also looking to try and ensure that the
research that the government’s issuing is or can be made subject to peer review, which
implies an ability to share the data that underlies the analysis, and share, to the
maximum extent possible, the methodology used by a particular agency in doing so. And so one question that, you know, comes
to mind for us from a research perspective is to the extent we’re committing to conduct
research in a particular area, we ought to think about the type of information that we’re
using to conduct that research, because there may be restraints on our ability to disclose
that information later on which may constrain academics on the outside in their ability
to actually build on the research that we’re issuing to the extent that they can’t understand
the methodology or can’t obtain access to the information that we used in the first
instance to build on the research. So I guess my question is to what extent do
you all find that important for purposes of peer review? What are the considerations that, you know,
ex ante that we should have in terms of thinking about the type of information we’re using
to inform our research? And how we can maximize the utility for outside
researchers on ex post to build on the research that we’ve done so it’s not just simply, you
know, a single report at a point in time that, you know, is quickly forgotten, but that can
form a basis for continued research and activity on the research topic? Well, I think the peer review process, you
know, results in really good work because everybody who has published a peer reviewed
paper knows what you’ve published isn’t what you started out with. And the referees will be, you know, they’re
obviously just pernicious and, you know, mean-spirited and everything, but at the end of the day,
it does sharpen the research and it makes it replicable and reliable, so I encourage
peer review. I don’t know how to do that. I’m not sure. I mean, we know that when we submit to a journal,
the referees are chosen by the editor and it’s double blind. We don’t know who they are, although in the
days of the internet, if you post one of your papers on the internet, the referee says,
you know, “Oh, it’s this guy.” But still, I think the process is pretty reasonable
and it’s rough. I mean, those referees are rough on you, but
it sharpens the result at the end of the day. So I’m going to respond to Brian’s question,
but first turn back to what Director Kraninger mentioned with regard to discrimination in
credit markets and raise that as an important agenda topic, especially given rising income
and wealth inequality and what that means for limited economic mobility that Karen mentioned,
and the ways in which credit is changing, that that be an important topic. And can I clarify your question, Brian? What I hear you saying is sometimes you can’t
disclose the full methodology of the data that you have. Was that — and does that limit academic participation
in the peer review process? Is that —
Yes, so to the extent there are limits on the type of information we can disclose or
a methodology employed, does that limit in some way the utility of the research itself
or, you know, researchers looking at that particular study and trying to build on it? And to the extent that there is a limitation,
how much of that should we care about and should we build that into, you know, our research
agenda at the outset? So, is it relevant to our consideration of
which topics to study, and would particular limitations on the later publication of that
data, should that be relevant to the topics that we’re choosing from among a varied, you
know, kind of universe of potential topics for research? And Terri, I think the issue is, I mean, it’s
more about, well, we can’t put out the data itself, so we — you know, it’s rare that
we can’t describe what we did with the data, but if we describe what we did with the data,
but the data is not out for anybody else to be able to replicate it, knowing, you know,
they can say that you can find methodological flaws, but you can’t actually test to make
sure that what we did actually was done well without being able to see the data, and that’s
where we run into these issues, I think, more than the methodology itself. The actual replication of the exact data that
you used? I think that’s fair, right, Brian? I don’t want to —
I mean, I think I see increasingly in peer review journals the request to include your
data in the publication of that, but I also don’t think that that is, you know, 100 percent
widespread. There are specific places where it’s important
to do that and would limit kind of work in those areas where it is important, but not
full scale and not across all academic disciplines is my experience with that. Just building on what Terri said, I think
there’s peer review, which is really important, but that’s, the issue about being able to
replicate it is a kind of separate issue, and I just think in the economic space, we
are increasingly moving towards people doing work with proprietary data and getting really
important, really interesting results. And I think kind of researchers accept that
as a tradeoff, that in an ideal world, the data would be out there and everyone would
be able to kind of test and try to replicate your results. But I think it’s like perfectly within kind
of what’s normal in the profession right now to have people doing work on important questions
with really good data and, you know, this be recognized as good, high quality research
because it’s gone through a peer review process, but not necessarily sharing the data. Brian, I mean, it’s a fantastic question. I mean, I would just say in terms of the sort
of ex ante research agenda design, the thing to think about from my perspective and then
echoing some of what Karen was landing there, this is a portfolio of work, right? There’s going to be a paper where you can’t
do peer review because you can’t disclose the data, and there’s going to be eight others
that are okay to go through the peer review process, and the agency’s research agenda,
and team, and staff get a bunch of capital, right. The intellectual capital of the agency increases
because you’re doing peer reviewed work. You’re making academic partnerships with other
— you’re sort of part of the academic community. And I think there’s sort of positive externalities
that flow from the publication of these other things that go through peer review. So when you put out the thing that’s — you
know, you do the important work first, and if you can’t share it because you can’t share
it, you know, if it’s in a sort of fleet of work, you know, I think the most important
thing is to do the work and, you know, share it to the extent you can. And I think it’s perfectly fine that some
of that — especially it’s certainly true in the economic spaces I work in, that it’s
pretty common now for a lot of papers to depend on some form of proprietary data, but I do
think that’s right sort of in terms of the —
It is a thoughtful observation that you have about kind of thinking about what that agenda
looks like as a whole, because if it’s eight pieces where you can’t put anything into peer
review ever, right, and there’s three years between the next two CFPB peer review publications,
I think that the tradeoffs sort of probably cut the other way. The other thing is to link that discussion
with the earlier one about collaborating with academics. The academic’s coin of the realm is publishing
in top journals. Some journals have policies around data sharing
that will allow for certain exceptions, but increasingly, there is this so-called replication
crisis in the biomedical and the social sciences, and so people want to know that things are
replicable, and so that has led– In some cases, the lack of replicability is
due to outright fraud, and that’s led to part of why, like in my field, there’s a big movement
toward posting your data, and posting your instruments, and so on. One of the things though, if you have proprietary
data, it’s pretty sure that there’s not outright fraud in the data themselves, and so I would
imagine that journals would be more accommodating to you in believing that there’s not been
something improper done with the data themselves if the data can’t be shared in their entirety. I appreciate all of your responses. It’s been a fantastic session, and at this
point, we will adjourn modulo. My apology to Chris Mufarrige who, I noticed
as I was looking down at Karen, I realized I didn’t introduce you, and I apologize for
that, an oversight on my part. Chris is a senior advisor to the deputy director,
so great to have you participate. So with that, we will adjourn for lunch, and
I look forward to reconvening after lunch. Well, thank you all. Welcome back. Great to see you folks back, as well. We’re now going to start the last session
of the day, and I’m pleased to welcome a couple of folks in the Bureau that are going to talk
about defining consumer harm and consumer injury, and in my opinion, that’s an incredibly
important task in protecting consumers is to make sure we’re measuring harm, right? Because that’s going to reflect the potential
benefits of enforcement or policies that we put in place. I’d like to welcome back Jason Brown, who’s
the Assistant Director for the Office of Research, and I’d like to welcome Paul Rothstein, the
Section Chief for Financial Institutions and Regulatory Policy for the Office of Research. Appreciate you taking the time to enlighten
us, Paul, and I’ll turn it over to you folks. I’m color blind. Thank you very much. So, good, standard disclaimer. All right. So thank you for proposing this session. The Dodd-Frank Act, what is states and one
of the primary functions of the Bureau is to identify risks to consumers in markets
for consumer financial products and services. We’ve discussed this a few times today, but
just to note, these financial products and services include most consumer credit products,
certain savings products, and certain services related to consumer credit, like debt collection
and loan servicing, and the payment system. Dodd-Frank cites risks to consumers as a reason
or factor to be considered in many specific Bureau activities, including market monitoring,
supervision, research, and the size of civil penalties, and Dodd-Frank also cites benefits
to competition, along with a substantial consumer harm as a factor to be considered when evaluating
whether an act or practice is unfair. So the Bureau routinely enforces consumer
law and further seeks and obtains a variety of monetary remedies for violations of these
laws. Dodd-Frank raises questions about risks and
consumer harms, and the Bureau considers — continues to consider these issues in our day-to-day
work. The Bureau has never articulated a fully general
definition of what consumer harm is. The FTC and state AGs have considered consumer
harm in enforcement and rule-making for decades, in both defining and remedying unfair and
deceptive acts and practices. So, that’s the initial setup through Dodd-Frank. In thinking about trying to ground this, I
then thought, well, how about getting real specific in our space, a little bit of a contrast
from the Federal Trade Commission’s space, and just really think about consumer credit
here for a moment, because at the end of the day, it’s not typically a one-and-done trade. It’s a contract, and a lot goes on. So credit allows individuals to consume more
than their income at certain times, and then less than their income at others. The main purpose, as economists describe it,
is to smoothen consumption, but credit cannot even be discussed without mentioning time,
risk, or uncertainty. As with any transaction, at the moment of
transaction, an individual may not obtain the best price available, or the right amount
for their needs. They may get too much, they may get too little,
or to fully understand the good or service, so relevant there is the information available
at origination, search behavior, disclosure, of course, is very relevant to all of that. But then, the story continues. So unlike many transactions, initial exchange
is followed by subsequent exchange or exchanges, repayment, and possibly refinancing, with
their own risks and opportunities. So what’s going on? Future income and individuals’ circumstances
are not fully known at origination. Certain features of credit products, such
as adjustable rates, balloons, repricing mechanisms, back-end fees triggered by usage patterns
may make it difficult to predict how much is owed or how a consumer will manage what
is owed. And then, additional actors, such as loan
servicers, debt collectors, may be involved, with a mix of contractual and private incentives,
which also raises sort of principle and aging considerations. What is the relationship of the creditor to
the debt collector or the servicer, principle agents? It’s a whole other set of issues when you
start to think about credit. The result can be a range of negative financial
and non-financial outcomes. So in specific terms, we think about delinquency,
default, foreclosure, loss of access to the credit system. These are the nouns that we use to describe
concrete kinds and countable kinds of harms, and in some of the — for example, the health
literature, you’ve got a computation of things like the value of a statistical life as how
to think about the risks of these negative outcomes, and one could think about computing
measures like the value of a statistical foreclosure avoided, or delinquency avoided in a similar
kind of framework, but these are the tangible, countable things. There’s also more general kinds of harms that
we often talk about in this work. Financial distress, failure to smooth consumption,
misallocation of household resources, these are a bit less tangible, but again, a part
of our vocabulary. And then, there are the downstream consequences
for health and family that can occur with particularly negative outcomes with credit. So, not to debate whether these are negative,
it’s more about are all of these negative outcomes necessarily consumer harms within
one of the disciplines that all of you bring to the table here? One way to address that would be to focus
on that discipline and ask about that, so think about economic harm, legally recognized
harm, policy-relevant harm. And ultimately, all of that is relevant, but
in setting up this session, we thought that it would be useful to solicit from you just
your view of these from the disciplines that you come from, and take what we characterize
here as an open and inductive approach to defining consumer harm. And so then, the questions that we get to
are designed to elicit your thinking about consumer harms. We really want to hear from you, and really
hope that you can bring just your perspective without worrying about what box it may or
may not fall into at this point, in thinking about how you define these harms and how you
go about measuring them, and so beginning with the personal in your own work. So we have four questions, and I do want to
get to all four, because I think all four touch on a different part of the question. So I’m going to watch the time, but I hope
you will all jump in on every one. So first, have you defined or measured consumer
harms associated with consumer financial products or services in your own research or other
professional work? And if so, how have you done this and what
data have you relied on? So think about your own research, and the
outcomes that you’ve measured, and some might be benefits, which have allayed a harm or
mitigated a harm, but you still have, again, harm as the central concept. There’s sort of a mirror image there, and
so a benefit, harm avoided, however you framed it. And then we’d like to hear from each of you
on this point. So, with that, I will open the floor. Okay, great, Paul. Thanks so much. I think this is a great opportunity for us
to kind of just maybe just have an open conversation about the various things we do, because I
know we all come from different areas within social science. So let me just begin with a couple of papers
that I’ve written that I believe are relevant for this. And let me just initially say that the specific
questions that I and my co-authors have attempted to address are very narrow questions, and
I think one has to first of all recognize that, you know, how you’re trying to measure
the benefits or costs of a particular policy or the, you know, more generally considered
utility is going to depend upon the underlying environment in which you’re doing it. So the first paper I’d like to talk about
is a paper that John Morgan and I published in the American Economic Review. It was in the early 2000s at the dawn of the
internet, and the question we were trying to address is, what’s the value of information
to online buyers of mortgages and other products as a result of information that’s available
about interest rates, prices, and so on, on the internet right? And so in our simple framework, and I just
want to stress that you’ve got to be very careful when you look in the academic literature
and try to apply a particular methodology to a real-world problem, they’re typically
going to be very narrowly focused in the academic literature. The specific question we were examining was
purely the impact of price information. No issue of fraud, no issue of deception,
no issues like that. Purely, what’s the value of price information
that arises in a market? In our model, consumers had rectangular demand,
and therefore, the measure of consumer welfare was the difference between the price they
would pay in a world of information versus the world without information. Demand for what? It could be a mortgage, the —
Oh, specifically mortgages, okay. — it could be you’re buying a consumer electronics
product on the internet. Right. The leading example we used was mortgages. The data we had at that time was just casual
empirical data, it was mortgage data. Okay. Subsequently, we published other papers that
looked at consumer electronics products and so forth. So that measure is just looking at, okay,
what prices would consumers pay in a world with information, what prices would they pay
in a world without information, and that difference, that differential, is a measure of the welfare
associated with the movement of the information regime, or alternatively, the harm the consumers
would suffer if you moved away from the information regime. Right. So that’s one example. I’ve got another paper with Maria Arbatskaya
that looks in more detail at online mortgage rate data from a company called mortgagequotes.com,
which no longer exists. And we did a similar type of methodology there
to look at the pure price issues. And were you thinking in terms of — this
was just relying on that data? So it wasn’t, like, a formal search model
in that there was — It’s a formal equilibrium model —
Oh, it is, okay. — of a platform that’s in the business of
offering price information to consumers. And so, you know, a couple of things that
come out of that, and I think this is an important thing to keep in mind as one is contemplating
measuring harm, our model explicitly recognized that there were multiple types of consumers. Consumers who were able to access the price
information, and then consumers who were on the wrong side of the digital divide who may
not have access to that information, similar to the environment Terri is talking about. And they’re differential welfare effects associated
to the presence of information. It could benefit people on one side of the
digital divide, and harm people on the other side of the digital divide, and I think oftentimes
in public policy, particularly when the CFPB or other consumer protection agencies put
forth press releases talking about results. It’s always kind of consumers benefitted. My experience, some consumers may benefit
from a policy, some consumers may be harmed. Like Terri’s discussion at lunch, when we
look at the impact of technological changes in online banking, people that are connected
to the internet may get different advantages than people who are disadvantaged, who live
in rural areas, and so forth. So I would encourage you to be thinking not
only about how do you measure consumer harm, but whose consumer harm are you measuring
and are the effects identical for all consumers in the market? If not, how do you weigh the benefits to one
group of consumers off against the potential harm to another group? One clarifying question. In your data, were the folk who were on the
wrong side of the digital divide actually harmed, or they just didn’t get as good a
deal as the group on the right side? Well, it depends on which question you’re
asking about the value of information, but prices went up for people who did less research. Who did less research, yes. Okay. Thank you. Others? I didn’t want to interrupt anyone talking
about their specific papers, because I have something to say that’s sort of a more general
point that I think at least is a little bit related. I know that at work, sort of both in the FTC
capacity, at least, I think it’s important just as a sort of framing mechanism — and
this is more comment than anything else — to think about the idea — I mean, there are
different ways to think about consumer harm for the purposes of an agency. One might be, I need to know if the — I need
to know about the magnitude of the harm for sort of damages, or fines, or sort of deterrence
calculation purposes, right? So the function might be there, and I might
be really interested in measurement questions if that’s the function of the harm inquiry. It might be a legal requirement, and so the
thing I’m really interested in, is there harm or not, right? So I need a — I’ve got a more binary question,
and so conceptually, I’m interested in is there a difference in the consumer outcome
with and without the behavior? The price difference, in Mike’s example, but,
you know, either it’s, you know, some sort of a unfair act or practice allegation, or
a deception allegation. I think, you know, oftentimes, at least in
FTC work, this takes the form of a materiality analysis of some sort. I need a difference in outcomes in the sort
of — in the counterfactual world. So what I really need are methods to sort
of do counterfactual analysis. The magnitude is important, but less important
for answering the binary question. And all sorts of measurements are okay. It could be products or output, it could be
the types of direct harms that we’ve talked about in some other context to sort of demonstrate
that the consumer is worse off in the world with the conduct than in the counterfactual. And sort of the third that comes to mind is
just from the perspective of an agency economizing on scarce resources that depend where — to
decide where it’s going to spend its time, right? I want to know, you know, there’s behavior
out there, if it looks really bad, but I put my sort of hands on the analysis, what happens
with and without, right? I can get a sense of the rate of return of
spending Agency dollars on solving a problem. It is very difficult to do that without doing
some sort of counterfactual analysis. Again, the measurement is going to matter
more, but in a — I think in a more conceptual way than if I’m doing sort of damages. I mean, for a lot of these questions, I don’t
know, in my mind, at least, when I think about the relevance of consumer harm and injury
to what an agency like the CFPB does, it does all of those things, and all of those things
are going to be important in different contexts. But I think kind of focusing in on the importance
of counterfactual analysis, is the broad, overarching theme, but in what detail, or
with what measurement, or exactly how you’re going about doing that in a specific context
depends a lot on whether you’re talking about a specific enforcement action, or rule, or
a high level prioritizing of where you’re going to spend Agency dollars. And I don’t think that’s even close to an
answer to question one, but it’s the first thing I’m thinking about in answering all
the questions, so I thought I’d say it because I have the microphone. That’s good, but just to press on that just
a little, we are very much in a space where, you know, the facts and circumstances of a
matter kind of determine, as a practical sense, what our conception of harm is, and then how
well we can measure it. I’m just speaking within my office, RMR, I’m
not talking about enforcement here, but I did enforcement at the FTC, so I was the same
there, too. I don’t know that that’s the world we necessarily
have to be in. There can be a more worked out, overarching
methodology that we can dip into in practical ways, but to have that further developed would
be very useful. And in these sorts of matters that we deal
with here, where time and risk are so intrinsic all the time, it would be good to be able
to move beyond comparative demand analysis of — that is very useful for certain kinds
of harm analysis, but is not the complete story of the complicated products and world
that we work with, I don’t think. Next? Does anyone else want to speak to question
one? Sure. Thank you for posing such a, like, a broad
and challenging question to us. And then I have a question in response, which
is a little bit rhetorical, but I wonder to whom is the consumer that you are concerned
about being harmed? So I would think — I would weigh more heavily
the consumers that may be already in jeopardy of having detrimental consequences of experiencing
some sort of harm in my analysis, but I also — I appreciate the question about in your
own research or other professional work, are there other ways beyond economic measures
of harm, I think perhaps is some of the question that’s being raised here, and measuring that
in financial or economic terms is one way to approach that question. I think there are mental health effects of
some of — of harms that consumers experience. Mental health effects from, such as depression,
anxiety, and suicide, as a result of carrying, for example, high cost burdens, some debt,
which are pretty significant and severe harms that can be quantified. But in and of themselves, death seems like
a pretty poor outcome of a harm as a result of, you know, mental health considerations. But I appreciate the, you know, kind of the
Reserve’s regulatory framework for understanding or approving mergers, and they have, like,
a kind of four categories that they think about with regard to bank branch mergers,
and considering whether the consolidation of a bank could result — you know, what kinds
of outcomes could those be? And they consider different harms, things
like competition, financial stability — if I can remember them all — public interest,
and, like, managerial and maybe human resources effects? I think those are the four ways that they
try and interpret and understand the effects of a bank merger, and anticipate any sorts
of harms. And so it seems like applying , you know,
various categories to understand what types of harms could occur, economic, individual,
consumer, broader market, and perhaps mental health effects, you know, or some sort of
other category that you might consider weighing on those harms. I’ll just put it out as a general question. I would be very interested in knowing how
much quantification is done on each of those factors versus more qualitative or intuitive,
as Josh was describing in it. Yes, I think the public interest one, in particular,
is maybe where — and the human managerial kind of resources is where there’s maybe more
qualitative input, and the extent to which all of those categories are weighted equally. You know, they’re not all weighted equally
in the analysis, but for example, public comments are things that are weighted in public interest,
as are quantitative measures like CRA — potential CRA effects of mergers. So this is great, Paul. I mean, I think it’s terrific that you’re
pursing it. It is really challenging, which means you
get even more credit for doing it. I have not quantified harm in my own research. Just a couple of broad points, which is, to
the extent that you can capture — I mean, so having point estimates is important, but
also kind of knowing what the confidence intervals are around those point estimates can be valuable. I mean, at the end of the day, you have to
make a decision, which is presumably going to be factoring in some point estimate, but
knowing whether — how precisely estimated, you know, how confident you are in the harms
that you’re measuring is important. And then, just along those lines, that kind
of a more specific point, that statistical — recognizing that statistical insignificance
is very different from a precisely estimated zero corollary. For the less technical, would you like to
just explain — unpack that, just for a second? Sorry, that you may — so we have ways to
use statistics to determine, you know, if you find, you know, some estimate that harm
is kind of on average going to be whatever, but if there’s a lot of variability, you may
decide you can’t really precisely measure that. You can’t determine that — statistically,
that kind of your estimate, which, you know, suggests that there’s positive harm, actually
is a reliable estimate. So kind of understanding whether you’ve estimated
harm and, you know, it’s a very reliable estimate is very different from when there’s so much
noise in the data, your program is kind of popping out a positive number, but you really
don’t know. It could be that number, but the confidence
interval around that goes from some negative number to some number that’s twice as large. All right, we’ll move on to the next question,
unless somebody wants to speak to this one. Very good. All right. So here’s something very concrete, drawing
on some of the Bureau’s research done actually pursuant to statutory requirements. So under Dodd-Frank, we have an obligation
to conduct an assessment of our significant rules five years after they take effect, and
we’ve completed three such assessments. One of those assessments was for the mortgage
servicing rules that we had issued in 2013 and took effect in 2014, and these are large
documents. I gave you Chapter Four. These are intensely empirical documents in
which we used really all available data we had in-house and collected some critical additional
data. This particular couple of numbers actually
come from a couple of data sets that we bought. These aren’t, like, proprietary from mortgage
servicers. We did use that, as well, but this is McDash
data, and another source, which is to say that others could’ve done this analysis, but
nobody ever saw fit to do it. Maybe it’s expensive data, maybe not others
really did, and that’s another thing about being at the Bureau is that we — sometimes
we will use data that others in principle have available and could do these analyses,
but for whatever reason, just don’t. And what we found here is that at least 26,000
additional borrowers who became delinquent in 2014 would have experienced foreclosure
within three years of becoming delinquent, had the rule not gone into effect in 2014
to the best of our ability to identify that. There’s no natural experiment here, there’s
no RCT. What we had, though, was a lot of data from
which we could then — monthly data from which you could then sort to pick out before and
after effects of the rule, and control for other conflating factors. There were other things going on with mortgage
servicing, there was a national mortgage settlement, of course, there were trends in the macro-economy. All of that, we could control for, and then
still look to see if there was something happening around the break points that then created
a persistent effect and affected the three-year foreclosure rate, both before the rule took
effect and after the rule took effect. Technically, for those who became 90 days
delinquent, that was the start of this. And for those of you who don’t know much about
foreclosure and servicing data, it’s actually surprising that — you might think if somebody
ever became 90 days delinquent in their mortgage, they were for sure going to get foreclosed
or have to sell the house, or they would not be in the home three years later. In fact, in our data, that’s a terrible outcome
and it happened to about half the people, but somehow, the other half, that was not
the outcome. Now, in our data, it wasn’t — it didn’t necessarily
mean they cured. They might’ve prepaid, or they might’ve dropped
out of the data, but they’re — it’s not as if it was a done deal and there wasn’t anything
one could do for such people. So in fact, we found 26,000 additional borrowers,
to the best that we could identify, did not become — did not experience foreclosure,
because of the rule, and we also found that at least 127,000 fewer borrowers, because
of the rule, who became delinquent — in this case, only 30 days delinquent, not 90 days
delinquent, would’ve recovered from delinquency within three years of becoming delinquent,
if not for the rule. So these are sizeable effects, and — but
I put it out there, both to show off some of the Bureau’s work, but also to ask the
question, do you view these effects as mitigation of a consumer harm, within whatever lens you
want to apply to thinking about that problem. Obviously, it’s a nice outcome, but thinking
more broadly about consumer harm, is this the mitigation of consumer harm, as you want
to think more broadly about the mortgage process. Why or why not? And if so, can you suggest ways that the Bureau
might monetize the value of this mitigation? Because we only took it up to counts. We didn’t put dollar values for the consumers
who were not foreclosed upon. These are the kinds of questions we have to
wrestle with, so you can all go back to your universities, but we’re still going to be
here thinking about it. So, let’s hear. Of course, living in an ivory tower, I’m not
bashful. So, I mean, I guess what I would say is that
other things equal, this suggests that the rule improved consumer welfare, and the question
then would become how much, if it’s an other things equal question, right? So I’d want to know something about, you know,
I’m assuming that this 26,000 is the incremental number, and, you know, what’s delinquency
cost an individual versus a financial institution? I mean, there’s two sides to a delinquency. It seems like, to me, just, you know, thinking
out of the blue here that, you know, the welfare effects — there can be welfare effects on
both sides of the transaction. It might benefit some lenders, it might benefit
some borrowers, and so that’s — I’d want to quantify that. I’d want to make sure that there weren’t any
unintended consequences of this rule, for example. I mean, you could create a rule as such that
no one is delinquent, but it might impose such onerous costs on banks and/or individuals
that, you know, the number of people that actually get loans shrinks as a result of
that. And so then, I’d want to know, okay, well
what’s the disutility a borrower gets that is deprived of a loan because of these more
onerous rules or because of the higher mortgage rates that a lender might charge to service
a loan or whatever, and, you know? So there’s just all those types of issues
that you’d want to factor in, and I think — but, you know, the other things equal,
comparison is nice here, but it doesn’t really shed light on whether, as a matter of public
policy, you know, the costs of that rule generated benefits sufficient to cover the cost, right? From a public policy standpoint. So I think that’s — I mean, we’re dismal
scientists, right? I mean, we’re always pointing out these, you
know, unintended consequences, but the reason for that is, you know, presumably scarce taxpayer
resources were put in place to make this happen. And you guys are competing with many other
things that could be done to improve consumer welfare, right? And so the economic analysis is designed to
kind of quantify that in dollars so that we can compare the value of a dollar you spend
on an action like this, versus a dollar that might be spent hypothetically by the EPA to
make the environment cleaner, right? And so it’s hard to quantify the benefits
of a clean environment or the benefits to a borrower from being less likely to be delinquent,
but effective public policy, you know, requires — and as I recall, you were in the FDA, weren’t
you, before you were — before you were at the FTC, you weren’t at the FDA? Okay, but they’ve got difficult problems there,
as well, I’ll point out, so — Just to clarify one point. The tradeoff you mentioned, that’s great,
that’s exactly the kind of open-ended point we want, but Dodd-Frank did provide for amendments
to the mortgage servicing requirements in RESPA, and we wrote rules pursuant to that. So it wasn’t like we could then say, we’re
not doing it because we think the EPA would be better off writing rules on something about
the environment. That’s not a margin we could operate on —
Right. — that, but it’s a good point, yes. I would say if the question is to extend understanding
of whether there are, you know, harms or unintended consequences kind of on both sides of that
equation, so consumers and financial institutions, then, you know, this is an opportunity where
measuring harm in a number of ways — hardship, you know, the cost of foreclosure, of a family
moving, the mental health effects of that, dislocation from schools and from communities,
and the societal effects of families being uprooted from foreclosure, and then, you know,
prevented access from entry again into credit markets seems to me — and this question would
also raise additional significant potential costs to borrowers who would have otherwise
experienced foreclosure. On the face of it, this seems to me what I
would have in mind in thinking about mitigating consumer harm, thinking of consumer as the
borrower and the person in that interaction, and monetizing value is one way to compare
whether that happened, but for borrowers, right, the monetary effects can be extensive
if they’re thinking of uprooting a household to adjust for that experience. And there is research in your space that monetizes
this, to some extent? No, I think that’s my suggestion, although
I can’t say that there’s not research that would monetize that. But my suggestion is, if it is being monetized,
then for borrowers, what is the cost of being foreclosed upon and needing to relocate, place
kids in school, you know, what effects does that have kind of long-term educationally,
and then for economic mobility? I think there are a number of ways of extending
what’s — you know, how value is being monetized there. So we did actually look at some of that literature
when we were writing the thing about the impact analyses of the rule. We did find some counts and information about
foreclosure and relocation, and there are even some research on children’s grades and
performance, that kind of thing. But it is also the case that for many of those
— foreclosure is a technical process. These folks were on track to have to move,
maybe do it more voluntary, more timely, you know, sell the house, and then move to another
location. So there was arguably a move in the offing
there, no matter — for most of these — not for the ones that we mitigated, but for a
lot of these folks. But there can be an incremental harm when
it’s forced like that versus just a bit more planned. We were thinking along those lines, as well,
but, yes, we — thank you. Someone else? So Paul, I thought you said at the very outset
that you really didn’t have a definition of harm, or the profession doesn’t have a definition
of harm, but then we’re talking about, are these harms or not? It’s like, I just don’t know. I mean, if we don’t have a definition for
what a harm is, how can we talk about harm? Well, and we’ll get to the question three
in a moment, but — I promise I didn’t look at that. — but it’s still open terrain, and so I agree. We want to have this precise as we can, but
it does seem to be difficult to come up with a good one size fits all. That’s why, in many practical matters, we
deal with factors, just as we do for deception and unfairness, factors you look at, and in
evaluating a merger, there will be factors. So, yes. So I’m assuming, you asked the question, you
have your own answer already and you’re trying to see that we say something —
Not at all. — no, not at all? Okay, then along the lines of what Mike was
suggesting, I wonder if we just take a look at litigations that are, like, class action
lawsuits, et cetera, and see what the experts offer on both sides, because you can imagine,
you know, a plaintiff offering some analysis that greatly inflates the nature of the harm,
and then you look at what the defendants say on the opposite side and just look at how
that works through. It’s probably likely to at least point to
some good factors to look at if you’re trying to quantify the harms, because that’s always
a part of the damages part. I mean, I think — I get that we’re going
to go to the really tough question next, and then even later, but I guess as I think about
this, just in the interest of full disclosure, I’m thinking of the utility or satisfaction
of a consumer, right? If that goes up, consumers are better off,
if that goes down, consumers are worse off. That’s why I live in an ivory tower. Now, if you want me to make that utility depend
upon the impact on household functionality, and where your kids go to school, boy, I’m
not smart enough to do that. That’s why I live in an ivory tower. But in principle, one can structurally estimate
a model, if you believe the model, and try to — you know as well as I do, Paul, you
can do that kind of thing. But my guess is, I’m looking at this question,
and maybe I’m not looking at the question in the same light everyone else is, but if
the reason we have fewer delinquencies is because 5,000,000 people that would’ve had
homes prior to 2014 no longer have homes, okay, because it’s tougher to get a mortgage,
because they’ve got to put more money in their escrow, or — I mean, I don’t know enough
details here to talk intelligently about it, but I can imagine a way where you can reduce
the number of delinquencies or foreclosures, right? In a way that, in my mind, you’ve got to ask
the question, well, okay, how much utility would the household have received from having
a house from 1999 to 2014, send the kids off to college, and now have their house foreclosed
on? That’s the but-for world without the policy,
and it’s a thorny — I mean, it’s easy, as an academic, to frame the question you have
to ask. It’s darn tough to quantify that, and so it’s
not clear the way this is framed that, you know, you’ve reduced delinquencies by just
flat telling 26,000 borrowers that they can’t — you can’t borrow money to buy a house. I just picked those people out of a hat and
say, you can’t buy a house, and I picked 26,000 of them, I’ve satisfied exactly this criteria,
and, you know, if I made those people better off — other things equal, I have, but other
things isn’t equal if by doing this, I’m taking them out of a house. So, but that’s why I live in an ivory tower. It’s really hard to do what you’re asking. So the data shows that delinquency servicing,
while always — maybe before 2007 was maybe double the cost of non-delinquency servicing,
delinquency servicing is not only more, like, four times the rate now of non-delinquency
servicing, but has also grown by a huge factor. It’s easily, I forget if it’s $1,000 or $2,000
a loan, but it’s very expensive, and those are the other kinds of effects that you’re
talking about. We don’t attribute them to the rule because
the timing doesn’t quite line up in the reports, but there’s no question that if you’re going
to go into the business of being a mortgage loan servicer, you are now looking at a much
more expensive investment, before the return you’re going to need is very different. But the business has changed, and we may have
changed it. And that translates into higher rates for
consumers that want to get loans, which by the law of demand reduces the number of consumers
that are going to get mortgages that are serviced, right? All right. Do the data show that? Do the data show what? Did the rates change with the regulation? Because it should follow, if Mike is correct. Well, you have to control for a lot, because
the economy was steadily improving, and rates, you know, interest rates have just cycled
down, down, down, so frankly, I don’t recall if one of the chapters we actually looked
at that. I think, Paul, in the ATRQM report, which
— and that rule to effect at the same time as the servicing report, we looked at mortgage
interest rates there, because we wanted to see whether there was any evidence that that
rule had affected interest rates. I think we couldn’t find any — couldn’t control
for all factors we wanted to control for, but no, no evidence of increase, but that
doesn’t say that it wouldn’t have been better had there not been these rules. That was the thing we couldn’t tease out,
as I recall, the ATRQM report. Yes. Thanks. Those are exactly the right things for you
guys to be looking at, and that’s great that you’re looking at them. So here’s an abstract one for you, Mike. In principle, it’s possible to measure the
amount of money a consumer is willing to pay to avoid a negative outcome or reduce the
change of a negative outcome. Measure of willingness to pay could perhaps
be used to measure consumer harm. Obviously, I’m not writing down demand curves
and indirect utility functions here, but Mike know — you know, everybody knows that. So there’s a formal way to approach this. What do you consider to be the pros and cons
of this approach? Do you consider such an approach practical,
say, through laboratory experiments? Should the measure of consumer harm be the
amount of money over and above the financial cost of the harm since the financial cost
was a transfer to other agents, or the total amount itself? And this gets to broader conversations about
how we think about transfers from consumers to providers, or even among consumers, but
just focus on — from consumers to the providers of good or service in this world. Paul, I’ll say that the amount of money I’m
willing to pay for car insurance is a lot more than I have to pay for car insurance. So I don’t know how you can measure — I would
think you would want to measure what people actually pay versus what they’re willing to
pay, this idea of consumer surplus, but — Well, as you know, if you can estimate a demand
curve, you can estimate consumer surplus. Yes, right, but my point is, why don’t we
estimate the actual price, not the consumer surplus? I’ll jump in because, again, this is my world. So, in general, these kinds of measures are
not very reliable. So if you ask people about their willingness
to pay, it’s not straight out, it’s not very reliable. People use methods to try to make the elicitation
more incentive compatible, but they don’t really match the real market. I was mentioning class action lawsuits, again,
in the world of class action lawsuits, it’s a super common methodology to use a technique
called conjoint analysis, I don’t know if you’re familiar with that idea. Conjoint analysis is a little bit — okay,
so the idea is that people are making tradeoffs in multi-attribute products and they’re able
to estimate how someone would tradeoff a feature of a product, say, something that’s got the
harmful aspect versus not, in a — say it’s got 10 attributes and price is one of them,
and you can back out from that analysis what the implied incremental price a given individual
is willing to pay to get that — to move from one level of feature, let’s say the harmful
level, to a different level of the feature. And there’s a really nice critique of that
approach in class action written by a guy named Olivier Toubia in the business school
at Columbia. Usually, when they do that — you know, often,
it’s the case that it’s a product that has, like — let’s say you’re doing litigation
about an automobile. It’s got thousands of features, and they pick
out, you know, 10 of them and they do this exercise. That typically greatly inflates the estimate
of what the economic value is of changing that one attribute if it’s supposedly the
harmful attribute. And so, like, the main problem with these
elicitation methods is that they — to the extent that the product is one with multiple
features, you call attention to a subset of features and put therefore in the background
the remaining features, and greatly overestimate the real-world sensitivity to the features
in question. When you say in the background, you mean in
the person’s mind? Yes, exactly, because the methodology doesn’t
bring those other features as part of the method, and so even though in the real world,
they might affect the person’s tradeoffs, in that method that’s often used in litigation
it’s — those things are suppressed. Okay, but if it was that or nothing, just
to press you a little — okay, so we don’t have perfect methodologies, what’s your take
on this, in general? Promising, worth pursuing, should I — should
we all be reading this paper, should we — related literature? Yeah, I mean, it sounds like you’re only — it
sounds like you nodded your head that you sort of were familiar with that methodology. It’s a good methodology to study, conjoint
analysis, I can give you some references. Okay. Others? Anyone want to just take a bite at this more
technical question? Of course, willingness to pay is the bread
and butter of welfare economics. It’s been that way for nearly a century, so
— and it is the go-to concept for — in all aspects of welfare economics, and I would
think that there would be a form of it relevant to these credit products, taking into account
the scope of what they are. To know that you won’t be — to know that
you have error resolution rights, is there a willingness to pay for that even if it is
a higher mortgage rate? To know that the agents who will be collecting
debt down the road are subject to certain restrictions that might also raise the costs
up front, is there a willingness to pay for that? I mean, this is kind of the thinking that
goes on — needs to go on. I think it’s an important question, that’s
why I’m going to — it’s not that I think I have an important answer, but I think it’s
an important question. I’d like you guys to participate here, because
I think, you know, Dodd-Frank talks about risk to consumers, and, you know, we — I
cross the street here in Washington, D.C. and risk my life every time I do so, right? And you laugh at that, but I work at the University,
and to walk from the parking garage to my office, you cross a street called Fee Lane. Openly walk there, now, you think about how
you could reduce risks of traffic accidents in Washington, D.C. You could literally put up barriers that prevented
anyone from crossing any street in Washington, D.C. at some cost, correct? We don’t do that, correct? Well, at Indiana University, we had a student
die, and guess what the university did? They erected barriers at that location to
prevent that. And again, love my university and my employer,
but I would argue that is just a reaction to an event that happened, right? Building a barrier in that one location. Maybe it’ll help, but I worry that those types
of public funds that are dispersed are emotional expenditures rather than rational expenditures,
and that’s why I encourage you to think about the question. It’s not that the University made a good or
a bad decision, or that, you know, consumers aren’t at risk if their loans are foreclosed
on. But if we don’t better understand what the
real costs are and what the real benefits are of the solutions that we propose, then
we’re not basing public policy on science and, you know, I’m a scientist. I like to think I am, anyway, so —
Are any transportation engineers watching or any from Department of Transportation? I think they would say, Mike, we knew that. That’s a tough call. They have to write regulations, too for Federal
projects. Okay. So —
I just wanted to jump in for one second. I just wanted to note how fun this is to turn
the tables on a group of professors, put them on the spot. This is student’s revenge for me. Thanks. So I just wanted to add, you know, my profession
has a kind of totally different kind of view on questions like this, where we have often
used, like, restorative justice circles to determine what would be the response to a
harm that was experienced that could be identified and resolved, which is completely different
than — you know, would frame the question completely differently and therefore require
a different set of approaches to understanding potential solutions. But, you know, that — in thinking of Mike’s
example of, you know, the death in the sidewalk crossing and the erection of barriers, that
has been for me in some of my work, you know, for juveniles that have been sentenced to
state care for some sort of crime that was committed, engaging them in a process with
the victims or the survivors of the crime to discuss what would be appropriate for both
parties to consider. You know, what is the appropriate kind of
restitution, and sometimes, that is — it is simply that conversation. So I was observing conflict resolution mediations
in Pittsburgh, and there was a group of young kids that were just — you know, they were
playing, they were throwing rocks, not at people or buildings, but they were just kids
and playing, and one broke through the window of a car. There was a baby in the car, the baby was
fine, but the kids were taken in for questioning and were diverted through, you know, kind
of this conflict resolution mediation process, and the harm that they were seeking to mitigate
was, like, the parents of the baby were very concerned, right, clearly that this had happened. But the process of that conversation was allowing
the kids to realize what could have been the outcome of their actions, and that conversation
was sufficient, as decided by all parties, that could, in that case, you know, serve
as a restitution or a response to that experience. So I think, you know, there are, you know,
clearly kind of the economic approaches — the foundation of economics, it sounds, this question
is the foundation of economics, but I think there are other kinds of disciplinary approaches
to some of those questions. So one quick response to that. At the height of the financial crisis, for
those who were paying attention to servicing issues, there was a call at one point, you
know, by people who were dealing with their servicers and were saying, this person doesn’t
have an interest in working out with me, I want to talk to the creditor. I want to talk to the person who owns my loan,
because I think that person I could work this out with, but I want to talk to that person. But there is, in our world, in the world of
finance, there actually was no such person. It’s in a securitized pool. There’s a trustee for the pool, the trustee
doesn’t have that responsibility, there’s really not another human being, but the homeowners
were pushing — were asking for this. And there was a lot of perplexed — people
were perplexed and felt that there just wasn’t a counter-party of the sort that you’re describing
to talk to, and markets work through these anonymous mechanisms. It’s kind of how it has to — how it is. But it also limits certain kinds of conversations
that even in our world, people sometimes call for. So Paul, let me just add that in the non-bank
lending space, those people do exist. I mean, you can work out — you can do workouts
with various small dollar lending products. So those aren’t sold, those aren’t securitized,
and sold. Just point of clarification. Okay. I do want to make sure we got to the very
last question. In regard to enforcement cases and remedies,
should the Bureau use standard economic approaches, such as before and after studies, randomized
controlled treatment experiments, or difference in difference methodologies to evaluate — this
is specifically about enforcement cases and remedies, or are other approaches at least
as useful? So this one does drill down specifically to
the enforcement context. We’ve got, you know, just a little time for
it, but for those with experience in this, maybe you’d like to speak to this. Josh and Mike? I’m happy to go first. Yes, with respect to the first question. I mean, of course there are other valuable
approaches for understanding, you know, whether we’re — this sort of goes back to the opening
remark — whether we’re sort of trying to identify materiality in some context, or the
presence of consumer injury. I don’t know how you do that without having
some sort of model constructed of the counterfactual world, and I think all of these tools — just
what you described as the standard economic approaches are, you know, getting as close
as we can in a context-specific setting to, you know, a reasonable and reliable counterfactual. I mean, sometimes we can do it really well,
and other times, some of the questions require sort of more imperfect counterfactuals, in
which case, we want to run all of the approaches and see if they’re getting us robust answers
or not. You know, I’ve been in an agency and voted
for cases. I understand sometimes we can’t do all the
approaches because we have to vote tomorrow, right? Or yesterday, as the case may be. But I think as a methodological commitment,
whether it’s enforcement cases, remedies, et cetera, I think the real strength — and
of course, there are drawbacks, as well in terms of data requirements and all of that
— but the real strength of the economic approach is the analysis is centered around the compared
to what question. What would the impact be on the population
that we’re interested in protecting with and without the conduct? You know, economics has been including with
the consumer welfare question, I mean, economic methodology is sort of — it’s built for that. I mean, that’s what all of these methods are
attempting to do in different ways, so certainly not to the exclusion of other approaches,
but I don’t know how one is a — can be confident that they were a good steward of the resources
that they are sort of granted to go out and protect consumers if they don’t also answer
the question, what would happen if we didn’t do this? You invited me, so I’m an academic, I’m never
going to turn down an invitation, unless it’s an invitation to collude, then I will not
accept. No, I agree with what Josh said. I think, you know, relevant but-for analysis
is the appropriate way. If you guys don’t do it as a bureau, I guarantee
you when you get to court, your adversaries are going to be using those techniques, and
you want to make sure that you stress test your analysis in that regard. The only other thing I will say is, you know,
saying but-for analysis is easy, but I think there needs to be important dialogue between
the economists and lawyers in the Bureau. So just to give you a simple example, you
can think of — imagine you’ve got a deception case, and as a matter of economics, the relevant
but-for question is, okay, what would — you know, for a mortgage instrument, you deceive
someone about how much you were going to pay or something like that. The relevant question is, okay, in the but-for
world of that deception, how would consumers have behaved? And that differential in their behavior and
the amount of money that the lender obtained as a result of that deceptive activity is
the harm, right? The ill-gotten gains, if you will. So that’s the economic way of doing it. It might be that only five percent of borrowers
change their behavior and as a result, the lender, you know, had $1 million more in loans
than it would’ve otherwise had. You can calculate harm that way. But-for analysis in the context of breach
of contract might say, well, how much would the party have collected had they not breached
the contract? You said the mortgage was free, but now you’re
charging us five percent interest. That’s a breach of contract. Well, there are two different — I’m not a
lawyer, but my understanding is, there’s two different ways to think about the benefit
of the bargain in the case of a breach of contract versus the way I would contemplate
harm in the context of deception. So if you come to me as an economist and say,
gee, Mike, we’re thinking of a deception case, what’s harm? I’ll give you one answer. If you come to me and say, gee, Mike, this
is a breach of contract case, how do you compute damages or harm? That’s benefit of the bargain, I may come
up with a different answer using that, and that’s why it’s important, I think for your
legal teams and your economists to be on the same page so that the lawyers don’t think
they know what the appropriate measure of harm is in a matter that is claiming something
other than that measure of harm. Just to push a little on that, you know, the
economists often — we do want to work within our framework, as our discipline instructs,
and then if it has to get cabined in for other reasons, that’s fine, nobody objects to that. But, you know, we might say, hey, here’s a
benefit or here’s a harm from, you know, an economic perspective, it may not be cognizable
under whatever practice we have to fit into. But we want to — definitely think there’s
value in making sure there’s a good hearing on the comprehensive — on the economic concepts
here — I know you can’t bill for that, but — Absolutely, and I’m not just — maybe I didn’t
make myself clear. At the end of the day, that’s not unusual,
I will add. But what I’m suggesting is that there are
oftentimes two different ways to legitimately measuring harm depending upon what the alleged
harm is. If you allege one type of harm — I mean,
the but-for analysis is but for the harm, and if you have two different legal theories
of harm, that could in principle give rise to two different measures of harm. And so all I’m suggesting is that, don’t forget
that you folks at the Bureau are on the same team. I know some of you are economists, some of
you are other stripes, social scientists, most of you are lawyers, I’m going to speculate,
but I think there’s value in making sure that whatever — even if you agree on how you go
about measuring harm, make sure that measure of harm is linked to the alleged acts in the
complaint. And there will be no discussion — I’m just
joking. I’ve actually exceeded our time, and that’s
because I am a monopolist. I think John was worried about monopolies
earlier, but we’ve concluded who the monopolist is, I think so. Anyway, thanks so much for the Bureau, first
of all, for allowing us to participate in these conversations. They’ve been absolutely enlightening to me,
and hopefully they were of value to you, as well. I’d like to thank Director Kraninger, Deputy
Director David Silberman, I’d like to thank Tom Pahl, Christopher Mufarrige, Matt Cameron,
Jason Brown, and everyone else on the Bureau staff, especially all the Kims and Kimberlys
that have helped us along the way. Hopefully, these discussions are kind of a
good preview for what ARC will be able to do over the course of the next year, and maybe
even the year following that, and we just thank you for that opportunity, and now formally
adjourn the meeting.

Leave a Reply

Your email address will not be published. Required fields are marked *