UBI in Denver - how's it going?
They're giving the homeless money, is it working?
Axios: “Denver's universal basic income project reports early success”
If we squint, we can see three errors in that headline - I’m reaching a little, but come on, it’s a short headline.
Errors:
Not universal, the money is going to a selected group of homeless people without drug or mental health issues.
“Success” implies some defined criteria that the project is meeting - that’s not the case.
The headline makes you think the basic income project shows promising results - in fact, the results are mixed.
What’s going on in Denver.
The Denver Basic Income Project (DBIP) selected a sample of 807 homeless people and split them into 3 groups - A, B, and C and gave them money.
A - $1,000 a month
B - $6,000 up front and $500 a month
C - $50 a month
Group C is an “active control” - they are getting enough money to stay in the program but so little money that we would expect the effects of receiving the basic income to be minimal. To determine the effects of unconditional basic income we can compare groups A and B to C.
The income is “unconditional” - participants get it no matter what. If they get a job and start earning lots of money, or if they receive other social services, or if they are no longer homeless, or anything - they just get the money unconditionally for a year.
This project has been going on for over 6 months and a preliminary 6 month report is available. That report is what the Axios article is based on.
Is it successful?
“Successful” in the sense that they gave out the money and wrote a report? Sure. Successful in the sense that it’s an effective tool to address homelessness? That’s not so clear.
First, let’s establish something important: most people aren’t homeless for very long. A report from Housing and Urban Development tells us that about 30% of the homeless are “chronically homeless” - that is, homeless for more than 1 year. That implies that most homeless are homeless for less than 1 year.
That’s important to establish because the DBIP report is going to show signs of improvement among participants - but we should expect lots of improvement anyway, the vast majority of homeless turn their situation around within a year.
Second, the DBIP participants are not a representative sample of homeless people. They intentionally exclude people with drug addiction or mental illness - maladies which are disproportionately prevalent among the chronically homeless.
Eligibility criteria for DBIP participation included being 18 years old or older, accessing services from one of the partner agencies, not having severe and unaddressed mental health or substance use needs, and experiencing homelessness, as defined by DBIP
DBIP Report Sampling and Methods Page 4 - Emphasis added
So, when we ask “Was this successful” it’s hard to know what that means. On most measures, things like “Did you have shelter when sleeping last night” or “Are you employed” or “What’s your financial situation”, all participants improved. On some measures A and B participants outperformed C participants. But, on other measures, especially the most important measures, C was the same as, or, in some cases, better than A or B.
We’ll review these measures in the next section. For me, it’s hard to conclude that UBI had much of an impact comparing Group C to B and A. But even if UBI did have an impact that wouldn’t tell us it was “successful” because we would have to compare the benefits from the UBI program to the costs of it and neither the DBIP report nor the Axios article make any effort to do this.
$1,000 a month is better than $50 a month - right?
The DBIP report considers many different measures. There are 15 figures illustrating changes between enrollment time and the 6-month follow-up time and dozens of additional statistics across 30 tables. Within this data we can find measures where groups A and B did better than C, but we can also find measures where C, the active control, did as well or better - and that’s confusing.
I say “confusing” because it’s not easy to synthesize the many different outcomes into some conclusion like “UBI is a cost-effective program for addressing homelessness.” And the DBIP report makes absolutely no effort to do this. Without that effort how could the program be a success?
The DBIP presentation of their findings makes me skeptical of the group’s objectivity. Their website summarizes their preliminary findings with 3 figures - all of which show a benefit for the treatment groups A and B over the active control C.
All 3 figures show A and B doing better than C - but marginally better. Is a few percentage points increase in the likelihood of staying in a place that you own or rent, or in being employed full time, really worth $12,000 a year? How do we know? How do we measure that?
Figure 2 is misleading. It’s the percentage of participants who said they stayed outside the previous night. At enrollment time participants take a survey “Did you sleep outside last night?” and the same at the 6-month follow up. Those results are what Figure 2 shows, and the reason I say it’s misleading is because you might think that UBI is reducing the likelihood of staying outside, but a different figure from the report conflicts with this.
Figure 6 shows average number of unsheltered nights, not just whether or not you were unsheltered the previous night, but, over the past 6 months, how frequently were you unsheltered on average. (It’s not clear to me if this is a weekly, monthly, or total number).
We can plot those numbers a different way to see percentage change from enrollment to follow-up.
Here, Group C, -41% average unsheltered nights, did better than Group B despite getting much less money.
I can’t explain why B does better on “last night unsheltered” and C does better on average. The DBIP report doesn’t address this.
It’s strange to me the DBIP webpage shows the “last night unsheltered” figure without even discussing the complication from the “average nights unsheltered” figure. The only way I can interpret that decision, especially in the context of the other figures shown, is that DBIP authors are heavily biased towards showing positive results - and that should cast a dubious light on everything they say and do in this matter.
Here are 2 other figures that seem really important that the DBIP webpage ignores - financial well-being and food insecurity.
Figure 8 shows financial well-being according to the CFBP Short Scale and it complicates Figure 3 from above. If Groups A and B are more likely to work full time, but C is doing equally well from a financial well-being standpoint, doesn’t that mean things are worse for A and B? They working more and they are getting $12,000 and their financial well-being is the same as Group C? Huh?
Figure 13 seems to illustrate a lack of effect from UBI. Giving a homeless person $1,000 a month doesn’t improve their food security?
So…. What?
This is a preliminary report halfway through an experiment. We cannot fault DBIP for not having firm conclusions. To me, the results seem underwhelming. Where the UBI shows an improvement it’s small compared to the active control and on many important measures UBI does not show an improvement.
DBIP is designed to show improvements. They select a non-representative sample of the homeless population by excluding people with mental illness and substance abuse problems. Because this selects against the chronically homeless that means participants are likely to improve over time without any intervention.
Participants were not randomly selected even after meeting those criteria but were instead recruited by partner organizations who intentionally oversampled women. Using a non-representative sample to study the effects of a possible intervention on a population seems ill-advised.
On the DBIP website and in popular reporting, as in Axios, only the positive results are shown while the confusing or negative results are ignored. That suggests there is more activism than social science happening here.
Having read the DBIP preliminary report my opinion is the improvements are much less than I would expect, and the cost-benefit analysis is entirely absent. I will read the final report, but I’m not optimistic. I would hope to see cost-benefit analysis (e.g. “Here is how much a Group C participant costs society compared to A or B”) and I would be interested in knowing mortality between the groups.
Can you check figure 8 on a smaller timeframe? I wonder if UBI is getting them out of homelessness faster.
These figures indicate, to me, some 'hard' spots created by our existing systems that aren't being surpassed by the level of basic income provided by the study. Over 6 months, these non-chronic homeless are reaching these points anyway. But it's likely that Groups A and B are getting there faster, and the existing charts are hiding this.
It's quite likely that the UBI used in this study isn't high enough for people to attain food security or financial security past a certain point, but it's enough to pull them up off the streets faster and more efficiently than without.
For example, your charts are averaged over 6 months as bar graphs. What about a line chart that compares on a daily basis how many members of each group were homeless on each day? I'd hypothesize that chart would show that members of group A and B became housed sooner than members of group C - and if your presentation of the target group being those that were going to be rescued from homelessness anyway (non-chronic) is accurate, then we see value in UBI by bringing groups A and B off the streets sooner!
Also, we're not comparing the level of quality of housing here, and we're relying heavily on self reporting (especially in your last two figures) as opposed to objective measurements. Might groups A and B have higher standards of living, even if their self reported financial security and food security is the same? Perhaps they're still maladjusted to having security, or aren't yet perceiving the security they've acquired in this short timeframe. We aren't objectively measuring those things in these results, I think that's a blind spot.
I hope the final study addresses these points though, because I agree that it's important to be transparent about the sample contents and the results of the study.
I will never understand those who say "Gee, they only selected homeless people with a lack of money to give money to, instead of homeless people with that problem and drug problems."
It's clear from this experiment and other experiments like it, that for a large percentage of the homeless population, they only need money to not be homeless. Providing basic income would therefore help all of those people.
The people who remain homeless after they have basic income are then also much easier to help because we know they have problems aside from money problems. We can then more easily help them because we more easily see who they are.
And don't give me this, "Gee, homeless people are only homeless for awhile anyway" crap. It's also been shown that having basic income can prevent a lot of homelessness from ever happening, because again, for many people, it's just a lack of money problem.
Homelessness is extremely expensive. If we can spend less by just giving people money than spending on everything that surrounds and emerges from homelessness, then we should do that.