Prosper.com hired Doug Fuller to lead "operations" some time around mid September. He's been there 90 days, so we're gonna take a look at how one of Prosper's biggest problems - collections - is doing under Doug's rule. Summary: Improving, but slowly.
You might think its strange to discuss an individual employee. A bit of history may make that more understandable. In the beginning, Prosper thought of themselves as the E-bay of loans. They built an incredible web-based infrastructure to match lenders with borrowers and process payments. Unfortunately, they didn't think enough about the loan business. Not enough attention was paid to loan "operations", identity verification, fraud checks, ensuring valid contact information, handling various payment situations, customer service, and collection of late payments. Prosper may look like E-bay on the outside, but inside a lot of machinery is needed to make the loan business work. They got off to a bad start with that machinery, and how quickly they recover may well determine the future of Prosper.
Example: a year ago, when I first started writing about these problems, the cure rate (the fraction of loans that recovered after going 1 month late) was only 7%. In other words, once a loan went late, you could pretty much write it off. And about 20% of prosper loans per year went 1 month late. The resulting huge default rate discouraged many early lenders, and threatened the future of Prosper and P2P lending. In addition, many lenders were angered by how slowly Prosper took visible steps to fix these problems.
Operations is important. Its the core of the business.
So it was an important event when Prosper two years after it started Prosper openly recognized the importance of loan operations by hiring a VP of Operations, Doug Fuller.
When Doug arrived, he promised a hard line on collections. Its been 90 days. Lender's money is at stake. How's he doin'?
Lets look at the history of Prosper's cure rate, that is the fraction of 1-month-late loans that are eventually broght current. (The rest of them go on to default.)
Good news: getting better all the time. Bad news: Not good enough yet.
As often occurs, there are some technical problems with interpretation of this data. The statistic I'm plotting above is the fraction of all loans (over the entire history of Prosper) that have cured while at the collection agency Penncro. In other words, an average over all time. This statistic is weighted down by a lot of old loans that have already defaulted or nearly so. I'd really like to have data showing collections performance on recent loans. Because this average moves up with time, we know that performance on recent loans must be significantly better than the average. But how much better?
Looking at the chart, you'll note that it looks like there's a little "bend" in the curve two months ago around the end of October. Doug Fuller arrived around mid-September, so I'm gonna call the end of October change the "Doug Fuller bend". I don't know that this change came from his efforts, but is seems a resonable possibliity. Things are better, but how much better? ...and what changes has Doug made to cause this improvement?
Besides this "lifetime" statistic, Prosper also provides a "last 3 months" statistic. At first I thought great, that's the thing I should use. The first thing I noticed was that the "last 3 months" numbers are lower than the "lfetime" stats. We know that recent collections performance is better than in the past, so this didn't make sense. I contacted Prosper for clarification. They told me that "last 3 months" doesn't mean loans that have entered the collections process within the last 3 months, as I had presumed. It means all loans that were in work at Penncro at any time during the last 3 months. Suddenly it became obvious why the "last 3 months" numbers are low. They are weighted down by all the bad loans that are stuck in collections, sitting around Penncro, 4 months, 5 months, 6 months, 7 months, etc late just waiting to be sold off. Its a roach motel statistic! That's too difficult to interpret. At least the "lifetime" statistic is unbiased. It contains every loan that has gone to collections exactly once, so the relative weightings of good and bad loans is exactly the relative weighting of same that arrived at collections.
I returned to the "lifetime" statistics. But I really wanted to get a quantitative measure of what fraction of recent loans are getting cured. Starting 10/30/07, in addition to copying the % cured from Prosper's web page each day, I also copied the # of loans in collections. Now that I have this data for 2 months, I can calculate the number of loans that were added to collections during this period and the number of loans that recovered. If I assume that all the loans that recovered were loans that were sent to collections during this period, I can calculate the cure rate that would be required to produce the numbers Prosper gives us. I'll spare you the equations. The number comes out 23.6% .
Now that number is almost surely on the high side, as it presumes that all loans recently cured were recently sent to collections. There were almost surely a few old ones that cured as well. Now we have bounds. The cure rate on recently-gone-late loans is between 15.65% and 23.6%. Its probably around 20%. That's a lot better, but it still means that about 80% of loans that go 1 month late will go on to default. That's still not very good.
We don't know what actions caused these improvments in collections performance. Several times Prosper employees have told me that low contact rates were a big problem. In other words, when the collection agency tried to contact the borrower, a large fraction of the time they would find that the phone numbers and addresses didn't work. Scary. We know that Prosper implemented phone checks and address verification post cards to help with this problem. We don't really know much, because they've been tight-lipped about both their understanding of the problems and ongoing improvements.
Seems to me that if address verification post cards helped, then fraud of some sort was a big problem, because it seems to me that this is the only problem an address verification card can fix. After all, Prosper already used Experian's database to check addresses given by borrowers. If the address matches the name in Experian's database, but the postcard fails, then the name and address matched, but the actual borrower doesn't live there. That's fraud, yet Prosper denies that identiy fraud has been significant. Curious. We just don't know, because Prosper hasn't shared with us any details of the ongoing battle against fraud, or even against nonfraudulent bad addresses and phone numbers.
In a traditional loan company it makes sense to keep this sort of info private, but Prosper isn't a traditional loan company. Because of the poor performance of Prosper's loans, there's a crisis of confidence. To fix that one would need to be more open, but Prosper's relationship with lenders has becme much less open during the last few months. In some cases Prosper has become downright hostile.
I tried to communicate with Doug Fuller once. Sent him an email message. I got back a response from Prosper's Shira Levine, saying that Doug had received my message. I have never had that happen anywhere before. I haven't tried to talk with him since. (If you've had better luck, let me know.)
Prosper began with an incredible philosophy of openness. Who else ever made public the incredible amount of information about loans that Prosper does? Nobody. Somehow things took a turn. It will be impossible to restore lender confidence without openness. That's my advice.
Doug Fuller did say in his 10/2/07 "interview" that there was a "40% increase" in contact rates during September 07. Doug Fuller Interview He said this improvement came from getting Penncro to start their autodialer at different times of day. How could the prior management of collections be so dumb as to dial at the same time every day? Its not like they didn't know collections was an important issue. We told 'em so, and they said they agreed.
In the early days of Prosper, lenders used to debate the causes of the high default rate and low collections cure rate. Some felt that there were just a lot of bad loans, either due to fraud or lender stupidity. Others believed that the collections effort was inept. Maybe it was (or is) all of the above. How are these things changing? How can we regain our confidence if we don't know?
My apologies. This discussion has run downhill. I started with numbers and have descended to speculation about tidbits that have escaped from the perimeter of the castle. To some this speculation may seem rude, but remember these are our loans. The lenders own them. Prosper merely originates and services them for our benefit.
- Collections is indeed improving, but very slowly
- The drawbridge is up; little information about collections is being shared
- Doug Fuller might be improving things if the little bend is for real
- Wish he were more open
A message from Patrick Gannon of Lendingclub ...
From: Patrick Gannon ...
Subj: Re: Lendingclub - bad math
Thank you for your analysis of our ROI calculations. We are taking down the ROI section of the statistics page for now while we review your comments, track down potential errors in the calculation and improve the help page. We will blog a response in the next few days after we relaunch that page.
I tried to open an account on your blog to post a public comment but couldn’t sign up.
We appreciate your interest in Lending Club, and we hope that we can win you over as a lender.
Thanks again for your analysis,
Patrick from Lending Club
Thought I'd post this, both to make their response public, and also to unpuzzle anyone who went looking for the page I was complaing about and did not find it.
I do hope they succeed. Prosper needs a competitor .
Lendingclub now provides an estimated "Lender ROI" calculation on their web site. Unfortunately, their mathemematics is incompetent, and the resulting numbers are meaningless.
You might say "Gee, why is he so hard on them?" The answer is simple. They want my money. This forces me to set a standard.
I guess us P2P lenders are spoiled because Prosper.com got much of this right. It looks like LendingClub tried to copy the presentation on the Prosper performance web page without understanding the calculation.
Lendingstats performance statistics can be found at http://www.lendingclub.com/info/statistics.action#
Here's a picture of their ROI chart...
The page containing this table comes with a popup help page that explains the calculation. Lets try to use it to understand what they have done. Sorry if this is a long slog. There's no other way to show you how utterly bogus this is.
Estimated Lenders ROI table
Average rate: The average interest rate for all loans that have closed within the loan grade (A-G)
Losses: The sum of the principal amount of the loans that are expected to default, expressed as a percentage of lenders estimated ROI. A loan is "expected to default" for purposes of this estimated ROI calculation (i) at a 50% clip with respect to loans that are at least 15 days late and (ii) at a 90% clip for loans that are at least 120 days late
Processing Fee: Fee paid by the lenders expressed as a percentage of lenders estimated ROI
Estimated ROI: The net estimated return on investment after subtracting estimated losses and fees
Average rate makes sense.
But when we come to "losses" ... Whoa. "sum of principal amount of the loans" ... That I understand. When they say "that are expected to default" they mean late loans multiplied by a factor, 50% or 90%. That I understand. They go on "expressed as a percentage of lenders estimated ROI." I stared at that for quite some time trying to figure out what the heck they could be doing. You see, we're building up to the ROI calculation. These losses are one component of it. We don't know the ROI yet, so we can't possibly divide by it. This is just wrong.
I reverse engineered the numbers in the table. What they actually have done is to add up all the estimated losses (dollars) and then divide by the total principal amount of the loans in that credit grade. So the guy who wrote the explanation didn't understand the thing he was trying to explain. Ok. The losses numbers they actually calculated are closer to being right, but are not right, for two reasons.
First, they are not properly normalized. Second, they include only the loss in principal. They forgot the interest which is lost when a loan defaults. I'll spend a few sentences explaining each of these errors a bit later, so you can understand how big these problems are. For now lets continue reading the help popup.
The processing fee explanation in the help popup is also wrong. The words "expressed as a percentage of lenders estimated ROI" simply don't belong there. After computing these various percentages, they subtract them, which means they had better all be percentages of the same thing: principal.
It isn't easy to describe a calculation precisly in english. Equations are better. So it seemed good that LendingClub's popup went on to provide equations. However, these equations are all wrong. (And by all, I mean each one of them is wrong.)
Estimated ROI = Average Interest Rate - (Loss due to Late Loans - Loss due to Default Loans) - Lending Club Processing Fee
Loss due to Late Loans = Sum (50% * (unpaid percentage) * (interest rate of the late loan))
Loss due to Default Loans = Sum (90% * (unpaid percentage) * (interest rate of the default loan))
Lending Club Processing Fee = 1% of all payments received by lenders.
Ok. Lets start at the top. The equation for Estimated ROI says we should subtract some numbers. I expected that. But look at that minus sign inside the parenthesis. This equation would have us subtract one kind of loss but then add (instead of subtract) another kind of loss. That's wrong. Either the minus sign inside the parens should be a plus, or the parens should be dropped.
Next lets look at the equations for losses. They're similar, so lets just look at the first one. It has us sum some things. We'll assume we're summing over the late loans. Lets look at what's inside the sum.
50% * unpaid percentage * interest rate of the late loan
There are many things wrong with this. First, they don't say percentage of what. Second, the numbers shown in the table are clearly the fraction of principal that is late, which has nothing to do with "interest rate of the late loan". I think what they meant to say was
sum(50%* late principal) / total principal
Finally, there's an equation for processing fee. Well, its not quite an equation. It is the definition of the LendingClub processing fee. Unfortunately, its in units of dollars. 1% of some dollars is going to be some other amount of dollars. The number in the table on the other hand is a percentage rather than dollars, so this equation can't represent what is in the table. Some calculation was done to get from dollars to percent. Not specified. Another problem is that they haven't told us how the dollar amount was calculated. Did they calculate the payments due during a year for each loan, multiply by 1% and then sum? The numbers in the table look too large for that. What they actually calculated is unknown. A proper equation would leave us with none of those questions.
Enough with this unhelpful help file. Lets go back to the two big errors in the actual calculation. These are: 1. the loss rate is not properly normalized. 2. They counted only principal and forgot lost interest.
To understand the normalization problem, we have to look at the units of the numbers in the table. The first line contains the average interest rate of the portfolio. That interest rate is a rate per year. The second line contains the loss rate, which they have forgotten to convert into a per year rate. They're going to subtract these two numbers, but they aren't in the same units!
To understand why this is important, imagine that you borrow some money from me at 3% per month, and lend it to Joe at 5% per year. Now you subtract 3% from 5% and think you are getting a return of 2%. Not so. You subtracted two numbers in different units. To do the calculation correctly, you would convert the 3% per month interest rate into an annual rate and then after you have two numbers in the same units you would subtract them. You would discover that you're losing a lot of money.
That "losses" rate in Lendingclub's table is not a per year rate. It shows loans that have gone bad (late or default) during their life so far. The Lendingclub portfolio is very young. "So far" has not been much time. To convert that number to an annual rate we're going to have to increase it a lot.
I downloaded the Lendingclub database, and calculated the average age of the entire Lendingclub portfolio. The loans in the portfolio are 84 days old on average. A lot less than a year! But that's not the entire story. Loans can't possibly go bad during their first 30 days, because no payment has yet become due. At the very least, we should subtract 30 days. But there's more. Lendingclub uses "15 days late" as the threshold where a loan becomes "bad" for the purposes of our loss calculation. This event cannot occur until a loan is 45 days old. Wow. So we should really subtract 45 days from the 84 day average loan age. You get 39 days, and that's just 0.11 of a year! To normalize the "losses" that Lendingclub shows in their table, we should divide them by 0.11, which is the same as multiplying them by approximately 9. Ouch.
Look at the column for loans of grade "B". The 0.51% losses they show, when properly normalized (annualized) becomes 4.64% per year losses. Quite a difference.
I should mention that this simple normalization method makes several assumptions about the behavior of loans. It is certainly not the only way the normalization can be done. I'd be open to Lendingclub using some other method, as long as they tell us what they've done and it makes sense. Just using 39 days worth of losses to represent a years losses is not acceptable.
The second error in this calculation is more subtle. When a loan defaults. Everybody realizes that you lose principal. You also lose the interest that you expected to get. I can hear people saying "but you didn't get it yet so how could you lose it?" Lets look at a simple example. Suppose I lend you $1 at 5% interest, and to keep this simple you're gonna pay me back in 1 year. My return will be 5% if you pay me back, or -100% if you don't. The difference between those two numbers is 105%. Using the kind of ROI formula that Lendingclub used on their stats page, I would start with the 5% interest rate I expected to receive (that's the "average rate" on the first line of their table), then I would subtract something called "losses", and I want the answer to come out to be my ROI. If I want to come out with an ROI of -100%, I'm going to have to show -105% on the losses line because that's what I need to subtract from 5% to get -100%. That -105% represents loss of both principal and interest.
If you look closely at Prosper's performance table, you will see two losses lines, labelled "net defaults" (which is where they put the estimated principal loss) and "adjustment (interest and fees)" where they put the estimated interest loss.
Lendingclub left out the 2nd line.
In the early days of Prosper.com, they got this one wrong too. They used to explain that all you needed to do was add the ROI you wanted to obtain to the default rate you expected on a certain grade of loan, and that sum was the interest rate you should bid. That calculation ignores the lost interest term. This wrong calculation for a long time was coded into the Prosper.com standing orders web page.
Having trashed their attempt at performance calculation, I don't want to end without saying a few things about some positives at Lendingclub. The principals seem like nice chaps. They seem very open to communication with the lender community. That's a great big positive. Like Prosper, they've also decided to make their loan database available. This is also a great positive. (Its not as extensive as Prosper. They only have loans, no bids or info about lenders. It turns out that in Prosper's data, the information about lenders' behavior and performance is the most useful part.)
I imagine the fine folks at Lendingclub will straigthen all this out. Once they do that, and their portfolio ages enough to be measurable, I'll be very interested to see how the returns numbers look. Before I invest in loans, I need some way to know (or believe, or have a hunch...) that the interest rates on the loans are reasonable. Investing thru Prosper.com, many of us learned that the historical default rates prosper used to give us from Experian were inapplicable. Prosper's default rates turned out to be much higher. I would assume its going to be the same way on Lendingclub. Lendingclub started by giving investors a set of historical default rates from Transunion. It seems likely to me that these will end up being inapplicable to the Lendingclub situation. We need good data from Lendingclub's own operation before we can decide whether these investments make sense.
The folks running Lendingclub set the interest rates for us. To do that properly they need to understand the mathematics of loans. I need to have confidence in their ability to do that math correctly. I'm not there.
If I could wish for just one thing this holiday season when it comes to Prosper.com it would be redacting any changes that lead to the appearance of censorship.
It is my opinion that the appearance of censorship on the part of Prosper.com and its management will be the downfall of Prosper. PMI needs to remember that this is a place where lenders invest their hard earned dollars and where borrowers ask for same. Everything that is done needs to be above board and honest to a fault. This appearance of censorship started when Prosper erased, completely erased, the old forums and installed a new one.
I have kept track and nine out of my last ten posts to the new forums did not meet approval and were not posted. Add to that that Prosper is sending out letters from attorneys to various members, banning anyone who mentions an outside website on Prosper.com, and erasing profiles that do the same and it looks like Prosper operates under a police state.
I want nothing more than for Prosper to succeed and it is my most fervent wish that I can work within Prosper to affect change.
As it stands now until or unless Prosper takes some action to remember who their customers are I won't be borrowing or lending again. That's a sad statement for me to make and I hope Prosper realizes I'm not the only one who feels this way. Many of us feel that Prosper has squandered not only the goodwill we once had but also the tremendous talents of their lender and borrower base are the customers who have been with them since the beginning. We all share one common goal: to see Prosper succeed and become the premier site for P2P lending. We hope Prosper shares this goal with us.
The words above were written by long-time Prosper member Cushie06. She submitted them to the official Prosper.com blog, where they met the following response:
Unfortunately Prosper is not able to accept your post for to the Prosper Blog. Many of the sentiments expressed are shared, but we fell it is inappropriate for the blog..
Here's my 12/15/07 update to the prosper.com late loan statistics, with some notes on default rates. You've heard me say it before. The default rate of prosper.com loans is repeatedly misrepresented in the press. This trend continues.
First, today's chart...
A larger more readable version of the chart is available here
An article about Prosper.com appeared on Yahoo today. It contains a misleading statement which has appeared in many articles about prosper. "The default rate on Prosper loans is a meager three percent." http://news.yahoo.com/s/afp/20071216/bs_afp/lifestyleusbritaininternetfinancemoney_071216024815
Too bad the author, Glenn Chapman didn't do a little research. Seems likely that he just copied from the last misleading article, the November 27, 2007 Associated Press article http://www.msnbc.msn.com/id/21993720/
Its pretty easy to see that the default rate on Prosper.com loans is about 20%/year. You can see it in the chart above. Doing an estimate using all the loans on Prosper requires some tricky math because loans are of various ages. That is difficult to explain. To make this simple, lets look at a group of loans that are all almost exactly 1 year old. That way there's no age adjustment to do. We're just gonna estimate how bad things got during that first year. I've written about this before. This time I'm gonna talk numbers instead of theory.
Lets look at all the loans that Prosper.com originated in October 2006. I'm going to consider, as I do in the chart above all those loans that have gone at least 1 month delinquent. The information about such delinquency comes to us with a built-in 2 month delay. In other words, a loan's first payment isn't due until the loan is 1 month old, and that payment can't possibly be 1 month late until a 2nd month has gone by. That's why the curves jump up from zero at 2 months after origination. With that in mind, I chose Oct'06 loans, because they're just now at the point where we have observed 12 months during which it was possible to be 1 month late.
Luckily, Prosper makes all the numbers available via their performance web page. This is a good thing! Now according to Prosper, 740 loans originated in Oct '06. Of those, 81 loans have already defaulted. If we counted only loans that have already gone all the way to default, we would divide 81 by 740, giving us 10.98% of these loans defaulted in the first year.
However, we know a bit more. In Prosper's vernacular, "default" actually means that they have already disposed o the bad loan. There are a great many loans that have gone more than 4 months late, Prosper's criteria for default, but have not yet been sold to junk dealers. These loans are just as bad, but haven't yet been given the name. While it is theoretically possible for one of them to recover, we know 99% will default. In fact, this will happen within the next few days, as Prosper has scheduled a loan sale before the end of the year. Prosper tells us there are 73 of the Oct'06 loans i nthe "4+ late" category. If we add those in, we get (81+73)/740 = 20.81% of Oct'06 loans are defaulting in the 1st year. Wow. That's a lot different than the 3% mentioned in the article, isn't it?
A better estimate can be made by using more of the late data. Based on historical performance, we know that once a Prosper.com loan goes 1 month late, it almost always defaults. It is legitimate to count these as associated with the loan's first year, because there is such a high probability that the "stop paying" event has already occurred. Prosper tells us that of the Oct'06 loans, 13 are 3 months late, 12 are 2 months late, and 16 are 1 month late. Therefore the total number of loans that are 1 month late or worse is 81+72+13+12+16 = 195. Each of these 195 loans was sent to Prosper's collection agency when the loan was 1 month late. We know from the collection agency statistics that Prosper also publishes that about 15% of loans that have gone to the agency have recovered. We can therefore estimate that about 85% of such loans will default. That gives us a way to estimate from loans that have already gone late, how many will default. That estimate is 85% x (81+72+13+12+16) = 165.75 . Finally, if we divide this by the total number of loans, 740, we get 22.4% . Stop pay events in the first year of these loans will cause 22.4% of them to default.
If you look at the chart above, you will see that there's nothing very special about Oct'06. The curves for most other months have the same slope and general shape.
Prosper loans, so far, have been defaulting at about 20%/year, not 3% as the article claims.
This number is important for Prosper lenders to know. The simplest takeaway is this: If you want your portfolio to have a default rate better than 20%, you'd better be selective!
There are many other things I've learned from these curves. Example: I used to see people say things like "After a borrower makes 4 or 5 payments, I think the loan will be ok." Turns out, that's nonsense. If you look at the curves, you see that nonpayment events are well spread out over time. We know so far that they're pretty uniform over the first 18 months or so of the loan, and may well continue thruought the life of the loan. (ie the curves are pretty straight) Nothing magic happens after a few good months. That was a lender's fantasy.
In the last update of the charts, I noted that the Aug'07 curve took a big jump. I wondered whether this was a data glitch. Now confirmed. It was a glitch. A few days later, Prosper revised the numbers. The Aug'07 curve looks perfectly ordinary now.