At Airbnb, Where Every Stay Is Above Average

On Airbnb, the hosts are uniformly great, the apartments amazing, the locations ideal. Even the dogs attached to the properties are cute and sweet. At least, that’s the impression you get from browsing reviews on the hugely popular online accommodations rental platform. Founded in 2008 by three friends who had air mattresses and a hefty rent burden on a house in San Francisco, Airbnb swears by its “honest, transparent” two-way review system, in which hosts and guests get to rate each other.

What does it mean when all the hosts are great—and all the guests have a wonderful time?

That’s one of the questions about the fast-growing online sharing economy that Georgios Zervas, an assistant professor of marketing at Boston University’s Questrom School of Business and a fellow at BU’s Rafik B. Hariri Institute for Computing and Computational Science & Engineering, has been exploring with two colleagues from BU College of Arts & Sciences: John W. Byers, a professor of computer science who is also a Hariri fellow, and Davide Proserpio, a PhD student in computer science. The title of their recent working paper on Airbnb ratings is telling: “A First Look at Online Reputation on Airbnb, Where Every Stay is Above Average.” In an analysis of over 600,000 Airbnb properties, they found that nearly 95 percent of the listings scored either 4.5 or 5 stars, the highest possible rating. They contrasted this with the ratings of about half a million hotels around the world listed on TripAdvisor, where the average is a much-lower 3.8 stars, and there is more variation across individual reviews. One important distinction the BU researchers took into account: unlike Airbnb, TripAdvisor allows guests to rate hotels, but not vice versa.

Considering properties by accommodation type and location, Zervas (GRS’12) and his colleagues found considerable variability in Airbnb ratings. They also analyzed several thousand properties listed on both platforms and found that even though the average ratings on Airbnb and TripAdvisor are similar, proportionally more properties receive the highest ratings on Airbnb than on TripAdvisor. They found only weak correlation in the ratings of individual cross-listed properties across the two platforms. The number of cross-listed properties rated 4.5 stars or above is 14 percent higher on Airbnb than on TripAdvisor. The number that receive 5.0, the maximum rating, is 18 percent higher, and Airbnb guests were much more enthusiastic and willing to overlook problems.

Case in point: the budget-friendly Hotel Tropica, in San Francisco’s Mission District. On TripAdvisor, the reviews were hardly raves. “Bug bites!” one guest complained. “Extremely expensive for the low quality,” another wrote. Of the 39 reviewers, 18 ranked the Hotel Tropica as “terrible.”

On Airbnb, 45 people who stayed at the very same Hotel Tropica gave three different rooms an average of four stars. They downplayed any complaints. “Beds were comfortable and relatively clean (some hairs on pillows etc.),” one guest wrote. Another review: “Rooms are very, very noisy, but comfortable.” Yet another: “It was very hot at night so we left our windows open so we heard a lot of noise, but it was manageable!”

Why the difference in ratings between Airbnb and TripAdvisor? Zervas and his colleagues suggest that when guests and hosts are rating each other—and when the transaction feels more personal, as it does on Airbnb, or in the case of Uber, the online ridesharing platform—both parties tend to be more charitable. On Airbnb, the hosts have the option of rejecting a potential guest, so a guest needs favorable reviews just as much as the host does. Reputations are at stake on both sides of the transaction.

The researchers recently completed another study examining Airbnb’s impact on the hotel industry in Austin, Tex., a center of Airbnb activity. They found that a 10 percent increase in the number of available Airbnb accommodations results in a 0.35 percent decrease in hotel room revenue, and lower-priced hotels and hotels that don’t cater to business travel are reducing prices to stay competitive. In earlier research on Yelp, the online restaurant review platform, Zervas and his co-author, Michael Luca, an assistant professor of business administration at Harvard Business School, found that at least 16 percent of the reviews are deemed by Yelp as suspicious and not published on the site. They found that the worst offenders are restaurants seeking to offset negative write-ups.

BU Research sat down with Zervas to talk about how consumers should view Airbnb ratings, how staying in someone’s space—or sharing their car—is much more intimate than, say, buying someone’s computer on eBay, and why a TripAdvisor hotel guest might trumpet bedbugs in their online review headline while the Airbnb-goer at the same hotel notices only “a few hairs on the pillow” and tucks them discreetly inside a parentheses.

BU Research: Can you explain the part of your study where you looked at cross-listed properties on both Airbnb and TripAdvisor?

Zervas: Some hotels—a small fraction—list their properties by the room on Airbnb. So, in our study, we looked for properties that are cross-listed on Airbnb and TripAdvisor and then we compare and contrast their ratings. And we found first that on Airbnb, they get higher ratings—and, second, that somehow these ratings are not correlated.

What do you mean they’re not correlated?

Suppose that we take all these cross-listed properties and we rank them by average rating on Airbnb. Maybe Hotel Tropica is first, some other place is second. Then we look at the same properties on TripAdvisor. Well, the rankings are completely different.

It’s as if I give my students random grades for the midterm—this is hypothetical—and I also give them random grades for the final and then I ask ‘Did students who did well on the midterms do well on the final?’ Is there a correlation? Of course not, I gave them grades randomly.

So I’m not saying ratings on Airbnb and TripAdvisor are random, but this is more or less what we observed—a very weak association. So the good places on Airbnb are not necessarily the good places on TripAdvisor—and the bad places on Airbnb are not necessarily the bad places on TripAdvisor.

So, are the rankings on Airbnb meaningless?

They’re not meaningless, but it does raise some concerns. One thing we have not ruled out is that people with different tastes use different sites. This is something we’re exploring. I do believe that Airbnb users have different tastes than TripAdvisor users, but I don’t think that explains every observation in our data. I cannot believe that Airbnb users would like dirty places or bedbugs, so the question now is: Are these exceptions? Is the phenomenon more widespread? We don’t have an answer yet, but we do notice this pattern where the ratings are really all over the place. These are the same properties, but they’re ranked in a different way across the two sites. Can this be attributed only to differences in tastes between Airbnb users and TripAdvisor users, or is there something else going on?

Is it possible that on one day, the place really was better than on the other day—like with restaurants?

But why would it be consistently better on days when Airbnb users are visiting?

Oh, I see your point. So, you bring this out in your study: if people are rating each other—if the host is rating the guest and the guest is rating the host—are they preempting each other?

Yes. This is what happened on eBay. Sometimes on eBay, you would hear stories that there was communication between the two parties after the transaction—possibly, both parties were saying, ‘Okay, you better give me a good rating if you want a good rating back.’ So I think, depending on how pushy you are, you can play that strategy on Airbnb.

So do you think we need professional reviewers—like, say, Pete Wells, the restaurant critic for the New York Times? It’s his job to be objective.

I don’t think Pete Wells could review every room on Airbnb. We would need many Pete Wellses….The goal of any reputation system, whether it’s Pete Wells or Yelp, is to provide an accurate, unbiased evaluation of the product or service for future consumers. That’s what we use reviews for. I don’t think anybody would disagree with that. And here, what is puzzling is the fact that the properties are rated in a different way across two sites. So, which ratings should you trust?

You point out that unlike the individual Airbnb hosts, or entrepreneurs as they call them, a big company can hire a marketing person…

Yes. Unlike firms with large marketing budgets, few of these individuals have an outside source of reputation. If you’re traveling, and I’ve booked you a room at a Hilton, you know what you’re getting, right?

Of course.

Why is that? You’ve seen their ads, maybe you’ve stayed at a Hilton before, right? There’s brand recognition. If I tell you I’ve booked you an Airbnb at the corner of Commonwealth and Mass Avenue, what does that tell you?

Nothing!

That’s the point we’re making here. So, for people on Airbnb, reviews are more important. This point is nicely illustrated in a paper by my co-author Mike Luca, of Harvard Business School, on Yelp. Mike studied the impact of Yelp ratings on restaurants. He showed, as you would expect, that restaurants with higher ratings get more consumers. But the effect is not nearly as strong for chain restaurants because, again, are you really going to check the reviews for Starbucks or McDonald’s?

No.

Right. These products are standardized, they’re advertised heavily, we see them everywhere. You know that quality is pretty much fixed. For these companies, online reviews don’t really matter as much. That’s the point we’re making here. For a Hilton, one damaging review—okay, fine, they can deal with that. For a single person running a small Airbnb listing, one damaging review might make them exit the market.

So, I hate to ask you this again, but how do you figure out what the ratings and rankings on Airbnb really mean?

I think the meaning is more hidden and more nuanced. Probably it’s best to invest some time reading a bunch of reviews on Airbnb to get a sense of the language Airbnb reviewers use to describe Airbnb properties, and how hints they leave in their reviews should be interpreted. At the same time, there are other ways in which you can encourage more honest reviewing.

How?

Maybe you don’t need bilateral reviews, or at least you don’t need them in that particular form. So here is something that Airbnb actually did: previously, the process was sequential. Let me explain. You come, you stay at my Airbnb listing, the transaction is complete. You go back and within 14 days, I can rate you and you can rate me. If you rate me first, I can see your rating and then if I don’t like your rating, then I can give you a negative rating back….

Even if you don’t deserve a bad rating?

Even if you don’t deserve it. Nobody is going to question that. It’s 100 percent punitive—one star for you, just because you gave me a low rating. I can do that. So, what’s an easy way to solve this problem?

Tell me.

Well, maybe don’t tell me what rating you left for me before I rate you back, and when we both rate each other, reveal our ratings simultaneously. So that’s what Airbnb did in July 2014. It remains to be seen how well that works out for them. This is something we’re measuring and we want to see how reviews and ratings evolve under this new scheme. Of course, there are other problems…

Like what?

There are other issues that this arrangement might not solve. For one, on eBay, it was harder for us to communicate, but now you’re in my car, you’re staying in my place. We can make an arrangement in advance. It’s very easy for me to guess what sort of rating you’re going to give me.

How?

I can see you. You’re in my car; I’m giving you a ride. I know if things are going well—maybe there’s too much traffic, maybe I was a bit late, maybe I got lost.

Maybe the car was dirty?

Maybe the car was dirty. I may not be very impartial in the way I rate you, but maybe I have an objective view of how things went. One thing that surprised us— even on TripAdvisor, which uses a unilateral review system and you don’t have this concern about strategizing, reviews of vacation rentals, similar properties to those found on Airbnb, are rated a lot higher than hotels.

What does that mean—that it’s about the experience?

The average vacation rental on TripAdvisor has something like 4.5, 4.6 stars. The average hotel has 3.8 stars, which is kind of puzzling, and we think what might be happening here—and this is complete speculation—is that people might feel a little bit worse rating some sole proprietor or some freelance driver, or whatever, rather than some big global brand. You know that it’s this guy’s business, he’s driving Uber, or he’s renting out his place on Airbnb. Things may have gone wrong, but you’re more charitable in your evaluations. If it’s a big brand—an airline, or Comcast, or something like that you—you don’t care so much.

I don’t care at all.

Exactly. But I think when we rate other people, we are more charitable. Sometimes when we’re rating a service, we’re actually rating the person. Again, this is purely speculative. You read Airbnb reviews and what do they always mention? The host—they say so-and-so was a wonderful hostess, so-and-so was a wonderful host, a great person. They were so warm, they welcomed us, they gave us wonderful advice, they told us where to go, we had such a great time, blah, blah, blah. So it’s more about the personal experience. This is what Airbnb advertises; it’s all about making friends all over the world.

Do you have any plans to look at Uber and how their reviews work?

We’ve thought of that, but collecting data to study Uber is a lot more difficult because it’s an app. It’s not on the web. It’s not like you can go to a website where they list reviews of every single driver.

What are you working on next?

This study is far from complete; it’s just an extended abstract. We want to take this further. We’re trying to look at how the particular tastes of Airbnb and TripAdvisor users differ—and if this taste differentiation alone can explain the ratings differences across the two sites. Another thing we looked at two years ago, and we got nowhere and we want to revisit now, is the following: On Airbnb, and really on any peer-to-peer platform, there are hosts with good ratings and hosts with bad ratings, and there are guests with good and bad ratings. Is it the case that hosts with high ratings are matched with guests with high ratings? Then, if you have a bad rating as a guest, do you have to book the crappy room? Is there assortativity—matching people of the same status?

If you have a bad reputation, do you essentially have to pay more for the experience? Remember, on Airbnb, if you’re the host, it’s not like I can automatically book your listing. You can reject me at no cost. How do you make this determination? Maybe by my picture, some basic things about me, maybe I have had bad reviews. Who is most likely to reject me if I have had bad reviews? The host who has very good reviews. They get a lot of requests. So where do I end up staying?

The Hotel Tropica! Speaking of hotels, does your most recent study—on Airbnb competition driving down hotel rates in Austin—show that Airbnb competition can be good for hotel goers?

Some travelers substitute Airbnb for hotel rooms. This brings hotel prices down. As a result, other travelers who prefer hotels also benefit.

There has been a lot in the news here about Boston and the 2024 Olympics. People are already saying—and they’re not entirely joking—that if the city does get the Olympics, they’re going to personally cash in by listing their houses or apartments on Airbnb for a lot of money. Do you think there will be an Airbnb frenzy if Boston gets the Olympics? How high do you think the rates could go?

I’m not sure how high rates could go, but I definitely expect a surge in Airbnb use. This is something we saw with the most recent World Cup in Brazil. Many fans used Airbnb as an alternative to hotels.

What about you? When are you going to try Airbnb?

Most of my travel these days is conference travel and it doesn’t make sense to stay outside the conference hotels in an Airbnb. And when I travel for vacation, I usually go to Greece and stay with my parents.