At Airbnb, Where Every Stay Is Above Average
Hosts and guests can rate each other—and they do, highly
On Airbnb, the hosts are uniformly great, the apartments amazing, the locations ideal. Even the dogs attached to the properties are cute and sweet. At least, that’s the impression you get from browsing reviews on the hugely popular online accommodations rental platform. Founded in 2008 by three friends who had air mattresses and a hefty rent burden on a house in San Francisco, Airbnb swears by its “honest, transparent” two-way review system, in which hosts and guests get to rate each other.
What does it mean when all the hosts are great—and all the guests have a wonderful time?
That’s one of the questions about the fast-growing online sharing economy that Georgios Zervas, an assistant professor of marketing at Boston University’s Questrom School of Business and a fellow at BU’s Rafik B. Hariri Institute for Computing and Computational Science & Engineering, has been exploring with two colleagues from BU College of Arts & Sciences: John W. Byers, a professor of computer science who is also a Hariri fellow, and Davide Proserpio, a PhD student in computer science. The title of their recent working paper on Airbnb ratings is telling: “A First Look at Online Reputation on Airbnb, Where Every Stay is Above Average.” In an analysis of over 600,000 Airbnb properties, they found that nearly 95 percent of the listings scored either 4.5 or 5 stars, the highest possible rating. They contrasted this with the ratings of about half a million hotels around the world listed on TripAdvisor, where the average is a much-lower 3.8 stars, and there is more variation across individual reviews. One important distinction the BU researchers took into account: unlike Airbnb, TripAdvisor allows guests to rate hotels, but not vice versa.
Considering properties by accommodation type and location, Zervas (GRS’12) and his colleagues found considerable variability in Airbnb ratings. They also analyzed several thousand properties listed on both platforms and found that even though the average ratings on Airbnb and TripAdvisor are similar, proportionally more properties receive the highest ratings on Airbnb than on TripAdvisor. They found only weak correlation in the ratings of individual cross-listed properties across the two platforms. The number of cross-listed properties rated 4.5 stars or above is 14 percent higher on Airbnb than on TripAdvisor. The number that receive 5.0, the maximum rating, is 18 percent higher, and Airbnb guests were much more enthusiastic and willing to overlook problems.
Case in point: the budget-friendly Hotel Tropica, in San Francisco’s Mission District. On TripAdvisor, the reviews were hardly raves. “Bug bites!” one guest complained. “Extremely expensive for the low quality,” another wrote. Of the 39 reviewers, 18 ranked the Hotel Tropica as “terrible.”
On Airbnb, 45 people who stayed at the very same Hotel Tropica gave three different rooms an average of four stars. They downplayed any complaints. “Beds were comfortable and relatively clean (some hairs on pillows etc.),” one guest wrote. Another review: “Rooms are very, very noisy, but comfortable.” Yet another: “It was very hot at night so we left our windows open so we heard a lot of noise, but it was manageable!”
Why the difference in ratings between Airbnb and TripAdvisor? Zervas and his colleagues suggest that when guests and hosts are rating each other—and when the transaction feels more personal, as it does on Airbnb, or in the case of Uber, the online ridesharing platform—both parties tend to be more charitable. On Airbnb, the hosts have the option of rejecting a potential guest, so a guest needs favorable reviews just as much as the host does. Reputations are at stake on both sides of the transaction.
The researchers recently completed another study examining Airbnb’s impact on the hotel industry in Austin, Tex., a center of Airbnb activity. They found that a 10 percent increase in the number of available Airbnb accommodations results in a 0.35 percent decrease in hotel room revenue, and lower-priced hotels and hotels that don’t cater to business travel are reducing prices to stay competitive. In earlier research on Yelp, the online restaurant review platform, Zervas and his co-author, Michael Luca, an assistant professor of business administration at Harvard Business School, found that at least 16 percent of the reviews are deemed by Yelp as suspicious and not published on the site. They found that the worst offenders are restaurants seeking to offset negative write-ups.
BU Research sat down with Zervas to talk about how consumers should view Airbnb ratings, how staying in someone’s space—or sharing their car—is much more intimate than, say, buying someone’s computer on eBay, and why a TripAdvisor hotel guest might trumpet bedbugs in their online review headline while the Airbnb-goer at the same hotel notices only “a few hairs on the pillow” and tucks them discreetly inside a parentheses.
BU Research: Can you explain the part of your study where you looked at cross-listed properties on both Airbnb and TripAdvisor?
Zervas: Some hotels—a small fraction—list their properties by the room on Airbnb. So, in our study, we looked for properties that are cross-listed on Airbnb and TripAdvisor and then we compare and contrast their ratings. And we found first that on Airbnb, they get higher ratings—and, second, that somehow these ratings are not correlated.
What do you mean they’re not correlated?
Suppose that we take all these cross-listed properties and we rank them by average rating on Airbnb. Maybe Hotel Tropica is first, some other place is second. Then we look at the same properties on TripAdvisor. Well, the rankings are completely different.
It’s as if I give my students random grades for the midterm—this is hypothetical—and I also give them random grades for the final and then I ask ‘Did students who did well on the midterms do well on the final?’ Is there a correlation? Of course not, I gave them grades randomly.
So I’m not saying ratings on Airbnb and TripAdvisor are random, but this is more or less what we observed—a very weak association. So the good places on Airbnb are not necessarily the good places on TripAdvisor—and the bad places on Airbnb are not necessarily the bad places on TripAdvisor.
So, are the rankings on Airbnb meaningless?
They’re not meaningless, but it does raise some concerns. One thing we have not ruled out is that people with different tastes use different sites. This is something we’re exploring. I do believe that Airbnb users have different tastes than TripAdvisor users, but I don’t think that explains every observation in our data. I cannot believe that Airbnb users would like dirty places or bedbugs, so the question now is: Are these exceptions? Is the phenomenon more widespread? We don’t have an answer yet, but we do notice this pattern where the ratings are really all over the place. These are the same properties, but they’re ranked in a different way across the two sites. Can this be attributed only to differences in tastes between Airbnb users and TripAdvisor users, or is there something else going on?
Is it possible that on one day, the place really was better than on the other day—like with restaurants?
But why would it be consistently better on days when Airbnb users are visiting?
Oh, I see your point. So, you bring this out in your study: if people are rating each other—if the host is rating the guest and the guest is rating the host—are they preempting each other?
Yes. This is what happened on eBay. Sometimes on eBay, you would hear stories that there was communication between the two parties after the transaction—possibly, both parties were saying, ‘Okay, you better give me a good rating if you want a good rating back.’ So I think, depending on how pushy you are, you can play that strategy on Airbnb.
So do you think we need professional reviewers—like, say, Pete Wells, the restaurant critic for the New York Times? It’s his job to be objective.
I don’t think Pete Wells could review every room on Airbnb. We would need many Pete Wellses….The goal of any reputation system, whether it’s Pete Wells or Yelp, is to provide an accurate, unbiased evaluation of the product or service for future consumers. That’s what we use reviews for. I don’t think anybody would disagree with that. And here, what is puzzling is the fact that the properties are rated in a different way across two sites. So, which ratings should you trust?
You point out that unlike the individual Airbnb hosts, or entrepreneurs as they call them, a big company can hire a marketing person…
Yes. Unlike firms with large marketing budgets, few of these individuals have an outside source of reputation. If you’re traveling, and I’ve booked you a room at a Hilton, you know what you’re getting, right?
Why is that? You’ve seen their ads, maybe you’ve stayed at a Hilton before, right? There’s brand recognition. If I tell you I’ve booked you an Airbnb at the corner of Commonwealth and Mass Avenue, what does that tell you?
That’s the point we’re making here. So, for people on Airbnb, reviews are more important. This point is nicely illustrated in a paper by my co-author Mike Luca, of Harvard Business School, on Yelp. Mike studied the impact of Yelp ratings on restaurants. He showed, as you would expect, that restaurants with higher ratings get more consumers. But the effect is not nearly as strong for chain restaurants because, again, are you really going to check the reviews for Starbucks or McDonald’s?
Right. These products are standardized, they’re advertised heavily, we see them everywhere. You know that quality is pretty much fixed. For these companies, online reviews don’t really matter as much. That’s the point we’re making here. For a Hilton, one damaging review—okay, fine, they can deal with that. For a single person running a small Airbnb listing, one damaging review might make them exit the market.
So, I hate to ask you this again, but how do you figure out what the ratings and rankings on Airbnb really mean?
I think the meaning is more hidden and more nuanced. Probably it’s best to invest some time reading a bunch of reviews on Airbnb to get a sense of the language Airbnb reviewers use to describe Airbnb properties, and how hints they leave in their reviews should be interpreted. At the same time, there are other ways in which you can encourage more honest reviewing.
Maybe you don’t need bilateral reviews, or at least you don’t need them in that particular form. So here is something that Airbnb actually did: previously, the process was sequential. Let me explain. You come, you stay at my Airbnb listing, the transaction is complete. You go back and within 14 days, I can rate you and you can rate me. If you rate me first, I can see your rating and then if I don’t like your rating, then I can give you a negative rating back….
Even if you don’t deserve a bad rating?
Even if you don’t deserve it. Nobody is going to question that. It’s 100 percent punitive—one star for you, just because you gave me a low rating. I can do that. So, what’s an easy way to solve this problem?
Well, maybe don’t tell me what rating you left for me before I rate you back, and when we both rate each other, reveal our ratings simultaneously. So that’s what Airbnb did in July 2014. It remains to be seen how well that works out for them. This is something we’re measuring and we want to see how reviews and ratings evolve under this new scheme. Of course, there are other problems…
There are other issues that this arrangement might not solve. For one, on eBay, it was harder for us to communicate, but now you’re in my car, you’re staying in my place. We can make an arrangement in advance. It’s very easy for me to guess what sort of rating you’re going to give me.
I can see you. You’re in my car; I’m giving you a ride. I know if things are going well—maybe there’s too much traffic, maybe I was a bit late, maybe I got lost.
Maybe the car was dirty?
Maybe the car was dirty. I may not be very impartial in the way I rate you, but maybe I have an objective view of how things went. One thing that surprised us— even on TripAdvisor, which uses a unilateral review system and you don’t have this concern about strategizing, reviews of vacation rentals, similar properties to those found on Airbnb, are rated a lot higher than hotels.
What does that mean—that it’s about the experience?
The average vacation rental on TripAdvisor has something like 4.5, 4.6 stars. The average hotel has 3.8 stars, which is kind of puzzling, and we think what might be happening here—and this is complete speculation—is that people might feel a little bit worse rating some sole proprietor or some freelance driver, or whatever, rather than some big global brand. You know that it’s this guy’s business, he’s driving Uber, or he’s renting out his place on Airbnb. Things may have gone wrong, but you’re more charitable in your evaluations. If it’s a big brand—an airline, or Comcast, or something like that you—you don’t care so much.
I don’t care at all.
Exactly. But I think when we rate other people, we are more charitable. Sometimes when we’re rating a service, we’re actually rating the person. Again, this is purely speculative. You read Airbnb reviews and what do they always mention? The host—they say so-and-so was a wonderful hostess, so-and-so was a wonderful host, a great person. They were so warm, they welcomed us, they gave us wonderful advice, they told us where to go, we had such a great time, blah, blah, blah. So it’s more about the personal experience. This is what Airbnb advertises; it’s all about making friends all over the world.
Do you have any plans to look at Uber and how their reviews work?
We’ve thought of that, but collecting data to study Uber is a lot more difficult because it’s an app. It’s not on the web. It’s not like you can go to a website where they list reviews of every single driver.
What are you working on next?
This study is far from complete; it’s just an extended abstract. We want to take this further. We’re trying to look at how the particular tastes of Airbnb and TripAdvisor users differ—and if this taste differentiation alone can explain the ratings differences across the two sites. Another thing we looked at two years ago, and we got nowhere and we want to revisit now, is the following: On Airbnb, and really on any peer-to-peer platform, there are hosts with good ratings and hosts with bad ratings, and there are guests with good and bad ratings. Is it the case that hosts with high ratings are matched with guests with high ratings? Then, if you have a bad rating as a guest, do you have to book the crappy room? Is there assortativity—matching people of the same status?
If you have a bad reputation, do you essentially have to pay more for the experience? Remember, on Airbnb, if you’re the host, it’s not like I can automatically book your listing. You can reject me at no cost. How do you make this determination? Maybe by my picture, some basic things about me, maybe I have had bad reviews. Who is most likely to reject me if I have had bad reviews? The host who has very good reviews. They get a lot of requests. So where do I end up staying?
The Hotel Tropica! Speaking of hotels, does your most recent study—on Airbnb competition driving down hotel rates in Austin—show that Airbnb competition can be good for hotel goers?
Some travelers substitute Airbnb for hotel rooms. This brings hotel prices down. As a result, other travelers who prefer hotels also benefit.
There has been a lot in the news here about Boston and the 2024 Olympics. People are already saying—and they’re not entirely joking—that if the city does get the Olympics, they’re going to personally cash in by listing their houses or apartments on Airbnb for a lot of money. Do you think there will be an Airbnb frenzy if Boston gets the Olympics? How high do you think the rates could go?
I’m not sure how high rates could go, but I definitely expect a surge in Airbnb use. This is something we saw with the most recent World Cup in Brazil. Many fans used Airbnb as an alternative to hotels.
What about you? When are you going to try Airbnb?
Most of my travel these days is conference travel and it doesn’t make sense to stay outside the conference hotels in an Airbnb. And when I travel for vacation, I usually go to Greece and stay with my parents.
Extreme positive bias is symptomatic of both Airbnb, and marketing in general. Even to the point of self-marketing platforms such as Instagram.
But it’s worrying that Airbnb can’t provide an accurate review system, given how important reputation is on that platform.
Plus there are a great number of risks that people rarely even pause to consider before using it, swept along by the assurances of safety and fun.
We’ve compiled a few of the steps hosts can take to ensure their property doesn’t turn into the next Airbnb online horror story: https://legaltemplates.net/airbnb-horror-stories-how-to-avoid-being-next-in-line/
I am a USNA grad and an Operations Analyst by degree. I didn’t see the key ingredient that I believe addresses the question of why Airbnb guests give much higher ratings than you see on hotel sites and others.
The issue is the “Accuracy” measurement. I travel a great deal and the killer for hotels is they always include pictures of their best features, dressed up and well lit. When you arrive, the room is poorly lit. The room isn’t as clean as represented. On item after item, the entire experience doesn’t match the presentation.
I can offer two experiences that I believe demonstrate exactly what happens on Airbnb.
The first was an Airbnb stay near the Arc de Triomphe in Paris. The apartment was not fancy and could have been in quite a bit better condition. Nevertheless, the host described it very accurately and the experience was great because it matched our expectations. Result – 5 stars across the board. In short, we gave a pass on many things because of the integrity of the hosts representation.
The second experience was a Airbnb listed bed and breakfast in Maryland. The host had represented the bed and breakfast much like the hotels. The experience totally failed to match our expectations. We ranked the experience down on every applicable issue. We gave top marks for all items that matched our expectation (cleanliness for example). Had the host been honest and not included photos that seemed to lie about the home, we likely would have given generous ratings.
In my experience, hotels and the sites that generally support them aren’t long on accuracy as you move down the star rating. This is in my opinion the root cause issue.
I can offer yet another view of the incredibly high AirBnb rating. A bad rating is bad for both the host and the guest. So people tend to book places with great score and as well hosts tend to pick up people with a great score.
Even if my staying was just OK, if the host had been reasonbly nice and responisve, I will tend to give him an over-avarage review. I do as well compromise on a lot of issues that I will not compromise on in a hotel.
I as well aim to have no issues with boocing an AirBnb so I try to have a good score. But the score is actually more important to the host, so host tend to prefer people who always give good socres to people who had never gotten a suspiciously sounding remark. So even if I know that things had not been perfect and the host will very likely addess an issue on which we didnt exactly get on, I will still tend to give a better review unless the stay had been truly horrible.
So, the main problem is the psychology of AirBnb reviews. It results in useless rating which actually plays against the guests: it is much easier for a host to avoid a bad rating as it is for a guest and the guests need to invest time screening reviews written in different languages, or in English but from foreigners and try to read between the lines how brilliant the place actually was.
I was fascinated to read this article. I am retired and have been traveling around Mexico for the last three years and have stayed in almost 40 Airbnb’s. I read the reviews but view them with a very small grain of salt. Standards are completely different in Mexico and other Latin American countries than that of the US and many other places. So when the listing says that COVID precautions are taken, I pretty much know that facemasks are worn when you check-in, but after that their not used. Also, a comfortable workspace is nothing more than a childlike desk with a hard uncomfortable chair. And a firm mattress in Mexico is almost rock hard in the US. Sanitation standards would receive a “D” rating in other countries, and when the listing says that they enforce a quiet and tranquil atmosphere, well, that tells me just the opposite exists. When the reviews are written by locals they are almost always favorable toward both the accomodations and the host. When foreigners write the reviews, very seldom are the accomodations mentioned, but instead they give high ratings to the host. I have read some negative reviews, and I am surprised that they even show up. The negative reviews I have given in the past were either deleted or summarized to the point where they lacked any meaning. The point made about the two-way review system is totally valid. I will not lie on a review, but I can’t afford to have a lot of negative reviews in retaliation which would make a potential host reject my stay request. So I stopped giving reviews quite a while ago. I have come to the conclusion that Airbnb is not legally held to the same standards as a hotel would be. The listings will quite often use wide angled pictures to make the rooms look bigger, and they will omit pictures of areas that should be condemned. The descriptions of both the dwelling and the area are mostly false. If a hotel did that, they could lose their license and a blanket lawsuit would occur. The only reason I have stayed in so many Airbnbs for so long in Mexico is because of the pandemic. Now that other countries are opening up again I will be leaving.