How Yelp reviewed competing LLMs for correctness, relevance and tone to develop its user-friendly AI assistant

View application Windpian Religions and other consumers gave us useful information for decades. Practice with machine learning since the first year. During the final explosion in AI technology, he still faced the stumbling blocks as he worked to hire major language models to strengthen some features.

Yelp, clients, especially sometimes, have realized that those who use the application Contact the AI ​​featuresFor example, an AI-power assistant.

“One of the open lessons we see is very easy to build something cool, but it is very difficult to build something cool and very useful,” said Craig Saldanha, Head Product Officer in Yelp, said in an interview.

Of course, everything was not easy. In April 2024, Yelp Assistant, Assistant of the EU service, the figures used by customers to a wider interrogation, the figures used for AI tools began to decline.

“Something that surprised us, when it launches consumers like a beta – it was a few users and people who are very familiar with the application. [and they] loved. We have received such a strong signal that we will be successful and then rounded to everyone, [and] The spectacle just fell, “said Saldanha.” It took a long time to understand why. “

It turned out that Yelp’s more casual users, sometimes visiting the application to find a new tailor or a new tailor or plumber, did not expect him to talk immediately with the AI ​​representative.

Of the AI ​​features involved more than

Most people provide information as a website and application to search for restaurant reviews and menu pictures. I use Yelp to find pictures of cookies in new canteets and seeing others that others have a food especially a dish. It is a place where I plan to use a coffee shop, WiFi, chips and sites, which I plan to use as a work area.

Saldanha remembered Yelpin invested in the AI ​​for “for a ten-year part.”

“When I returned, I would say in 2013-2014, a part of a meaningful connection helps people to determine their search intentions,” he said.

As AI continues to develop, Yelpin’s needs did. To identify popular foods to AI, to identify the food in the pictures provided by users and then launched new ways to connect to businessputes and services and Help users search for platform.

Yelp assistant helps to find the correct “Pro” to work on Yelp users. People can hit the chat box or use the desires or write the necessary task. Then it provides an explanation to narrow potential service providers before preparing a message for advantages to offer to offer for auxiliary work.

Although Saldanha admits that the larger brands are often managing the messages that manage messages created by Yelp AI Assistant.

In addition to the Yelp Assistant, Yelp examined research and moments. This analyzes the feelings of LLS, user and reviewer, which is also gathered in sentiment scores. Yelp uses details GPT-4O offer Create a database for a list of topics. Then, it is well set up with a GPT-4o-mini model.

Overview Summary, the feature that provides information from comments, also uses an LLM offer to create a database. However, in GPT-4, GPT-3.5 is based on GPT-4 with subtle regulation from Turbo. Yelp said the feature would update with GPT-4O and O1.

Yelp joined many other companies Using llms to enhance the benefits of Reviews by adding better search functions based on customer comments. For example, Amazon Rufus has launchedAi-Energy Assistant to help people find the recommended items.

Great Models and Performance Needs

The new AI features, including AI Assistant Yelp, turned to Openai’s GPT-4Os and other models, but Saldanha is a hidden sauce, regardless of the model. Yelp did not want to close himself with a model and which LLS would be the best service for customers.

“We use Openai, Anthropic and other models in AWS Bedrock,” Saldanha said.

Saldanha explained that Yelp created a rubric to test the performance of models in correctness, relevance, consciousness, customer safety and compatibility. He said, “It’s really best models” performed the best. The company wants the ITERATY decision and the answer to the result of the answer.

Teach users

Yelp made an agreed effort to teach both casual and power users to be comfortable with new AI features. Saldanha, especially with the AI ​​Assistant, especially one of the first things that man had to feel man, he said. Failed to respond very fast or too slowly; Excessive encouraging or not very brusque.

“We made a bunch of efforts to help people feel comfortable, especially in this first answer. As soon as we did, this hockey stick can see that the hockey stick was engaged,” Saldanha said.

Part of this process has prepared a helper assistant to use certain words and sound positive. After all this wonderful adjustment, Saldanha said they finally saw higher use of Yelp for AI features.