How Good Is Google Gemini Advanced?

Some people say it’s GPT-4 level. Others are deeply disappointed. Who’s right?

The two faces of Gemini Advanced

This article is a selection from The Algorithmic Bridge, an educational project to bridge the gap between AI and people.

Google has released Gemini Ultra. They’ve also rebranded Bard into Gemini. What was Gemini Ultra is now Ultra 1.0 (the language model). What was Bard Advanced is now Gemini Advanced (the chatbot).

Here’s a quick review of the official info in case you’ve missed the launch.

Gemini Advanced costs $19.99/month (like ChatGPT with GPT-4) and it’s free for the first two months, which is more than enough time to test it thoroughly and decide if you like it or not (try it here). You can also stick to Gemini Pro, which is free.

Gemini is available through Google One. The subscription includes other benefits like storage and seamless integration (to come soon) with Google’s services, like Gmail, Docs, Sheets, etc. (formerly known as Duet AI). Google’s goal is to integrate its best AIs with its widely-used services to outcompete OpenAI.

Gemini Advanced is also available as an app for Android (if you opt-in through Google Assistant, you can access the chatbot that way) and iOS. It’s been launched in 150 countries in English.

Google’s blog post says Gemini Advanced is the preferred option “in blind evaluations with our third-party raters,” which sounds similar to the LMSys arena but without the important part: no transparency. Like GPT-4, Gemini Advanced is multimodal, has data analysis capabilities, and improved reasoning compared to Gemini Pro.

The strange contrast among Gemini users

That was the objective part, taken directly from Google’s announcement. For a subjective overview of Gemini’s behavior, you can take a look at Ethan Mollick’s notes.

Mollick was given early access to Gemini Advanced and tested it over six weeks, comparing it with GPT-4 across prompt settings and task categories (so far he’s written about Gemini’s linguistic and reasoning skills, not multimodality or code).