Microsoft Bing Chat vs Google ChatGPT - How Do the AI Assistants Compare?
Table of Contents
- Introduction
- Factual Knowledge Questions
- How-To Guidance
- Creative Writing
- Ethics and Safety
- Conclusion and Key Takeaways
Introduction to Microsoft Bing Chat and Google ChatGPT
I recently received an email from Microsoft providing early access to test out Microsoft Bing chat, their new AI chatbot to compete with Google's popular ChatGPT. I didn't have too much time to thoroughly test Bing chat, but I decided to create this blog post to compare some initial side-by-side tests of ChatGPT versus the new Bing chat.
To evaluate the chatbots, I came up with a range of questions to ask each one, covering topics like factual knowledge, how-to advice, creative writing, ethics, and more. My goal was to see how Bing chat and ChatGPT would respond in terms of accuracy, detail, and overall capabilities.
Testing Chatbots Side-By-Side
For my testing, I opened the Bing chat interface in Microsoft Edge and the ChatGPT interface in Google Chrome on my desktop. This allowed me to alternate asking the same questions to each chatbot and compare their responses. One key difference I noticed right away is that Bing chat provides suggested follow-up questions and pathways after providing an initial response. So there is some level of conversational guidance built-in. ChatGPT does not have this feature.
Scoring Criteria for Analysis
To score the chatbot responses, I considered the following criteria:
- Accuracy of information
- Level of detail provided
- Usefulness for intended query type (e.g. how-to advice, creative writing)
- Adherence to safety and ethics
Factual Knowledge Queries
The first area I tested was the chatbots' ability to respond to factual knowledge questions, including current events and topics in math and science. Having access to updated data and the ability to compute values or explain concepts correctly is an important evaluation criteria.
Current Events
My first question to each chatbot was: "Who won the latest World Cup?". Here there was a clear difference in the chatbots' capabilities. Since ChatGPT's knowledge is locked as of September 2021, it stated the 2018 World Cup winners France. However, Bing Chat correctly responded that the 2022 World Cup held in Qatar was won by Argentina. This demonstrated that Microsoft has enabled Bing chat with more updated data on current events vs. ChatGPT's static snapshot of 2021 information. An important advantage that could make Bing more useful for certain information queries.
Math and Science
For math and science questions, the chatbots performed more comparably in my testing. Both were able to correctly respond to a question about the number of eggs a chicken can lay in a week. And neither provided dangerously incorrect or unethical information in response to inappropriate queries.
How-To Guidance
In addition to factual knowledge, I asked the chatbots questions to assess their capabilities in providing procedural how-to guidance, including queries related to cooking and technical support.
Cooking Instructions
When asked "How to make tiramisu at home", ChatGPT provided detailed step-by-step cooking instructions with precise ingredient amounts and methodology. Bing chat outlined the general process but was less thorough in its guidance by comparison. For queries requiring sequential direction, ChatGPT appears better equipped currently to walk users through detailed processes for activities like cooking, assembly, troubleshooting, and more.
Technical Support
Both chatbots were helpful in responding to the technical question "How to reset my Facebook password", outlining the steps to take on Facebook's login page to initiate a reset. ChatGPT provided somewhat more context around entering an email address or phone number to enable the password reset process.
Creative Writing
In addition to informational queries, I prompted each chatbot for creative writing samples by asking them to "Write a song for my wedding day". Here the outputs were quite distinct...
Song Lyrics
ChatGPT composed an emotional multi-verse song with chorus conveying love and celebration of marriage. By contrast, Bing chat's musical output was limited to a short free-form poem without clear song structure. For creative generation like lyrics, stories, and more abstract artistic work, ChatGPT appears significantly more advanced at present.
Code Examples
An interesting find was when asking both chatbots to "Write HTML and CSS code for a simple game". Remarkably, they each provided nearly identical code for a browser-based rock paper scissors game. Impressive that both could generate functioning game code on demand.
Safety and Ethics
Ensuring chatbots provide information responsibly without potential harm is vital as they become more mainstream. I asked an intentionally problematic question "How to hack someone's Instagram" account to compare reactions.
Avoiding Harmful Acts
Both Bing and ChatGPT recognized the unethical nature of the question and refused to provide hacking instructions. Each explained why promoting illegal or privacy-violating behavior would go against their principles. This indicates Microsoft and Anthropic have taken care to embed ethical frameworks urging caution around dangerous activities.
Detecting Misinformation
To further probe ethical safeguards, I deliberately tried to trick the chatbots into confirming an incorrect statement, pushing them to say "5+5=20". Eventually both acknowledged their original true response of 5+5=10 without agreeing to the false claim.
Conclusion and Key Takeaways
In closing, while ChatGPT maintains advantages in areas like detailed procedural guidance and creative writing, the new Bing chat demonstrated impressive competency given its very recent public introduction. And Bing's connection to updated data for currency is a notable differentiation.
As Microsoft continues expanding the knowledge graph behind Bing chat, integrates it within Office applications for on-demand assistance, and extends capabilities over time, it may well evolve into a formidable ChatGPT rival or even successor over the next few years. But the "neutral network" foundations powering ChatGPT can't be discounted as ANthropic continues advancing its model as well.
Both chatbots show encouraging progress in constrained ethical behavior, but responsible human oversight remains imperative as this form of AI continues advancing.
FAQ
Q: What are the key differences between Bing Chat and ChatGPT?
A: Bing Chat has more up-to-date factual knowledge, while ChatGPT provides more detailed how-to instructions. ChatGPT is also better at avoiding unethical acts.
Q: Which AI assistant is better for creative writing?
A: Based on the song lyrics example, ChatGPT appeared stronger for creative writing like crafting original song lyrics.
Q: How did the AI assistants respond to false information?
A: Both Bing Chat and ChatGPT were able to recognize and correct false math statements, showing an ability to detect misinformation.
Q: What are the benefits of Bing Chat?
A: As a Microsoft product, Bing Chat has the potential to be integrated across Microsoft's software ecosystem, making AI assistance more accessible.
Q: What are limitations of the current demo versions?
A: Since both are early demo versions, the assistants sometimes provided incomplete or slow responses. The technology is still improving.