Do you ever get frustrated when you search for something on Google, but end up getting confused by different answers from different websites? Well according to many news outlets, ChatGPT is able to save you the hassle of searching and just present you with the correct answer to your question. But, how good is it really?

To check, UK Radiators asked their team of heating experts to ask ChatGPT the most searched heating questions in the UK. Here’s what they found out.

Testing ChatGPT On The UK’s Most Searched Heating Questions

The easiest way to test whether ChatGPT can replace a heating expert, is by asking the most common questions our experts get asked so we used Ahrefs to gather the most-searched-for heating-related questions in the UK and fed them to ChatGPT. Here were the questions: 

  • How Much Does Underfloor Heating Cost? 
  • How Do Heat Pumps Work & Are They Worth It?
  • How Much Is The Heating Allowance & How Do I Claim It?
  • How Can I Heat My House For Free?
  • How To Heat Your Home Cheaply?
  • Should I Leave the Heating On All Night In Winter?
  • Why Is My Radiator Not Heating Up?
  • What Is The Best Time To Turn On The Heating?
  • What Is Central Heating?
  • Why Is My Heating Working But I Have No Hot Water?

Analysis

ChatGPT surpassed our expectations by providing a lot of very accurate information in a digestible way. However, we found that ChatGPT did not yet have an understanding of exactly what information a person asking the question would need, while our heating expert was able to add a bit more context to the questions and answers. 

ChatGPT’s Strengths

When presented with definition questions, ChatGPT is able to explain any heating topic in a simple and understandable manner. The study found that the chatbot is able to present very accurate information on topics like heat pumps and central heating and explain those topics to absolute beginners. 

ChatGPT is pretty good at providing the point of view held by the majority of experts on a topic. While answering the question about underfloor heating costs, it provided a decent estimate given how vague the question was. It also does a pretty good job of explaining the reasoning behind the answer as it did with the heating at night question. A lot of experts debate whether it’s best to leave the heating on at night or not and ChatGPT did a good job of explaining the debate and which viewpoint is more widely held.

It also showed an understanding of complex concepts such as the dynamic between money and heating and provided excellent actionable tips to save money on heating. Overall ChatGPT showcased that it was able to provide very quick and easy definitions or tips on a variety of heating topics which means that it can definitely replace Google in that regard, but can it replace a heating expert? The following points show that maybe not.

ChatGPT’s Weaknesses

One major issue ChatGPT has is that it currently can’t comment on timely events as it has limited knowledge of any events that took part after 2021. This means that if any advice has been disproven or if you’re trying to find out the solutions to a current problem, it will not show the correct answer.

ChatGPT is very direct with its answers, which can be a good thing in many scenarios, but when someone is new to a topic it is important to get a more fully rounded answer as people might not know what to ask. That’s why when we compared ChatGPT’s to our expert’s answer ChatGPT was concise and accurate but did not offer as much detail as the expert’s answer. In areas such as heating, need an expert to be able to understand the client’s problem fully before they offer advice which makes ChatGPT harmful in some cases.

Another major issue with ChatGPT is the lack of sources and citations which means you can’t verify how accurate the information is or where it came from. This is very dangerous when it comes to areas such as heating as this might lead to wasted money or even damaged heating equipment due to misinformation which could easily be avoided if you know the right expert to contact.

Is ChatGPT ReadyTo Replace Experts?

ChatGPT scored 68/120 on our test which is slightly over 57% so we asked an expert in AI to evaluate how ChatGPT did compare to our expert to understand more about AI and its limitations.

Scott Jones, web expert and CEO of the UX agency Illustrate Digital, had this to say:

“Conversing with ChatGPT can be a dauntingly realistic experience. Ask it for help on how to save money on energy bills, and you’ll get some surprisingly human-like replies. But, of course, the limitation of any AI-based tool is that it’s merely artificial; learning primarily from its conversations with real humans and growing its database to provide more accurate answers over time. The question remains though – what is the source of this data, and how relevant is the information provided?”

“When it comes to getting advice from ChatGPT on saving money, it’s worth considering that its responses are only as reliable as what it believes humans would say. So the best chance of finding accurate savings tips still lies with the likes of MoneySavingExpert, who have built their reputation on giving sound financial advice. They continue to provide detailed guidance on ways to cut your costs and make sure you’re getting the most out of your finances – something ChatGPT may never be able to match due to its ever-evolving nature as an artificial learning tool.”

“ChatGPT is also pre-trained, which means it’s not connected to the internet and doesn’t have access to the most recent updates in the news. Just try asking it a question about anything that happened in the last week or even month, it’ll likely tell you that its data only runs up to 2021 and that you should check news sources for the most recent information.”

“Whether AI can reach a point where it can fully replace all our experts is the great debate around ChatGPT right now, but I would argue not… Whilst it’s easily the most powerful chatbot ever created, it’ll only ever be as good as the data it’s fed. Experts combine knowledge, relevance and context to provide expert opinions. Whilst ChatGPT will become increasingly accurate with facts and handling common conversations such as support requests, it’s unlikely to ever reach a point that it replaces experts entirely. After all, someone still needs to feed AI the information it needs to reach greater levels of accuracy.”

In conclusion, we think ChatGPT is a great tool that people can use to find quick definitions or tips, but when it comes to making investments in your house or understanding more about your heating system, it is still best to ask a heating expert.