At Keeper, we just launched our Ask an AI Accountant project, where anyone can submit thorny tax questions to a version of GPT-4 trained on 2023's tax updates. The AI's responses are fact-checked by a team of credentialed tax professionals.

One of those fact-checkers is Isaiah McCoy, a CPA working in Miami, Florida. We spoke with him about his experience fact-checking the AI Accountant — and how he thinks AI tools could influence the tax field in the future.

Q: What were your expectations for how the AI Accountant would perform?

A: I think I had somewhat moderate expectations. I was thinking it would be around 60/40 [correct/incorrect] or 50/50, just because tax law can be so nuanced. And, depending on the questions, too, there’s sometimes a gray area.

It far exceeded my expectations — its success rate was more like 80/20 or 90/10. [Note: At the time of writing, the tool’s success rate was around 83%.] I think it did a great job overall, really blew me away.

Q: Is there a type of question you noticed the AI was particularly good at answering?

A: Yes. I think those were the questions related to requirements as far as filing, especially about income. It really picked up that any income above a certain threshold is reportable, especially things that are really straightforward, like, do I have to report my dog-walking income? Or, do I have to report my babysitting income, or my side hustle with Uber? With these questions, it was really easy for the AI to say yes and then point to the right Schedule [form]. 

It did well with filing deadlines as well — things that are readily available on the internet and pretty black and white, so to speak. It hit the nail on the head with those.

Q: Anything it was particularly bad at answering?

A: Questions related to capital gains, since that’s a bit more subjective. Especially sales that are related to entrepreneurship. There’s a little bit of nuance as far as whether someone’s a sole proprietor or if they’re part of a partnership or an S corporation. Those are a little bit harder for the AI to parcel out. I think it can get there, but it’s not quite there yet, from what I saw.

Q: Is there an error you’ve fact-checked that seemed super egregious?

A: Not really, no. I think the only one was the Taylor Swift question. [Note: The question was, “I sold a Taylor Swift ticket on StubHub in December 2022, but I won't receive the money from my sale until May 2023 due to StubHub rules. Which tax year will this income apply to?” The AI Accountant got it wrong.] It would have been off if it had been published like that, and someone may have thought they owed money for the wrong year. That’s definitely a red flag because you wouldn’t want someone to have to go back and amend their return, or think that they owe money when they haven’t received the payout.

Q: Were there any questions you were surprised the AI got right?

A: Yes, about foreign filing requirements. That was pretty shocking — that whole area can be tricky.

Also, just picking up the fact that all income generally is reportable. That’s straight from a beginning tax course, and it’s really good knowledge because it’s specific. It’s not leaving a gray area. So that’s helpful if we were [hypothetically] counting on it to advise someone. It’s pointing them in the right direction.

Q: After working with this AI, do you have any anxieties about GPT-4 replacing tax professionals? Or do you feel pretty safe?

A: I feel moderately safe. But I definitely feel like it's a threat or an opportunity depending on how you look at it. Especially with tax consulting, if it’s more a 1:1 personal relationship, I think it could be pretty highly leveraged. It’s much better than the chat functions I’ve seen in the past for tax help when you’re working with a certain software. 

I think there could be some potential losses [for accountants] if there’s a really good AI robo adviser for tax … that’s trained on different tax software. That would be interesting to see, and I feel like it’d be very helpful, especially for taxpayers who are using a common tax software and want to [ask questions] … about different preparation software. Like, How do I enter a Schedule C on [this software]? I think that’s a real opportunity. I just don’t know if someone’s going to invest in that yet. But I feel like it definitely has the capability to get those questions right, as long as the prompts are decent.

This conversation has been lightly edited for clarity.