The Licensing Problem:

Unknowingly using potentially copyrighted code from an LLM in a FLOSS project opens it up to being sued for infringement, which most FLOSS devs can’t afford to fight, especially in the US’s current extremely pro-corporate courts.

It’s putting a target on your back for down the road when it becomes profitable for patent trolls to use AI to try to scan for copyrighted code on public code bases. Big tech companies could do the same to squash an open source competitor.

Both Haiku OS and BSD are not allowing AI code contributions for this very reason.

The Ethical problem:

Simply using an AI that's run on a corporate data centers encourages the construction of yet more data centers, with all of the environmental/climate negatives they bring, as well as local harms they induce on the people living near them, such as increased electricity rates.

Using corporate AI directly helps the financial situations of the big tech companies that host them (by boosting usage/user numbers, they are able to attract more investment capital), most of which are owned by right-wing CEO's who are more than willing to collaborate with and fund fascist governments to ensure that they are not regulated in search of both maximum profits. Some of these companies, such as Nvidia, Palantir and Oracle, genuinely appear to be seeking to use these tools for what would previously be considered crackpot conspiracy theory levels of public control and surveillance.

But if someone is absolutely determined to use them regardless, I would hope they either use a locally run LLM, or use a distributed model like AI Horde, as at the very least that does not encourage the construction of corporate data centers nor boost their financial prospects.