Lawfare did a nice analysis of whether Section 230 shields model hosts (likely no): https://www.lawfareblog.com/section-230-wont-protect-chatgpt Professor Volohk has a public draft article on liability for output of AI models, with an emphasis on defamation: https://www2.law.ucla.edu/volokh/ailibel.pdf

Both suggest that this is a real area of concern. Folks have gotten complacent that Section 230 protects platform providers, but it is not clear that the protection extends to outputs from generative models, indeed one of the Supreme Court Justices implied that it would not in oral arguments for one of the recent Section 230 cases.