Oh that's surprising. I uploaded content generated by overleaf (they provide a arXiv compatible dump of your project) a month or so ago and it worked smoothly (after uploading to arXiv I deleted the bib file of course).
I tried about a year ago and about half a year ago, respectively. Both times, the exported bbl file hat a different version than the one accepted by arXiv. First time, the bbl version was too old. Second time, the bbl version was too new. First time, I ended up printing the latex generated pdf file to a new pdf, so that arXiv wouldn't recognize it as a Latex produced file. That kinda sucked because in the printed pdf, you couldn't select text or access references. Second time that didn't work anymore because arXiv added a check to prevent users from uploading printed pdfs. I ended up simply hosting it at our institute page and abandoning arXiv.
