The definition of what you're allowed to do with `unsafe`, however nebulous its specifics may be at the moment, is that such a situation is a bug in one or both of those libraries, not their composition.
To put it another way, if you can't observe unsafety with X or Y alone, but you can with both together, then at least one of them has given you a new capability that you did not have before. Either that new capability is not truly safe, and thus the bug is providing that capability, or it exposes the other library relying on something not truly safe, and thus the bug is relying on that property.
The important point here is that, by definition, at least one of X or Y will have to change when such a situation is discovered, in order to preserve the property that composing safe interfaces is safe.
To put it another way, if you can't observe unsafety with X or Y alone, but you can with both together, then at least one of them has given you a new capability that you did not have before. Either that new capability is not truly safe, and thus the bug is providing that capability, or it exposes the other library relying on something not truly safe, and thus the bug is relying on that property.
The important point here is that, by definition, at least one of X or Y will have to change when such a situation is discovered, in order to preserve the property that composing safe interfaces is safe.