I specialize in this area and build a product for self hosted inference.

The challenge to support a new model architecture is about coding the preprocessing for inputs (like tokenization or image resizing and color feature extraction) and post processing the outputs (for example entity recognition needs to lookup the entities and align the text).

Once an architecture is coded for the pre/post processing, then serving a new model for inference with that architecture is easy!