I run Jellyfin on my home server, and this has been a problem for me as well. Even within my home network I'll have occasional stuttering while watching movies, despite the fact that I'm using a dedicated server with 128gb of RAM and 12 CPUs, with a 10 gigabit network card in a 10 gigabit switch, and I'll still get stuttering.
I've debated moving to Plex but I'm broadly against running proprietary software on a server (for reasons I cannot honestly articulate terribly well).
During transcoding, but I'm having a hell of a time disabling that. I've looked around for a configuration to just broadly disable transcoding, since pretty much everything I use to play JellyFin is libavcodec or ffmpeg based anyway and can happily decode stuff client side.
It depends on the client's (the one you are streaming to) range of codec support. If you are streaming to the TV that your TV don't have specific codecs that it can decode, then it will transcode to the TV specification. If the file is in AVC (2.654) but the audio codec is not supported by the TV, then Jellyfin will transcode it. If you are watching the stream via the browser (HTML5 player), it have a limited range of codec it can support hence the transcoding. If your source file are in HEVC/AV1, browsers don't have a way to decode it. Jellyfin's ExoPlayer also don't a way to decode HEVC/AV1 and limited range of audio codecs.
If you want DirectPlay, then it best to use the external player for it. You have to set it up in the setting in Jellyfin to use the external player. For Android, Jellyfin can DirectPlay to MX Player Pro. I use Jellyfin in my Android tablet and set MX Player Pro as external player, it works flawless without shuttering.
Also I learn that it is best to enable QuickSync if you have Intel. With QS, it can transcode effectively this way.
Edit I saw your reply to other commenter. Browsers ("web-native") does not support MKV, unfortunately. If you want to run it in the browser without transcoding, the file need to be in MP4.
Use the Desktop app or mobile app. Chrome/edge don't support the codecs needed, and jellyfin transcodes. The desktop app is chromium, but with the right video codecs enabled, on it. It's pretty nice.
I'm not certain, but I think it will avoid transcoding for certain "Web-native" formats that your client device can already play. For example, if you download an mp4 then you should be able to watch it on your iPhone without Jellyfin needing to transcode it.
Every file on my server is a raw blu-ray rip using an MKV container. I don't compress or transcode anything myself, everything is a raw rip from MakeMKV. I suppose I could write an ffmpeg batch job to convert them to MP4 if I wanted, but MKV is really convenient since it'll keep all the subtitle and multiple audio tracks within the file instead of a bunch of SRT files floating around.
Also, while a majority of blu-rays are h264, a non-zero number are encoded with VC-1 or Mpeg2, which as far as I'm aware are definitely not "web-native".
I really wish there was just a checkbox that I could click, or a setting in a config file I could change, that would just avoid transcoding and just send the file.
Apparently the overarching goal of Jellyfin is to always be able to "direct play" (no transcoding), but whether or not that's possible depends on each file and the client device. They have a comprehensive table of which formats require transcoding on which devices: https://jellyfin.org/docs/general/clients/codec-support/
Apple TV 4K (129 USD) + Infuse (10 USD/yr) is the perfect player for me. The only con is that it does not support anything atmos other than Dolby Digital Plus, but my setup is 5.1 only, so I don't mind.
Honestly, that sounds quite weird to me. I've been remotely running my instance with only 100Mb/s connection and it hasn't even struggled running 4K film. All this on a budget i3 9100. And I'm quite sure it should be able to run on even slower hardware if need be.
I misused the word "dedicated"; the server is actually used for lots of stuff. What I meant is that I didn't repurpose an old desktop or anything, it's an actual rack mounted server.
I run NixOS on there with a bunch of different services, though the only thing that comes anywhere near mattering resource-wise is JellyFin when it's transcoding.
I've debated moving to Plex but I'm broadly against running proprietary software on a server (for reasons I cannot honestly articulate terribly well).