When I bought my B650 Lightning, Wendell from Level1Techs recommendations over the years played a big part. I haven't found a single thing to complain about.
"Mindfactory's sales information in repeated articles over several years, and its Ryzen 7 5800X, 5600X, and 5900X RMA rates are 0.58 percent, 0.52 percent, and 0.33 percent, respectively."
Based on some very rough estimates using mindfactory and amazon sales data, the 9800x3d is in line with or below 5000 series RMA rates.
If there was real cause for concern, the RMA rate should be much higher.
Any low power device is going to struggle with video playback unless you're doing direct streaming. The exception being the new Intel N line of processors that have capable enough GPUs to do the transcoding. If all you're after is decent transcoding performance for H264/H265 video, anything newer than Intel's 7-8th gen CPUs will do the job. $100-$200 used optiplexes and thinkcentres can easily handle this type of workload.
Can you be more specific about your performance objectives? The Pi5 can handle jellyfin and nextcloud relatively well, so if you're having issues with them, identifying the bottlenecks can help narrow down the hardware recommendations.
Unfortunately, this is going to be a bit of a journey. You'll probably end up going through a few of these options until you find one that works for you and fits your workflow.
I primarily use antennapod but tried abs out of curiousity. How did you get around the queue issue? I found it to be a dealbreaker