Mesa 21 Broke VAAPI Hardware Encoding On Machines With AMD Graphics Cards. A Fix Is Coming.
Those of you who have tried to hardware encode video using AMD hardware after upgrading to Mesa 21 may have noticed that the encoded video is, in fact, not a video - it's some kind of odd slide-show with one new frame every 5 seconds or so. Mesa 20.3 didn't have that problem, Mesa 21 and the newly released 21.0.2 bug-fix do. The Mesa developers are aware of the problem we are referring to and a patch has been merged into the upcoming 21.0.3 version scheduled to be released around two weeks from now.
FFmpeg hardware encoding a video using the VAAPI interface the Mesa graphics library provides.
The streaming/recording studio OBS Studio and the handy command-line video encoding utility FFmpeg can both be used to encode video using hardware encoding via VAAPI if the hardware supports it. The newly released FFmpeg 4.4 added support for even more hardware encoding options. The software/driver-side of hardware encoding support is provided by the Mesa graphics library if you are using AMD or Intel graphics hardware.
A video file can be encoded to MPEG-AVC (x264) video with a fine FFmpeg command such as:
ffmpeg -i inputfile.mkv \ -vaapi_device /dev/dri/renderD128 \ -c:v h264_vaapi \ -qp 21 \ -vf 'format=nv12,hwupload' \ -b:a 128k \ -c:a aac -strict experimental \ outputfile.mp4
Hardware encoding with FFmpeg by using a
-vaapi_device /dev/dri/renderD128 device and
-c:v h264_vaapi as the codec worked just fine on AMD hardware with Mesa versions up to 20.3.5. That is, sadly, not the case with the shiny new Mesa 21 release and the recent Mesa 21.0.2 bug-fix release for it didn't include a bug fix for this particular issue.
We hardware encoded a few videos using the command shown above on a machine with Intel graphics hardware and Mesa 21.0.2. They turned out just fine, so this is a very AMD-specific problem.
The good news is that a patch that fixes the problem was merged to Mesa master late last night. Mesa 21.0.3 won't have this problem. The bad news is that Mesa has a bi-weekly bug-fix release schedule, and Mesa 21.0.2 was released this week. Those of you who have AMD hardware and a distribution shipping Mesa 21 as the default graphics library will, therefore, just have to live without hardware video encoding for a few weeks. You are, on the other hand, completely unaffected if you are using a distribution shipping Mesa 20.3.x since that branch doesn't have this problem, and Mesa 21 will likely have been fixed for a long time when your distribution eventually updates Mesa to the 21.x.y branch.