Google Stadia Dev Brings Spinlocks Back From The Dead, Blames The Linux Kernels Scheduler For Performance Problems
If you use use spinlocks where mutexes are needed, you suffer all the performance problem you deserve. The Linux kernel provides futexes (fast user-space mutexes) and you should be using those to do locking in games and other applications that require thread synchronization. Don't blame the Linux kernel if you are doing it wrong and your programs performance suffers as a result.
written by Öyvind Sæther 2020-01-06 - last edited 2020-01-07. © CC BY
A Google developer who can't code for Linux caused some drama around the Linux kernel which was picked up by some corporate blogs.
Google is in the process of porting numerous games to the Linux-based Stadia gaming-as-a-service platform. Google Stadia developer Malte Skarupke tried to investigate why a game using a spinlock implementation he wrote was stalling and wrote a rather long and tiresome blog post titled "Measuring Mutexes, Spinlocks and how Bad the Linux Scheduler Really is" explaining his self-inflicted problems with the Linux scheduler. The blog post blaming the Linux kernels scheduler for bad code has been widely publicized by some larger technology publications like Tom's Hardware ("Google Stadia Port Troubles Blamed on the Linux Kernel Scheduler").
Linux inventor and chief architect Linus Torvalds responded with two long and very detailed forums posts over at the real world technologies forum. His first response titled "No nuances, just buggy code (was: related to Spinlock implementation and the Linux Scheduler)" starts out by pointing out that the measurements done by the Google Stadia developer are meaningless:
" First off, spinlocks can only be used if you actually know you're not being scheduled while using them. But the blog post author seems to be implementing his own spinlocks in user space with no regard for whether the lock user might be scheduled or not. And the code used for the claimed "lock not held" timing is complete garbage.
It basically reads the time before releasing the lock, and then it reads it after acquiring the lock again, and claims that the time difference is the time when no lock was held. Which is just inane and pointless and completely wrong.
(..)
So the code in question is pure garbage. You can't do spinlocks like that. Or rather, you very much can do them like that, and when you do that you are measuring random latencies and getting nonsensical values, because what you are measuring is "I have a lot of busywork, where all the processes are CPU-bound, and I'm measuring random points of how long the scheduler kept the process in place"."
Torvalds further elaborated that the right thing to do on Linux is to use a kernel-aware lock like the futex system call:
"Use a lock where you tell the system that you're waiting for the lock, and where the unlocking thread will let you know when it's done, so that the scheduler can actually work with you, instead of (randomly) working against you.
Notice, how when the author uses an actual std::mutex, things just work fairly well, and regardless of scheduler. Because now you're doing what you're supposed to do. Yeah, the timing values might still be off - bad luck is bad luck - but at least now the scheduler is aware that you're "spinning" on a lock.
Or, if you really want to use use spinlocks (hint: you don't), make sure that while you hold the lock, you're not getting scheduled away. You need to use a realtime scheduler for that (or be the kernel: inside the kernel spinlocks are fine, because the kernel itself can say "hey, I'm doing a spinlock, you can't schedule me right now").
(..)
This has absolutely nothing to do with cache coherence latencies or anything like that. It has everything to do with badly implemented locking.
I repeat: do not use spinlocks in user space, unless you actually know what you're doing. And be aware that the likelihood that you know what you are doing is basically nil.
There's a very real reason why you need to use sleeping locks (like pthread_mutex etc)."
Google's Stadia developer responded to that post which prompted Linus Torvalds to reply and further explain that you should not be using spinlocks at all and placing a lot of sched_yield in your code will not help:
"And all because you did locking fundamentally wrong.
(..)
The fact is, doing your own locking is hard. You need to really understand the issues, and you need to not over-simplify your model of the world to the point where it isn't actually describing reality any more.
And no, any locking model that uses "sched_yield()" is simply garbage. Really. If you use "sched_yield()" you are basically doing something random. Imagine what happens if you use your "sched_yield()" for locking in a game, and somebody has a background task that does virus scanning, updates some system DB, or does pretty much anything else at the time?
Yeah, you just potentially didn't just yield cross-CPU, you were yiedling to something else entirely that has nothing to do with your locking.
sched_yield() is not acceptable for locking. EVER. Not unless you're en embedded system running a single load on a single core."
Developers who are writing or porting games and other heavily multi-threaded can learn a thing or two by reading the entire discussion. The Google Stadia developer Malte Skarupke admitted that:
"this is my first experience with the Linux scheduler as a developer."
January 1st, 2020
Older Linux programmers learned that spinlocks have no place in userspace code back in the 1990s. They have no business being used outside of the kernel and they should only be used in the kernel if you are absolutely sure you will be waiting a very short period of time. Everyone has to learn sometime, it is not strange that younger programmers need to learn concepts that may seem obvious to seasoned Linux developers. It is also worth remembering that Linux is not Windows, they are very different and you can't use what is a bad approach on both operating systems and expect the result to work well on Linux. Making mistakes and learning from them is fine, but don't blame the Linux scheduler if you are doing it wrong.
Enable comment auto-refresher