December 12, 2022

BSD TCP/IP for Kyu - Race conditions, SPL and locks in retrospect

The BSD code uses the venerable "spl_net()" along with "splx(s)" calls. These manipulate the processor "interrupt level". On the VAX (where BSD was first developed) the interrupt level was set in a "processor control register" and various levels were allowed.

On the ARM, you can either enable interrupts or not. Any differentiation among levels is handled in a fairly complex (and very flexible) interrupt controller called the "GIC". This makes emulating the VAX style spl() calls problematic (but not impossible, see below).

On the other hand, in Kyu, it did not seem that we ought to even be thinking about hardware interrupts within TCP code. That sort of thing should be restricted to network drivers and the TCP code should be a component that should exist and work at a higher level. This philosophy in large part motivated a desire to use Kyu semaphores for locking in the TCP code.

Semaphores however have a different semantics than the spl calls. In particular, the spl calls can be freely nested, whereas with semaphores this will lead to deadlock.

Kyu uses INT_lock() and INT_unlock() to either disable or enable semaphores. These also behave differently when nested. We do not get deadlock, but we get an inadvertant release of the lock when the inside pair "releases the lock".

The VAX style spl and splx nest nicely, because the splx() call that exits the critical section returns to whatever state things were in when the spl() call entered the critical section (since the spl() call returns the state and it is saved and restored.

It would certainly be possible on the ARM to produce something with semantics like spl() and splx(). We can extract the interrupt state from the CPSR register inside the spl() macro, save it, and restore it in splx(). It might be interesting to do something like this as an experiment in lieu of "big locks" as is now done.

It is also worth nothing that the BSD code uses two levels of locking. It has spl_net() and spl_imp(). My impression is that these could be made the same (i.e. lock all interrupts on the ARM). Having two levels allowed a more fine grained control about what was being locked.

Two other options using semaphores come time mind. The first is to carefully study the code and have a lot of different locking semaphores, one for each resource that needs to be locked. In many ways this is the best of all solutions. The second option would be to work up some flavor of semaphore with different semantics (that allows nesting without deadlock by saving and restoring state in the same way as spl/splx.

Big locks

I am not sure I ever explained this, but this is how this works. Whenever we are in BSD code, we must hold the "big lock". There are really only a small set of different players in this game: The best part about this is that it works. The bad part is that many time we are holding "the lock" and don't really need to.


Have any comments? Questions? Drop me a line!

Kyu / [email protected]