Prashant's FreeBSD Blog

Status at end of GSoC

Mon, 24 Aug 2009 20:54:16 +0530

So GSoC has officially come to an end, it’s time to summarize what has been and has not been achieved during this time.

Two of the goals were achieved, while the third is quite far off right now.

The callout “wheel” was replaced with a binary heap. This forms the basis of the suggested improvements to the callout subsystem. The binary heap was implemented as a set of macros which provide a min-heap as well as a max-heap, for various general-purpose priority queues. More information can be found in the comments in the kernel source file sys/sys/bheap.h and in the previous blog posts.
The new API was implemented from the “client-side”, i.e. the functionality is usable by callout consumers now, instead of the existing API. It was greatly simplified, though there could be some more modifications to make it more useful. The API stands as follows :

callout_handle_t — The handle (actually a typedef’s struct) used to identify a single callout (can be reused). The client should malloc a callout_handle_t and zero it out before using it.
hwclocktick_t — A semi-opaque hardware-dependent clock tick count. Clients need to use auxiliary functions to convert to/from real time and hardware-dependent time. The hwclocktick_t is limited to nanosecond resolution (because the conversion functions currently use timespec)
absolute_time_to_hw_time(const struct timespec* a, hwclocktick_t* t) — This function converts a timespec to a hardware dependent tick count. The timespec structure can hold time in the format of seconds + nanosecond.
hw_time_to_absolute_time(const hwclocktick_t* t, struct timespec* a) — This function converts given hardware-dependent ticks into a timespec.
time_interval_to_hw_time(int interval, hwclocktick_t* h, enum time_scale ts) — This is probably the function which will be used the most. It converts a given number of real time units into a hwclocktick_t which can be later used to schedule a callout. The time_scale is used to describe what units interval is specified in, and can be one of: TIMESCALE_SECOND, TIMESCALE_MS, TIMESCALE_US (microsecond), TIMESCALE_NS or TIMESCALE_MINUTE.

The above 3 functions can be used to create a hwclocktick_t of the appropriate duration. If the same duration will be used frequently, it makes sense to do the conversion once and cache the resultant hwclocktick_t.

Behind the scenes, the callout subsystem will “ask” the current timer hardware to do the conversion for it. This will use the Kobj kernel object system with a common interface that each timer hardware will need to specify. Currently this is not in a working state yet, so the functions use traditional hz arithmetic to do the conversions. In any case, this is transparent to the callout client.

Once the required hardware-dependent tick count is generated, a new callout can be armed by calling:

callout_arm(callout_handle_t* c, hwclocktick_t when, void (*func)(void*), void* arg, int flags, mtx* mutex) — Pass the allocated callout handle, the required callback time as a hw tick, a pointer to the callback function, an argument to pass to it, some flags (same as old API) and any mutex as required. This combines the functionality of callout_init(), callout_init_mtx(), callout_reset() etc. all in one function. This might be a good thing or a bad thing.

There is no callout_init[_mtx]() anymore. The callout_arm() function will handle initialization. There is no reason to call callout_cancel() on an unarmed callout, so explicitly initializing it is not needed. But if your design does require it, make sure to zero out the callout_handle_t structure after memory for it is acquired. Calls to callout_cancel() with an unarmed callout handle will simply fail. Results of calling callout_cancel() without a zeroed-out, unarmed callout are undefined!

If you frequently need to re-arm callouts, for which only the callout time changes, callout_rearm() can be used as a shorthand. Obviously callout_rearm() should only be called with a callout handle that has been armed before (else results are undefind).

Calling callout_arm() on an already pending callout will also simply reschedule it, so if you need a different callback function or have to pass another argument in addition to rescheduling it, this method will suffice. Note that this is different from the current behaviour where the function and argument must also match for most functions.

There is also callout_[re]arm_within() — this function can be used to arm or re-arm a callout with an imprecise time-out. That is to say, one can specify a time period during which the callout can be serviced. This will allow for optimizations in the future. Currently we take the lazy approach and schedule such callouts to the farthest time specified.

callout_cancel() and callout_drain() remain basically the same. When the underlying implementation will be rewritten, atomic operation of callout_cancel() will be guaranteed - i.e. if the callback function has not already been executed, upon return of callout_cancel(), the callout will be guaranteed to have been stopped. There will not be a grey-zone during which if callout_cancel() is called, the callout may still run. This is not implemented yet!

That is about it for the new API. As mentioned before they are currently simply wrappers around existing implementation, and both implementations currently exist in the kernel (in my experimental branch).

I modified the iwi driver to use the new API as a quick test and demonstration. It works, but we’re not using it in the most efficient way yet. I’ll be checking the driver’s logic in more detail so we can replace most instances of callout_arm() with callout_rearm() instead and avoid internally re-initializing the callout structure.

On the Kobj side, I’ve started writing the interface and implemented a timer_hz class which will source off the traditional periodic timer via hardclock(). The idea is that various timer hardware can implement this timer interface to vend its services, and register with the callback system at boot-time (or even run-time). The user (or system) will then select whichever timer it needs. This part has also not been implemented yet.

So to summarize:

The callout internally uses a binary heap which should be more efficient specially when re-arming existing callouts.
The new API is basically usable, although still works as a wrapper layer over the existing implementation.
Multiple hardware timer support is still in its very early implementation stages, Kobj will be used for this.
Tickless capability is dependent on point 3 being finished.

I plan to continue working on the project so hopefully in the next month or two, #3 and subsequently #4 will be done and we can have a high-resolution, tickless capable callout subsystem in FreeBSD!

Re-arming existing callouts slightly optimized

Sun, 28 Jun 2009 22:44:00 +0530

After general testing, I narrowed down most of the missed-callout problems to my modification of callout_reset_on() in which I remove and reinsert (the old way) callouts that are pending and still on the callout queue.

EDIT: Well turns some interrupts are being missed, not callouts. :-\ Will have to find better ways to test.

Instead of removing and re-inserting the callouts, I’m now using the MINHEAP_KEY_CHANGE() macro to simply modify and rearrange the callout queue as appropriate. In the limited time I’ve been testing it, everything seems to be working much better. I still have doubts that some callouts are being delivered later than they should, though ktrdump(8) doesn’t show anything obvious. No callouts are being missed, though, and the wifi LED as well as the CPU frequency scaling are no longer getting stuck.

From the kernel trace dumps I also see that on a newly-booted system, the callout queue is pretty small, only about 65 callouts are pending at any given time. After starting X, a window manager, a browser and a few terminals, the queue becomes around 85 in size. With heavy network transfers going on, it can go up to 120 pending callouts. I’ve not been able to create a situation where the number of pending timeouts is higher. It seems like such a waste to pre-allocate c. 18000 (!! as my kern.ncallout key says at the moment) when only a couple hundred are going to be used at any given time.

I also re-tested the binary heap implementation, with over 1200 callouts in the queue and it passes all tests. So there can’t be any more bugs in it.

So barring any significant discoveries or malfunction, I consider this phase of the project completed. Next week I’ll be adding new callout api functions as well as (still) modifying kern_timeout.c to have my own locking scheme (having a queue implementation on which performing any operation leaves it in a consistent state means we can get away with much more finer grained locking). During this phase I’m hoping to implement some of the suggestions that rwatson has: more info on this mailing list post.

To wrap up, here is another example of why I dislike perforce:

/usr/home/pvaibhav/p4/calloutapi/src/sys$ p4 open kern/kern_timeout.c

Path ‘/usr/home/pvaibhav/p4/calloutapi/src/sys/kern/kern_timeout.c’ is not under client view ‘/home/pvaibhav/p4’

/usr/home/pvaibhav/p4/calloutapi/src/sys$ cd ~/p4/calloutapi/src/sys/
~/p4/calloutapi/src/sys$ p4 open kern/kern_timeout.c

//depot/projects/soc2009/calloutapi/src/sys/kern/kern_timeout.c#4 - opened for edit

Lame.

Issues with hardware: LAPIC et al

Wed, 24 Jun 2009 20:06:27 +0530

I did some more “naive” testing: using the system for general and not-so-general tasks, trying to identify any ill effects. The first problem I encountered (which I had been simply enduring for 2+ months) was sluggish system response. Turns out it was because my powerd configuration was set to adaptive instead of hiadaptive. I had always thought the sluggish response was due to bad Intel xorg drivers. Anyway, with that out of the way, and a relatively fast system (graphics as bad as always, however), I resumed testing other parts.

First off I noticed that the periodic blinks of the wireless card LED were sometimes taking longer than usual, and sometimes they were totally stuck in the “on” state. I had never paid attention to this earlier, but with the new kernel and new callout system, I was trying to find every single clue that might point at a problem.

I found something very frustrating: The local APIC, the hardware that delivers the preferred timer interrupts, apparently loses ticks when the processor is in a deep sleep state (C-state >= C2). As if a variable TimeStamp Counter (TSC) losing ticks in slower P-states was not enough trouble on a Pentium M, even the LAPIC had to lose ticks too — in C-states.

After setting the lowest C-state to only C1 (hw.acpi.cpu.cx_lowest=C1), and the corresponding increase in CPU temperature in this scorching 43 degree C Indian summer, the wireless LED started behaving.

I also made a few minor changes in softclock() -

Reverted to using a cached value of cc_monoticks (our per-CPU monotonically increasing tick counter — I could just use the global variable ticks but I decided to keep the timers per-CPU. This is kinda irrelevant now as this whole infrastructure will be replaced soon). Earlier I had switched to using cc_monoticks directly, despite the fact that it could be increased by hardclock when it calls callout_tick(). The reasoning was that softclick() expires present and past callouts, so even if cc_monoticks had been increased while we were still processing the callouts, any newly-“missed” callouts would still get expired anyway, and this will save us having to reschedule softclock() again on the next HW tick. However, I changed to using cached value and allowing reschedules, because: not doing so would mean softclock() could potentially run for a long duration, in case it gets behind hardclock, and will continue “chasing” newly-missed callouts. Since during this process, the cc_lock is held, it makes things really nasty. So: idea dropped.
I put back the idea of checking how man callouts softclock() has to examine during one tick, and temporarily unlocking and relocking cc_lock to “give interrupts a chance.” It’s made easier for us as in general, the callout queue cannot be changed in such a way that unlocking/relocking will make resuming normal operation impossible. In other words, even if during the time we drop the cc_lock, some other thread inserts or removes a callout, the callout queue will still always be in a consistent state (because each insertion/removal will preserve the heap property). There are no links and pointers to manage — internally the heap uses an array and each callout knows its own “selfindex.” So no harm done. However, the important part is that we must extract head only after we have reacquired the callout queue lock, which brings me to the last change :
I made a mistake when adding the previous feature: I made softclock() extract the head first, and then drop the callout lock temporarily. This was wrong order, since during the time the lock was dropped, someone could have inserted a callout which ended up as the head, so the head we already extracted was no longer the “real” head. So I changed it to do this dropping/relocking business first, and only extract the head after we have reacquired the lock. In other words, only operate on a callout while holding the callout queue lock.

I’ll continue doing more casual tests, although finally I have to devise a proper plan to make sure things are working before I can move on and start adding new timer hardware interfaces. I’ve just read up on DTrace and am thinking of ways to use it for testing the callout system (which already implements a couple of DTrace probes). But I’m also planning to continue using ktr(9) — as the old saying goes

Real programmers don’t use Dtrace, they use printf(..)

Hehe.

Done: Prototype implementation of binary-heap based callout system

Wed, 24 Jun 2009 05:05:21 +0530

After a couple hours of debugging and analyzing ktr(9) traces in ddb, I managed to figure out the problem which was causing kernel panic just 3 hardclock ticks into bootup. I had conveniently forgotten to unlock a mutex in callout_tick() — not exactly forgotten, but overlooked the case when there were no callouts to expire during the current hardclock tick. The mutex should have been unlocked regardless of whether we are scheduling softclock() to run or not.

As soon as I fixed this, the kernel booted up well and everything seems to be working so far:

[pvaibhav@matrix:src/sys]$ uptime
 4:58AM  up  2:07, 2 users, load averages: 0.13, 0.32, 0.39

Although my laptop’s wireless LED seems to get stuck every now and then, when it should be blinking depending on traffic. I’m not sure if this is because the signal is weak right now, or whether the callouts needed by the LED taskqueue in the iwi(4) driver are not being serviced properly.

Testing has also only been done on my uniprocessor setup, so it remains to be seen how it performs in SMP environment. I’ve left the locking semantics largely untouched from the original implementation so there shouldn’t be any problems. I also need to devise a plan to test the whole thing more thoroughly than simply.. using the kernel. Perhaps it’ll be interesting to see how (if at all) the performance is affected by using the binary heap and not having to loop through all the callouts stored in each “bucket” of the old callout wheel, during a hardware interrupt.

First post!

Wed, 24 Jun 2009 01:17:16 +0530

I decided to maintain a development blog for my Google Summer of Code 2009 project with FreeBSD. The main intent is to document my thought process, successes and failures through this project: mostly as a scratch pad. I hope this will also be useful for those curious to track the progress.

Background information for the project can be found on the FreeBSD wiki. In short, the project’s goal is to redesign the way the FreeBSD kernel handles callouts/timeouts. The changes include a new API for clients, underlying data structure changes for better efficiency, and the most important of all (in my opinion): the possibility of making the kernel tickless by doing away with periodic timer interrupts, and using programmable deadline timers exclusively.

Notes about the current status and implementation details:

A generic binary heap has been implemented, and priority queue based on it now works. Both MIN and MAX heaps are working, although we will only consume the MINHEAP variant. The design is similar to the SLIST, TAILQ and friends found in sys/queue.h. The advantage of using this implementation of a priority queue is that the binary heap, on which this PQ is based, provides O(1) time complexity on an average for most operations including inserting and removing nodes or changing their priority. Although the upper bound time complexity is still O(log n), in general this will only occur occasionally when a new callout is scheduled which must bubble up to the root (ie., have the smallest timeout).
callout_clock() was modified to peek at the first pending callout, and schedule the software clock handler softclock() only if the expiry time has arrived (or is past, if ticks were missed). Peeking at the first item is O(1) worst-case, so this operation is extremely efficient.
softclock(), the software interrupt handler, will extract items in priority-order and if their expiry time has come or is in the past, will run the associated function and remove the callout from the queue. It then checks the next callout in line and repeats this process until the expiry time of the next callout is found to be in the future. At this point, softclock()’s job is done (for now).
Auxiliary functions to reset/stop callouts have been implemented, and they all use the new priority queue.

The current issues to note are:

The kernel doesn’t boot to desktop! Although the build is successful and the kernel can run until the time the first callout is scheduled, the kernel panics with the message mi_switch: switch in a critical section. I am currently trying to debug this, although it’s made difficult by the fact that I can’t obtain a dump of the crash because the harddrives are not even ready when this occurs. I’m looking to use ktr(9), the kernel trace facility to see if we can figure out where the problem is.
The kernel still has to service periodic interrupts. The tickless-work has not yet started — and won’t be until the current implementation works with binary heaps as its queue instead of the call “wheel.” Because of this, we’re still using hardclock ticks as the timing unit, and not the opaque hwclocktick_t that I have proposed in the plan. Once the binary heap priority queue is working, I’ll start to implement the common timer interface (most of this already exists, just have to add functions to get frequency and convert between walltime and hardware time). Only after this can we begin implementing the new functions which work with opaque time values.
The extremely old timeout() and untimeout() calls are still supported, but they are a pain to manage. These functions do not mandate clients to allocate their own callout structures, so the kernel must do it for them. Right now, the existing method of allocating a fixed number of callouts locally, is used. We are manually managing the callouts that have been used and freed, using a linked list. In the future I want to use uma(9) zone allocator which is meant for this kind of work. However, I couldn’t find a way to create a zone if I already have some memory preallocated (although this is mentioned in the man pages!). So this part has been postponed for later.

That’s about it for now. Later I’ll post more info about the heavyweight debug session I’m about to begin. Let’s hope I’m able to track it down!