After a few months of blissfully doing precious little hacking in our spare time, Johan and I have returned to work on the new event manager for GHC.
I spent the past few days writing a paper about the motivation, design, and internals of the new event manager, which I submitted to this year’s Haskell Symposium at about 2am this morning.
Independently, last week I started the work of integrating the event manager into GHC’s runtime libraries, and Johan picked up the pace this week while I concentrated on writing the paper. We’ve reached the point of having the code compile, but it’s still a few weeks from actually working.
While you wait for the fruits of our labours, please indulge in some vicarious enjoyment of the California summer:
It’s really exciting to see this corner of the runtime get some attention. The paper is great.
You mention using edge-triggered epoll as a potential improvement in section 10.1. If I’m understanding, a program going to sleep on a recv() would be doing the
following syscalls:
recv() -> EWOULDBLOCK
eventfd_write()
* event manager wakes from epoll_wait()
* eventfd_read()
* epoll_ctl(EPOLL_CTL_ADD)
* epoll_wait()
* epoll_ctl(EPOLL_CTL_DEL)
* event manager wakes runtime thread somehow
recv() -> returns data
(* meaning it’s performed in the event manager thread)
To do edge-triggered, you’ll need to leave FDs active in the epoll (i.e. not do an immediate DEL), right? So that starts getting into the question of when you’re done with an FD; definitely once it’s closed, but possibly you need an API hint to inform the IO manager that the FD won’t be needed for a while.
Once you aren’t doing immediate DELs, you don’t have to go all the way to edge-triggered mode; instead you could have each recv would trigger an EPOLL_CTL_MOD with the EPOLLONESHOT flag. The second epoll_ctl syscall would be unnecessary. Also, MOD just has to set some flags whereas ADD/DEL needs to insert into the epoll’s internal red-black tree (though possibly the cost of the userspace->kernel->user transition dominates the rbtree magic, I don’t know).
But when you say edge-triggered, that really means you’re avoiding both syscalls, right?
Another random idea: in the above syscall trace, the event manager thread wakes just so it can do the first epoll_ctl and then go back to sleep. Did you ever consider trying to do the epoll_ctl() from the thread that called recv()? As far as I’m aware epoll should cooperate when used multithreaded in that kind of way. Don’t quote me though.
Just some random thoughts. The new event manager looks like a huge improvement already.
Brian, those are good suggestions, thanks, and I’ll give them a try. If you want to participate in the conversation, please feel welcome to join us over at http://groups.google.com/group/ghc-io-manager