On 20/09/2015 11:12, Laurent Bercot wrote:
> Interesting, thanks for the notice. I'll have to download OpenRC and
> perform experiments to see exactly what it's doing.
So, I downloaded OpenRC, compiled it - the build process makes a lot
of annoying assumptions, such as "ncurses is there and has been
installed with pkg-config") - and experimented with it.
I'll spare you the details, but give the essential results, which
are... interesting, to say the least.
* Strictly speaking, I was wrong: on a high level, rc_parallel
does honor the dependency graph. OpenRC only starts a subprocess
for a service when all the subprocesses for the service's dependencies
have exited. Functionally, it's looking good. Which is strange,
because with a fundamentally serial design, it shouldn't be able to
do that... so I explored more.
* The way OpenRC implements rc_parallel is this: instead of starting
one subprocess at a time like in the serial case, it starts all the
subprocesses for the whole dependency chain at the same time, and lets
them sort themselves out.
* The way a subprocess knows when it can run: it attempts to take a
lock on a specific file for that subprocess. When it manages to grab
the lock, it means that all its dependencies have completed.
But it doesn't attempt to grab the lock in blocking way. Instead...
it uses a nonblocking flock() call that it loops around with a 20 ms
sleep in-between two calls.
In other words, for every service that is going to be brought up
(or down) but is waiting for a dependency to complete, OpenRC spawns
a new process that polls on a lock file every 20 milliseconds.
This is EVIL and irresponsible. One poll is bad enough, but N
parallel polls, N being the number of processes waiting for
dependencies? This can quickly amount to a non-negligible load,
for big dependency graphs.
And it's not even correct! You can't poll a lock. Another OpenRC
invocation at the wrong time may very well grab the lock that has
just been released by the dependencies, and the process polling on
that lock will fail to take it, again. Race conditions like this is
how you corrupt a database. This is Oracle-level coding, boys, I
hope I'm not infringing a EULA here.
But of course, you can't block on a lock either, because if you do,
you can't implement a timeout. There's a solution to that: fork a
thread, or a process. The s6lock library does that. Among all the
processes spawned by an OpenRC invocation, including at least one
shell script per service, forking one to handle timed locks would
not have been a bad idea. But polling the lock? No, just no.
Even with a correct solution such as "use s6lock_* functions or
similar to attempt a lock grab", this would not entirely solve the
problem, because a concurrent OpenRC invocation could also attempt
to grab the same lock and win, keeping the service in an unfinished
state! Lock-grabbers have no priority, there's no guaranteed order
in which they will get the lock.
"One lock per service, released when the service has no more pending
dependencies" is just not the right mechanism. It's not powerful
enough to guarantee correct behaviour. Especially when you poll the
lock, which you should *never* do.
So, even if I was wrong in my initial assessment, and OpenRC does
honor the dependency graph when the rc_parallel option is used, I
still cannot recommend the use of that option, because as for now,
its implementation is an ugly, ugly hack. And after what I've seen
of OpenRC, and the bloated straces that result from invoking it,
I'm even reluctant to recommend it as a serial service manager. :(
--
Laurent
Received on Mon Sep 21 2015 - 02:07:17 UTC