Re: Consensus on process "instances"?

From: Avery Payne <avery.p.payne_at_gmail.com>
Date: Sat, 29 Nov 2014 21:41:29 -0800

On Sat, Nov 29, 2014 at 4:01 AM, Laurent Bercot <ska-supervision_at_skarnet.org
> wrote:

> If you have a fixed number of instance, it's not significantly harder.
> The main problem if is you have a dynamic number of instances. Changing
> the
> main service tree dynamically is probably abusing the framework, and some
> additional tool is probably needed here.
>

I agree on a practical level. Each "service" consumes one for the
supervisor and one (or more) for the process proper. That alone is enough
reason to avoid the existing method for launching instances, as you would
quickly fill the process table when launching hundreds or thousands of
instances. Because there is no way to do this with the existing
arrangement, it implies a new tool is needed.

 A possible solution would be to create a separate svscan directory, use the
>> same script for each instance with separate per-instance settings, and
>> start all of the instances as a process sub-tree of the parent svscan.
>>
>
> Yes.
> As soon as you think of using a supervision tree to do that kind of thing,
> more glue work is needed. Creating a specialised subtree sounds like a good
> idea, because it gives you separate places to 1. factor all the
> commonalities
> and 2. instantiate.
>

Another thing you gain is better control. A master process that tends the
tree could be used to bring down all instances cleanly.


> There are more advantages to this than you give it credit for. For
> instance,
> you can run all your subtree under a dedicated user, with dedicated quota.
> You
> can factor more than the run script - most of the configuration can be done
> at the "master" level.
>

All good points, I didn't think of those.


> Is there a better way to accomplish this, or a common consensus about how
>
> to handle this situation? Or do people simply deal with the pain of
>> writing all of the instances as they are needed?
>>
>
> So far, I'm not aware of any consensus. Dynamic instantiation isn't
> something
> lots of thoughts have been given about yet; thanks for bringing it to our
> attention.
>

I'm sure there will be more as I go along. Writing the scripts continues
to be not only practical but educational as well.


> What kind of utilities would you need to make it easier to manage and
> maintain
> dynamic instantiation ?


I have given this some thought. I ask everyone on the mailing list to be
patient with me as the thought process for me is exploratory; so while it
is long-winded and will cover some things that don't work, I want to at
least provide my thoughts as to why they don't work. It is open to ideas,
criticism, and of course improvement.

The first method is the obvious one: just use the existing tools and add
instances as needed, with each instance in the main supervision tree. This
has already been discussed above and has already been dismissed.

Pros: No new tools are required. Uses the existing framework and
knowledge. Easily done for static instances.
Cons: Consumes PIDs rapidly on larger installations. Not as easy to
administer as each instance has to be started and stopped separately,
instead of starting them all and stopping them all. Difficult to
automate, as you need to create custom solutions to spawn instances
dynamically.

The second method would be to span a separate svscan and create a sub-tree,
but use the first arrangement. This was also discussed above, and
dismissed because while it solves some of the management issues it still
has several of the drawbacks.

Pros: Solves the administration issue by allowing both fine-grain control
per-instance, and course-grain control over the entire tree. Allows the
administrator to apply limits and other settings across all instances.
Cons: Still has the same issue with PID consumption. Same issue with
automation.

The third method would be to alter svscan to keep its "find-and-spawn"
behavior, but instead of running a supervisor plus process, it runs just
the process. This would be accomplished by giving a command line option
that changes its behavior.

Pros: Fixes the PID consumption issue.
Cons: Automation is still a problem. With no supervisor to capture
logging, how do you handle log guarantees?

The fourth method would be to alter the supervisor to have a behavior
change. It will use a single service directory entry, and a single run
script, but through a command line switch, it will spawn more processes as
needed depending on some external factor.

Pros: Also fixes the PID consumption issue. Can be automated. Logging can
be included, and is left as-is.
Cons: All instances go to the same log as there is only one logging
process. There is no existing specification as to what the "External
factor" is that dynamically spawns new processes.

The fifth method would be like the fourth, but instead of introducing a
change of behavior, a new tool is written using the supervisor code base as
a starting point, and svscan is only changed to launch a different tool
that would take the place of the supervision process. The new tool would
be purpose-built for launching instances and would be used just like the
supervisor. For the sake of discussion, I will just name it as
"instantiate".

Pros: Keeps all of the fixes for PID consumption, automation, and logging.
Keeps the supervision code base as-is, allowing us to stick with "each tool
serves a specific purpose".
Cons: requires a slight change of semantics - the svscan tool now needs to
know if it is running a supervisor, or if it is running the "instantiate".
Perhaps a new file in the service directory signals that supervisor isn't
to be used. The "External factor" still remains unresolved, so we still
don't know how to start/stop instances on the fly.

Summary of Issues:
- Both course and fine controls: start/stop all instances, or start/stop
individual instances
- PID Consumption must be considered for large values of n instances, where
n > 2^15
- svscan must be able to differentiate between regular service definitions,
and instances
- Automation of new instances by messaging or signalling the "instantiate"
process
- Logging currently appears to capture logs from all instances, they are
not separate.
- A separate tool is suggested to keep existing code base aligned with
other design goals/issues

My suggestion:

Create a new "instantiate" tool, that takes the place of supervisor where
needed. To create a process tree with instances, we would create a regular
service directory, but instead of a ./run file, we would perhaps have an
./instance file that fulfills the same role. When svscan sees the
directory, it will first attempt to find ./run, and failing that, will look
for ./instance instead. If it finds ./instance, it launches "instantiate"
instead of supervisor. Instantiate would then behave similar to supervisor
with regard to logging, etc. But instead of running just a single process,
it would then determine how many processes to spawn and with what
parameters. Each one would spawn as a child of "instantiate" and have its
stdout attached to the logging process (if present). This would be our
"External factor" and I haven't had time to think about how to do this; for
now, please excuse some hand-waving on my part due to a lack of
information, and simply picture it as some kind of signal or data feed that
instructs "instantiate" to spawn a new child with a set of conditions, or
stop an existing child. A roughly drawn process tree follows:

init
  -> svscan /etc/sv
    -> supervise /etc/sv/a-logged-process
      -> my-logger /var/log/a-logged-process
      -> a-logged-process
    -> supervise /etc/sv/a-simple-process
      -> a-simple-process
    -> instantiate /etc/sv/bunch-of-instances
      -> my-logger /var/log/bunch-of-instances
      -> bunch-of-instances configfile-1
      -> bunch-of-instances configfile-2
      -> bunch-of-instances configfile-3
      -> bunch-of-instances configfile-1
--with-an-option-that-makes-it-a-variant
      -> bunch-of-instances configfile-4

We can of course start and stop all instances like any other process with
this arrangement. And instantiate behaves like supervisor when shutting
down, sending TERM to all children when it is time to stop. By default,
all processes must be a member of the same process group as instantiate
(otherwise they could stray away) so this is probably an inherent
limitation. Semantically, not a lot changes for the systems
administrator. Program-wise it may or may not be feasible - I don't do a
lot of C coding.

I'm trying to go for simplicity with this arrangement. I noticed that the
resulting design did away with the 2nd tier of svscan, so unless it is
needed I didn't see a reason to continue down that line of thought. If it
is desirable to keep a 2nd tier of svscan in this arrangement for reasons
that I have missed, it could certainly be added back in. So for now, I
don't see the missing svscan as a big issue because it can be done with or
without.

So that leaves the *last* question: how do we tell "instantiate" when to
start new processes, or stop existing ones?
Received on Sun Nov 30 2014 - 05:41:29 UTC