On Wed, Jun 22, 2016 at 8:08 AM, Avery Payne <avery.p.payne_at_gmail.com> wrote:
> It almost sounds like you need to chain-load memcached using chpst. If
> memcached has internal code to change its process group then it is
> "escaping" supervision, which means that runsv is not in direct control of
> it. To fix this, your ./run script would be similar to:
>
> #!/bin/sh
> exec 2>&1
> exec chpst -P memcached
>
> See http://smarden.org/runit/chpst.8.html for details. This would cause
> memcached to be "captive" to the runsv process. Try the change with chpst
> and see what happens. You may find other issues you're not seeing after you
> make this change; check the log with tail -f /path/to/log/file and see if it
> is restarting over and over (a "restart loop").
>
Memcached doesn't do anything fancy like that, at least not if you run
it in foreground mode. Testing against an unused memcached instance at
work, the problem really does seem to be the one I described: runsv is
catching some non-SIGTERM signal, exiting, and orphaning the memcached
that it's managing.
Thomas, can you post your entire run script? Make sure there's nothing
glaring. At the least your last line should be something like:
`exec /usr/bin/memcached -v -m 512 -p 11211 -u nobody -c 1024'
It does seem suspicious that runsv is getting killed but memcached is
surviving. I've seen systems under impressive amounts of duress that
have had their runsvdir and runsv processes survive (they are unlikely
to be OOM targetted, they don't grow, etc).
Cheers!
--
"If the doors of perception were cleansed every thing would appear to
man as it is, infinite. For man has closed himself up, till he sees
all things thru' narrow chinks of his cavern."
-- William Blake
Received on Wed Jun 22 2016 - 15:51:56 UTC