synchronizing concurrent shell scripts

Michael L Torrie torriem at chem.byu.edu
Mon Feb 12 15:15:17 MST 2007


On Mon, 2007-02-12 at 12:29 -0800, Michael Brailsford wrote:
> If I had to synchronize multiple scripts, I think I would stop thinking about shell scripting.  The noclobber thing is bash specific, and most unix distributions you see in the real world will not have bash, but instead they all have korn shell.  Korn shell is considered the de facto standard.  Unless, bash gets installed manually, which is generally frowned upon in places where standard builds and standard configurations are more important than the user's preference.  I would suggest using something like perl.  Every unix machine I have ever seen in any setting in my experience has perl.  

Well my scripts are bash scripts, not korn (sh).  And for my particular
job, short of writing a client server application, the file locking
method works very well.  My scripts all have #!/bin/bash at the top, so
I'm not worried about anything in the real world or other such things.
And given my problem, I don't see any benefit of perl over bash.

> 
> Of course, if you are stuck supporting an existing system and you cannot change it, then good luck. 

This is a recently created system and, yes, synchronizing asynchronous
bash scripts is one of the better solutions for the job at hand.  In
fact, the problem of synchronization is the same whether I'm doing it in
perl or bash.  To humor you, though, I will state the problem.

Server A sshes to Server B and runs a script.  For security reasons
server A uses a ssh key to a non-privileged account on Server B, where
is uses sudo to run a script that is limited in what it does (limit the
root scope).  The script's job is to tar up a particular folder and then
delete the folder.  For certain reasons, the script then uses the atd to
schedule itself to run in 5 minutes.  When the script finally runs, it
checks to see if the folder really should be archived (in case the admin
changed his mind during the last 5 minutes), and then proceeds the
archive the folder and then delete the folder.  The problem is that this
archiving can take a long time.  Further, server A can ssh in at any
time and ask for another folder to be archived.  Worst case server A
sends up to 1500 folder archive requests over a period of 20 minutes.
This is a rare event.  Nonetheless, after the 5 minute sleep, it's
possible to have hundreds of scripts trying to run tar and gzip
simultaneously on folders ranging anywhere from 1 to 500 MB.  Without
creating something complicated, the solution is to get the scripts to
synchronize themselves so that only one tar (current solution) is
running at a time.  I forcibly serialize them to one at a time only
because it's the simplest and the need to serialize is rare anyway.  So
the locking solution works very very well.  Completion time is of no
consequence.

The problem for me was that the main script was already coded up and
functioning before the concurrency problem was thought about.  And it's
not a true concurrency problem either; it's just a matter of keeping the
server load down.  If I was doing it again, maybe setting up some kind
of job queue and then polling the queue every minute or two with cron
may have worked.  But this file locking turned out to be a very simple
and effective solution, given that concurrency would only be an issue
once a year or so.

> 
> -Michael
> 
> ----- Original Message ----
> From: Bart Whiteley <bart.plug at whiteley.org>
> To: Provo Linux Users Group Mailing List <plug at plug.org>
> Sent: Monday, February 12, 2007 2:20:03 PM
> Subject: Re: synchronizing concurrent shell scripts
> 
> On 2/12/07, Michael L Torrie <torriem at chem.byu.edu> wrote:
> >
> > >
> > > However, touch works well.  Something like this has always worked for
> > me.
> > >
> > > while [ -d /var/somelockfile ]; do sleep 1; done
> > > touch /var/somelockfile
> > > # do something
> > > rm /var/somelockfile
> >
> > Sadly this will not work, at least if you goal is to synchronize
> > multiple scripts running asynchronously.  It would have worked in my
> > case, but it's not a semaphore.  Who wins when several processes make it
> > past the while loop at about the same time?
> >
> >
> 
> Right.  This is why you should use the redirect '>' with noclobber instead.
> With noclobber, the redirect works like an atomic test-and-set.
> 
> /*
> PLUG: http://plug.org, #utah on irc.freenode.net
> Unsubscribe: http://plug.org/mailman/options/plug
> Don't fear the penguin.
> */
> 
> 
> 
> 
> /*
> PLUG: http://plug.org, #utah on irc.freenode.net
> Unsubscribe: http://plug.org/mailman/options/plug
> Don't fear the penguin.
> */
> 




More information about the PLUG mailing list