0 0 1
default
Jean-Francois Pieronne - 4 years ago 2016-11-30 15:39:11
jf.pieronne@laposte.net
Docs/tsshbatch.rst initial version
1 file changed with 1520 insertions and 0 deletions:
↑ Collapse Diff ↑
 
new file 100644
1
 
.. contents::
2
 

	
3
 
NAME
4
 
----
5
 

	
6
 
**tsshbatch** - Run Commands On Batches Of Machines
7
 

	
8
 
.. WARNING:: ``tsshbatch`` is a powerful tool for automating activities
9
 
             on many servers at a time.  This also gives you to power
10
 
             *to make many mistakes at a time!* This is especially
11
 
             true if you have ``sudo`` privilege promotion
12
 
             capabilities on the systems in your care.  *So be careful
13
 
             out there!*
14
 

	
15
 
             We therefore STRONGLY recommend you do the following
16
 
             things to mitigate this risk:
17
 

	
18
 
                 - Read This Fine Manual from beginning to end.
19
 

	
20
 
                 - Practice using ``tsshbatch`` on test machines or
21
 
                   VMs that can easily be recovered or reimaged if you
22
 
                   break someting.
23
 

	
24
 
                 - Make heavy use of test mode (which is the default)
25
 
                   to see what the program *would* do if it actually
26
 
                   ran in execution mode.
27
 

	
28
 

	
29
 
SYNOPSIS
30
 
--------
31
 
::
32
 

	
33
 
  tsshbatch.py [-EKLNSTWaehkqrstvxy -G 'file dest' -P 'file dest' -f cmdfile -l logfile -n name -p pw ] -H 'host ..' -i 'hostfile ...' [command arg ... ]
34
 

	
35
 
DESCRIPTION
36
 
-----------
37
 

	
38
 
``tsshbatch`` is a tool to enable you to issue a command to many
39
 
hosts without having to log into each one separately.  When writing
40
 
scripts, this overcomes the ``ssh`` limitation of not being able to
41
 
specify the password on the command line.
42
 

	
43
 
You can also use ``tsshbatch`` to ``GET`` and ``PUT`` files
44
 
from- and to many hosts at once.
45
 

	
46
 
``tsshbatch`` also understands basic ``sudo`` syntax and can be used
47
 
to access a host, ``sudo`` a command, and then exit.
48
 

	
49
 
``tsshbatch`` thus allows you to write complex, hands-off scripts that
50
 
issue commands to many hosts without the tedium of manual login and
51
 
``sudo`` promotion.  System administrators, especially, will find this
52
 
helpful when working in large server farms.
53
 

	
54
 

	
55
 
OPTIONS
56
 
-------
57
 

	
58
 
``tsshbatch`` supports a variety of options which can be specified
59
 
on either the command line or in the ``$TSSHBATCH`` environment
60
 
variable:
61
 

	
62
 
  -B               Print start, stop, and elapsed execution time
63
 
                   statistics.  This does not include any time
64
 
                   spent for interactive prompting and response,
65
 
                   but reflects actual program runtime.  (Default: Off)
66
 

	
67
 

	
68
 
  -C configfile    Specify the location of the ssh configuration
69
 
                   file.  (Default: ``~/.ssh/config``)
70
 

	
71
 
  -E               Normally, ``tsshbatch`` writes it's own errors
72
 
                   to ``stderr``.  It also writes the ``stderr``
73
 
                   output from each host it contacts to the local
74
 
                   shell's ``stderr`` (unless the ``-e`` option has
75
 
                   been selected).
76
 

	
77
 
                   The ``-E`` option redirects any such ``tsshbatch``
78
 
                   output intended for ``stderr`` to ``stdout``
79
 
                   instead.  This avoids the need to do things like
80
 
                   ``2>&1 | ...` on the command line when you want to
81
 
                   pipe all ``tsshbatch`` output to another program.
82
 

	
83
 
  -F strings       This will examine every file on the host- or
84
 
                   command paths, looking for matching strings within
85
 
                   these files.  Matches will report the file name,
86
 
                   the location within the file, and the line
87
 
                   containing any of the specified strings.
88
 

	
89
 
                   This is a simple, case-insensitive string literal
90
 
                   match and does not support regular expressions.
91
 

	
92
 
                   This is handy when you're looking for a host name
93
 
                   or command string, say like, ``sudo`` and you don't
94
 
                   want to have to manually go through all your
95
 
                   support files.
96
 

	
97
 
  -K               Force prompting for passwords.  This is used
98
 
                   to override a prior ``-k`` argument.
99
 

	
100
 
  -G spec          GET file on host and write local dest directory.
101
 
                   ``spec`` is a quoted pair of strings.  The first
102
 
                   specifies the path of the source file (on the
103
 
                   remote machine) to copy.  The second, specifies
104
 
                   the destination *directory* (on the local
105
 
                   machine)::
106
 

	
107
 
                     tsshbatch.py -G "/foo/bar/baz /tmp" -i hostlist
108
 

	
109
 
                   This copies ``/foo/bar/baz`` from every machine in
110
 
                   host file to the local ``/tmp/`` directory.
111
 
                   Since all the files have the same name, they would
112
 
                   overwrite each other if copied into the same
113
 
                   directory.  So, ``tsshbatch`` prepends the string
114
 
                   ``hostname-`` to the name of each file it saves
115
 
                   locally.
116
 

	
117
 
  -H hostlist      List of hosts on which to run the command.  This should
118
 
                   be enclosed in *quotes* so that the list of hosts
119
 
                   is handed to the -H option as a single argument::
120
 

	
121
 
                     -H 'host1 host2 host3'
122
 

	
123
 
                   This option may appear on the command line multiple
124
 
                   times.  The hosts will be processed in the order
125
 
                   in which they appeared on the command line.
126
 

	
127
 
  -L               List names of all (if any) host- and command files
128
 
                   found on their respective search paths.  These are
129
 
                   listed in the order they are found on those paths.
130
 

	
131
 
  -N               Force interactive username dialog.  This cancels
132
 
                   any previous request for key exchange authentication.
133
 

	
134
 
  -P spec          PUT file from local machine to remote machine destination
135
 
                   directory.  ``spec`` is a quoted pair of strings.
136
 
                   The first specifies the path of the source file (on
137
 
                   the local machine) to copy.  The second, specifies
138
 
                   the destination *directory* (on the remote machine)::
139
 

	
140
 
                     tsshbatch.py -P "/foo/bar/baz /tmp" -i hostlist
141
 

	
142
 
                   This copies ``/foo/bar/baz`` on the local
143
 
                   machine to ``/tmp/`` on every host in  ``hostlist``.
144
 

	
145
 
  -S               Force prompting for ``sudo`` password.
146
 

	
147
 
  -T seconds       Set timeout for ssh connection attempts. (Default: 15 seconds)
148
 

	
149
 
  -V strings       Similar to the ``-F`` command but with the inverse logic.
150
 
                   Reports the names of all host- and command files that
151
 
                   do not contain any of the specified strings.
152
 

	
153
 
  -W               Print out a single line list of the inventory that would
154
 
                   be processed and exit. (Test mode only - Ignored in
155
 
                   execution mode.)
156
 

	
157
 
                   This allows you to embed ``tsshbatch`` in external
158
 
                   shell scripts like this::
159
 

	
160
 
                     for server in $(tsshbatch.py -i devserverlist -i uatserverlist -W)
161
 
                     do
162
 
                       ssh $server
163
 
                     done
164
 

	
165
 
                   Why?  Because tsshbatch has lots of powerful ways
166
 
                   to maintain inventories of hosts and combine them
167
 
                   through includes and multiple command line
168
 
                   arguments.  The ``-W`` option makes it convenient for
169
 
                   external programs to make use of those inventory
170
 
                   features.
171
 

	
172
 
  -a               Don't abort program after failed file transfers.
173
 
                   Continue to next transfer attempt.  (Default: Abort)
174
 

	
175
 
  -b               Don't abort program after failed ``sudo`` command.  Normally, any
176
 
                   ``sudo`` failure causes immediate program termination.  This
177
 
                   switch tells ``tsshbatch`` to continue processing on the
178
 
                   next host even if such a failure occurs.  This allows
179
 
                   processing to continue for those hosts where ``sudo`` does
180
 
                   work correctly.  This is helpful in large environments where
181
 
                   ``sudo`` is either improperly configured on some hosts or
182
 
                   has a different password.  This can also be used to discover
183
 
                   where ``sudo`` does- and does not work correctly.
184
 

	
185
 
  -e               Don't report remote host ``stderr`` output.
186
 

	
187
 
  -f cmdfile       Read commands from a file.  This file can be commented
188
 
                   freely with the ``#`` character.  Leading- and
189
 
                   trailing whitespace on a line are ignored.
190
 

	
191
 
  -h               Print help information.
192
 

	
193
 
  -i hostfiles    Specify which files to read to get a list of desired
194
 
                  hosts to target.  This option may be repeated on the
195
 
                  command line and may also be followed with a quoted
196
 
                  list of such files.  The following are equivalent::
197
 

	
198
 
                     tsshbatch.py -i devservers -i uatservers ...
199
 

	
200
 
                     tsstbatch.py -i "devservers uatservers" ...
201
 

	
202
 
                  The ``-H`` and ``-i`` options can be freely
203
 
                  combined and repated on the command line to create
204
 
                  custom host lists made up of both known inventory
205
 
                  and specific, individual hosts.
206
 

	
207
 
  -k               Use ssh keys instead of name/password credentials.
208
 

	
209
 
  -l logfile       Log diagnostic output to ``logfile``. (Default: ``/dev/null``)
210
 

	
211
 
  -n name          Login name to use.
212
 

	
213
 
  -p pw            Password to use when logging in and/or doing ``sudo``.
214
 

	
215
 
  -q               Quiet mode - produce less noisy output.  Turns off ``-y``.
216
 

	
217
 
  -r               Suppress reporting of start/stop statistics. This
218
 
                   allows you to make statistics reporting the
219
 
                   default, say via the ``$TSSHBATCH`` environment
220
 
                   variable, but override it when you need to.
221
 

	
222
 
  -s               Silence all program noise - only return command output.
223
 
                   Applies only to command operations.  File transfer
224
 
                   and error reporting, generally, are unaffected.
225
 

	
226
 
  -t               Test mode: Only show what *would* be done but don't
227
 
                   actually do it.  This also prints diagnostic
228
 
                   information about any variable definitions, the list
229
 
                   of hosts, any ``GET`` and ``PUT`` requests, and final
230
 
                   command strings after all variable substitutions have
231
 
                   been applied.  This is the default program behavior.
232
 

	
233
 
  -v               Print detailed program version information and exit.
234
 

	
235
 
  -x               Override any previous ``-t`` specifications and
236
 
                   actually execute the commands.  This is useful
237
 
                   if you want to put ``-t`` in the ``$TSSHBATCH``
238
 
                   environment variable so that the default is
239
 
                   always run the program in test mode.  Then, when
240
 
                   you're ready to actually run commands, you can
241
 
                   override it with ``-x`` on the command line.
242
 

	
243
 
  -y               Turn on 'noisy' reporting for additional detail on
244
 
                   every line, instead of just at the top of the
245
 
                   ``stdout`` and ``stderr`` reporting.  This is
246
 
                   helpful when you are filtering the output through
247
 
                   something like ``grep`` that only returns matching
248
 
                   lines and thus no context information.  Turns off ``-q``.
249
 

	
250
 

	
251
 
The last entry on the command line is optional and defines a command
252
 
to run.  ``tsshbatch`` will attempt to execute it on every host you've
253
 
specified either via ``-H`` or a host file::
254
 

	
255
 
  tsshbatch.py -Hmyhost ls -al /etc
256
 

	
257
 
This will do a ``ls -al /etc`` on ``myhost``.
258
 

	
259
 
Be careful when using metacharacters like ``&&, <<, >>, <, >`` and so
260
 
on in your commands.  You have to escape and quote them properly or
261
 
your local shell will interfere with them being properly conveyed to
262
 
the remote machine.
263
 

	
264
 
``tsshbatch`` does all the ``GETs``, then all the ``PUTs`` before
265
 
attempting to do any command processing.  If no ``GETs``, ``PUTs``, or
266
 
commands have been specified, ``tsshbatch`` will exit silently, since
267
 
"nothing to do" really isn't an error.
268
 

	
269
 

	
270
 
ENVIRONMENT
271
 
-----------
272
 

	
273
 
``tsshbatch`` respects the ``$TSSHBATCH`` environment variable.  You
274
 
may set this variable with any options above you commonly use to avoid
275
 
having to key them in each time you run the program.  For example::
276
 

	
277
 
  export TSSHBATCH="-n jluser -p l00n3y"
278
 

	
279
 
This would cause all subsequent invocations of ``tsshbatch`` to
280
 
attempt to use the login name/password credentials of ``jluser`` and
281
 
``l00n3y`` respectively.
282
 

	
283
 
``tsshbatch`` also supports searching for files over specified
284
 
paths with the ``$TSSHBATCHCMDS`` and ``$TSSHBATCHHOSTS`` environment
285
 
variables.  Their use is described later in this document.
286
 

	
287
 

	
288
 
SSH CONFIGURATION FILE PROCESSING
289
 
---------------------------------
290
 

	
291
 
``tsshbatch`` has limited support for ssh configuration files.  Only the
292
 
``HostName`` and ``IdentityFile`` directives are currently supported.
293
 

	
294
 
By default, ``tsshbatch`` will look in ``~/.ssh/config`` for this
295
 
configuration file.  However, the location of the file can be
296
 
overriden with the ``-C`` option.
297
 

	
298
 

	
299
 
MANAGING INVENTORY: HOST FILES
300
 
------------------------------
301
 

	
302
 
Although you can specify a list of target hosts with one or more ``-H
303
 
"host host host"`` command line options, this gets cumbersome when you
304
 
have to manage a large inventory of machines.  This is what "host files"
305
 
are intended to do.
306
 

	
307
 
Host files are files that name a host, one per line.  They may
308
 
contain comments, include other host files, and make reference to
309
 
previously defined variables.  Here's an example, let's call it
310
 
``stage-servers``::
311
 

	
312
 
  # Example staging host list
313
 

	
314
 
  .define __EXTERNAL__ = splat.com
315
 

	
316
 
  stage1.example.com
317
 
  stage2.example.com
318
 
  stage3.__EXTERNAL__
319
 

	
320
 
Say you'd like to get the uptime for each of these servers::
321
 

	
322
 
  tsshbatch.py -x -i stage-servers uptime
323
 

	
324
 
You can have more than one of these on a command line::
325
 

	
326
 
  tsshbatch.py -x -i dev-servers -i stage-servers -i prod-servers uptime
327
 

	
328
 
But ... that's kind of clumsy.  Instead let's create a new host file
329
 
called ``all-servers`` like this::
330
 

	
331
 
  # Master host list
332
 

	
333
 
  .include dev-servers
334
 
  .include stage=servers
335
 
  .include prod-servers
336
 

	
337
 
Now our command looks like this::
338
 

	
339
 
   tsshbatch.py -x -i all-servers uptime
340
 

	
341
 
You can put as many ``-i`` arguments as you wish on the command line.
342
 
The contents of these files will be run in the order they appear from
343
 
left-to-right on the command line.
344
 

	
345
 
You can organize your host files in any hierarchy you like by making
346
 
use of search paths, as described later in this document.
347
 

	
348
 
The use of host files, the ``-H`` command line option, ``.includes``,
349
 
and variable substitution, give ``tsshbatch`` a very powerful way to
350
 
manage complex host inventories.  There may be times when you'd like
351
 
to select a subset of that inventory but *run some other program with
352
 
those host names*.  This is most commonly the case when you'd like to
353
 
interactively ``ssh`` into each host for some reason.
354
 

	
355
 
That's what the ``-W`` command line option is for.  ``tsshbatch`` computes
356
 
the inventory you've specified, spits it out as single line of text and exits,
357
 
thereby enabling things like this::
358
 

	
359
 
  for server in $(tsshbatch.py -i dev-servers -i dr-servers -W)
360
 
  do
361
 
    ssh $server
362
 
  done
363
 

	
364
 
This allows you to do all your intentory management via ``tsshbatch``
365
 
constructs, but make use of it with any other program of your
366
 
choosing.
367
 

	
368
 

	
369
 
MANAGING JOBS: COMMAND FILES
370
 
----------------------------
371
 

	
372
 
Although ``tsshbatch`` accepts command to be run on each host on its
373
 
own command line, this gets cumbersome for multi-step activities.  The
374
 
``-f`` option allows you to specify one or more "command files" that
375
 
list the things you want done, in order.
376
 

	
377
 
Command files may include other command files, make use of variables,
378
 
and even specify sudo commands.  For example::
379
 

	
380
 
  # This is a comment
381
 
  .include master_commands  # optional, can be repeated
382
 
  command1                  # a command to run on each host
383
 
  sudo foo                  # run foo with sudo promotion on each host
384
 

	
385
 
Notice that this is not the same thing as a shell script.  There are
386
 
no conditionals or other programming features.  The contents of a
387
 
command file are more-or-less a "to do" list for each host.  If your
388
 
job requires the complexity of a "real" language, write script in
389
 
the language of your choice, and then make reference to that script
390
 
in a command file.
391
 

	
392
 
If you've specified a command file containing the commands you want run
393
 
via the ``-f`` option, these commands will run *before* the command
394
 
you've defined on the command line.  That one is always the last
395
 
command run on each host.
396
 

	
397
 
Command files may freely make use of comments and variable definitions
398
 
as described below.  They also have a special directive available,
399
 
``.notify``.  This directive tells ``tsshbatch`` to print descriptive
400
 
messages to ``stdout`` as the commands in the file are executed on a
401
 
remote host.  This is helpful when running long, complex jobs::
402
 

	
403
 
  # Example cmdfile
404
 

	
405
 
  .include /my/fine/options
406
 
  .define __JOB_NAME__
407
 
  .notify starting JOB_NAME processing
408
 
  /usr/local/__JOB_NAME__
409
 
  .notify starting phase 2 of __JOB_NAME__
410
 
  do_another_command
411
 

	
412
 
Notifications are entirely optional *and do not run on the remote
413
 
host*.  Think of them as runtime comments emitted by ``tsshbatch`` to
414
 
help you know what's going on.  Notifications are supressed if you
415
 
select silent operation (``-s``).
416
 

	
417
 
You can also specify file transfers within a command file.  See the
418
 
section below on file transfers for all the details.
419
 

	
420
 
You can put as many ``-f`` arguments as you wish on the command line.
421
 
The contents of these files will be run in the order they appeare from
422
 
left-to-right on the command line.
423
 

	
424
 
You can organize your command files in any hierarchy you like by making
425
 
use of search paths, as described later in this document.
426
 

	
427
 

	
428
 
SEARCHING YOUR HOST AND COMMAND FILES
429
 
-------------------------------------
430
 

	
431
 
Over time, you will probably build up a large set of host files for
432
 
describing your inventory and command files for standard jobs you
433
 
run often.  It's convenient to search through them quickly when
434
 
you're looking for something specific.
435
 

	
436
 
The ``-L`` command line option just lists every host file and command
437
 
file ``tsshbatch`` knows about on all defined search paths.  This is
438
 
handy if you want to examine your hierarchy of files.
439
 

	
440
 
The ``-F string string ...`` command line option looks through all
441
 
your host *and* command files and returns the names of those that
442
 
contain any of the strings you've listed.  The match is
443
 
case-insensitive and literal - no regular expressions are supported.
444
 

	
445
 
The ``-V string string ...`` command line option is the inverse
446
 
of ``-F``.  It returns a list of host and command files that do
447
 
*not contain* the strings you specify.  Again, the match is
448
 
literal and case-insensitive.
449
 

	
450
 

	
451
 
FEATURES AND USE CASES
452
 
----------------------
453
 

	
454
 
The sections below describe the various features of ``tsshbatch`` in
455
 
more detail as well as common use scenarios.
456
 

	
457
 
Different Ways To Specify Targeted Hostnames
458
 
============================================
459
 

	
460
 
There are two ways to specify the list of hosts on which you want
461
 
to run the specified command:
462
 

	
463
 
  - On the command line via the ``-H`` option::
464
 

	
465
 
      tsshbatch.py -H 'hostA hostB' uname -a
466
 

	
467
 
    This would run the command ``uname -a`` on the
468
 
    hosts ``hostA`` and ``hostB`` respectively.
469
 

	
470
 
    Notice that the list of hosts must be separated by spaces but
471
 
    passed as a *single argument*.  Hence we enclose them in single
472
 
    quotes.
473
 

	
474
 
  - Via a host list file::
475
 

	
476
 
      tsshbatch.py myhosts df -Ph
477
 

	
478
 
    Here, ``tsshbatch`` expects the file ``myhosts`` to contain a list
479
 
    of hosts, one per line, on which to run the command ``df -Ph``. As
480
 
    an example, if you want to target the hosts ``larry``, ``curly``
481
 
    and ``moe`` in ``foo.com``, ``myhosts`` would look like this::
482
 

	
483
 
      larry.foo.com
484
 
      curly.foo.com
485
 
      moe.foo.com
486
 

	
487
 
    This method is handy when there are standard "sets" of hosts on
488
 
    which you regularly work.  For instance, you may wish to keep a
489
 
    host file list for each of your production hosts, each of your
490
 
    test hosts, each of your AIX hosts, and so on.
491
 

	
492
 
    You may use the ``#`` comment character freely throughout a host
493
 
    list file to add comments or temporarily comment out a particular
494
 
    host line.
495
 

	
496
 
    You can even use the comment character to temporarily comment out
497
 
    one or most hosts in the list given to the ``-H`` command line
498
 
    argument.  For example::
499
 

	
500
 
      tsshbatch.py -H "foo #bar baz" ls
501
 

	
502
 
    This would run the ``ls`` command on hosts ``foo`` and ``baz`` but
503
 
    not ``bar``.  This is handy if you want to use your shell's
504
 
    command line recall to save typing but only want to repeat the
505
 
    command for some of the hosts your originally Specified.
506
 

	
507
 

	
508
 
Authentication Using Name And Password
509
 
======================================
510
 

	
511
 
The simplest way to use ``tsshbatch`` is to just name the hosts
512
 
can command you want to run::
513
 

	
514
 
  tsshbatch.py linux-prod-hosts uptime
515
 

	
516
 
By default, ``tsshbatch`` uses your login name found in the ``$USER``
517
 
environment variable when logging into other systems.  In this
518
 
example, you'll be prompted only for your password which ``tsshbatch``
519
 
will then use to log into each of the machines named in
520
 
``linux-prod-hosts``.  (*Notice that his assumes your name and
521
 
password are the same on each host!*)
522
 

	
523
 
Typing in your login credentials all the time can get tedious after
524
 
awhile so ``tsshbatch`` provides a means of providing them on the
525
 
command line::
526
 

	
527
 
  tsshbatch.py -n joe.luser -p my_weak_pw linux-prod-hosts uptime
528
 

	
529
 
This allows you to use ``tsshbatch`` inside scripts for hands-free
530
 
operation.
531
 

	
532
 
If your login name is the same on all hosts, you can simplify this
533
 
further by defining it in the environment variable::
534
 

	
535
 
  export TSSHBATCH="-n joe.luser"
536
 

	
537
 
Any subsequent invocation of ``tsshbatch`` will only require a
538
 
password to run.
539
 

	
540
 
HOWEVER, there is a huge downside to this - your plain text password
541
 
is exposed in your scripts, on the command line, and possibly your
542
 
command history.  This is a pretty big security hole, especially if
543
 
you're an administrator with extensive privileges.  (This is why the
544
 
``ssh`` program does not support such an option.)  For this reason, it
545
 
is strongly recommended that you use the ``-p`` option sparingly, or
546
 
not at all.  A better way is to push ssh keys to every machine and use
547
 
key exchange authentication as described below.
548
 

	
549
 
However, there are times when you do have use an explicit password,
550
 
such as when doing ``sudo`` invocations.  It would be really nice to
551
 
use ``-p`` and avoid having to constantly type in the password.  There
552
 
are two strategies for doing this more securely than just entering it
553
 
in plain text on the command line:
554
 

	
555
 
  - Temporarily store it in the environment variable::
556
 

	
557
 
      export TSSHBATCH="-n joe.luser -p my_weak_pw"
558
 

	
559
 
    Do this *interactively* after you log in, not from a script
560
 
    (otherwise you'd just be storing the plain text password in a
561
 
    different script).  The environment variable will persist as long
562
 
    as you're logged in and disappear when you log out.
563
 

	
564
 
    If you use this just make sure to observe three security
565
 
    precautions:
566
 

	
567
 
      1) Clear your screen immediately after doing this so no one
568
 
         walking by can see the password you just entered.
569
 

	
570
 
      2) Configure your shell history system to ignore commands
571
 
         beginning with ``export TSSHBATCH``.  That way your plain
572
 
         text password will never appear in the shell command history.
573
 

	
574
 
      3) Make sure you don't leave a logged in session unlocked so
575
 
         that other users could walk up and see your password by
576
 
         displaying the environment.
577
 

	
578
 
    This approach is best when you want your login credentials
579
 
    available for the duration of an *entire login session*.
580
 

	
581
 
  - Store your password in an encrypted file and decrypt it inline.
582
 

	
583
 
    First, you have to store your password in an encrypted format.
584
 
    There are several ways to do this, but ``gpg`` is commonly used::
585
 

	
586
 
      echo "my_weak_pw" | gpg -c >mysecretpw
587
 

	
588
 
    Provide a decrypt passphrase, and you're done.
589
 

	
590
 
    Now, you can use this by decrypting it inline as needed::
591
 

	
592
 
      #!/bin/sh
593
 
      # A demo scripted use of tsshbatch with CLI password passing
594
 

	
595
 
      MYPW=`cat mysecretpw | gpg`   # User will be prompted for unlock passphrase
596
 

	
597
 
      tsshbatch.py -n joe.luser -p $MYPW -i hostlist1 command1 arg
598
 
      tsshbatch.py -n joe.luser -p $MYPW -i hostlist2 command2 arg
599
 
      tsshbatch.py -n joe.luser -p $MYPW -i hostlist3 command3 arg
600
 

	
601
 
    This approach is best when you want your login credentials
602
 
    available for the duration of *the execution of a script*.  It
603
 
    does require the user to type in a passphrase to unlock the
604
 
    encrypted password file, but your plain text password never
605
 
    appears in the wild.
606
 

	
607
 

	
608
 
Authentication Using Key Exchange
609
 
=================================
610
 

	
611
 
For most applications of ``tsshbatch``, it is much simpler to use
612
 
key-based authentication.  For this to work, you must first have
613
 
pushed ssh keys to all your hosts.  You then instruct ``tsshbatch`` to
614
 
use key-based authentication rather than name and password.  Not only
615
 
does this eliminate the need to constantly provide name and password,
616
 
it also eliminates passing a plain text password on the command line
617
 
and is thus far more secure.  This also overcomes the problem of
618
 
having different name/password credentials on different hosts.
619
 

	
620
 
By default, ``tsshbatch`` will prompt for name and password if they
621
 
are not provided on the command line.  To force key- authentication,
622
 
use the ``-k`` option::
623
 

	
624
 
  tsshbatch.py -k AIX-prod-hosts ls -al
625
 

	
626
 
This is so common that you may want to set it in your ``$TSSHBATCH``
627
 
environment variable so that keys are used by default.  If you do
628
 
this, there may still be times when you want for force prompting for
629
 
passwords rather than using keys.  You can do this with the ``-K``
630
 
option which effectively overrides any prior ``-k`` selection.
631
 

	
632
 

	
633
 
Executing A ``sudo`` Command
634
 
============================
635
 

	
636
 
``tsshbatch`` is smart enough to handle commands that begin with the
637
 
``sudo`` command.  It knows that such commands *require* a password no
638
 
matter how you initially authenticate to get into the system.  If you
639
 
provide a password - either via interactive entry or the ``-p``
640
 
option - by default, ``tsshbatch`` will use that same password for
641
 
``sudo`` promotion.
642
 

	
643
 
If you provide no password - you're using ``-k`` and have not provided
644
 
a password via ``-p`` - ``tsshbatch`` will prompt you for the password
645
 
``sudo`` should use.
646
 

	
647
 
You can force ``tsshbatch`` to ask you for a ``sudo`` password with
648
 
the ``-S`` option.  This allows you to have one password for initial
649
 
login, and a different one for ``sudo`` promotion.
650
 

	
651
 
Any time you a prompted for a ``sudo`` password and a login password
652
 
has been provided (interactive or ``-p``), you can accept this as the
653
 
``sudo`` password by just hitting ``Enter``.
654
 

	
655
 
.. NOTE:: ``tsshbatch`` makes a reasonable effort to scan your command
656
 
          line and/or command file contents to spot explicit
657
 
          invocations of the form ``sudo ...``.  It will ignore these
658
 
          if they are inside single- or double quoted strings, on the
659
 
          assumption that you're quoting the literal string ``sudo
660
 
          ...`` for some other purpose.
661
 

	
662
 
          However, this is not perfect because it is not a full
663
 
          reimplementation of the shell quoting and aliasing features.
664
 
          For example, if you invoke an alias on the remote machine
665
 
          that resolves to a ``sudo`` command, or you run a script
666
 
          with a ``sudo`` command in it, ``tsshbatch`` has no way to
667
 
          determine what you're trying to do.  For complex
668
 
          applications, it's best to write a true shell script, push
669
 
          it all the machines in question via ``-P``, and then have
670
 
          ``tsshbatch`` remotely invoke it with ``sudo myscript`` or
671
 
          something similar.
672
 

	
673
 
          As always, the best way to figure out what the program
674
 
          thinks you're asking for is to run it in test mode and look
675
 
          at the diagnostic output.
676
 

	
677
 

	
678
 
Precedence Of Authentication Options
679
 
====================================
680
 

	
681
 
``tsshbatch`` supports these various authentication options in a
682
 
particular heirarchy using a "first match wins" scheme.  From highest
683
 
to lowest, the precedence is:
684
 

	
685
 
   1. Key exchange
686
 

	
687
 
   2. Forced prompting for name via -N. Notice this cancels any
688
 
      previously requested key exchange authentication.
689
 

	
690
 
   3. Command Line/$TSSHBATCH environment variable sets name
691
 

	
692
 
   4. Name picked up from $USER  (Default behavior)
693
 

	
694
 
If you try to use Key Exchange and ``tsshbatch`` detects a command
695
 
beginning with ``sudo``, it will prompt you for a password anyway.
696
 
This is because ``sudo`` requires a password to promote privilege.
697
 

	
698
 
File Transfers
699
 
==============
700
 

	
701
 
The ``-G`` and ``-P`` options specify file ``GET`` and ``PUT``
702
 
respectively.  Both are followed by a quoted file transfer
703
 
specification in the form::
704
 

	
705
 
  "path-to-source-file path-to-destination-directory"
706
 

	
707
 
Note that this means the file will always be stored under its original
708
 
name in the destination directory.  *Renaming isn't possible during
709
 
file transfer*.
710
 

	
711
 
However, ``tsshbatch`` always does ``GETs`` then ``PUTs`` *then* any
712
 
outstanding command (if any) at the end of the command line.  This
713
 
permits things like renaming on the remote machine after a ``PUT``::
714
 

	
715
 
  tsshbatch.py -P "foo ./" -i hostlist mv -v foo foo.has.a.new.name
716
 

	
717
 
``GETs`` are a bit of a different story because you are retrieving a
718
 
file of the same name on every host.  To avoid having all but the last
719
 
one clobber the previous one, ``tsshbatch`` makes forces the files you
720
 
``GET`` to be uniquely named by prepending the hostname and a "-" to
721
 
the actual file name::
722
 

	
723
 
  tsshbatch.py -H myhost -G "foo ./"
724
 

	
725
 
This saves the file ``myhost-foo`` in the ``./`` on your a local
726
 
machine.
727
 

	
728
 
These commands do not recognize any special directory shortcut symbols
729
 
like ``~/`` like the shell interpreter might.  You must name file and
730
 
directory locations using ordinary pathing conventions.  You can put
731
 
as many of these requests on the command line as you like to enable
732
 
``GETs`` and ``PUTs`` of multiple files.  You cannot, however, use
733
 
filename wildcards to specify multi-file operations.
734
 

	
735
 
You can put multiple ``GETs`` or ``PUTs`` on the command line for the
736
 
same file.  They do not override each other but are *cummulative*. So
737
 
this::
738
 

	
739
 
  tsshbatch.py -P"foo ./" -P"foo /tmp" ...
740
 

	
741
 
Would put local file ``foo`` in both ``./`` and ``/tmp`` on each host
742
 
specified. Similarly, you can specify multiple files to ``GET`` from
743
 
remote hosts and place them in the same local directory::
744
 

	
745
 
  tsshbatch.py -G"/etc/fstab ./tmp" -G"/etc/rc.conf ./tmp" ...
746
 

	
747
 
You may also put file transfer specifications into a command file via
748
 
the ``.getfile`` and ``.putfile`` directives.  This is handy when you
749
 
have many to do and don't want to clutter up the command line.  Each
750
 
must be on its own line in the command file and in the same form as if
751
 
it were provided on the command line::
752
 

	
753
 
    .getfile /path/to/srcfile destdir    # This will get a file
754
 
    .putfile /path/to/srcfile destdir    # This will put a file
755
 

	
756
 
File transfers are done in the order they appear.  For instance, if
757
 
you have a file transfer specification on the command line and then
758
 
make reference to a command file with a file transfer specification in
759
 
it, the one on the command line gets done first.
760
 

	
761
 
.. NOTE:: Keep in mind that ``tsshbatch`` always processes file
762
 
          transfers *before* executing any commands, no matter what
763
 
          order they appear in the command file.  If you have this in a
764
 
          command file::
765
 

	
766
 
              echo "Test"
767
 
              .putfile "./myfile /foo/bar/baz/"
768
 

	
769
 
          The file will be transferred *before* the ``echo`` command
770
 
          gets run.  This can be counterintuitive.  It's therefore
771
 
          recommended that you put your file transfers into a single
772
 
          file, and ``.include`` it as the first thing in your
773
 
          command file to make it obvious that these will be run first.
774
 

	
775
 

	
776
 
By default, ``tsshbatch`` aborts if any file transfer fails.  This is
777
 
unlike the case of failed commands which are reported but do *not*
778
 
abort the program.  The rationale' for this is that you may be doing
779
 
both file transfer and command execution with a single ``tsshbatch``
780
 
invocation, and the commands may depend on a file being transfered
781
 
first.
782
 

	
783
 
If you are sure no such problem exists, you can use the ``-a`` option
784
 
to disable abort-after-failure semantics on file transfer.  In this
785
 
case, file transfer errors will be reported, but ``tsshbatch`` will
786
 
continue on to the next transfer request.
787
 

	
788
 
``tsshbatch`` does preserve permissions when transferring files.
789
 
Obviously, for this to work, the destination has to be writable by the
790
 
ID you're logging in with.
791
 

	
792
 
.. NOTE:: The file transfer logic cannot cope with filenames that
793
 
          contain spaces.  The workaround is to either temporarily
794
 
          rename them, or put them in a container like a tarball or
795
 
          zip file and transfer that instead.
796
 

	
797
 

	
798
 
Commenting
799
 
==========
800
 

	
801
 
Both the command file and host file can be freely commented
802
 
using the ``#`` character.  Everything from that character to the end
803
 
of that line is ignored.  Similarly, you can use whitespace freely,
804
 
except in cases where it would change the syntax of a command or host
805
 
name.
806
 

	
807
 

	
808
 
Includes
809
 
========
810
 

	
811
 
You may also include other files as you wish with the ``.include
812
 
filename`` directive anywhere in the command file or host file.
813
 
This is useful for breaking up long lists of things into smaller
814
 
parts.  For example, suppose you have three host lists, one for each
815
 
major production areas of your network::
816
 

	
817
 
    hosts-development
818
 
    hosts-stage
819
 
    host-production
820
 

	
821
 
You might typically run different ``tsshbatch`` jobs on each of these
822
 
sets of hosts.  But suppose you now want to run a job on all of them.
823
 
Instead of copying them all into a master file (which would be
824
 
instantly obsolete if you changed anything in one of the above files),
825
 
you could create ``hosts-all`` with this content::
826
 

	
827
 
  .include hosts-development
828
 
  .include hosts-stage
829
 
  .include hosts-production
830
 

	
831
 
that way if you edited any of the underlying files, the
832
 
``hosts-all`` would reflect the change.
833
 

	
834
 
Similarly you can do the same thing with the command file to group
835
 
similar commands into separate files and include them.
836
 

	
837
 
``tsshbatch`` does not enforce a limit on how deeply nested
838
 
``.includes`` can be.  An included file can include another file and
839
 
so on.  However, if a circular include is detected, the program will
840
 
notify you and abort.  This happens if, say, file1 includes file2,
841
 
file2 includes file3, and file3 includes file1.  This would create an
842
 
infinite loop of includes if permitted.  You can, of course, include
843
 
the same file multiple times, either in a single file or throughout
844
 
other included files, so long as no circular include is created.
845
 

	
846
 
The target of a ``.include`` directive can also contain variable
847
 
references.  Note, however, that references to builtin variables will
848
 
fail unless you have overriden them.  Why?  Because builtins don't get
849
 
defined until a host connection is attempted.  This doesn't happen
850
 
until *after* all global variable processing and file includes
851
 
have been done.  So::
852
 

	
853
 
  .define MYINCLUDE = /some/fine/file                    # OK
854
 
  .include MYINCLUDE
855
 

	
856
 
  .define MYINCLUDE = ! find /some/path -name includeski # OK
857
 
  .include MYINCLUDE
858
 

	
859
 
  .include __HOSTNAME__   # Nope, not defined yet -
860
 
                          # tries to include file called '__HOSTNAME__'
861
 

	
862
 
  .define __HOSTNAME__ ! hostname  # Override the builtin
863
 
  .include __HOSTNAME__                                   # OK
864
 

	
865
 
As a matter of keeping things simple, stick to global variables
866
 
as part of an ``.include`` target.
867
 

	
868
 

	
869
 
Search Paths
870
 
============
871
 

	
872
 
``tsshbatch`` supports the ablity to search paths to find files you've
873
 
referenced.  The search path for ``cmdfiles`` is specified in the
874
 
``$TSSHBATCHCMDS`` environment variable.  The ``hostlistfiles`` search
875
 
path is specified in the ``$TSSHBATCHHOSTS`` environment variable.
876
 
These are both in standard path delimited format for your operating
877
 
system.  For example, on Unix-like systems these look like this::
878
 

	
879
 
  export TSSHBATCHCMDS="/usr/local/etc/.tsshbatch/commands:/home/me/.tsshbatch/commands"
880
 

	
881
 
And so forth.
882
 

	
883
 
These paths are honored both for any files you specify on the command
884
 
line as well as for any files you reference in a ``.include``
885
 
directive.  This allows you to maintain libraries of standard commands
886
 
and host lists in well known locations and ``.include`` the ones you
887
 
need.
888
 

	
889
 
``tsshbatch`` will always first check to see if a file you've
890
 
specified is in your local (invoking) directory and/or whether it is a
891
 
fully qualified file name before attempting to look down a search
892
 
path.  If a file exist in several locations, the first instance found
893
 
"wins".  So, for instance, if you have a file called ``myhosts``
894
 
somewhere in the path defined in ``$TSSHBATCHHOSTS``, you can override
895
 
it by creating a file of same name in your current working directory.
896
 

	
897
 
``tsshbatch`` also checks for so-called "circular includes" which
898
 
would cause an infinite inclusion loop.  It will abort upon
899
 
discovering this, prior to any file transfers or commands being
900
 
executed.
901
 

	
902
 
An Overview Of Variables
903
 
========================
904
 

	
905
 
As you become more sophisticated in your use of ``tsshbatch``, you'll
906
 
begin to see the same patterns of use over and over again.  Variables
907
 
are a way for you to use "shortcuts" to reference long strings
908
 
without having to type the whole string in every time.  So, for example,
909
 
instead of having to type in a command like this::
910
 

	
911
 
  myfinecommand -X -Y -x because this is a really long string
912
 

	
913
 
You can just define variable like this::
914
 

	
915
 
  .define __MYCMD__ =  myfinecommand -X -Y -x because this is a really long string
916
 

	
917
 
From then on, instead of typing in that long command on the command line or in
918
 
a command file, you can just use ``__MYCMD__`` and ``tsshbatch`` will substitute
919
 
the string as you defined it whenever it encounters the variable.
920
 

	
921
 
Variables can be used pretty much everwhere:
922
 

	
923
 
  - In ``hostlistfiles`` or in the hostnames listed with ``-H``::
924
 

	
925
 
      .define __MYDOMAIN__ = stage.mydomain.com
926
 
      #.define __MYDOMAIN__ = prod.mydomain.com
927
 

	
928
 
      host1.__MYDOMAIN__
929
 
      host2.__MYDOMAIN__
930
 

	
931
 
    Now you can switch ``tsshbatch`` operation from stage to prod simply
932
 
    by changing what is commented out at the beginning.
933
 

	
934
 
  - In file transfer specifications::
935
 

	
936
 
      tsshbatch.py -xP"./fstab-__MYHOSTNAME__  ./" -i hostlist
937
 
      tsshbatch.py -xG"/etc/__OSNAME__-release ./" -i hostlist
938
 

	
939
 
  - In ``cmdfiles``::
940
 

	
941
 
      .define __SHELL__ = /usr/local/bin/bash
942
 

	
943
 
      __SHELL__ -c myfinescript
944
 

	
945
 
.. NOTE:: A variable can have pretty much any name you like excepting
946
 
          the use of metacharacters like ``<`` or ``!``.  But if
947
 
          you are not careful, you can cause unintended errors::
948
 

	
949
 
            .define foo = Slop
950
 

	
951
 
            myfoodserver.foods.com
952
 

	
953
 
          When you run ``tsshbatch`` it will then turn the server name
954
 
          into ``mySlopdserver.Slopds.com`` - probably not what you
955
 
          want.
956
 

	
957
 
          So, it's a Really Good Idea (tm) to use some kind of naming
958
 
          scheme to make variables names stand out and make them
959
 
          unlikely to conflict accidentally with command- and host
960
 
          strings.
961
 

	
962
 

	
963
 
Types Of Variables
964
 
==================
965
 

	
966
 
``tsshbatch`` has four different kinds of variables:
967
 

	
968
 
  - *Global Variables* are the kind in the example above.
969
 
    You, the user, define them as you wish in a command file or
970
 
    host file.
971
 

	
972
 
  - *Local Variables* work exactly the same a global variables, except
973
 
    that they can only be used in the file (command- or host) in which
974
 
    they are defined.  These variables are defined with the ``.local
975
 
    name = value`` syntax.
976
 

	
977
 
  - *Execution Variables* run any program or script of your choosing
978
 
    (on the same machine you're running ``tsshbatch``) and assign
979
 
    the results to either a global- or locala variable.
980
 

	
981
 
  - *Builtin Variables* are variables the ``tsshbatch`` itself
982
 
    defines.  You can override their default values by creating
983
 
    a Global Variable of the same name.
984
 

	
985
 

	
986
 
Where And When Do Variables Get Processed?
987
 
==========================================
988
 

	
989
 
Global, Local, and Execution Variables are defined in either a
990
 
host file or command file.
991
 

	
992
 
Builtin Variables are defined within ``tsshbatch`` itself unless you
993
 
override them.
994
 

	
995
 
Global Variables are *all* read in and *then* used.  If you do
996
 
something like this::
997
 

	
998
 
   .define __FOO__ = firstfoo
999
 
   echo __FOO__
1000
 
   .define __FOO__ = secondfoo
1001
 

	
1002
 
You'll get an output of ... ``secondfoo``!  Why?  Because before
1003
 
``tsshbatch`` tries to run anything, it has to process all the
1004
 
``cmdfiles``, host file, and the command line content.  So,
1005
 
before we ever get around to doing an ``echo __FOO__`` on some host,
1006
 
the second definition of __FOO__ has been read in ... and last
1007
 
definition wins.
1008
 

	
1009
 
Local Variables are read in as a command- or host file is
1010
 
processed.  They can only be referenced *within the same
1011
 
file* - i.e., They are not visible to any other file.
1012
 

	
1013
 
Execution Variables are processed a single time *at the time they're
1014
 
read in* from a command file or host file.
1015
 

	
1016
 
Builtin Variables get evaluated *every time ``tsshbatch`` prepares to
1017
 
connect to a new host* (unless you've overriden them).  That way, the
1018
 
most current value for them is available for use on the next host.
1019
 

	
1020
 
Keep in mind that ``tsshbatch`` isn't a programming language.  It's
1021
 
"variables" are simple string substitutions with "last one wins"
1022
 
semantics (for globals).  If you define the same global variable in,
1023
 
say, a command file and also in the host file, the latter will "win".
1024
 
Why?  Because ``hostlistfiles`` are always read in after any
1025
 
``cmdfiles``.
1026
 

	
1027
 
Finally, variable references in a definition are *ignored*.  Say you
1028
 
do this in a command file::
1029
 

	
1030
 
  .define __CLEVER      __ = __REALLYCLEVER__
1031
 
  .define __REALLYCLEVER__ = Not That Smart
1032
 
  echo __CLEVER__
1033
 

	
1034
 
You will get this output, ``__REALLYCLEVER__``!  Why?  Because, the
1035
 
variable references on the right side of a definition statement are
1036
 
never replaced.  This is a concious design choice to keep variable
1037
 
definition and use as simple and obvious as possible.  Allowing such
1038
 
"indirect" definitions opens up a treasure trove of maintenance pain
1039
 
you really want to avoid.  Trust us on this one.
1040
 

	
1041
 

	
1042
 
Global Variables
1043
 
================
1044
 

	
1045
 
``tsshbatch`` allows you to define variables which will then be used
1046
 
to replace matching strings in ``cmdfiles``, ``hostlistfiles``, and
1047
 
file transfer specifications.  For example, suppose you have this in a
1048
 
host file::
1049
 

	
1050
 
  .define DOMAIN=.my.own.domain.com
1051
 

	
1052
 
  host1DOMAIN
1053
 
  host2DOMAIN
1054
 
  host3DOMAIN
1055
 

	
1056
 
At runtime, the program will actually connect to
1057
 
``host1.my.own.domain.com``, ``host2.my.domain.com``, and so on.  This
1058
 
allows for ease of modularization and maintenance of your files.
1059
 

	
1060
 
Similarly, you might want define ``MYCMD=some_long_string`` so you
1061
 
don't have to type ``some_long_string`` over and over again in a
1062
 
command file.
1063
 

	
1064
 
There are some "gotchas" to this:
1065
 

	
1066
 
  - The general form of a variable definition is::
1067
 

	
1068
 
      .define name = value
1069
 

	
1070
 
    You have to have a name but the value is optional.  ``.define
1071
 
    FOO=`` simply replaces any subsequent ``FOO`` strings with
1072
 
    nothing, effectively removing them.
1073
 

	
1074
 
    Any ``=`` symbols to the right of the one right after ``name`` are
1075
 
    just considered part of the variable's value.
1076
 

	
1077
 
    Whitespace around the ``=`` symbol is optional but allowed.
1078
 

	
1079
 
  - Variables are substituted in the order they appear::
1080
 

	
1081
 
      .define LS = ls -alr
1082
 
      LS /etc               # ls -alr /etc
1083
 
      .define LS = ls -1
1084
 
      LS /foo               # ls -1 /foo
1085
 

	
1086
 
  - Variable names and values are *case sensitive*.
1087
 

	
1088
 
  - Variables may be defined in either ``cmdfiles`` or
1089
 
    ``hostlistfiles`` but they are *visible to any subsequent file
1090
 
    that gets read*.  For instance, ``cmdfiles`` are read before any
1091
 
    ``hostlistfiles``.  Any variables you've defined in a command file
1092
 
    that happen to match a string in one of your hostnames will be
1093
 
    substituted.
1094
 

	
1095
 
    This is usually not what you want, so be careful.  One way to
1096
 
    manage this is to use variables names that are highly unlikely to
1097
 
    ever show up in a hostname or command.  That way your commands and
1098
 
    hostnames will not accidentally get substrings replaced with
1099
 
    variable values.  For example, you might use variable names like
1100
 
    ``--MYLSCOMMAND--`` or ``__DISPLAY_VGS__``.
1101
 

	
1102
 
  - Variable substitution is also performed on any host names
1103
 
    or commands passed on the command line.
1104
 

	
1105
 

	
1106
 
Local Variables
1107
 
===============
1108
 

	
1109
 
As we saw previously, global variables have "last one wins"
1110
 
semantics.  So, if the same variable name appears in different
1111
 
files, the last instance read will be the value assigned to
1112
 
that variable.  Sometimes, you don't want this behavior. You
1113
 
want to define a variable in a file and *only use it there
1114
 
without changing a global variable with the same name*.
1115
 

	
1116
 
Say we have three files::
1117
 

	
1118
 
  # First command file
1119
 
  .define __DEFAULT__ = ls -al  # global variable
1120
 

	
1121
 
  # Second command file
1122
 
  .local __DEFAULT__ = ls -1    # variable is local to 2nd file
1123
 
  __DEFAULT__
1124
 

	
1125
 

	
1126
 
  # Third command file
1127
 
  __DEFAULT__
1128
 

	
1129
 

	
1130
 
Now, say we run this command::
1131
 

	
1132
 
  tsshbatch.py -xH"somehost" -f file1 -f file2 -f file3
1133
 

	
1134
 

	
1135
 
The net commands to be run on ``somehost`` will be::
1136
 

	
1137
 
  ls -1
1138
 
  ls -al
1139
 

	
1140
 
The reference to ``__DEFAULT`` in ``file2`` is resolved with the
1141
 
*local* definition, and the reference in ``file3`` is resolved with
1142
 
the *global* definition.
1143
 

	
1144
 
Local variables accompish several things.  First, they "insulate" a
1145
 
host- or command file from prior- or subsequent variable definitions
1146
 
found in other files.  You are guaranteed that the local copy of a
1147
 
varaible always "wins".
1148
 

	
1149
 
Conversely, local variables also protect global variables of the same
1150
 
name from changing.  In the example above, we have default behavior
1151
 
which can be overriden in a case-by-case basis, without changing the
1152
 
"master" definition.
1153
 

	
1154
 

	
1155
 
Execution Variables
1156
 
===================
1157
 

	
1158
 

	
1159
 
Execution Variables are actually a special case of Global and Local
1160
 
Variables.  That is, they are evaluated at the same time and in the
1161
 
same manner as any other variable.  The difference is that a normal
1162
 
variable describes a *literal string replacement*.  But an Execution
1163
 
Variable *runs a command, program, or script and assigns the results
1164
 
to the variable*.
1165
 

	
1166
 
For example, suppose you want create a file on many machines, and you
1167
 
want that file to be named based on who ran the ``tsshbatch`` job.
1168
 
You might do this in a command file::
1169
 

	
1170
 
  .define __WHOAMI__ = ! whoami
1171
 
  touch __WHOAMI__-Put_This_Here.txt
1172
 

	
1173
 
So, if ID ``luser`` is running ``tsshbatch``, a file called
1174
 
``luser-Put_This_Here.txt`` will be created (or have its timestamp
1175
 
updated) on every machine in the host file or named with
1176
 
``-H``.
1177
 

	
1178
 
Similarly, you can do the same thing with local variables::
1179
 

	
1180
 
  .local __TYPE__ = ! uname
1181
 

	
1182
 
Notice it is the ``!`` character that distinguishes an Execution
1183
 
Variable from a Global Variable.  It is this character that
1184
 
tells ``tsshbatch``, "Go run the command to the right of me and return
1185
 
the results."  The trailing space is optional and the definition could
1186
 
be written as::
1187
 

	
1188
 
  .define __WHOAMI__ = !whoami
1189
 

	
1190
 
If the command you specify returns multiple lines of output, it's up
1191
 
to you to process it properly.  ``tsshbatch`` does no newline
1192
 
stripping or other postprocessing of the command results.  This can
1193
 
make the output really "noisy".  ``tssbatch`` normally reports
1194
 
a summary of the command and its results.  But if you do something
1195
 
like this::
1196
 

	
1197
 
  .define __LS__ = ! ls -al
1198
 
  echo __LS__
1199
 

	
1200
 
You will get a multiline summary of the command and then the actual
1201
 
output - which is also multiline.  This gets to be obnonxious pretty
1202
 
quickly.  You can make a lot of this go away with the ``-q``, or
1203
 
"quiet" option.
1204
 

	
1205
 
.. NOTE:: It's important to remember that the program you are invoking
1206
 
          *runs on the same machine as* ``tsshbatch`` *itself*, NOT each
1207
 
          host you are sending commands to.  In other words, just
1208
 
          like Builtin Variables, Execution Variables are *locally
1209
 
          defined*.
1210
 

	
1211
 

	
1212
 
Builtin Variables
1213
 
=================
1214
 

	
1215
 
As noted previously, Builtin Variables are created by ``tsshbatch``
1216
 
itself.  They are created for each new host connection so that things
1217
 
like time, host number, and hostname are up-to-date.
1218
 

	
1219
 
As of this release, ``tsshbatch`` supports the following Builtins:
1220
 

	
1221
 
        ====================    ===============================================
1222
 
         ``__DATE__``           Date in YYYYMMDD format
1223
 
         ``__DATETIME__``       Date and time in YYYYMMDDHHMMSS format
1224
 
         ``__HOSTNAME__``       Full name of current host as passed to ``tsshbatch``
1225
 
         ``__HOSTNUM__``        Count of host being processed, starting at 1
1226
 
         ``__HOSTSHORT__``      Leftmost component of hostname as passed to ``tsshbatch``
1227
 
         ``__LOGINNAME__``      User name used for remote login.  For key auth, name of tsshbatch user.
1228
 
         ``__TIME__``           Time in HHMMSS format
1229
 
        ====================    ===============================================
1230
 

	
1231
 

	
1232
 
Using Builtin Variables
1233
 
=======================
1234
 

	
1235
 

	
1236
 
There are times when it's convenient to be able to embed the name of
1237
 
the current host in either a command or in a file transfer
1238
 
specification.  For example, suppose you want to use a single
1239
 
invocation of ``tsshbatch`` to transfer files in a host-specific way.
1240
 
You might name your files like this::
1241
 

	
1242
 
  myfile.host1
1243
 
  myfile.host2
1244
 

	
1245
 
Now, all you have to do is this::
1246
 

	
1247
 
  tsshbatch.py -xH "host 1 host2" -P "myfile.__HOSTNAME__ ./"
1248
 

	
1249
 
When run, ``tsshbatch`` will substitute the name of the current host
1250
 
in place of the string ``__HOSTNAME__``.  (*Note that these are
1251
 
**double** underbars on each side of the string.*)
1252
 

	
1253
 
You can do this in commands (and commands within command files) as
1254
 
well::
1255
 

	
1256
 
  tsshbatch.py -x hosts 'echo I am running on __HOSTNAME__'
1257
 

	
1258
 
Be careful to escape and quote things properly, especially from the
1259
 
the command line, since ``<`` and ``>`` are recognized by the shell as
1260
 
metacharacters.
1261
 

	
1262
 
There are two forms of host name substitution possible.  The first,
1263
 
``__HOSTNAME__`` will use the name *as you provided it*, either as an
1264
 
argument to ``-H`` or from within a host file.
1265
 

	
1266
 
The second, ``__HOSTSHORT__``, will only use the portion of the name
1267
 
string you provided up to the leftmost period.
1268
 

	
1269
 
So, if you specify ``myhost1.frumious.edu``, ``__HOSTNAME__`` will be
1270
 
replaced with that entire string, and ``__HOSTSHORT__`` will be
1271
 
replaced by just ``myhost1``.
1272
 

	
1273
 
Notice that, in no case does ``tsshbatch`` do any DNS lookups to
1274
 
figure this stuff out.  It just manipulates the strings you provide as
1275
 
hostnames.
1276
 

	
1277
 
The symbols ``__HOSTNAME__`` and ``__HOSTSHORT__`` are like any other
1278
 
symbol you might have specified yourself with ``.define``.  *This
1279
 
means you can override their meaning*.  For instance, say you're doing
1280
 
this::
1281
 

	
1282
 
  tsshbatch.py -x myhosts  echo "It is: __HOSTNAME__"
1283
 

	
1284
 
As you would expect, the program will log into that host, echo the
1285
 
hostname and exit.  But suppose you don't want it to echo something
1286
 
else for whatever reason.  You'd create a command file with this
1287
 
entry::
1288
 

	
1289
 
  .define __HOSTNAME__ = Really A Different Name
1290
 

	
1291
 
Now, when you run the command above, the output is::
1292
 

	
1293
 
  It is: Really A Different Name
1294
 

	
1295
 
In other words, ``.define`` has a *higher precedence* than the
1296
 
preconfigured values of ``HOSTNAME`` and ``HOSTSHORT``.
1297
 

	
1298
 

	
1299
 
Noise Levels
1300
 
============
1301
 

	
1302
 
``tsshbatch`` defaults to a medium level of reporting as it runs.
1303
 
This includes connection reporting, headers describing the command
1304
 
being run on every host,and the results written to ``stdin`` and
1305
 
``stdout``.  Each line of reporting output begins with ``--->`` to
1306
 
help you parse through the output if you happen to be writing a
1307
 
program that post-processes the results from ``tsshbatch``.
1308
 

	
1309
 
This output "noise" is judged to be right for most applications of the
1310
 
program.  There are times, however, when you want more- or less
1311
 
"noise" in the output.  There are several ``tsshbatch`` options that
1312
 
support this.
1313
 

	
1314
 
These options *only affect reporting of
1315
 
commands you're running*.  They do not change the output of file
1316
 
transfer operations.  They also do not change error reporting, which
1317
 
is always the same irrespective of current noise level setting.
1318
 

	
1319
 
``-q`` or "quiet" mode, reduces the amount of output noise in
1320
 
two ways.  First, it silences reporting each time a successful
1321
 
connection is made to a host.  Secondly, the command being run
1322
 
isn't reported in the header.   For example, normally, running
1323
 
``ls -l`` is reported like this::
1324
 

	
1325
 
  --->  myhost:    SUCCESS: Connection Established
1326
 
  --->  myhost (stdout) [ls -l]:
1327
 
  ...
1328
 
  --->  myhost (stderr) [ls -l]:
1329
 

	
1330
 
In quiet mode, reporting looks like this::
1331
 

	
1332
 
   --->  localhost (stdout):
1333
 
   ...
1334
 
   --->  localhost (stderr):
1335
 

	
1336
 
The main reason for this is that some commands can be very long.
1337
 
With execution variables, it's possible to create commands that
1338
 
span many lines.  The quiet option gives you the ability to
1339
 
suppress echoing these long commands for each and every host
1340
 
in your list.
1341
 

	
1342
 
``-y`` or "noisy" mode, produces normal output noise but
1343
 
also replicates the hostname and command string *for
1344
 
every line of output produced*.  For instance, ``ls -1``
1345
 
might normally produce this::
1346
 

	
1347
 
  --->  myhost:    SUCCESS: Connection Established
1348
 
  --->  myhost (stdout) [ls -1]:
1349
 

	
1350
 
         backups
1351
 
         bin
1352
 

	
1353
 
But in noisy mode, you see this::
1354
 

	
1355
 
  --->  myhost:    SUCCESS: Connection Established                                                                                                                     --->  myhost (stdout) [ls -1]:
1356
 
  [myhost (stdout) [ls -1]]        backups
1357
 
  [myhost (stdout) [ls -1]]        bin
1358
 

	
1359
 
Again, the purpose here is to support post-processing where you might
1360
 
want to search through a large amount of output looking only for
1361
 
results from particular hosts or commands.
1362
 

	
1363
 
``-s`` or "silent" mode returns *only the results from running the
1364
 
commands*.  No headers or descriptive information are produced.  It's
1365
 
more-or-less what you'd see if you logged into the host and ran the
1366
 
command interactively.  For instance, ``ls -l`` might look like this::
1367
 

	
1368
 
  total 44
1369
 
  drwxr-xr-x  2 splot splot 4096 Nov  5 14:54 Desktop
1370
 
  drwxrwxr-x 39 splot splot 4096 Sep  9 14:57 Dev
1371
 
  drwxr-xr-x  3 splot splot 4096 Jun 14  2012 Documents
1372
 

	
1373
 

	
1374
 
The idea here is to use silent mode with the various variables
1375
 
described previously to customize your own reporting output.  Imagine
1376
 
you have this in a command file and you run ``tsshbatch`` in silent
1377
 
mode::
1378
 

	
1379
 
  .define __USER__ = ! echo $USER
1380
 
  echo "Run on __HOSTNAME__ on __DATE__ at __TIME__ by __USER__"
1381
 
  uname -a
1382
 

	
1383
 
You'd see output along these lines::
1384
 

	
1385
 
  Run on myhost on 20991208 at 141659 by splot
1386
 
  Linux myhost 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
1387
 

	
1388
 

	
1389
 

	
1390
 

	
1391
 
OTHER
1392
 
-----
1393
 

	
1394
 
- Comments can go anywhere.
1395
 

	
1396
 
- Directives like ``.define`` and ``.include`` must be the first
1397
 
  non-whitespace text on the left end of a line.  If you do this in a
1398
 
  command file::
1399
 

	
1400
 
      foo .include bar
1401
 

	
1402
 
  ``tsshbatch`` thinks you want to run the command ``foo`` with an
1403
 
  argument of ``.include bar``.  If you do it in a host file,
1404
 
  the program thinks you're trying to contact a host called ``foo
1405
 
  .include bar``.  In neither case is this likely to be quite what you
1406
 
  had in mind.  Similarly, everything to the right of the directive is
1407
 
  considered its argument (up to any comment character).
1408
 

	
1409
 
- Whitespace is not significant at the beginning or end of a line but
1410
 
  it is preserved within ``.define`` and ``.include`` directive
1411
 
  arguments as well as within commmand definitions.
1412
 

	
1413
 
- Strictly speaking, you do not have to have whitespace after a
1414
 
  directive.  This is recognized::
1415
 

	
1416
 
      .includesomefileofmine
1417
 
      .definemyvar=foo
1418
 

	
1419
 
  But this is *strongly* discouraged because it's really hard to read.
1420
 

	
1421
 
- ``tsshbatch`` writes the ``stdout`` of the remote host(s) to
1422
 
  ``stdout`` on the local machine.  It similarly writes remote
1423
 
  ``stderr`` output to the local machine's ``stderr``.  If you wish to
1424
 
  suppress ``stderr`` output, either redirect it on your local command
1425
 
  line or use the ``-e`` option to turn it off entirely.  If you want
1426
 
  everything to go to your local ``stdout``, use the ``-E`` option.
1427
 

	
1428
 
- You must have a reasonably current version of Python 2.x installed.
1429
 
  It almost certainly will not work on Python 3.x because it uses the
1430
 
  deprecated ``commands`` module.  This decision was made to make the
1431
 
  program as backward compatible with older versions of Python as
1432
 
  possible (there is way more 2.x around than there is 3.x).
1433
 

	
1434
 
- If your Python installation does not install ``paramiko`` you'll
1435
 
  have to install it manually, since ``tsshbatch`` requires these
1436
 
  libraries as well.
1437
 

	
1438
 
- ``tsshbatch`` has been run extensively from Unix-like systems (Linux,
1439
 
  FreeBSD) and has had no testing whatsoever on Microsoft Windows.  If
1440
 
  you have experience using it on Windows, do please share with the
1441
 
  class using the email address below.  While we do not officially
1442
 
  support this tool on Windows, if the changes needed to make it work
1443
 
  properly are small enough, we'd consider updating the code
1444
 
  accordingly.
1445
 

	
1446
 

	
1447
 
BUGS AND MISFEATURES
1448
 
--------------------
1449
 

	
1450
 
- You will not be able to run remote ``sudo`` commands if the host in
1451
 
  question enables the ``Defaults requiretty`` in its ``sudoers``
1452
 
  configuration.  Some overzealous InfoSec folks seem to think this is
1453
 
  a brilliant way to secure your system (they're wrong) and there's
1454
 
  nothing ``tsshbatch`` can do about it.
1455
 

	
1456
 
- When ``sudo`` is presented a bad password, it ordinarily prints a
1457
 
  string indicating something is wrong.  ``tsshbatch`` looks for this
1458
 
  to let you know that you've got a problem and then terminates
1459
 
  further operation.  This is so that you do not attempt to log in
1460
 
  with a bad password across all the hosts you have targeted.  (Many
1461
 
  enterprises have policies to lock out a user ID after some small
1462
 
  number of failed login/access attempts.)
1463
 

	
1464
 
  However, some older versions of ``sudo`` (noted on a RHEL 4 host
1465
 
  running ``sudo`` 1.6.7p5) do not return any feedback when presented
1466
 
  with a bad password.  This means that ``tsshbatch`` cannot tell the
1467
 
  difference between a successful ``sudo`` and a system waiting for
1468
 
  you to reenter a proper password.  In this situation, if you enter a
1469
 
  bad password, the *the program will hang*.  Why?  ``tsshbatch``
1470
 
  thinks nothing is wrong and waits for the ``sudo`` command to
1471
 
  complete.  At the same time, ``sudo`` itself is waiting for an
1472
 
  updated password.  In this case, you have to kill ``tsshbatch`` and
1473
 
  start over.  This typically requires you to put the program in
1474
 
  background (```Ctrl-Z`` in most shells) and then killing that job
1475
 
  from the command line.
1476
 

	
1477
 
  There is no known workaround for this problem.
1478
 

	
1479
 

	
1480
 
OTHER, SIMILAR PRODUCTS
1481
 
-----------------------
1482
 

	
1483
 
It's always interesting to see how other people approach the same
1484
 
problem.  If you're interested in this general area of IT automation,
1485
 
you may want to also look at ``Ansible``, ``Capistrano``, ``Cluster
1486
 
SSH``, ``Fabric``, ``Func``, and ``Rundeck``.
1487
 

	
1488
 

	
1489
 
COPYRIGHT AND LICENSING
1490
 
-----------------------
1491
 

	
1492
 
**tsshbatch** is Copyright (c) 2011-2016 TundraWare Inc.
1493
 

	
1494
 
For terms of use, see the ``tsshbatch-license.txt`` file in the
1495
 
program distribution.  If you install **tsshbatch** on a FreeBSD
1496
 
system using the 'ports' mechanism, you will also find this file in
1497
 
``/usr/local/share/doc/tsshbatch``.
1498
 

	
1499
 

	
1500
 
AUTHOR
1501
 
------
1502
 

	
1503
 
::
1504
 

	
1505
 
   Tim Daneliuk
1506
 
   tsshbatch@tundraware.com
1507
 

	
1508
 

	
1509
 
DOCUMENT REVISION INFORMATION
1510
 
-----------------------------
1511
 

	
1512
 
::
1513
 

	
1514
 
  $Id: '345a9ed tundra Sat Oct 15 16:12:01 2016 -0500'
1515
 

	
1516
 
This document was produced with ``emacs``, ``RestructuredText``, and ``TeX Live``.
1517
 

	
1518
 
You can find the latest version of this program at:
1519
 

	
1520
 
  http://www.tundraware.com/Software/tsshbatch
0 comments (0 inline, 0 general)