0 0 1
default
Jean-Francois Pieronne - 4 years ago 2016-11-30 15:39:17
jf.pieronne@laposte.net
Docs/tsshbatch.txt initial version
1 file changed with 1448 insertions and 0 deletions:
↑ Collapse Diff ↑
 
new file 100644
1
 
()                                                                          ()
2
 

	
3
 

	
4
 

	
5
 
NAME
6
 
        -
7
 

	
8
 
   Contents
9
 
       o NAME
10
 

	
11
 
       o SYNOPSIS
12
 

	
13
 
       o DESCRIPTION
14
 

	
15
 
       o OPTIONS
16
 

	
17
 
       o ENVIRONMENT
18
 

	
19
 
       o SSH CONFIGURATION FILE PROCESSING
20
 

	
21
 
       o MANAGING INVENTORY: HOST FILES
22
 

	
23
 
       o MANAGING JOBS: COMMAND FILES
24
 

	
25
 
       o SEARCHING YOUR HOST AND COMMAND FILES
26
 

	
27
 
       o FEATURES AND USE CASES
28
 

	
29
 
         o Different Ways To Specify Targeted Hostnames
30
 

	
31
 
         o Authentication Using Name And Password
32
 

	
33
 
         o Authentication Using Key Exchange
34
 

	
35
 
         o Executing A sudo Command
36
 

	
37
 
         o Precedence Of Authentication Options
38
 

	
39
 
         o File Transfers
40
 

	
41
 
         o Commenting
42
 

	
43
 
         o Includes
44
 

	
45
 
         o Search Paths
46
 

	
47
 
         o An Overview Of Variables
48
 

	
49
 
         o Types Of Variables
50
 

	
51
 
         o Where And When Do Variables Get Processed?
52
 

	
53
 
         o Global Variables
54
 

	
55
 
         o Local Variables
56
 

	
57
 
         o Execution Variables
58
 

	
59
 
         o Builtin Variables
60
 

	
61
 
         o Using Builtin Variables
62
 

	
63
 
         o Noise Levels
64
 

	
65
 
       o OTHER
66
 

	
67
 
       o BUGS AND MISFEATURES
68
 

	
69
 
       o OTHER, SIMILAR PRODUCTS
70
 

	
71
 
       o COPYRIGHT AND LICENSING
72
 

	
73
 
       o AUTHOR
74
 

	
75
 
       o DOCUMENT REVISION INFORMATION
76
 

	
77
 
NAME
78
 
       tsshbatch - Run Commands On Batches Of Machines
79
 

	
80
 
       WARNING:
81
 
          tsshbatch  is  a  powerful  tool  for  automating activities on many
82
 
          servers at a time.  This also gives you to power to make  many  mis-
83
 
          takes  at a time! This is especially true if you have sudo privilege
84
 
          promotion capabilities on the systems in your care.  So  be  careful
85
 
          out there!
86
 

	
87
 
          We therefore STRONGLY recommend you do the following things to miti-
88
 
          gate this risk:
89
 

	
90
 
              o Read This Fine Manual from beginning to end.
91
 

	
92
 
              o Practice using tsshbatch on test machines or VMs that can eas-
93
 
                ily be recovered or reimaged if you break someting.
94
 

	
95
 
              o Make heavy use of test mode (which is the default) to see what
96
 
                the program would do if it actually ran in execution mode.
97
 

	
98
 
SYNOPSIS
99
 
          tsshbatch.py [-EKLNSTWaehkqrstvxy -G 'file dest' -P 'file dest' -f cmdfile -l logfile -n name -p pw ] -H 'host ..' -i 'hostfile ...' [command arg ... ]
100
 

	
101
 
DESCRIPTION
102
 
       tsshbatch is a tool to enable you to issue  a  command  to  many  hosts
103
 
       without  having to log into each one separately.  When writing scripts,
104
 
       this overcomes the ssh limitation of not  being  able  to  specify  the
105
 
       password on the command line.
106
 

	
107
 
       You can also use tsshbatch to GET and PUT files from- and to many hosts
108
 
       at once.
109
 

	
110
 
       tsshbatch also understands basic sudo syntax and can be used to  access
111
 
       a host, sudo a command, and then exit.
112
 

	
113
 
       tsshbatch  thus  allows  you  to  write complex, hands-off scripts that
114
 
       issue commands to many hosts without the tedium  of  manual  login  and
115
 
       sudo  promotion.   System  administrators,  especially,  will find this
116
 
       helpful when working in large server farms.
117
 

	
118
 
OPTIONS
119
 
       tsshbatch supports a variety of  options  which  can  be  specified  on
120
 
       either the command line or in the $TSSHBATCH environment variable:
121
 

	
122
 
          -B     Print  start,  stop,  and  elapsed execution time statistics.
123
 
                 This does not include any time spent for interactive  prompt-
124
 
                 ing  and  response,  but  reflects  actual  program  runtime.
125
 
                 (Default: Off)
126
 

	
127
 
          -C configfile
128
 
                 Specify  the  location  of  the   ssh   configuration   file.
129
 
                 (Default: ~/.ssh/config)
130
 

	
131
 
          -E     Normally,  tsshbatch  writes  it's  own errors to stderr.  It
132
 
                 also writes the stderr output from each host it  contacts  to
133
 
                 the  local  shell's  stderr  (unless  the  -e option has been
134
 
                 selected).
135
 

	
136
 
                 The -E option redirects any such  tsshbatch  output  intended
137
 
                 for  stderr  to  stdout  instead.  This avoids the need to do
138
 
                 things like 2>&1 | ...` on the command line when you want  to
139
 
                 pipe all ``tsshbatch output to another program.
140
 

	
141
 
          -F strings
142
 
                 This  will  examine every file on the host- or command paths,
143
 
                 looking for matching strings  within  these  files.   Matches
144
 
                 will  report the file name, the location within the file, and
145
 
                 the line containing any of the specified strings.
146
 

	
147
 
                 This is a simple, case-insensitive string literal  match  and
148
 
                 does not support regular expressions.
149
 

	
150
 
                 This  is handy when you're looking for a host name or command
151
 
                 string, say like, sudo and you don't want to have to manually
152
 
                 go through all your support files.
153
 

	
154
 
          -K     Force  prompting  for  passwords.  This is used to override a
155
 
                 prior -k argument.
156
 

	
157
 
          -G spec
158
 
                 GET file on host and write local dest directory.  spec  is  a
159
 
                 quoted  pair of strings.  The first specifies the path of the
160
 
                 source file (on the remote machine)  to  copy.   The  second,
161
 
                 specifies the destination directory (on the local machine):
162
 

	
163
 
                     tsshbatch.py -G "/foo/bar/baz /tmp" -i hostlist
164
 

	
165
 
                 This  copies  /foo/bar/baz from every machine in host file to
166
 
                 the local /tmp/ directory.  Since all the files have the same
167
 
                 name, they would overwrite each other if copied into the same
168
 
                 directory.  So, tsshbatch prepends the  string  hostname-  to
169
 
                 the name of each file it saves locally.
170
 

	
171
 
          -H hostlist
172
 
                 List  of  hosts  on which to run the command.  This should be
173
 
                 enclosed in quotes so that the list of hosts is handed to the
174
 
                 -H option as a single argument:
175
 

	
176
 
                     -H 'host1 host2 host3'
177
 

	
178
 
                 This  option  may  appear on the command line multiple times.
179
 
                 The hosts will be  processed  in  the  order  in  which  they
180
 
                 appeared on the command line.
181
 

	
182
 
          -L     List  names  of all (if any) host- and command files found on
183
 
                 their respective search paths.  These are listed in the order
184
 
                 they are found on those paths.
185
 

	
186
 
          -N     Force interactive username dialog.  This cancels any previous
187
 
                 request for key exchange authentication.
188
 

	
189
 
          -P spec
190
 
                 PUT file from local machine  to  remote  machine  destination
191
 
                 directory.   spec  is  a  quoted  pair of strings.  The first
192
 
                 specifies the path of the source file (on the local  machine)
193
 
                 to copy.  The second, specifies the destination directory (on
194
 
                 the remote machine):
195
 

	
196
 
                     tsshbatch.py -P "/foo/bar/baz /tmp" -i hostlist
197
 

	
198
 
                 This copies /foo/bar/baz on the local  machine  to  /tmp/  on
199
 
                 every host in  hostlist.
200
 

	
201
 
          -S     Force prompting for sudo password.
202
 

	
203
 
          -T seconds
204
 
                 Set  timeout  for  ssh connection attempts. (Default: 15 sec-
205
 
                 onds)
206
 

	
207
 
          -V strings
208
 
                 Similar to  the  -F  command  but  with  the  inverse  logic.
209
 
                 Reports  the names of all host- and command files that do not
210
 
                 contain any of the specified strings.
211
 

	
212
 
          -W     Print out a single line list of the inventory that  would  be
213
 
                 processed  and  exit.  (Test mode only - Ignored in execution
214
 
                 mode.)
215
 

	
216
 
                 This allows you to embed tsshbatch in external shell  scripts
217
 
                 like this:
218
 

	
219
 
                     for server in $(tsshbatch.py -i devserverlist -i uatserverlist -W)
220
 
                     do
221
 
                       ssh $server
222
 
                     done
223
 

	
224
 
                 Why?  Because tsshbatch has lots of powerful ways to maintain
225
 
                 inventories of hosts and combine them  through  includes  and
226
 
                 multiple command line arguments.  The -W option makes it con-
227
 
                 venient for external programs to make use of those  inventory
228
 
                 features.
229
 

	
230
 
          -a     Don't abort program after failed file transfers.  Continue to
231
 
                 next transfer attempt.  (Default: Abort)
232
 

	
233
 
          -b     Don't abort program after failed sudo command.  Normally, any
234
 
                 sudo  failure  causes  immediate  program  termination.  This
235
 
                 switch tells tsshbatch to continue  processing  on  the  next
236
 
                 host  even  if such a failure occurs.  This allows processing
237
 
                 to continue for those hosts where sudo does  work  correctly.
238
 
                 This  is  helpful  in large environments where sudo is either
239
 
                 improperly configured on some hosts or has a different  pass-
240
 
                 word.  This can also be used to discover where sudo does- and
241
 
                 does not work correctly.
242
 

	
243
 
          -e     Don't report remote host stderr output.
244
 

	
245
 
          -f cmdfile
246
 
                 Read commands from a file.  This file can be commented freely
247
 
                 with  the # character.  Leading- and trailing whitespace on a
248
 
                 line are ignored.
249
 

	
250
 
          -h     Print help information.
251
 

	
252
 
          -i hostfiles
253
 
                 Specify which files to read to get a list of desired hosts to
254
 
                 target.   This option may be repeated on the command line and
255
 
                 may also be followed with a quoted list of such  files.   The
256
 
                 following are equivalent:
257
 

	
258
 
                     tsshbatch.py -i devservers -i uatservers ...
259
 

	
260
 
                     tsstbatch.py -i "devservers uatservers" ...
261
 

	
262
 
                 The  -H  and -i options can be freely combined and repated on
263
 
                 the command line to create custom host lists made up of  both
264
 
                 known inventory and specific, individual hosts.
265
 

	
266
 
          -k     Use ssh keys instead of name/password credentials.
267
 

	
268
 
          -l logfile
269
 
                 Log diagnostic output to logfile. (Default: /dev/null)
270
 

	
271
 
          -n name
272
 
                 Login name to use.
273
 

	
274
 
          -p pw  Password to use when logging in and/or doing sudo.
275
 

	
276
 
          -q     Quiet mode - produce less noisy output.  Turns off -y.
277
 

	
278
 
          -r     Suppress  reporting of start/stop statistics. This allows you
279
 
                 to make statistics reporting the default, say via the  $TSSH-
280
 
                 BATCH environment variable, but override it when you need to.
281
 

	
282
 
          -s     Silence  all  program  noise  -  only  return command output.
283
 
                 Applies only to command operations.  File transfer and  error
284
 
                 reporting, generally, are unaffected.
285
 

	
286
 
          -t     Test mode: Only show what would be done but don't actually do
287
 
                 it.  This also prints diagnostic information about any  vari-
288
 
                 able  definitions,  the  list  of  hosts,  any  GET  and  PUT
289
 
                 requests, and final command strings after all  variable  sub-
290
 
                 stitutions  have  been  applied.  This is the default program
291
 
                 behavior.
292
 

	
293
 
          -v     Print detailed program version information and exit.
294
 

	
295
 
          -x     Override any previous -t specifications and actually  execute
296
 
                 the  commands.   This  is useful if you want to put -t in the
297
 
                 $TSSHBATCH environment variable so that the default is always
298
 
                 run  the  program  in  test mode.  Then, when you're ready to
299
 
                 actually run commands, you can override it  with  -x  on  the
300
 
                 command line.
301
 

	
302
 
          -y     Turn  on  'noisy'  reporting  for  additional detail on every
303
 
                 line, instead of just at the top of  the  stdout  and  stderr
304
 
                 reporting.  This is helpful when you are filtering the output
305
 
                 through something like grep that only returns matching  lines
306
 
                 and thus no context information.  Turns off -q.
307
 

	
308
 
       The last entry on the command line is optional and defines a command to
309
 
       run.  tsshbatch will attempt to execute it on every host you've  speci-
310
 
       fied either via -H or a host file:
311
 

	
312
 
          tsshbatch.py -Hmyhost ls -al /etc
313
 

	
314
 
       This will do a ls -al /etc on myhost.
315
 

	
316
 
       Be careful when using metacharacters like &&, <<, >>, <, > and so on in
317
 
       your commands.  You have to escape and  quote  them  properly  or  your
318
 
       local  shell  will  interfere  with them being properly conveyed to the
319
 
       remote machine.
320
 

	
321
 
       tsshbatch does all the GETs, then all the PUTs before attempting to  do
322
 
       any command processing.  If no GETs, PUTs, or commands have been speci-
323
 
       fied, tsshbatch will exit silently, since "nothing to do" really  isn't
324
 
       an error.
325
 

	
326
 
ENVIRONMENT
327
 
       tsshbatch  respects  the  $TSSHBATCH environment variable.  You may set
328
 
       this variable with any options above you commonly use to  avoid  having
329
 
       to key them in each time you run the program.  For example:
330
 

	
331
 
          export TSSHBATCH="-n jluser -p l00n3y"
332
 

	
333
 
       This  would cause all subsequent invocations of tsshbatch to attempt to
334
 
       use the login name/password credentials of jluser  and  l00n3y  respec-
335
 
       tively.
336
 

	
337
 
       tsshbatch  also  supports searching for files over specified paths with
338
 
       the $TSSHBATCHCMDS and $TSSHBATCHHOSTS  environment  variables.   Their
339
 
       use is described later in this document.
340
 

	
341
 
SSH CONFIGURATION FILE PROCESSING
342
 
       tsshbatch  has  limited  support for ssh configuration files.  Only the
343
 
       HostName and IdentityFile directives are currently supported.
344
 

	
345
 
       By default, tsshbatch will look in ~/.ssh/config for this configuration
346
 
       file.   However,  the location of the file can be overriden with the -C
347
 
       option.
348
 

	
349
 
MANAGING INVENTORY: HOST FILES
350
 
       Although you can specify a list of target hosts with  one  or  more  -H
351
 
       "host  host  host"  command line options, this gets cumbersome when you
352
 
       have to manage a large inventory  of  machines.   This  is  what  "host
353
 
       files" are intended to do.
354
 

	
355
 
       Host  files are files that name a host, one per line.  They may contain
356
 
       comments, include other host files, and make  reference  to  previously
357
 
       defined variables.  Here's an example, let's call it stage-servers:
358
 

	
359
 
          # Example staging host list
360
 

	
361
 
          .define __EXTERNAL__ = splat.com
362
 

	
363
 
          stage1.example.com
364
 
          stage2.example.com
365
 
          stage3.__EXTERNAL__
366
 

	
367
 
       Say you'd like to get the uptime for each of these servers:
368
 

	
369
 
          tsshbatch.py -x -i stage-servers uptime
370
 

	
371
 
       You can have more than one of these on a command line:
372
 

	
373
 
          tsshbatch.py -x -i dev-servers -i stage-servers -i prod-servers uptime
374
 

	
375
 
       But  ...  that's  kind of clumsy.  Instead let's create a new host file
376
 
       called all-servers like this:
377
 

	
378
 
          # Master host list
379
 

	
380
 
          .include dev-servers
381
 
          .include stage=servers
382
 
          .include prod-servers
383
 

	
384
 
       Now our command looks like this:
385
 

	
386
 
          tsshbatch.py -x -i all-servers uptime
387
 

	
388
 
       You can put as many -i arguments as you wish on the command line.   The
389
 
       contents  of  these  files  will  be  run in the order they appear from
390
 
       left-to-right on the command line.
391
 

	
392
 
       You can organize your host files in any hierarchy you  like  by  making
393
 
       use of search paths, as described later in this document.
394
 

	
395
 
       The use of host files, the -H command line option, .includes, and vari-
396
 
       able substitution, give tsshbatch a very powerful way to manage complex
397
 
       host  inventories.  There may be times when you'd like to select a sub-
398
 
       set of that inventory but run some other program with those host names.
399
 
       This  is  most  commonly  the case when you'd like to interactively ssh
400
 
       into each host for some reason.
401
 

	
402
 
       That's what the -W command line option is for.  tsshbatch computes  the
403
 
       inventory  you've  specified,  spits  it out as single line of text and
404
 
       exits, thereby enabling things like this:
405
 

	
406
 
          for server in $(tsshbatch.py -i dev-servers -i dr-servers -W)
407
 
          do
408
 
            ssh $server
409
 
          done
410
 

	
411
 
       This allows you to do all your intentory management via tsshbatch  con-
412
 
       structs, but make use of it with any other program of your choosing.
413
 

	
414
 
MANAGING JOBS: COMMAND FILES
415
 
       Although  tsshbatch  accepts  command to be run on each host on its own
416
 
       command line, this gets cumbersome for multi-step activities.   The  -f
417
 
       option  allows you to specify one or more "command files" that list the
418
 
       things you want done, in order.
419
 

	
420
 
       Command files may include other command files, make use  of  variables,
421
 
       and even specify sudo commands.  For example:
422
 

	
423
 
          # This is a comment
424
 
          .include master_commands  # optional, can be repeated
425
 
          command1                  # a command to run on each host
426
 
          sudo foo                  # run foo with sudo promotion on each host
427
 

	
428
 
       Notice that this is not the same thing as a shell script.  There are no
429
 
       conditionals or other programming features.  The contents of a  command
430
 
       file  are  more-or-less  a  "to  do"  list  for each host.  If your job
431
 
       requires the complexity of a "real" language, write script in the  lan-
432
 
       guage  of your choice, and then make reference to that script in a com-
433
 
       mand file.
434
 

	
435
 
       If you've specified a command file containing the commands you want run
436
 
       via  the  -f  option, these commands will run before the command you've
437
 
       defined on the command line.  That one is always the last  command  run
438
 
       on each host.
439
 

	
440
 
       Command  files may freely make use of comments and variable definitions
441
 
       as described below.  They also  have  a  special  directive  available,
442
 
       .notify.   This directive tells tsshbatch to print descriptive messages
443
 
       to stdout as the commands in the file are executed on  a  remote  host.
444
 
       This is helpful when running long, complex jobs:
445
 

	
446
 
          # Example cmdfile
447
 

	
448
 
          .include /my/fine/options
449
 
          .define __JOB_NAME__
450
 
          .notify starting JOB_NAME processing
451
 
          /usr/local/__JOB_NAME__
452
 
          .notify starting phase 2 of __JOB_NAME__
453
 
          do_another_command
454
 

	
455
 
       Notifications  are entirely optional and do not run on the remote host.
456
 
       Think of them as runtime comments emitted by tsshbatch to help you know
457
 
       what's  going  on.   Notifications  are  supressed if you select silent
458
 
       operation (-s).
459
 

	
460
 
       You can also specify file transfers within a  command  file.   See  the
461
 
       section below on file transfers for all the details.
462
 

	
463
 
       You  can put as many -f arguments as you wish on the command line.  The
464
 
       contents of these files will be run in  the  order  they  appeare  from
465
 
       left-to-right on the command line.
466
 

	
467
 
       You can organize your command files in any hierarchy you like by making
468
 
       use of search paths, as described later in this document.
469
 

	
470
 
SEARCHING YOUR HOST AND COMMAND FILES
471
 
       Over time, you will probably build up a large set  of  host  files  for
472
 
       describing  your  inventory and command files for standard jobs you run
473
 
       often.  It's convenient to search  through  them  quickly  when  you're
474
 
       looking for something specific.
475
 

	
476
 
       The  -L command line option just lists every host file and command file
477
 
       tsshbatch knows about on all defined search paths.  This  is  handy  if
478
 
       you want to examine your hierarchy of files.
479
 

	
480
 
       The  -F  string  string  ... command line option looks through all your
481
 
       host and command files and returns the names of those that contain  any
482
 
       of  the  strings you've listed.  The match is case-insensitive and lit-
483
 
       eral - no regular expressions are supported.
484
 

	
485
 
       The -V string string ... command line option is the inverse of -F.   It
486
 
       returns  a  list  of  host  and  command  files that do not contain the
487
 
       strings you specify.  Again, the match is literal and case-insensitive.
488
 

	
489
 
FEATURES AND USE CASES
490
 
       The sections below describe the various features of tsshbatch  in  more
491
 
       detail as well as common use scenarios.
492
 

	
493
 
   Different Ways To Specify Targeted Hostnames
494
 
       There  are  two  ways to specify the list of hosts on which you want to
495
 
       run the specified command:
496
 

	
497
 
          o On the command line via the -H option:
498
 

	
499
 
                tsshbatch.py -H 'hostA hostB' uname -a
500
 

	
501
 
            This would run the command uname -a on the hosts hostA  and  hostB
502
 
            respectively.
503
 

	
504
 
            Notice  that  the  list  of  hosts must be separated by spaces but
505
 
            passed as a single argument.  Hence  we  enclose  them  in  single
506
 
            quotes.
507
 

	
508
 
          o Via a host list file:
509
 

	
510
 
                tsshbatch.py myhosts df -Ph
511
 

	
512
 
            Here,  tsshbatch  expects  the  file  myhosts to contain a list of
513
 
            hosts, one per line, on which to run the command  df  -Ph.  As  an
514
 
            example,  if  you want to target the hosts larry, curly and moe in
515
 
            foo.com, myhosts would look like this:
516
 

	
517
 
                larry.foo.com
518
 
                curly.foo.com
519
 
                moe.foo.com
520
 

	
521
 
            This method is handy when there are standard "sets"  of  hosts  on
522
 
            which  you  regularly  work.  For instance, you may wish to keep a
523
 
            host file list for each of your production  hosts,  each  of  your
524
 
            test hosts, each of your AIX hosts, and so on.
525
 

	
526
 
            You  may use the # comment character freely throughout a host list
527
 
            file to add comments or temporarily comment out a particular  host
528
 
            line.
529
 

	
530
 
            You  can even use the comment character to temporarily comment out
531
 
            one or most hosts in the list given to the -H command  line  argu-
532
 
            ment.  For example:
533
 

	
534
 
                tsshbatch.py -H "foo #bar baz" ls
535
 

	
536
 
            This  would  run  the ls command on hosts foo and baz but not bar.
537
 
            This is handy if you want to use your shell's command line  recall
538
 
            to save typing but only want to repeat the command for some of the
539
 
            hosts your originally Specified.
540
 

	
541
 
   Authentication Using Name And Password
542
 
       The simplest way to use tsshbatch is to just name the hosts can command
543
 
       you want to run:
544
 

	
545
 
          tsshbatch.py linux-prod-hosts uptime
546
 

	
547
 
       By  default, tsshbatch uses your login name found in the $USER environ-
548
 
       ment variable when logging into other systems.  In this example, you'll
549
 
       be prompted only for your password which tsshbatch will then use to log
550
 
       into each of the machines named in linux-prod-hosts.  (Notice that  his
551
 
       assumes your name and password are the same on each host!)
552
 

	
553
 
       Typing  in  your  login  credentials all the time can get tedious after
554
 
       awhile so tsshbatch provides a means of providing them on  the  command
555
 
       line:
556
 

	
557
 
          tsshbatch.py -n joe.luser -p my_weak_pw linux-prod-hosts uptime
558
 

	
559
 
       This  allows  you to use tsshbatch inside scripts for hands-free opera-
560
 
       tion.
561
 

	
562
 
       If your login name is the same on all hosts, you can simplify this fur-
563
 
       ther by defining it in the environment variable:
564
 

	
565
 
          export TSSHBATCH="-n joe.luser"
566
 

	
567
 
       Any  subsequent invocation of tsshbatch will only require a password to
568
 
       run.
569
 

	
570
 
       HOWEVER, there is a huge downside to this - your plain text password is
571
 
       exposed in your scripts, on the command line, and possibly your command
572
 
       history.  This is a pretty big security hole, especially if  you're  an
573
 
       administrator  with extensive privileges.  (This is why the ssh program
574
 
       does not support such an option.)  For this reason, it is strongly rec-
575
 
       ommended that you use the -p option sparingly, or not at all.  A better
576
 
       way is to push ssh keys to every machine and use key exchange authenti-
577
 
       cation as described below.
578
 

	
579
 
       However,  there  are  times  when you do have use an explicit password,
580
 
       such as when doing sudo invocations.  It would be really nice to use -p
581
 
       and  avoid  having  to  constantly type in the password.  There are two
582
 
       strategies for doing this more securely than just entering it in  plain
583
 
       text on the command line:
584
 

	
585
 
          o Temporarily store it in the environment variable:
586
 

	
587
 
                export TSSHBATCH="-n joe.luser -p my_weak_pw"
588
 

	
589
 
            Do  this interactively after you log in, not from a script (other-
590
 
            wise you'd just be storing the plain text password in a  different
591
 
            script).   The environment variable will persist as long as you're
592
 
            logged in and disappear when you log out.
593
 

	
594
 
            If you use this just make sure to observe three  security  precau-
595
 
            tions:
596
 

	
597
 
                1. Clear  your  screen  immediately after doing this so no one
598
 
                   walking by can see the password you just entered.
599
 

	
600
 
                2. Configure your shell  history  system  to  ignore  commands
601
 
                   beginning  with export TSSHBATCH.  That way your plain text
602
 
                   password will never appear in the shell command history.
603
 

	
604
 
                3. Make sure you don't leave a logged in session  unlocked  so
605
 
                   that  other  users  could  walk up and see your password by
606
 
                   displaying the environment.
607
 

	
608
 
            This approach is best when you want your login credentials  avail-
609
 
            able for the duration of an entire login session.
610
 

	
611
 
          o Store your password in an encrypted file and decrypt it inline.
612
 

	
613
 
            First,  you  have  to  store your password in an encrypted format.
614
 
            There are several ways to do this, but gpg is commonly used:
615
 

	
616
 
                echo "my_weak_pw" | gpg -c >mysecretpw
617
 

	
618
 
            Provide a decrypt passphrase, and you're done.
619
 

	
620
 
            Now, you can use this by decrypting it inline as needed:
621
 

	
622
 
                #!/bin/sh
623
 
                # A demo scripted use of tsshbatch with CLI password passing
624
 

	
625
 
                MYPW=`cat mysecretpw | gpg`   # User will be prompted for unlock passphrase
626
 

	
627
 
                tsshbatch.py -n joe.luser -p $MYPW -i hostlist1 command1 arg
628
 
                tsshbatch.py -n joe.luser -p $MYPW -i hostlist2 command2 arg
629
 
                tsshbatch.py -n joe.luser -p $MYPW -i hostlist3 command3 arg
630
 

	
631
 
            This approach is best when you want your login credentials  avail-
632
 
            able  for  the  duration  of  the  execution of a script.  It does
633
 
            require the user to type in a passphrase to unlock  the  encrypted
634
 
            password  file,  but your plain text password never appears in the
635
 
            wild.
636
 

	
637
 
   Authentication Using Key Exchange
638
 
       For most applications of tsshbatch, it is much simpler to use key-based
639
 
       authentication.   For this to work, you must first have pushed ssh keys
640
 
       to all your hosts.   You  then  instruct  tsshbatch  to  use  key-based
641
 
       authentication rather than name and password.  Not only does this elim-
642
 
       inate the need to constantly provide name and password, it also  elimi-
643
 
       nates passing a plain text password on the command line and is thus far
644
 
       more secure.  This also  overcomes  the  problem  of  having  different
645
 
       name/password credentials on different hosts.
646
 

	
647
 
       By default, tsshbatch will prompt for name and password if they are not
648
 
       provided on the command line.  To force key- authentication, use the -k
649
 
       option:
650
 

	
651
 
          tsshbatch.py -k AIX-prod-hosts ls -al
652
 

	
653
 
       This  is so common that you may want to set it in your $TSSHBATCH envi-
654
 
       ronment variable so that keys are used by default.   If  you  do  this,
655
 
       there  may  still  be times when you want for force prompting for pass-
656
 
       words rather than using keys.  You can do this with the -K option which
657
 
       effectively overrides any prior -k selection.
658
 

	
659
 
   Executing A sudo Command
660
 
       tsshbatch  is  smart enough to handle commands that begin with the sudo
661
 
       command.  It knows that such commands require a password no matter  how
662
 
       you  initially  authenticate  to get into the system.  If you provide a
663
 
       password - either via interactive entry or the -p option - by  default,
664
 
       tsshbatch will use that same password for sudo promotion.
665
 

	
666
 
       If  you  provide  no password - you're using -k and have not provided a
667
 
       password via -p - tsshbatch will  prompt  you  for  the  password  sudo
668
 
       should use.
669
 

	
670
 
       You  can  force  tsshbatch  to  ask you for a sudo password with the -S
671
 
       option.  This allows you to have one password for initial login, and  a
672
 
       different one for sudo promotion.
673
 

	
674
 
       Any  time  you  a prompted for a sudo password and a login password has
675
 
       been provided (interactive or -p), you can  accept  this  as  the  sudo
676
 
       password by just hitting Enter.
677
 

	
678
 
       NOTE:
679
 
          tsshbatch makes a reasonable effort to scan your command line and/or
680
 
          command file contents to spot explicit invocations of the form  sudo
681
 
          ....   It  will  ignore  these  if they are inside single- or double
682
 
          quoted strings, on the assumption that you're  quoting  the  literal
683
 
          string sudo
684
 

	
685
 
          However,  this is not perfect because it is not a full reimplementa-
686
 
          tion of the shell quoting and aliasing features.   For  example,  if
687
 
          you  invoke  an  alias on the remote machine that resolves to a sudo
688
 
          command, or you run a script with a sudo command  in  it,  tsshbatch
689
 
          has  no  way  to  determine  what  you're trying to do.  For complex
690
 
          applications, it's best to write a true shell script,  push  it  all
691
 
          the  machines  in  question via -P, and then have tsshbatch remotely
692
 
          invoke it with sudo myscript or something similar.
693
 

	
694
 
          As always, the best way to figure out what the program thinks you're
695
 
          asking for is to run it in test mode and look at the diagnostic out-
696
 
          put.
697
 

	
698
 
   Precedence Of Authentication Options
699
 
       tsshbatch supports these various authentication options in a particular
700
 
       heirarchy  using  a "first match wins" scheme.  From highest to lowest,
701
 
       the precedence is:
702
 

	
703
 
          1. Key exchange
704
 

	
705
 
          2. Forced prompting for name via -N. Notice this cancels any  previ-
706
 
             ously requested key exchange authentication.
707
 

	
708
 
          3. Command Line/$TSSHBATCH environment variable sets name
709
 

	
710
 
          4. Name picked up from $USER  (Default behavior)
711
 

	
712
 
       If  you  try to use Key Exchange and tsshbatch detects a command begin-
713
 
       ning with sudo, it will prompt you for  a  password  anyway.   This  is
714
 
       because sudo requires a password to promote privilege.
715
 

	
716
 
   File Transfers
717
 
       The  -G and -P options specify file GET and PUT respectively.  Both are
718
 
       followed by a quoted file transfer specification in the form:
719
 

	
720
 
          "path-to-source-file path-to-destination-directory"
721
 

	
722
 
       Note that this means the file will always be stored under its  original
723
 
       name in the destination directory.  Renaming isn't possible during file
724
 
       transfer.
725
 

	
726
 
       However, tsshbatch always does GETs then PUTs then any outstanding com-
727
 
       mand (if any) at the end of the command line.  This permits things like
728
 
       renaming on the remote machine after a PUT:
729
 

	
730
 
          tsshbatch.py -P "foo ./" -i hostlist mv -v foo foo.has.a.new.name
731
 

	
732
 
       GETs are a bit of a different story because you are retrieving  a  file
733
 
       of  the  same name on every host.  To avoid having all but the last one
734
 
       clobber the previous one, tsshbatch makes forces the files you  GET  to
735
 
       be  uniquely  named  by prepending the hostname and a "-" to the actual
736
 
       file name:
737
 

	
738
 
          tsshbatch.py -H myhost -G "foo ./"
739
 

	
740
 
       This saves the file myhost-foo in the ./ on your a local machine.
741
 

	
742
 
       These commands do not recognize any special directory shortcut  symbols
743
 
       like  ~/  like  the  shell  interpreter  might.  You must name file and
744
 
       directory locations using ordinary pathing conventions.  You can put as
745
 
       many  of  these requests on the command line as you like to enable GETs
746
 
       and PUTs of multiple files.  You cannot, however,  use  filename  wild-
747
 
       cards to specify multi-file operations.
748
 

	
749
 
       You  can  put  multiple  GETs  or PUTs on the command line for the same
750
 
       file.  They do not override each other but are cummulative. So this:
751
 

	
752
 
          tsshbatch.py -P"foo ./" -P"foo /tmp" ...
753
 

	
754
 
       Would put local file foo in both ./ and /tmp on  each  host  specified.
755
 
       Similarly,  you can specify multiple files to GET from remote hosts and
756
 
       place them in the same local directory:
757
 

	
758
 
          tsshbatch.py -G"/etc/fstab ./tmp" -G"/etc/rc.conf ./tmp" ...
759
 

	
760
 
       You may also put file transfer specifications into a command  file  via
761
 
       the .getfile and .putfile directives.  This is handy when you have many
762
 
       to do and don't want to clutter up the command line.  Each must  be  on
763
 
       its  own  line  in  the command file and in the same form as if it were
764
 
       provided on the command line:
765
 

	
766
 
          .getfile /path/to/srcfile destdir    # This will get a file
767
 
          .putfile /path/to/srcfile destdir    # This will put a file
768
 

	
769
 
       File transfers are done in the order they appear.  For instance, if you
770
 
       have  a  file  transfer specification on the command line and then make
771
 
       reference to a command file with a file transfer specification  in  it,
772
 
       the one on the command line gets done first.
773
 

	
774
 
       NOTE:
775
 
          Keep  in  mind that tsshbatch always processes file transfers before
776
 
          executing any commands, no matter what order they appear in the com-
777
 
          mand file.  If you have this in a command file:
778
 

	
779
 
              echo "Test"
780
 
              .putfile "./myfile /foo/bar/baz/"
781
 

	
782
 
          The file will be transferred before the echo command gets run.  This
783
 
          can be counterintuitive.  It's therefore recommended  that  you  put
784
 
          your file transfers into a single file, and .include it as the first
785
 
          thing in your command file to make it obvious that these will be run
786
 
          first.
787
 

	
788
 
       By  default,  tsshbatch  aborts  if  any  file transfer fails.  This is
789
 
       unlike the case of failed commands which are reported but do not  abort
790
 
       the  program.   The  rationale'  for this is that you may be doing both
791
 
       file transfer and command execution with a single tsshbatch invocation,
792
 
       and the commands may depend on a file being transfered first.
793
 

	
794
 
       If  you  are  sure no such problem exists, you can use the -a option to
795
 
       disable abort-after-failure semantics on file transfer.  In this  case,
796
 
       file  transfer  errors will be reported, but tsshbatch will continue on
797
 
       to the next transfer request.
798
 

	
799
 
       tsshbatch does preserve permissions  when  transferring  files.   Obvi-
800
 
       ously,  for  this to work, the destination has to be writable by the ID
801
 
       you're logging in with.
802
 

	
803
 
       NOTE:
804
 
          The file transfer logic cannot cope with filenames that contain spa-
805
 
          ces.   The  workaround  is to either temporarily rename them, or put
806
 
          them in a container like a tarball or zip  file  and  transfer  that
807
 
          instead.
808
 

	
809
 
   Commenting
810
 
       Both the command file and host file can be freely commented using the #
811
 
       character.  Everything from that character to the end of that  line  is
812
 
       ignored.   Similarly,  you  can  use whitespace freely, except in cases
813
 
       where it would change the syntax of a command or host name.
814
 

	
815
 
   Includes
816
 
       You may also include other files as you wish with the .include filename
817
 
       directive  anywhere  in  the command file or host file.  This is useful
818
 
       for breaking up long lists of things into smaller parts.  For  example,
819
 
       suppose  you have three host lists, one for each major production areas
820
 
       of your network:
821
 

	
822
 
          hosts-development
823
 
          hosts-stage
824
 
          host-production
825
 

	
826
 
       You might typically run different tsshbatch jobs on each of these  sets
827
 
       of  hosts.   But  suppose  you  now  want  to run a job on all of them.
828
 
       Instead of copying  them  all  into  a  master  file  (which  would  be
829
 
       instantly  obsolete if you changed anything in one of the above files),
830
 
       you could create hosts-all with this content:
831
 

	
832
 
          .include hosts-development
833
 
          .include hosts-stage
834
 
          .include hosts-production
835
 

	
836
 
       that way if you edited any of the underlying files, the hosts-all would
837
 
       reflect the change.
838
 

	
839
 
       Similarly you can do the same thing with the command file to group sim-
840
 
       ilar commands into separate files and include them.
841
 

	
842
 
       tsshbatch does not enforce a limit on how deeply nested  .includes  can
843
 
       be.   An included file can include another file and so on.  However, if
844
 
       a circular include is detected, the program will notify you and  abort.
845
 
       This  happens  if, say, file1 includes file2, file2 includes file3, and
846
 
       file3 includes file1.  This would create an infinite loop  of  includes
847
 
       if  permitted.   You  can,  of  course,  include the same file multiple
848
 
       times, either in a single file or throughout other included  files,  so
849
 
       long as no circular include is created.
850
 

	
851
 
       The  target  of  a  .include directive can also contain variable refer-
852
 
       ences.  Note, however, that references to builtin variables  will  fail
853
 
       unless  you  have  overriden  them.   Why?   Because builtins don't get
854
 
       defined until a host connection  is  attempted.   This  doesn't  happen
855
 
       until  after all global variable processing and file includes have been
856
 
       done.  So:
857
 

	
858
 
          .define MYINCLUDE = /some/fine/file                    # OK
859
 
          .include MYINCLUDE
860
 

	
861
 
          .define MYINCLUDE = ! find /some/path -name includeski # OK
862
 
          .include MYINCLUDE
863
 

	
864
 
          .include __HOSTNAME__   # Nope, not defined yet -
865
 
                                  # tries to include file called '__HOSTNAME__'
866
 

	
867
 
          .define __HOSTNAME__ ! hostname  # Override the builtin
868
 
          .include __HOSTNAME__                                   # OK
869
 

	
870
 
       As a matter of keeping things simple, stick to global variables as part
871
 
       of an .include target.
872
 

	
873
 
   Search Paths
874
 
       tsshbatch supports the ablity to search paths to find files you've ref-
875
 
       erenced.  The search path for  cmdfiles  is  specified  in  the  $TSSH-
876
 
       BATCHCMDS environment variable.  The hostlistfiles search path is spec-
877
 
       ified in the $TSSHBATCHHOSTS environment variable.  These are  both  in
878
 
       standard path delimited format for your operating system.  For example,
879
 
       on Unix-like systems these look like this:
880
 

	
881
 
          export TSSHBATCHCMDS="/usr/local/etc/.tsshbatch/commands:/home/me/.tsshbatch/commands"
882
 

	
883
 
       And so forth.
884
 

	
885
 
       These paths are honored both for any files you specify on  the  command
886
 
       line  as  well  as for any files you reference in a .include directive.
887
 
       This allows you to maintain libraries of  standard  commands  and  host
888
 
       lists in well known locations and .include the ones you need.
889
 

	
890
 
       tsshbatch  will always first check to see if a file you've specified is
891
 
       in your local (invoking) directory and/or whether it is a fully  quali-
892
 
       fied file name before attempting to look down a search path.  If a file
893
 
       exist in several locations, the first instance found "wins".   So,  for
894
 
       instance,  if  you  have  a  file  called myhosts somewhere in the path
895
 
       defined in $TSSHBATCHHOSTS, you can override it by creating a  file  of
896
 
       same name in your current working directory.
897
 

	
898
 
       tsshbatch  also  checks  for  so-called "circular includes" which would
899
 
       cause an infinite inclusion loop.  It will abort upon discovering this,
900
 
       prior to any file transfers or commands being executed.
901
 

	
902
 
   An Overview Of Variables
903
 
       As you become more sophisticated in your use of tsshbatch, you'll begin
904
 
       to see the same patterns of use over and over again.  Variables  are  a
905
 
       way for you to use "shortcuts" to reference long strings without having
906
 
       to type the whole string in every time.  So, for  example,  instead  of
907
 
       having to type in a command like this:
908
 

	
909
 
          myfinecommand -X -Y -x because this is a really long string
910
 

	
911
 
       You can just define variable like this:
912
 

	
913
 
          .define __MYCMD__ =  myfinecommand -X -Y -x because this is a really long string
914
 

	
915
 
       From  then  on,  instead  of typing in that long command on the command
916
 
       line or in a command file, you can just  use  __MYCMD__  and  tsshbatch
917
 
       will substitute the string as you defined it whenever it encounters the
918
 
       variable.
919
 

	
920
 
       Variables can be used pretty much everwhere:
921
 

	
922
 
          o In hostlistfiles or in the hostnames listed with -H:
923
 

	
924
 
                .define __MYDOMAIN__ = stage.mydomain.com
925
 
                #.define __MYDOMAIN__ = prod.mydomain.com
926
 

	
927
 
                host1.__MYDOMAIN__
928
 
                host2.__MYDOMAIN__
929
 

	
930
 
            Now you can switch tsshbatch operation from stage to  prod  simply
931
 
            by changing what is commented out at the beginning.
932
 

	
933
 
          o In file transfer specifications:
934
 

	
935
 
                tsshbatch.py -xP"./fstab-__MYHOSTNAME__  ./" -i hostlist
936
 
                tsshbatch.py -xG"/etc/__OSNAME__-release ./" -i hostlist
937
 

	
938
 
          o In cmdfiles:
939
 

	
940
 
                .define __SHELL__ = /usr/local/bin/bash
941
 

	
942
 
                __SHELL__ -c myfinescript
943
 

	
944
 
       NOTE:
945
 
          A  variable can have pretty much any name you like excepting the use
946
 
          of metacharacters like < or !.  But if you are not careful, you  can
947
 
          cause unintended errors:
948
 

	
949
 
              .define foo = Slop
950
 

	
951
 
              myfoodserver.foods.com
952
 

	
953
 
          When  you  run tsshbatch it will then turn the server name into myS-
954
 
          lopdserver.Slopds.com - probably not what you want.
955
 

	
956
 
          So, it's a Really Good Idea (tm) to use some kind of  naming  scheme
957
 
          to make variables names stand out and make them unlikely to conflict
958
 
          accidentally with command- and host strings.
959
 

	
960
 
   Types Of Variables
961
 
       tsshbatch has four different kinds of variables:
962
 

	
963
 
          o Global Variables are the kind in  the  example  above.   You,  the
964
 
            user, define them as you wish in a command file or host file.
965
 

	
966
 
          o Local  Variables  work exactly the same a global variables, except
967
 
            that they can only be used in the file (command- or host) in which
968
 
            they  are  defined.   These  variables are defined with the .local
969
 
            name = value syntax.
970
 

	
971
 
          o Execution Variables run any program or script of your choosing (on
972
 
            the  same machine you're running tsshbatch) and assign the results
973
 
            to either a global- or locala variable.
974
 

	
975
 
          o Builtin Variables are variables the tsshbatch itself defines.  You
976
 
            can override their default values by creating a Global Variable of
977
 
            the same name.
978
 

	
979
 
   Where And When Do Variables Get Processed?
980
 
       Global, Local, and Execution Variables are defined  in  either  a  host
981
 
       file or command file.
982
 

	
983
 
       Builtin  Variables are defined within tsshbatch itself unless you over-
984
 
       ride them.
985
 

	
986
 
       Global Variables are all read in and then used.  If  you  do  something
987
 
       like this:
988
 

	
989
 
          .define __FOO__ = firstfoo
990
 
          echo __FOO__
991
 
          .define __FOO__ = secondfoo
992
 

	
993
 
       You'll  get an output of ... secondfoo!  Why?  Because before tsshbatch
994
 
       tries to run anything, it has to process all the cmdfiles,  host  file,
995
 
       and  the  command line content.  So, before we ever get around to doing
996
 
       an echo __FOO__ on some host, the second definition of __FOO__ has been
997
 
       read in ... and last definition wins.
998
 

	
999
 
       Local  Variables  are  read in as a command- or host file is processed.
1000
 
       They can only be referenced within the same file - i.e., They  are  not
1001
 
       visible to any other file.
1002
 

	
1003
 
       Execution  Variables  are  processed  a single time at the time they're
1004
 
       read in from a command file or host file.
1005
 

	
1006
 
       Builtin Variables get evaluated every time  ``tsshbatch``  prepares  to
1007
 
       connect  to  a  new host (unless you've overriden them).  That way, the
1008
 
       most current value for them is available for use on the next host.
1009
 

	
1010
 
       Keep in mind that tsshbatch isn't a programming language.  It's  "vari-
1011
 
       ables"  are  simple string substitutions with "last one wins" semantics
1012
 
       (for globals).  If you define the same global variable in, say, a  com-
1013
 
       mand  file  and  also  in  the  host file, the latter will "win".  Why?
1014
 
       Because hostlistfiles are always read in after any cmdfiles.
1015
 

	
1016
 
       Finally, variable references in a definition are ignored.  Say  you  do
1017
 
       this in a command file:
1018
 

	
1019
 
          .define __CLEVER      __ = __REALLYCLEVER__
1020
 
          .define __REALLYCLEVER__ = Not That Smart
1021
 
          echo __CLEVER__
1022
 

	
1023
 
       You  will  get this output, __REALLYCLEVER__!  Why?  Because, the vari-
1024
 
       able references on the right side of a definition statement  are  never
1025
 
       replaced.  This is a concious design choice to keep variable definition
1026
 
       and use as simple and obvious as possible.   Allowing  such  "indirect"
1027
 
       definitions  opens  up  a treasure trove of maintenance pain you really
1028
 
       want to avoid.  Trust us on this one.
1029
 

	
1030
 
   Global Variables
1031
 
       tsshbatch allows you to define variables which will  then  be  used  to
1032
 
       replace  matching strings in cmdfiles, hostlistfiles, and file transfer
1033
 
       specifications.  For example, suppose you have this in a host file:
1034
 

	
1035
 
          .define DOMAIN=.my.own.domain.com
1036
 

	
1037
 
          host1DOMAIN
1038
 
          host2DOMAIN
1039
 
          host3DOMAIN
1040
 

	
1041
 
       At    runtime,    the    program    will    actually     connect     to
1042
 
       host1.my.own.domain.com,  host2.my.domain.com,  and so on.  This allows
1043
 
       for ease of modularization and maintenance of your files.
1044
 

	
1045
 
       Similarly, you might want define MYCMD=some_long_string  so  you  don't
1046
 
       have to type some_long_string over and over again in a command file.
1047
 

	
1048
 
       There are some "gotchas" to this:
1049
 

	
1050
 
          o The general form of a variable definition is:
1051
 

	
1052
 
                .define name = value
1053
 

	
1054
 
            You  have  to have a name but the value is optional.  .define FOO=
1055
 
            simply replaces any subsequent FOO strings  with  nothing,  effec-
1056
 
            tively removing them.
1057
 

	
1058
 
            Any  =  symbols  to the right of the one right after name are just
1059
 
            considered part of the variable's value.
1060
 

	
1061
 
            Whitespace around the = symbol is optional but allowed.
1062
 

	
1063
 
          o Variables are substituted in the order they appear:
1064
 

	
1065
 
                .define LS = ls -alr
1066
 
                LS /etc               # ls -alr /etc
1067
 
                .define LS = ls -1
1068
 
                LS /foo               # ls -1 /foo
1069
 

	
1070
 
          o Variable names and values are case sensitive.
1071
 

	
1072
 
          o Variables may be defined in either cmdfiles or  hostlistfiles  but
1073
 
            they  are  visible  to  any  subsequent  file that gets read.  For
1074
 
            instance, cmdfiles are read before any hostlistfiles.   Any  vari-
1075
 
            ables  you've  defined  in  a  command file that happen to match a
1076
 
            string in one of your hostnames will be substituted.
1077
 

	
1078
 
            This is usually not what you want, so be careful.  One way to man-
1079
 
            age  this  is  to  use variables names that are highly unlikely to
1080
 
            ever show up in a hostname or command.  That way your commands and
1081
 
            hostnames will not accidentally get substrings replaced with vari-
1082
 
            able values.  For example,  you  might  use  variable  names  like
1083
 
            --MYLSCOMMAND-- or __DISPLAY_VGS__.
1084
 

	
1085
 
          o Variable  substitution is also performed on any host names or com-
1086
 
            mands passed on the command line.
1087
 

	
1088
 
   Local Variables
1089
 
       As we saw previously, global variables have "last one wins"  semantics.
1090
 
       So,  if  the  same  variable  name appears in different files, the last
1091
 
       instance read will be the value assigned to that variable.   Sometimes,
1092
 
       you  don't  want this behavior. You want to define a variable in a file
1093
 
       and only use it there without changing a global variable with the  same
1094
 
       name.
1095
 

	
1096
 
       Say we have three files:
1097
 

	
1098
 
          # First command file
1099
 
          .define __DEFAULT__ = ls -al  # global variable
1100
 

	
1101
 
          # Second command file
1102
 
          .local __DEFAULT__ = ls -1    # variable is local to 2nd file
1103
 
          __DEFAULT__
1104
 

	
1105
 

	
1106
 
          # Third command file
1107
 
          __DEFAULT__
1108
 

	
1109
 
       Now, say we run this command:
1110
 

	
1111
 
          tsshbatch.py -xH"somehost" -f file1 -f file2 -f file3
1112
 

	
1113
 
       The net commands to be run on somehost will be:
1114
 

	
1115
 
          ls -1
1116
 
          ls -al
1117
 

	
1118
 
       The  reference to __DEFAULT in file2 is resolved with the local defini-
1119
 
       tion, and the reference in file3 is resolved with  the  global  defini-
1120
 
       tion.
1121
 

	
1122
 
       Local  variables  accompish  several  things.  First, they "insulate" a
1123
 
       host- or command file from prior- or  subsequent  variable  definitions
1124
 
       found  in  other  files.   You  are guaranteed that the local copy of a
1125
 
       varaible always "wins".
1126
 

	
1127
 
       Conversely, local variables also protect global variables of  the  same
1128
 
       name  from  changing.   In  the example above, we have default behavior
1129
 
       which can be overriden in a case-by-case basis,  without  changing  the
1130
 
       "master" definition.
1131
 

	
1132
 
   Execution Variables
1133
 
       Execution  Variables  are  actually  a special case of Global and Local
1134
 
       Variables.  That is, they are evaluated at the same  time  and  in  the
1135
 
       same  manner  as  any  other variable.  The difference is that a normal
1136
 
       variable describes a literal  string  replacement.   But  an  Execution
1137
 
       Variable  runs a command, program, or script and assigns the results to
1138
 
       the variable.
1139
 

	
1140
 
       For example, suppose you want create a file on many machines,  and  you
1141
 
       want  that  file  to  be named based on who ran the tsshbatch job.  You
1142
 
       might do this in a command file:
1143
 

	
1144
 
          .define __WHOAMI__ = ! whoami
1145
 
          touch __WHOAMI__-Put_This_Here.txt
1146
 

	
1147
 
       So,   if   ID   luser   is   running   tsshbatch,   a    file    called
1148
 
       luser-Put_This_Here.txt will be created (or have its timestamp updated)
1149
 
       on every machine in the host file or named with -H.
1150
 

	
1151
 
       Similarly, you can do the same thing with local variables:
1152
 

	
1153
 
          .local __TYPE__ = ! uname
1154
 

	
1155
 
       Notice it is the ! character that distinguishes an  Execution  Variable
1156
 
       from a Global Variable.  It is this character that tells tsshbatch, "Go
1157
 
       run the command to the right of me and return the results."  The trail-
1158
 
       ing space is optional and the definition could be written as:
1159
 

	
1160
 
          .define __WHOAMI__ = !whoami
1161
 

	
1162
 
       If the command you specify returns multiple lines of output, it's up to
1163
 
       you to process it properly.  tsshbatch does  no  newline  stripping  or
1164
 
       other  postprocessing of the command results.  This can make the output
1165
 
       really "noisy".  tssbatch normally reports a summary of the command and
1166
 
       its results.  But if you do something like this:
1167
 

	
1168
 
          .define __LS__ = ! ls -al
1169
 
          echo __LS__
1170
 

	
1171
 
       You  will  get  a  multiline summary of the command and then the actual
1172
 
       output - which is also multiline.  This gets to  be  obnonxious  pretty
1173
 
       quickly.   You  can  make a lot of this go away with the -q, or "quiet"
1174
 
       option.
1175
 

	
1176
 
       NOTE:
1177
 
          It's important to remember that the program you are invoking runs on
1178
 
          the  same machine as tsshbatch itself, NOT each host you are sending
1179
 
          commands to.  In other words, just like Builtin Variables, Execution
1180
 
          Variables are locally defined.
1181
 

	
1182
 
   Builtin Variables
1183
 
       As noted previously, Builtin Variables are created by tsshbatch itself.
1184
 
       They are created for each new host connection so that things like time,
1185
 
       host number, and hostname are up-to-date.
1186
 

	
1187
 
       As of this release, tsshbatch supports the following Builtins:
1188
 

	
1189
 
          center;  |l|l|.   _ T{ __DATE__ T}   T{ Date in YYYYMMDD format T} _
1190
 
          T{ __DATETIME__ T}   T{ Date and time in YYYYMMDDHHMMSS format T}  _
1191
 
          T{ __HOSTNAME__ T}   T{ Full name of current host as passed to tssh-
1192
 
          batch T} _ T{ __HOSTNUM__ T}   T{ Count  of  host  being  processed,
1193
 
          starting  at  1  T} _ T{ __HOSTSHORT__ T}   T{ Leftmost component of
1194
 
          hostname as passed to tsshbatch T} _ T{ __LOGINNAME__  T}   T{  User
1195
 
          name  used  for remote login.  For key auth, name of tsshbatch user.
1196
 
          T} _ T{ __TIME__ T}   T{ Time in HHMMSS format T} _
1197
 

	
1198
 
   Using Builtin Variables
1199
 
       There are times when it's convenient to be able to embed  the  name  of
1200
 
       the  current  host in either a command or in a file transfer specifica-
1201
 
       tion.  For example, suppose you want to  use  a  single  invocation  of
1202
 
       tsshbatch  to  transfer  files  in a host-specific way.  You might name
1203
 
       your files like this:
1204
 

	
1205
 
          myfile.host1
1206
 
          myfile.host2
1207
 

	
1208
 
       Now, all you have to do is this:
1209
 

	
1210
 
          tsshbatch.py -xH "host 1 host2" -P "myfile.__HOSTNAME__ ./"
1211
 

	
1212
 
       When run, tsshbatch will substitute the name of  the  current  host  in
1213
 
       place  of  the  string  __HOSTNAME__.   (Note  that these are **double*
1214
 
       underbars on each side of the string.*)
1215
 

	
1216
 
       You can do this in commands (and  commands  within  command  files)  as
1217
 
       well:
1218
 

	
1219
 
          tsshbatch.py -x hosts 'echo I am running on __HOSTNAME__'
1220
 

	
1221
 
       Be careful to escape and quote things properly, especially from the the
1222
 
       command line, since < and > are recognized by the shell as  metacharac-
1223
 
       ters.
1224
 

	
1225
 
       There  are  two  forms  of host name substitution possible.  The first,
1226
 
       __HOSTNAME__ will use the name as you provided it, either as  an  argu-
1227
 
       ment to -H or from within a host file.
1228
 

	
1229
 
       The second, __HOSTSHORT__, will only use the portion of the name string
1230
 
       you provided up to the leftmost period.
1231
 

	
1232
 
       So, if you specify myhost1.frumious.edu, __HOSTNAME__ will be  replaced
1233
 
       with  that  entire  string,  and __HOSTSHORT__ will be replaced by just
1234
 
       myhost1.
1235
 

	
1236
 
       Notice that, in no case does tsshbatch do any  DNS  lookups  to  figure
1237
 
       this  stuff  out.  It just manipulates the strings you provide as host-
1238
 
       names.
1239
 

	
1240
 
       The symbols __HOSTNAME__ and __HOSTSHORT__ are like  any  other  symbol
1241
 
       you  might  have  specified  yourself with .define.  This means you can
1242
 
       override their meaning.  For instance, say you're doing this:
1243
 

	
1244
 
          tsshbatch.py -x myhosts  echo "It is: __HOSTNAME__"
1245
 

	
1246
 
       As you would expect, the program will log  into  that  host,  echo  the
1247
 
       hostname  and  exit.   But  suppose you don't want it to echo something
1248
 
       else for whatever reason.  You'd create a command file with this entry:
1249
 

	
1250
 
          .define __HOSTNAME__ = Really A Different Name
1251
 

	
1252
 
       Now, when you run the command above, the output is:
1253
 

	
1254
 
          It is: Really A Different Name
1255
 

	
1256
 
       In other words, .define has a higher precedence than the  preconfigured
1257
 
       values of HOSTNAME and HOSTSHORT.
1258
 

	
1259
 
   Noise Levels
1260
 
       tsshbatch  defaults  to  a  medium level of reporting as it runs.  This
1261
 
       includes connection reporting, headers describing the command being run
1262
 
       on  every  host,and the results written to stdin and stdout.  Each line
1263
 
       of reporting output begins with ---> to help you parse through the out-
1264
 
       put  if  you  happen  to  be  writing a program that post-processes the
1265
 
       results from tsshbatch.
1266
 

	
1267
 
       This output "noise" is judged to be right for most applications of  the
1268
 
       program.  There are times, however, when you want more- or less "noise"
1269
 
       in the output.  There are several tsshbatch options that support this.
1270
 

	
1271
 
       These options only affect reporting of commands you're  running.   They
1272
 
       do not change the output of file transfer operations.  They also do not
1273
 
       change error reporting, which is always the same irrespective  of  cur-
1274
 
       rent noise level setting.
1275
 

	
1276
 
       -q  or  "quiet"  mode,  reduces the amount of output noise in two ways.
1277
 
       First, it silences reporting each time a successful connection is  made
1278
 
       to  a  host.   Secondly,  the  command  being run isn't reported in the
1279
 
       header.   For example, normally, running ls -l is reported like this:
1280
 

	
1281
 
          --->  myhost:    SUCCESS: Connection Established
1282
 
          --->  myhost (stdout) [ls -l]:
1283
 
          ...
1284
 
          --->  myhost (stderr) [ls -l]:
1285
 

	
1286
 
       In quiet mode, reporting looks like this:
1287
 

	
1288
 
          --->  localhost (stdout):
1289
 
          ...
1290
 
          --->  localhost (stderr):
1291
 

	
1292
 
       The main reason for this is that some commands can be very long.   With
1293
 
       execution  variables,  it's  possible to create commands that span many
1294
 
       lines.  The quiet option gives you  the  ability  to  suppress  echoing
1295
 
       these long commands for each and every host in your list.
1296
 

	
1297
 
       -y  or  "noisy"  mode, produces normal output noise but also replicates
1298
 
       the hostname and command string for every line of output produced.  For
1299
 
       instance, ls -1 might normally produce this:
1300
 

	
1301
 
          --->  myhost:    SUCCESS: Connection Established
1302
 
          --->  myhost (stdout) [ls -1]:
1303
 

	
1304
 
                 backups
1305
 
                 bin
1306
 

	
1307
 
       But in noisy mode, you see this:
1308
 

	
1309
 
          --->  myhost:    SUCCESS: Connection Established                                                                                                                     --->  myhost (stdout) [ls -1]:
1310
 
          [myhost (stdout) [ls -1]]        backups
1311
 
          [myhost (stdout) [ls -1]]        bin
1312
 

	
1313
 
       Again,  the  purpose here is to support post-processing where you might
1314
 
       want to search through a  large  amount  of  output  looking  only  for
1315
 
       results from particular hosts or commands.
1316
 

	
1317
 
       -s or "silent" mode returns only the results from running the commands.
1318
 
       No headers or descriptive information are produced.  It's  more-or-less
1319
 
       what you'd see if you logged into the host and ran the command interac-
1320
 
       tively.  For instance, ls -l might look like this:
1321
 

	
1322
 
          total 44
1323
 
          drwxr-xr-x  2 splot splot 4096 Nov  5 14:54 Desktop
1324
 
          drwxrwxr-x 39 splot splot 4096 Sep  9 14:57 Dev
1325
 
          drwxr-xr-x  3 splot splot 4096 Jun 14  2012 Documents
1326
 

	
1327
 
       The idea here  is  to  use  silent  mode  with  the  various  variables
1328
 
       described  previously  to customize your own reporting output.  Imagine
1329
 
       you have this in a command file and you run tsshbatch in silent mode:
1330
 

	
1331
 
          .define __USER__ = ! echo $USER
1332
 
          echo "Run on __HOSTNAME__ on __DATE__ at __TIME__ by __USER__"
1333
 
          uname -a
1334
 

	
1335
 
       You'd see output along these lines:
1336
 

	
1337
 
          Run on myhost on 20991208 at 141659 by splot
1338
 
          Linux myhost 3.11.0-12-generic #19-Ubuntu SMP Wed Oct 9 16:20:46 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
1339
 

	
1340
 
OTHER
1341
 
       o Comments can go anywhere.
1342
 

	
1343
 
       o Directives like .define and .include must be the first non-whitespace
1344
 
         text on the left end of a line.  If you do this in a command file:
1345
 

	
1346
 
            foo .include bar
1347
 

	
1348
 
         tsshbatch  thinks you want to run the command foo with an argument of
1349
 
         .include bar.  If you do it in a host file, the program thinks you're
1350
 
         trying  to  contact a host called foo had in mind.  Similarly, every-
1351
 
         thing to the right of the directive is considered its argument (up to
1352
 
         any comment character).
1353
 

	
1354
 
       o Whitespace  is  not significant at the beginning or end of a line but
1355
 
         it is preserved within .define and .include  directive  arguments  as
1356
 
         well as within commmand definitions.
1357
 

	
1358
 
       o Strictly  speaking, you do not have to have whitespace after a direc-
1359
 
         tive.  This is recognized:
1360
 

	
1361
 
            .includesomefileofmine
1362
 
            .definemyvar=foo
1363
 

	
1364
 
         But this is strongly discouraged because it's really hard to read.
1365
 

	
1366
 
       o tsshbatch writes the stdout of the remote host(s) to  stdout  on  the
1367
 
         local machine.  It similarly writes remote stderr output to the local
1368
 
         machine's stderr.  If you wish to suppress stderr output, either  re-
1369
 
         direct  it on your local command line or use the -e option to turn it
1370
 
         off entirely.  If you want everything to go to your local stdout, use
1371
 
         the -E option.
1372
 

	
1373
 
       o You  must  have a reasonably current version of Python 2.x installed.
1374
 
         It almost certainly will not work on Python 3.x because it  uses  the
1375
 
         deprecated  commands module.  This decision was made to make the pro-
1376
 
         gram as backward compatible with older versions of Python as possible
1377
 
         (there is way more 2.x around than there is 3.x).
1378
 

	
1379
 
       o If  your Python installation does not install paramiko you'll have to
1380
 
         install it manually, since  tsshbatch  requires  these  libraries  as
1381
 
         well.
1382
 

	
1383
 
       o tsshbatch  has  been  run  extensively from Unix-like systems (Linux,
1384
 
         FreeBSD) and has had no testing whatsoever on Microsoft Windows.   If
1385
 
         you  have  experience  using  it on Windows, do please share with the
1386
 
         class using the email address below.  While we do not officially sup-
1387
 
         port  this  tool  on  Windows,  if the changes needed to make it work
1388
 
         properly are small enough, we'd consider updating  the  code  accord-
1389
 
         ingly.
1390
 

	
1391
 
BUGS AND MISFEATURES
1392
 
       o You will not be able to run remote sudo commands if the host in ques-
1393
 
         tion enables the Defaults requiretty in  its  sudoers  configuration.
1394
 
         Some  overzealous InfoSec folks seem to think this is a brilliant way
1395
 
         to secure your system (they're wrong) and there's  nothing  tsshbatch
1396
 
         can do about it.
1397
 

	
1398
 
       o When  sudo is presented a bad password, it ordinarily prints a string
1399
 
         indicating something is wrong.  tsshbatch looks for this to  let  you
1400
 
         know that you've got a problem and then terminates further operation.
1401
 
         This is so that you do not attempt to log  in  with  a  bad  password
1402
 
         across all the hosts you have targeted.  (Many enterprises have poli-
1403
 
         cies to lock out  a  user  ID  after  some  small  number  of  failed
1404
 
         login/access attempts.)
1405
 

	
1406
 
         However,  some older versions of sudo (noted on a RHEL 4 host running
1407
 
         sudo 1.6.7p5) do not return any feedback when presented  with  a  bad
1408
 
         password.   This  means  that  tsshbatch  cannot  tell the difference
1409
 
         between a successful sudo and a system waiting for you to  reenter  a
1410
 
         proper password.  In this situation, if you enter a bad password, the
1411
 
         the program will hang.  Why?  tsshbatch thinks nothing is  wrong  and
1412
 
         waits  for  the  sudo  command  to  complete.  At the same time, sudo
1413
 
         itself is waiting for an updated password.  In this case, you have to
1414
 
         kill  tsshbatch  and  start over.  This typically requires you to put
1415
 
         the program in background (`Ctrl-Z in most shells) and  then  killing
1416
 
         that job from the command line.
1417
 

	
1418
 
         There is no known workaround for this problem.
1419
 

	
1420
 
OTHER, SIMILAR PRODUCTS
1421
 
       It's always interesting to see how other people approach the same prob-
1422
 
       lem.  If you're interested in this general area of IT  automation,  you
1423
 
       may  want  to  also  look  at Ansible, Capistrano, Cluster SSH, Fabric,
1424
 
       Func, and Rundeck.
1425
 

	
1426
 
COPYRIGHT AND LICENSING
1427
 
       tsshbatch is Copyright (c) 2011-2016 TundraWare Inc.
1428
 

	
1429
 
       For terms of use, see the tsshbatch-license.txt  file  in  the  program
1430
 
       distribution.   If  you install tsshbatch on a FreeBSD system using the
1431
 
       'ports'   mechanism,   you   will    also    find    this    file    in
1432
 
       /usr/local/share/doc/tsshbatch.
1433
 

	
1434
 
AUTHOR
1435
 
          Tim Daneliuk
1436
 
          tsshbatch@tundraware.com
1437
 

	
1438
 
DOCUMENT REVISION INFORMATION
1439
 
          $Id: '0c4840a tundra Sat Oct 15 15:29:02 2016 -0500'
1440
 

	
1441
 
       This document was produced with emacs, RestructuredText, and TeX Live.
1442
 

	
1443
 
       You can find the latest version of this program at:
1444
 
          http://www.tundraware.com/Software/tsshbatch
1445
 

	
1446
 

	
1447
 

	
1448
 
                                                                            ()
0 comments (0 inline, 0 general)