Yeah, before I was heavily into automation (i.e. puppet) this was my go to command. Use a for loop and ssh to run things on a group of systems.

  for host in h-abc h-def h-ghi h-jkl h-mno h-pqr h-stu h-vwx h-yza; 
    do 
    ssh -l root $host "hostname; echo "1.1.1.1 server" >> /etc/hosts"; 
  done

This would iterate over hosts (h-abc .. h-yza) and run the commands "hostname; echo "1.1.1.1 server" >> /etc/hosts". Or maybe use a for loop and scp to push the files out and then run md5 to verify they are correct. It was a hack but useful for a small set of machines.

ps. if you are interested in learning about puppet, I have put together a screencast about it @ http://sysadmincasts.com/episodes/8-learning-puppet-with-vag...

make a small change:

  for host in ... ; do ssh -l root $host "..." & done; wait

and it's very parallel. All that key exchange takes a while, so it's not ridiculously fast, but it's pretty good. If you've got ControlMaster set up already to be per-host, this might very well be very fast the second time.

You may want to look into using GNU parallel with that.

Also check out xargs with the -P 10 -n 1 flags (for example) -- to limit at most 10 parallel tasks running at the same time. Useful if you don't have GNU parallel installed where you happen to be working (everyone has xargs installed).

dsh (distributed/dancer's shell) is a wrapper around ssh which provides that capability. Plus you can define hostlists.

The single most useful thing about ssh to me is that it connects stdout/in across the channel (this is behavior inherited from rsh). This allows for e.g.:

  % (cd /foo; tar cpf - .) | ssh bar \(cd /baz\; tar xpf -\)

Or:

  % cat ~/.ssh/id_rsa.pub | ssh bar tee -a .ssh/authorized_keys

This is a bit of a pet peeve of mine, but cd/tar should likely be joined with && - in other words, in what directory do you want tar to (not) run if cd fails? So consider:

    % (cd /foo && tar cpf - .) | ssh bar '(cd /baz && tar xpf -)'

or explore something like:

    % rsync -aSHx --update --delete -e ssh /foo/ user@bar:baz/.

GNU tar has evolved and can use simpler syntax: % tar cp -C /foo . | ssh bar tar xp -C /baz

It would be an interesting project to look at people's shell usage and try to determine when they started using Unix. I have loads of Irixisms that have only recently been crowded out of my head by Darwin BSDism.

Forgive my ignorance, but I don't know what what it means to connect "across the channel", but it sounds important. Do you know a link that explains the concept? I've failed to Google it.

It means that whatever you put into the ssh process's STDIN gets sent to the STDIN of the remote process. For example you can do this:

    tar cj $bigdir | ssh host 'tar xj'

That creates a tar.bz2 archive on your machine, sends it over ssh to the remote host, where it's unpacked. Ad hoc compressed and encrypted file transfers made easy.

That's a good illustration, but rsync, if available, would generally be preferred for that use case:

  rsync -az $bigdir host:$bigdir

Sure, of course. But I've used that tar|ssh combination so much that it was the first example of using the rsh-like stdout/in pairing that came to mind. As much as I bitch about Unix and POSIX and horrific shell brain damage, I still reach for crazy shell pipelines when I have data manipulation tasks up to some pretty large scale n.

What you pipe into ssh gets out of ssh on the host you're connected to.

So basically, you get :

      (local)            (remote)
    stdin -> ssh       ssh -> stdout

Hope that helps.

He's talking about forwarding stdin and stdout across the SSH connection to the program that SSH is running on the other end; such that the stdin of ssh is passed unmodified to the program and vice-versa.

Are you familiar with stdin/stdout/stderr on Unix and how they interact? What ssh does that I find so powerful is ensure that what you put into stdout on one side of a pipe flows out of stdin on the process that ssh starts on the remote host.

Of course, ssh also gives you the output of the remote process, so you can put an ssh command in the middle of a pipeline. My day to day life no longer involves interacting with lots of remote Unix systems, but when it did, it was really, really useful.

ssh-copy-id is made for exactly this:

  ssh-copy-id  -  install  your  public  key in a remote machine’s authorized_keys

the tip does work on OS X though, which doesn't have ssh-copy-id (by default, at least)

Still about ssh :

"typing a password on your local machine once, then using a secure identity to login to several remote machines without having to retype your password by using an ssh key agent. This is awesome. "

Sounds awesome, but I really didn't understand what he means. Is he talking about using a certificate (-i option) to log in ?

Yeah, you have a cert set up and put the public cert in the identities file for each server you want to connect to. Then just SSH into each box with agent forwarding ('-a' I think, but you're best to check that).

So you'll need to create an SSH cert first:

    ssh-keygen -b 4096 -t rsa -C "$USER created on $(hostname)" -f $USER.key

Keep the private key, that's essential you keep this safe as it's literally your key to access any servers you do the next step for.

On each server SSH, append your public key to the of /home/$USER/.ssh/authorized_keys (creating the file if it doesn't exist). It must be the public key!

There'll be better guides online for this stuff so I suggest you have a read through them before blindly pasting the stuff I've posted into your terminal. But it's all quite simple stuff once you've grasped the difference between a public and private key (if you weren't already aware - you may well be)

Keeping in mind that the ssh agent is root accessible on the machine in the /tmp directory looking something like this:

"ssh-ebT23030"

Meaning that anyone with root access who know the other machines that someone is accessing as well as their username can access those machines.

I've seen cases where in abnormally terminated sessions that file stays around instead of getting deleted when the session ends.

I am fairly sure they are referring to ssh-add

I think most people know about this, but it's also good to know about scp. You can easily copy files using a ssh connection.

Scp doesn't get all file types correct (fifos, etc), while tar does.

Piped tar commands do, but are generally inferior to rsync (at least if you ever need to copy more than once). I believe I've heard (here) that piped tar commands can be useful for the first sync (possibly better compression, no overhead of file checksums), then rsync thereafter.

'screen' is absolutely crucial with ssh, too.

I find tmux to be a bit more pleasant to work with. In the past it didn't do reflowing upon resize, but it does that now.

I lean towards

  nohup time whatever >whatever.out &

because so many programs behave differently when attached to a tty (like prompting for input) and I just want a plain text log that says when it succeeded or where it blew up.

Well that is fine, but that is only covering a very small sliver of what screen/tmux are used for.

Agreed, and my problem with even the usage above is that I often realize too late that "whatever" is not going to finish before I need to unplug. Then I have to kill "whatever" and restart with nohup and logging. tmux/screen always has me covered.

Obviously I think tmux/screen is the better option, but bash's builtin 'disown' may have you covered for that specific usecase.

Thanks for the tip, I didn't know about "disown".

sh.py leverages this in command baking http://amoffat.github.io/sh/#baking