Quelques digressions sous GPL...

Welcome AWS and its many EC2 instances. The idea of maintaining a large number of instances is nice, except when you realize that you forgot to create a user account and your environment is already running.

In that case, you have two choice: rebuild the AMI and terminate/re-initiate all your instances, or use SSH capabilities to perform the task on multiple systems.

I chose the later, essentially because, a few months back, I was interviewing with Google and one of their engineers asked me something similar (how would you edit a file on thousands of machine at once). At the time, I didn't fully know the answer. Now that I do, if anybody from Google is reading me, please call back, you have my number ;)

Back to our initial problem, the solution is a mix of SSH commands, sudo, and some bash fun.

The diagram above shows the infrastructure. The idea is to create the accounts without leaving the laptop. So the first barrier we have to deal with is how to connect to a server through a relay without interacting with the relay at all.

ssh-agent magic

ssh-agent is a mandatory tool for any sysadmin. It allows you to cache your SSH keys in your session, thus avoiding the need to unlock the key every time your connect to a destination. My ssh-agent is started with my session (more explanation here). So every time I log in, I just need to add my private key into ssh-agent. The command is simple:

julien@laptop:~$ ssh-add ~/.ssh/id_dsa_jve
Enter passphrase for /home/julien/.ssh/id_dsa_jve: 
Identity added: /home/julien/.ssh/id_dsa_jve (/home/julien/.ssh/id_dsa_jve)

And from there on, I can ssh into any destination without a password prompt

julien@laptop:~$ ssh security-relay.linuxwall.info
Last login: Fri Jul 22 14:43:22 2011 from 62.15.98.128
[julien@security-relay ~]$

But that doesn't give me the hability to ssh into the destination server directly. If I try to pipe a second SSH command in the first one, here is what happens:

julien@laptop:~$ ssh -t -t security-relay.linuxwall.info ssh 10.1.2.20
Permission denied (publickey,gssapi-with-mic).
Connection to security-relay.linuxwall.info closed.

I can successfully connect to security-relay.linuxwall.info but the connection to 10.1.2.20 fails with a Permission denied. This makes sense because the second commands is executed like any normal bash command on the security-relay machine, and because there is no ssh-agent running in my security-relay session, the command has no key to connect to 10.1.2.20.

The inefficient solution here would be to transfert my private key to the security relay, but I don't want that. SSH provides a more elegant way of doing this by forwarding your local agent status to your remote session. You can activate this feature by changing a parameter in the client configuration of ssh:

julien@laptop:~$ cat .ssh/config 

Host security-relay.linuxwall.info
    ForwardAgent yes
    ForwardX11 no
    SendEnv LANG LC_*
    HashKnownHosts no
    GSSAPIAuthentication yes
    GSSAPIDelegateCredentials no

ForwardAgent yes will do the trick and allow you to SSH in the remote server using your local private key.

julien@laptop:~$ ssh -t -t security-relay.linuxwall.info ssh -t -t 10.1.2.20
Last login: Fri Jul 22 15:13:58 2011 from 10.1.4.10

[julien@10.1.2.20 ~]$ 

Note: the -t option is mandatory, otherwise ssh complains about the tty.

     -t      Force pseudo-tty allocation.  This can be used to execute
             arbitrary screen-based programs on a remote machine, which
             can be very useful, e.g. when implementing menu services.
             Multiple -t options force tty allocation, even if ssh has no
             local tty.

Launch a root command remotely.

Now that we can automate the connection, there is another problem we have to deal with: useradd requires root permission. And to gain root permission, we can either enter the root password, or enter the sudo password.

sudo has the -S option to take the password from stdin. So we can pipe the password from the command line using:

julien@laptop:~$ echo "mysudopassword"|ssh -t -t security-relay.linuxwall.info sudo -S 'tail /var/log/messages'

[... output of /var/log/messages...]

But there are two problems with this method. First, the password become visible on the command line, and stored in bash_history afterward. You do not want that.

We can use read instead. read will prompt for the password and store it in a variable that we can echo to our command. The password will still be displayed in the local terminal, but not in the bash_history or list of processus.

julien@laptop:~$ read -s -p "password: " rootpassword && echo $rootpassword|ssh -t -t security-relay.linuxwall.info sudo -S 'tail /var/log/messages'

[... output of /var/log/messages...]

Now, the second problem: I haven't found a way to pass a variable through the SSH relay. I can pass one level of SSH but not the second one. I've looked around but didn't find an option for this, so if somebody knows a way, I'm all ears.

To overcome that, we have to open a little security hole, and temporarily store the password on the security-relay, then proceed in two steps instead of one:

1. launch an ssh command that store the password in a file on the relay
2. launch a second commant that cat the file and pass the variable to the destination sudo command

We don't need to cat the password file more than once, by doing it once only, we open the sudo session on the destination server and reuse that session afterward.

The first command looks like that

julien@laptop:~$ read -s -p "password: " rootpassword && ssh -t -t security-relay.linuxwall.info "env rootpassword=\"$rootpassword\" |echo $rootpassword > /home/julien/laptopsudo && chmod 400 /home/julien/laptopsudo"

We first store the password using read, then pass that environment variable to the remote session and store it in a local file called /home/julien/laptopsudo.

It's not ideal, and I'm aware of the security issue. but we reduce the risk by keeping that file for a few seconds only (the duration of the operation), and limiting it's permissions to 400.

[julien@security-relay ~]$ ls -al laptopsudo 
-r-------- 1 julien julien 11 Jul 22 17:16 laptopsudo

Note that if anybody has a better solution, I'm definitely interested.

The subsequent commands read the password from the local file and execute the sudo command in the destination server. However, sudo is a bit tricky here as well. By default, sudo refuses to execute a sudo command if the user doesn't own a proper tty. So to allow this behavior, we need to update /etc/sudoers. I must admit I haven't read too much about the security implications of this. But considering that my servers are not exposed, and no user is able to open a session on them, I imagine this is fine.

On the destination server, edit /etc/sudoers as follow:

[julien@10.1.2.20 ~]$ sudo sudoedit /etc/sudoers
[sudo] password for julien:

[...]
     55 # Disable "ssh hostname sudo <cmd>", because it will show the password in clear.
     56 #         You have to run "ssh -t hostname sudo <cmd>".
     57 #
     58 #Defaults    requiretty
     59 Defaults    !requiretty
[...]

With requiretty disabled, we can now run the following command

julien@laptop:~$ ssh -t -t security-relay.linuxwall.info "cat /home/julien/laptopsudo | ssh 10.1.2.20 sudo -S tail /var/log/messages"

Creating the user

It's a several step process, but now that we can launch sudo commands on the destination server, it's fairly straighforward.

1. sudo as root create the user with useradd
2. sudo as the user and create it's .ssh
3. copy the public key locally
4. move the public key to the user's authorized_keys file
5. destroy the sudo session

It can be done in a few commands. I put everything in a bash script with variables to make it look cleaner and reusable.

#!/usr/bin/env bash
# connect to a dest server through a ssh relay to create a new user
server_list="10.1.2.20 10.1.2.21 10.1.2.22 10.1.2.23 10.1.2.24"
user_to_create=spongebob
user_group=bobleponge
user_public_key='ssh-rsa AAxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxWt1RZ0n2ee3dzPepNODw== spongebob'
ssh_relay=security-relay.linuxwall.info


sudo_password_file=sudopassword$(date +%s)

echo "starting batch processing, please enter your sudo password"
read -s -p "password: " rootpassword && ssh $ssh_relay "env rootpassword=\"$rootpassword\" |echo $rootpassword > $sudo_password_file && chmod 400 $sudo_password_file"

for server in $server_list; do

    echo "creating $user_to_create in $server"
    if [ ! -z $user_group ]; then
        ssh -t -t $ssh_relay "cat $sudo_password_file | ssh $server \"sudo -S -u root /usr/sbin/useradd -d /home/$user_to_create -g $user_group -m -s /bin/bash $user_to_create\""
    else
        ssh -t -t $ssh_relay "cat $sudo_password_file | ssh $server \"sudo -S -u root /usr/sbin/useradd -d /home/$user_to_create -m -s /bin/bash $user_to_create\""
    fi

    ssh -t -t $ssh_relay "ssh $server \"sudo -u $user_to_create mkdir /home/$user_to_create/.ssh\""

    ssh -t -t $ssh_relay "ssh $server \"echo $user_public_key > $user_to_create.authorized_keys\""

    ssh -t -t $ssh_relay "ssh $server \"sudo mv $user_to_create.authorized_keys /home/$user_to_create/.ssh/authorized_keys\""

    ssh -t -t $ssh_relay "ssh $server \"sudo chown $user_to_create:$user_group /home/$user_to_create/.ssh/authorized_keys\""

    ssh -t -t $ssh_relay "ssh $server \"sudo chmod 740 /home/$user_to_create/.ssh/authorized_keys\""

    ssh -t -t $ssh_relay "ssh $server \"sudo -k\""
done

ssh $ssh_relay rm $sudo_password_file

And that's it ! It took me longer than expected to get to this result, I wasn't expecting sudo and ssh do be so complex to deal with, but the results is quite pleasant. And once you have the knowledge, reusing that code for other tasks is a piece of cake.

Cheers

Young impatient jedi: the disk cache in your image XML configuration file you must remove, or network bandwidth slower than jabba at the NewYork marathon experience you will !

root@jvegln:/etc/libvirt/qemu# cat OpenSolarisGLN-i386.xml

[...]

 <devices>
<emulator>/usr/bin/kvm</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='none'/>