Background

I recently had a task that required me to connect to an SFTP server, download particular files (if they existed), and then process the contents. We were given the option of using any language to fulfil the criteria, so I chose Python to tackle the problem. This blog outlines the problems I encountered when attempting to build the script, as well as a tutorial on how to use the library.

Why not an alternative lib something like pysftp

This StackOverflow answer lists the differences between the libraries.

Requirements

Verify Python

We need to have some version of python installed in our machine to use the library, in my case I am currently using python 3.8.

Install Paramiko

Run the following command in the terminal to install the paramiko library globally

pip install paramiko

SFTP Server

It is suggested that we have an SFTP server up and running in order to test the script. Our server can assist us in configuring the setup to our liking.

If you do not want to set up or have an SFTP server to test on, we can utilise several publicly available servers. A list of servers may be found here.

How to use the library

Now let's start discussing how to use the library. We'll be talking about the following in upcoming sections.

• Connecting to the remote server
• Executing commands on the server
• Transferring files (Upload/Download)
• The final script

Connecting to the remote server

First let's define the constants for connection properties like hostname, username, password, public key, private key etc.., In a real-world application, these properties should come from a configuration file/system.

# There might be others based on the SFTP configuration
HOST_NAME = "test.rebex.net"
USER_NAME = "demo"
PASSWORD = "password"
PRIVATE_KEY_FILE = '/some/file/path/private_key.txt'
PORT = 22

To connect to the server we create an instance client of paramiko.SSHClient(). Paramiko.SSHClient is the primary class used to make connections to the remote server and execute commands. This class wraps Transport, Channel, and SFTPClient to take care of most(not all as discussed below) aspects of authenticating and opening channels. The creation of an SSHClient object allows establishing server connections via the connect() method.

Password-based auth

client = paramiko.SSHClient()
client.connect(hostname=HOST_NAME, port=PORT, username=USER_NAME, password=PASSWORD)

connect() method has support for a wide variety of ways to authenticate before connecting to the server for a user. The above-mentioned method is for password-based authentication.

For private key-based auth, instead of passing the password parameter, we can instead pass pkey an object of type PKey or key_filename the list of private key file names.

Private key-based auth

# Passing the PKey object
client = paramiko.SSHClient()
pkey = paramiko.RSAKey.from_private_key_file(PRIVATE_KEY_FILE)
client.connect(hostname=HOST_NAME, port=PORT, username=USER_NAME, pkey=pkey)

# Passing private key file path
client = paramiko.SSHClient()
client.connect(hostname=HOST_NAME, port=PORT, username=USER_NAME, key_filename=PRIVATE_KEY_FILE)

However, connect() method doesn't support 2-Factor authentication. Following are a few examples where the above method might not work.

• SFTP server is enabled with both password and private key-based auth.
• SFTP server needs an OTP post entering the password to login into the server.

For use cases like these instead of creating an object of Paramiko.SSHClient we can create Paramiko.Transport instead. Transport method has a support to authenticate the connection after successful negotiation by the methods like auth_password, auth_publickey, auth_interactive etc..,

Password and Private key-based auth

Following are a few sample code snippets for the usage of above mentioned methods

transport = paramiko.Transport()
transport.start_client()
transport.auth_password(USER_NAME, PASSWORD)
pkey = paramiko.RSAKey.from_private_key_file(PRIVATE_KEY_FILE)
transport.auth_publickey(USER_NAME, pkey)

Password and OTP based auth

Here we use auth_interactive method which accepts a handler for passing the values to the prompt. The custom handler that we write has the logic to pass the OTP or Password etc.. based on the output in the prompt. If we want to get the input to the handler from stdin instead then we can use the auth_interactive_dumb method.

transport = paramiko.Transport()
transport.start_client()
transport.auth_interactive(username, handler)

def handler(self, title, instructions, prompt_list):
        answers = []
        for prompt_, _ in prompt_list:
            prompt = prompt_.strip().lower()
            if prompt.startswith('password'):
                answers.append(PASSWORD)
            elif prompt.startswith('verification'):
                answers.append(OTP)
            else:
                raise ValueError('Unknown prompt: {}'.format(prompt_))
        return answers

Executing commands on the server

After we have successfully logged into the server by any one of the above-mentioned methods executing the commands is as simple as passing your command to exec_command.

command = "ls -l"
stdin, stdout, stderr = client.exec_command(command)
print(stdout.read())

However paramiko has some very useful util methods to run SFTP commands on the server, following are a few of them

sftp_client = client.open_sftp()
sftp_client.chdir() # Changing the directory
sftp_client.cwd() # Get the current working directory
sftp_client.listdir() # List all the contents of the current working directory

For other supported commands, we can refer to this documentation.

File Transfers

Paramiko allows to programmatically send and receive files using the SFTP protocol. To upload/download file we need to create an SFTPClient session object by calling open_sftp(). This object allows to perform common SFTP operations like get(), put().

Upload File

sftp_client= client.open_sftp()
sftp_client.put(uploadlocalfilepath,uploadremotefilepath)
sftp_client.close()

Download File

sftp_client= client.open_sftp()
sftp_client.get(downloadremotefilepath,downloadlocalfilepath)
sftp_client.close()

The final script

Navigate to this link for the gist of the following script. Star ⭐️ the gist if it helped you.

References

• Official Paramiko documentation can be a good start for exploring other features of the library
• pysftp vs paramiko
• SSH using Python Paramiko