How to access EODATA and Object Storage using s3cmd on Linux on ESA HPC

You can install s3cmd using Python PIP or from Linux repository.

Installation from system repository on Debian/Ubuntu systems:

Check for updates:

$ sudo apt update

Installing from the repository:

$ sudo apt install s3cmd

Installation from Python repository

It is possible to install s3cmd with pip.

Installing s3cmd with PIP

Check if you have PIP installed:

$ pip

The program 'pip' is currently not installed. To run 'pip' please ask your administrator to install the package 'python-pip'

Installing on Ubuntu without PIP

$ sudo apt install python-pip

$ pip --version

pip 8.1.1 from /usr/lib/python2.7/dist-packages (python 2.7)

$ sudo pip install s3cmd

If you see the following:

Traceback (most recent call last):
File "/usr/bin/pip", line 11, in <module>
sys.exit(main())
File "/usr/lib/python2.7/dist-packages/pip/init.py", line 215, in main
locale.setlocale(locale.LC_ALL, '')
File "/usr/lib/python2.7/locale.py", line 581, in setlocale
return _setlocale(category, locale)
locale.Error: unsupported locale setting

add the following line:

export LC_ALL=en_US.UTF-8

to the file:

~/.profile

Now you can check .profile:

$ cat ~/.profile
export LC_ALL=en_US.UTF-8
$ source ~/.profile
$ s3cmd --version
s3cmd version 2.0.1

Configure s3cmd

$ s3cmd --configure

Enter new values or accept defaults in brackets with Enter.

Refer to the user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty to use the environment variables.

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: XXXXXXXXXXXXXXXXXXXXXXXXX
Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
Default Region [US]: RegionOne

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: eodata.cloudferro.com

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: eodata.cloudferro.com

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]:

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]:

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

New settings:
      Access Key: XXXXXXXXXXXXXXXXXXXX
      Secret Key: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
      Default Region: RegionOne
      S3 Endpoint: eodata.cloudferro.com
      DNS-style bucket+hostname:port template for accessing a bucket: eodata.cloudferro.com
      Encryption password:
      Path to GPG program: /usr/bin/gpg
      Use HTTPS protocol: True
      HTTP Proxy server name:
      HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to '/home/eouser/.s3cfg'

Now you can use s3cmd commands:

$ s3cmd ls

2017-12-11 15:30  s3://DIAS
2017-12-11 15:30  s3://EOCLOUD
2017-12-11 15:30  s3://EODATA

$ s3cmd ls s3://EODATA/

                       DIR   s3://EODATA/Envisat/
                       DIR   s3://EODATA/Landsat-5/
                       DIR   s3://EODATA/Landsat-7/
                       DIR   s3://EODATA/Landsat-8/
                       DIR   s3://EODATA/Sentinel-1/
                       DIR   s3://EODATA/Sentinel-2/
                       DIR   s3://EODATA/Sentinel-3/
                       DIR   s3://EODATA/Sentinel-5P/

In order to acquire access to Object Storage buckets via s3cmd, first you have to get your credentials from /etc/passwd-s3fs file. See article:

How to get credentials used for accessing EODATA on a cloud VM on ESA HPC.

After this, remove file .s3cfg from your Home folder and then reconfigure s3cmd by entering:

s3cmd --configure

using the following values:

New settings:
Access Key: (credentials from /etc/passwd-s3fs file)
Secret Key: (credentials from /etc/passwd-s3fs file)
Default Region: none
S3 Endpoint: eodata.cloudferro.com
DNS-style bucket+hostname:port template for accessing a bucket: eodata.cloudferro.com
Encryption password: (your password)
Path to GPG program: /usr/bin/gpg
Use HTTPS protocol: True
HTTP Proxy server name:
HTTP Proxy server port: 0

You should now be allowed to list and access your Object Storage.