How to access or list EODATA using boto3 on ESA HPC

Note

LEGACY ARTICLE

This article is marked as a legacy document and may not reflect the latest information. Please refer to the following article: How to access EODATA using boto3 on ESA HPC

Warning

We strongly recommend using virtualenv for isolating python packages. Configuration tutorial is available here: How to install Python virtualenv or virtualenvwrapper on ESA HPC.

If virtualenv is activated:

(myvenv) eouser@vm01:~$ pip3 install boto3

If we want to install the package globally:

eouser@vm01:~$ sudo pip3 install boto3

To get the credentials for your VM, see article How to get credentials used for accessing EODATA on a cloud VM on ESA HPC.

As an example, let us suppose that the credentials you got from that article are

BSIYKQYCZKUIMXXUBPDAADDAAAADDV:kksHunhhnTreMIxyzePGqMavSBuwDbOxHRAYhahkQex

Two examples for listing products:

Client

  • low-level service access

  • generated from service description

  • exposes botocore client to the developer

  • typically maps 1:1 with the service API

  • snake-cased method names (e.g. ListBuckets API => list_buckets method)

import boto3

access_key='<ACCESS_KEY_FROM_/etc/passwd-s3fs>'
secret_key='<SECRET_KEY_FROM_/etc/passwd-s3fs>'

host='https://eodata.cloudferro.com'

s3=boto3.client('s3',aws_access_key_id=access_key,
aws_secret_access_key=secret_key, endpoint_url=host,)

for i in s3.list_objects(Delimiter='/', Bucket="DIAS", Prefix='Landsat-8/OLI/L1TP/2022/05/15/',MaxKeys=30000)['CommonPrefixes']:
                print(i)

Resource:

  • higher-level, object-oriented API

  • generated from resource description

  • uses identifiers and attributes

  • has actions (operations on resources)

  • exposes subresources and collections

Now, substituting with the actual values of credentials, you get:

import boto3

access_key='BSIYKQYCZKUIMXXUBPDAADDAAAADDV'
secret_key='kksHunhhnTreMIxyzePGqMavSBuwDbOxHRAYhahkQex'

host='http://eodata.cloudferro.com'


s3  = boto3.resource('s3',aws_access_key_id=access_key,aws_secret_access_key=secret_key, endpoint_url=host,)
bucket = s3.Bucket('DIAS')

# list all files in /eodata
for bucket_object in bucket.objects.all():
       print(bucket_object.key)
#list all Sentinel-2 folders and products
for bucket_object in bucket.objects.filter(Prefix='Sentinel-2'):
       print(bucket_object.key)

Save your file with the .py extension and run it with python3 <filename.py> command in your terminal. For example:

(boto3) eouser@vm01:~$ python3 boto_list.py