How to mount eodata using s3fs in Linux on ESA HPC
In this tutorial, you will access EO DATA resources using a Linux VM running on ESA HPC EOHPC cloud using s3fs, which is a tool for mounting S3 buckets.
What We Are Going To Cover
Accessing eodata using s3fs on new VMs using preinstalled images
Accessing eodata using s3fs on older VMs or VMs using custom images
Gathering additional logs for s3fs
Prerequisites
No. 1 Account
You need a ESA HPC hosting account with access to the Horizon interface: https://horizon.eohpc.net/auth/login/?next=/.
No. 2 Linux VM
You must operate an Ubuntu, Debian or CentOS virtual machine running on the ESA HPC cloud. It must be connected to the eodata network.
You can create the VM using either one of the default images or uploading your own disk image of the operating system. See these articles:
How to create a Linux VM and access it from Linux command line on ESA HPC
How to create a Linux VM and access it from Windows desktop on ESA HPC.
How to upload your custom image using OpenStack CLI on ESA HPC.
No. 3 Credentials
This Prerequisite only applies if you are using an operating system image which does not come with access to EODATA repository via s3fs. It could be for example a custom image uploaded by you.
You need to obtain appropriate credentials to be able to configure s3fs. Follow this article to learn how to do that: How to get credentials used for accessing EODATA on a cloud VM on ESA HPC.
Accessing eodata using s3fs on new VMs Configured Using Default Images
New virtual machines running on ESA HPC EOHPC cloud that are configured using Ubuntu or CentOS default images have access to the eodata repository. It is mounted automatically at startup in the /eodata folder. The service eodata.mount is used for that purpose. To navigate to the eodata repository simply execute the following command:
cd /eodata
Now you can list its contents using the following command:
ls
You should see output similar to this:
What To Do Next
You can also try other methods of accessing EODATA, such as:
How to mount EODATA as a filesystem using Goofys in Linux on ESA HPC
How to download EODATA files using s3cmd on ESA HPC
If you find that the amount of disk space on your VM is insufficient for your EODATA processing, you can create and attach a volume to it. You can do so by following one of these articles: