Skip to main content

How to integrate AWS s3 bucket with AWS ec2 instance (Centos & Ubuntu)



In this tutorial we can check how to mount S3 bucket on your EC@ instance.
S3FS is a FUSE (File System in User Space) will mount Amazon S3 as a local file system. S3FS has an ability to manipulate Amazon S3 bucket in many useful ways. If you wish to access your Amazon S3 bucket without mounting it on your server, you can use s3cmd command line utility to manage S3 bucket.

What is an Amazon S3 bucket?
Amazon S3 is a cloud based web service interface that you can used to store and retrieve any amount of data. To upload your data, first you need to create an S3 bucket. S3 bucket doesn't need any region.
Creating a Bucket
S3 provides an API for creating and managing buckets. You can create a maximum of 100 buckets from your AWS console. When you create a bucket, you need to provide a name and AWS region where you want to create the bucket. In each bucket, you can store any number of objects. You can use your AWS account root credentials to create a bucket, but it is not recommended. Instead  just create an IAM user and add full permission to that user on S3 bucket. You can access your S3 bucket from your Amazon S3 console.
Please follow the below steps to mount s3 bucket on your server.

1) Remove Existing Packages
Before installing any package, first you need to check if you have any existing fuse or S3FS on your server. If it is already existing, then remove it from your server to avoid further conflicts. Use the following command to check if you have any existing fuse or S3FS on your server
CentOS users:
$ yum remove fuse fuse-s3fs
Ubuntu Users:
$ apt-get remove fuse

2) Install Packages
Install all dependency packages for fuse and s3cmd using the below command.
CentOS users:
$ yum install gcc libstdc++-devel gcc-c++ curl-devel libxml2-devel openssl-devel mailcap  
yum install automake fuse fuse-devel gcc-c++ git libcurl-devel libxml2-devel make openssl-devel

Ubuntu Users:
$ apt-get install build-essential libcurl4-openssl-dev libxml2-dev mime-support

3) Download and Compile Fuse
Move to /usr/src then download and compile fuse source code. After compiling, add fuse to kernel. In my case the latest version of fuse is fuse-3.0.0
$ cd /usr/src/
$ tar xzf fuse-3.0.0.tar.gz
$ cd fuse-3.0.0
$ ./configure –prefix=/usr/local
$ make && make install
$ export PKG_CONFIG_PATH=/usr/local/lib/pkgconfig
$ ldconfig
$ modprobe fuse

4) Download and compile S3FS
Navigate to /usr/src, Download and compile s3fs source code.
$ cd s3fs-fuse
$ ./autogen.sh
$ ./configure
$ make
$ make install

5) Setup Access Key
Both access key and secret key of your s3 AWS account is required for configuring S3FS. Replace the AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY with your actual key values.
$ vi /etc/passwd-s3fs
AWS_ACCESS_KEY_ID:AWS_SECRET_ACCESS_KEY
Make sure that the file has proper permission.
$ chmod 600 /etc/passwd-s3fs

6) Mount S3 Bucket
You can run the below command to mount s3fs.
$ s3fs mybucket /path/to/mountpoint -o passwd_file=/etc/passwd-s3fs
You can also mount the s3 bucket on boot by following below commands.
$ mkdir /tmp/cache
$ mkdir /path/to/mountpoint
$ chmod 777 /tmp/cache /path/to/mountpoint
$ vi /etc/fstab
s3fs# /path/to/mountpoint fuse allow_other,use_cache=/tmp/cache,uid=userid,gid=groupid 0 0
$ mount -a

Congratulations you have successfully mounted s3 bucket on your server.

Labels:

Comments

Post a Comment

Popular posts from this blog

Setup a Multi-Protocol VPN Server Using SoftEther on Centos7

  Introduction This article explains how to install and configure a multi-protocol VPN server using the SoftEther package. We enable and configure OpenVPN and L2TP over IPSec and SSTP VPN Servers on Linux. What is SoftEther SoftEther VPN is one of the world’s most powerful and easy-to-use multi-protocol VPN software, made by the good folks at the University of Tsukuba, Japan. It runs on Windows, Linux, Mac, FreeBSD and Solaris and is freeware and open-source. You can use SoftEther for any personal or commercial use free of charge. Step 1: Create a Virtual Server First, you need to create a DigitalOcean Droplet. As mentioned in SoftEther’s website, SoftEther will work on almost every Linux distro with kernel v2.4 or above,; however it’s recommended to choose one of these distributions: CentOS, Fedora, or Red Hat Enterprise Linux. Personally I have tried it on Ubuntu, CentOS and Fedora, both 32 and 64 bit editions, and it has worked perfectly. Step 2: Update your Server Software Usin...
Post a Comment
Read more

Setting Up MariaDB MaxScale Binlog Proxy for backup/failover

  Setting Up MariaDB MaxScale Binlog Proxy MariaDB MaxScale Binlog Server acts as an intermediate master replication proxy. By storing binary logs and serving them to replicas, it enables scalable replication architectures, reduces primary database load, and accelerates disaster recovery. Step 1: Install MariaDB MaxScale Install MaxScale using the package manager appropriate for your Linux distribution, then verify the installation. On Ubuntu/Debian: sudo apt-get update sudo apt-get install maxscale On RHEL/CentOS: sudo yum install maxscale Verify the installation: maxscale --version Step 2: Configure MaxScale as a Binlog Proxy Open MaxScale's core configuration file to establish the routing properties and specify your database backend configuration. sudo nano /etc/maxscale.cnf Add the following configurations for the moni...
Post a Comment
Read more

Deploying a laravel app using nginx alias

  server { listen 80; server_name subdomian.example.com; root /var/www/mainapp/; add_header X-Frame-Options "SAMEORIGIN"; add_header X-XSS-Protection "1; mode=block"; add_header X-Content-Type-Options "nosniff"; index index.html index.htm index.php; charset utf-8; error_log /var/log/nginx/error.log warn; location / { try_files $uri $uri/ /index.php?$query_string; } location = /favicon.ico { access_log off; log_not_found off; } location = /robots.txt { access_log off; log_not_found off; } error_page 404 /index.php; location ~ \.php$ { fastcgi_pass unix:/var/run/php/php8.1-fpm.sock; fastcgi_index index.php; fastcgi_param SCRIPT_FILENAME $realpath_root$fastcgi_script_name; include fastcgi_params; } location /laravelapp { alias /var/www/mainapp/laravelapp/public; index index.php index.html in...
Post a Comment
Read more