GLusterfs Replication Setup

GLusterfs Replication Setup

Note:- you can create a partition, mount it to the directory which we will create below and then do the following steps. If you don’t create a partition directly create a directory in / it will use the / partition size.
In this tutorial I use three systems, two servers and a client:
server1.example.com: IP address 192.168.0.100 (server)
server2.example.com: IP address 192.168.0.101 (server)
client1.example.com: IP address 192.168.0.102 (client)
Server setup……

#vi /etc/hosts
192.168.0.100 server1.example.com server1
192.168.0.101 server2.example.com server2
192.168.0.102 client1.example.com client1
:wq!
(It is also possible to use IP addresses instead of hostnames in the following setup. If you prefer to use IP addresses, you don't have to care about whether the hostnames can be resolved or not.)

#yum install libibverbs-devel fuse-devel
#wget http://ftp.gluster.com/pub/gluster/glusterfs/2.0/LATEST/glusterfs-2.0.9.tar.gz
Or
#wget http://pkgs.fedoraproject.org/repo/pkgs/glusterfs/glusterfs-2.0.9.tar.gz/0467740f423e32bdf2f66d2cf63467f3/glusterfs-2.0.9.tar.gz

#tar -xvfz glusterfs-2.0.9.tar.gz
#cd glusterfs-2.0.9
#./configure
#make && make install
#ldconfig
#glusterfs --version
#mkdir /data/
#mkdir /etc/glusterfs
#vi /etc/glusterfs/glusterfsd.vol
volume posix
type storage/posix
option directory /data
end-volume

volume locks
type features/locks
subvolumes posix
end-volume

volume brick
type performance/io-threads
option thread-count 8
subvolumes locks
end-volume

volume server
type protocol/server
option transport-type tcp
option auth.addr.brick.allow 192.168.0.102
subvolumes brick
end-volume

:wq!
Please note that it is possible to use wildcards for the IP addresses (like 192.168.*) and that you can specify multiple IP addresses separated by comma (e.g. 192.168.0.102,192.168.0.103).

Afterwards we create the following symlink...
#ln -s /usr/local/sbin/glusterfsd /sbin/glusterfsd
and then the system startup links for the GlusterFS server and start it
#chkconfig --levels 35 glusterfsd on
#/etc/init.d/glusterfsd start
Same configuration for server-2 also…………..

Client setup…….
#yum install libibverbs-devel fuse-devel
Then we load the fuse kernel module...
#modprobe fuse
and create the file /etc/rc.modules with the following contents so that the fuse kernel module will be loaded automatically whenever the system boots
#vi /etc/rc.modules
modprobe fuse
:wq!
# chmod +x /etc/rc.modules
#wget http://ftp.gluster.com/pub/gluster/glusterfs/2.0/LATEST/glusterfs-2.0.9.tar.gz
Or
#wget http://pkgs.fedoraproject.org/repo/pkgs/glusterfs/glusterfs-2.0.9.tar.gz/0467740f423e32bdf2f66d2cf63467f3/glusterfs-2.0.9.tar.gz
#tar -xvfz glusterfs-2.0.9.tar.gz
#cd glusterfs-2.0.9
#./configure
#make && make install
#ldconfig
#mkdir /mnt/glusterfs
#mkdir /etc/glusterfs
#vi /etc/glusterfs/glusterfs.vol
volume remote1
type protocol/client
option transport-type tcp
option remote-host server1.example.com
option remote-subvolume brick
end-volume

volume remote2
type protocol/client
option transport-type tcp
option remote-host server2.example.com
option remote-subvolume brick
end-volume

volume replicate
type cluster/replicate
subvolumes remote1 remote2
end-volume

volume writebehind
type performance/write-behind
option window-size 1MB
subvolumes replicate
end-volume

volume cache
type performance/io-cache
option cache-size 512MB
subvolumes writebehind
end-volume

:wq!
Make sure you use the correct server hostnames or IP addresses in the option remote-host lines!
That's it! Now we can mount the GlusterFS filesystem to /mnt/glusterfs with one of the following two commands:

#glusterfs -f /etc/glusterfs/glusterfs.vol /mnt/glusterfs
or
#mount -t glusterfs /etc/glusterfs/glusterfs.vol /mnt/glusterfs
#mount
#df -h
If you get the following error “Transport endpoint is not connected”. Check the glusterfsd.vol file at server “option auth.addr.brick.allow 192.168.0.102” ip is proper or not then again mount the file system
For permanent mounting make entry in fstab file
#vi /etc/fstab
/etc/glusterfs/glusterfs.vol /mnt/glusterfs glusterfs defaults 0 0

:wq!

For testing you create files and check the replication.. on each server as well as client……….
Testing GLuster…….
Now let's create some test files on the GlusterFS share:
client1.example.com:
#touch /mnt/glusterfs/test1
#touch /mnt/glusterfs/test2
Now let's check the /data/export directory on server1.example.com and server2.example.com. The test1 and test2 files should be present on each node:
server1.example.com/server2.example.com:
#ls -l /data/export
[root@server1 ~]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2010-02-22 16:50 test1
-rw-r--r-- 1 root root 0 2010-02-22 16:50 test2
[root@server1 ~]#
Now we shut down server1.example.com and add/delete some files on the GlusterFS share on client1.example.com.
server1.example.com:
#shutdown -h now
client1.example.com:
#touch /mnt/glusterfs/test3
#touch /mnt/glusterfs/test4
#rm -f /mnt/glusterfs/test2
The changes should be visible in the /data/export directory on server2.example.com:
server2.example.com:
#ls -l /data/export
[root@server2 ~]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2010-02-22 16:50 test1
-rw-r--r-- 1 root root 0 2010-02-22 16:53 test3
-rw-r--r-- 1 root root 0 2010-02-22 16:53 test4
[root@server2 ~]#
Let's boot server1.example.com again and take a look at the /data/export directory:
server1.example.com:
#ls -l /data/export
[root@server1 ~]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2010-02-22 16:50 test1
-rw-r--r-- 1 root root 0 2010-02-22 16:50 test2
[root@server1 ~]#
As you see, server1.example.com hasn't noticed the changes that happened while it was down. This is easy to fix, all we need to do is invoke a read command on the GlusterFS share on client1.example.com, e.g.:
client1.example.com:
#ls -l /mnt/glusterfs/
[root@client1 ~]# ls -l /mnt/glusterfs/
total 0
-rw-r--r-- 1 root root 0 2010-02-22 16:50 test1
-rw-r--r-- 1 root root 0 2010-02-22 16:53 test3
-rw-r--r-- 1 root root 0 2010-02-22 16:53 test4
[root@client1 ~]#
Now take a look at the /data/export directory on server1.example.com again, and you should see that the changes have been replicated to that node:
server1.example.com:
#ls -l /data/export
[root@server1 ~]# ls -l /data/export
total 0
-rw-r--r-- 1 root root 0 2010-02-22 16:50 test1
-rw-r--r-- 1 root root 0 2010-02-22 16:53 test3
-rw-r--r-- 1 root root 0 2010-02-22 16:53 test4
[root@server1 ~]#
Note:- there is no limit for server and client, configuration will be same……..