For the purpose of Distributed Operating Systems course, we want to manually configure load-balancing using Linux Virtual Server (LVS).

The architecture design presented below is far from ideal, but for the first step, we just need to get more familiar with Linux network and virtual server settings.

It shouldn’t be considered as pattern, but as a warm-up before getting to much nicer load-balanced and high-availability designs.

What we intend to achieve here may be partition into four steps:

1. Configure communication:
   1. Add new Network Interface Card (eth1) for local communication on LVS and attach it to local network VMNet2.
   2. Change attachment of RS1 and RS2 Network Interfaces Cards to local network VMNet2.
   3. Add new IP address for LVS on eth1 (192.168.1.1)
   4. Change the addresses of eth0 on RS1(192.168.1.200), RS2 (192.168.1.200).
2. Update package repositories and install additional software
   1. ipvsadm on LVS
   2. apache2, php5, php5-mysql, mysql-server on RS1 and RS2
3. Configure LVS
   1. Allow packet forwarding in IPv4 settings (ip_forward=1), so it can act as router,
   2. Disallow ICMP messages about better path existing to some destinations (send_redirects=0), so it won’t inform client that he’s not receiving traffic for VIP,
   3. Configure loadbanalcing (3 ipvsadm commands, -A for adding service, -a for adding server, -m for masquerading)
   4. Configure NAT firewall (iptables commands: *flush nat table – so it clean, *configure NAT, so we’ll do the NAT for traffic back)
4. Configure RS1 and RS2
   1. delete default route added by VMWare Player,
   2. delete IPs configured by VMWare Player,
   3. add default routing via LVS (192.168.1.1).

Update repositories and install additional software

On all servers (LVS, RS1, RS2) update information about available packages:

On LVS install ipvsadm package (for configiuring LVS):

On RS1 and RS2 install apache2, php5 libraries and mysql server:

Using VMWare Player we’ll modify Virtual Machine Settings.

LVS should have two Network Adapters:

• first attached to VMNet8 (NAT) – represented by eth0 in Linux,
• second attached to VMNet2 – represented by eth1 in Linux.

If you want to figure out which network interface is represented as eth0 you should:

• check the MAC addresses in VMWare Player by clicking “Virtual Machine Settings” the choose the Network Adapter and click “Advanced” button.
• check the MAC address in Linux by running “ip a s” command.

If MAC address attached to VMNet8 (NAT) is the same as MAC address on eth0 then everything is all rigth. Otherwise you need to reassign the network in VMWare Player Settings.

Linux Virtual Server and Real Servers config clearing

Put down and flush all the configuration of network interfaces on LVS and RS servers:

If error occurs during ifdown, you can probably ignore it and carry on with the work.

Linux Virtual Server interfaces

Change the /etc/network/interfaces – comment out the line with dhcp (dynamic ip configutration) and add the lines with eth0 and eth1 static ip configuration:

Save the file  and run:

Real Servers interfaces

Change the /etc/network/interfaces – comment out the line with dhcp (dynamic ip configutration) and add the lines with eth0 static ip configuration:

Save the file  and run:

Testing

You can try to check reachability:

• LVS-Windows Host – ping 192.168.47.100 from LVS
• LVS-RS1,2 – ping 192.168.47.100,192.168.1.100 and 192.168.1.200

(Right now this config doesn’t work, so take it as a hints, we used it for environments provided in 2013-2017)

Local communication between nodes

Adding IP address on server interefaces

Add new address 192.168.1.1, mask 255.255.255.0 (/24) to eth1 on LVS:

Add new address 192.168.1.100, mask 255.255.255.0 (/24) on RS1 :

Add new address 192.168.1.200, mask 255.255.255.0 (/24) on RS2:

Setting up RS1 and RS2

On RS1 and RS2, by running “ip a”, check what address was assigned to its network interface (eth0). Check what is the default gateway with “ip r” command. Example below shows configuration of RS2 (do the same on RS1):

We found that the IP address in ths example is 192.168.47.130, and default gateway is 192.168.47.2.

We want RS1 and RS2 to connect to external networks using LVS as the default gateway.We also want that LVS-RS1,2 communication to be taken on their local IP addresses (192.186.1.0/24).

To do so:

Remove the default gateway (previous route via 192.168.47.2 set up by VMware needs to be deleted).

we’ll add the default gateway via s192.168.1.1 (LVS)

We’ll remove address assigined by VMware.

From now on LVS is a default gateway for RS2.

Task:

• do analogically configuration for RS1.

LVS Nat configuration

With ipvsadm we’ve already made our LVS to pass and load-balance the connections from externall to RS1 and RS2. We’ve also changed RS1 and RS2 so they’re using LVS as default gateway for the connections coming back.

It will still not work though.

We need to set up NAT (Network Address Translation) of the source addresses in packet coming back from RS1 and RS2 to external network (CLIent).

The work will be done on LVS. Client connected and send HTTP request to external IP address of LVS (VIP), so we cannot send the response from RIP1/RIP2 directly to client. We need to translate source address in responeses from RIP1/RIP2 to VIP.

On LVS we need to first flush all the previous configuration and add SNAT rule:

From now on there’ll be proper communication working between external network and our real server.

Caution:

• all what was done here is ephemeral (i.e. it’ll not be available after reboot).
• the architecture design of our cluster is far from ideal, but it’s just the first step to get to the more real life example

Setting up LVS

We need to adjust parameters of networking on our virtual server

On LVS run commands as shown below (you may try to write this to a scrip lvs-rso.sh, so you’ll be able to recall it in the future):

On RS1 and RS2 disable packet forwarding:

On RS1 and RS2 websites server by apache http server are located in /var/www/. Change default index.html files so they differ (we’ll be able to recognize if cluster works).

It may happen that old configuration of nameserver is present in /etc/resolv.conf.  It should consist of nameserver 192.168.47.2 (you may have different value than 47).

Check the configuration by launching 192.168.47.111 on Firefox.

LVS:

• Czy ipvsadm jest zainstalowany na LVS?
• Czy ip_forward jest włączony (value set to 1) na LVS?
• CZy send_redirects jest wyłączony (value set to 0) na LVS?
• Czy iptables na LVS – pokazuje NAT na dobry adres VIP?

RS1:

• Czy na RS1 – jest ustawiony default routing via 192.168.1.1?
• Czy na RS1 – nie ma innego default routing?
• Czy na RS1 – nie ma routingu do starej trasy dodanej przez VMWare (tj. do komunikacji bezpośrednio z Windows)?

RS2:

• Czy na RS2 – jest ustawiony default routing via 192.168.1.1?
• Czy na RS2 – nie ma innego default routing?
• Czy na RS2 – nie ma routingu do starej trasy dodanej przez VMWare (tj. do komunikacji bezpośrednio z Windows)?