How to setup Interactive Applications

From UC Grid Wiki
Jump to: navigation, search

Overview

UGP provides users the ability to run interactive GUI applications on the cluster compute nodes and to interact with these applications via their web browsers. UGP impliments this by using VNC, a program that allows users to interact with interactive programs accross the network. To use VNC, UGP runs the VNC Server on the cluster compute node as the user and the VNC Viewer applet in the users web browser. All the tasks normally required to run VNC, such as the generation of a VNC password for the user, the creation of a VNC startup file, etc., are taken care of for the user by UGP.

The following figure summarizes how the interactive applications are run:

File:Vnc arch.jpg

When a users asks to run an interactive application, UGP running on the Grid Portal:


The following is the steps when the user runs xterm in the browser:

  • Runs the VNC Server as the user on the cluster compute node. It does this by using Globus GRAM to start up a fork job on the Grid Appliance conected to the cluster. The fork job, SSHs to the compute node where it starts up the VNC Server. As part of starting up the VNC Server, UGP generates a random password for VNC, and places all necessary files in a subdirectory of the .vnc directory under the user's login id.
  • Runs the VNC Viewer Applet in the user's web browser, passing it the VNC password and a port and address that it is to use to communicate with the VNC Server.

Because an applet can only communicate with the machine that starts it up, the VNC Viewer applet can only communicate with the Grid Portal. Because the VNC Server is running on the cluster compute node, iptables have to be set up in both the Grid Portal and the Grid Appliance machine to pass the communications through to the cluster compute node.

This is shown for a single connection in the following diagram:

File:Vnc single port arch.jpg

In this diagram port 90001 on the Grid Portal is forwarded to port 70001 on the Grid Appliance which is forwarded to port 5901 on the cluster compute node, which is where the VNC Server is running. Since the VNC Viewer applet can only connect to the Grid Portal, it must connect to the Grid Portal at port 90001, to talk to the VNC Server running on port 5901 on the cluster compute node.

Since the Grid Portal is connected to a number of Grid Appliances/clusters and each cluster may have a number of different compute nodes on which interactive jobs can be run and since multiple simultaneous interactive jobs can be run on each of these compute nodes, a large number of ports have be forwarded appropriately. For example you could:

  • Forward ports 90001 through 90100 from the Portal to ports 70001 through 70100 on the Appliance connected to cluster 1.
  • Forward ports 90101 through 90200 from the Portal to ports 70001 through 70100 on the the Appliance connected to cluster 2.
  • etc.
  • Forward ports 70001 through 70025 from the Appliance connected to cluster 1 to one of the compute nodes on that cluster
  • Forward ports 70026 through 70050 from the Appliance connected to cluster 1 to another compute node.
  • etc.

This would allow each of four compute nodes on cluster 1 to run 25 simultaneous interactive jobs.

Constraints

The number of simultaneous interactive jobs that can be run are limited by:

  • The number of compute nodes dedicated to running interactive jobs.
  • The number of licenses available for the applications to be run.
  • The number of ports that have been forwarded from the Grid Portal through to the compute nodes.

VNC sessions that are sitting idle consume little or no resources. Therefore, we allow users to start up interactive application, logout of the Grid Portal and come back later to check on them. Of course, an interactive application that is actually running and doing work will take whatever resources that application requires to run.

Because of the constraints, UGP limits interactive sessions by:

  • Allowing each user only a small number of simultaneous interactive sessions. A users in not allowed to start another interactive application if he/she has reached this limit without killing one that is already running.
  • Killing each VNC Server after a specific amount of idle time. Ports made available by timing out the idle VNC servers are periodically returned to the pool of available ports.

Setup

IPtables

On the Grid Portal

As root, modify /etc/sysconfig/iptables.

Assumptions:

  • The Grid Portal has IP Address 128.1.1.1.
  • The Grid Appliances have IP Addresses 129.1.1.1, 130.1.1.1, ...
  • The Grid Portal has a single network interface card at eth0.

Add the following lines into the nat queue of iptables:

*nat
...
#------------------------------------------------------------------------
# MASQUERADE causes any packet to that is passing through this machine
# destined for somewhere else to appear as if it came from this machine
#------------------------------------------------------------------------
-A POSTROUTING -o eth0 -j MASQUERADE
...
#------------------------------------------------------------------------
# Do Port Forwarding
#------------------------------------------------------------------------
-A PREROUTING -i eth0 -p tcp -d 128.1.1.1 --dport 90001 -j DNAT --to 129.1.1.1:70001
-A PREROUTING -i eth0 -p tcp -d 128.1.1.1 --dport 90002 -j DNAT --to 129.1.1.1:70002
...
-A PREROUTING -i eth0 -p tcp -d 128.1.1.1 --dport 90100 -j DNAT --to 129.1.1.1:70100
-A PREROUTING -i eth0 -p tcp -d 128.1.1.1 --dport 90101 -j DNAT --to 130.1.1.1:70001
...
-A PREROUTING -i eth0 -p tcp -d 128.1.1.1 --dport 90200 -j DNAT --to 130.1.1.1:70100
...
COMMIT

The nat queue is reponsible for network address translation. Prerouting is used to change the destination address of a packet and postrouting is used to change the source address of a packet.

Here we make the following changes:

  • For each port in the range 90001 to 90100 we forward the packets coming to this machine (Grid Portal) to that port, on to the corresponding port on one of the Grid Appliances.
  • We have packets sent to the Grid Appliances MASQUERADE so that they appear as if they were sent from the Grid Portal because otherwise the Grid Appliances might not accept them.

On the Grid Appliance

As root, modify /etc/sysconfig/iptables. Assumptions:

  • The Grid Appliance has IP Address 129.1.1.1.
  • It has two Network Interface Cards: eth0 is the private network interface; eth1 is the public network interface. The Grid Appliance communicates with the Grid Portal over the public network and it communicates with the cluster compute nodes over the private network.
  • The cluster compute nodes to be used for interactive applications have IP Addresses 10.0.0.2 and 10.0.0.3, 10.0.0.4 and 10.0.0.5.

Add the following lines between:

-A INPUT -j RH-Firewall-1-INPUT

and

-A FORWARD -j RH-Firewall-1-INPUT

in the section of the file for the filter queue as follows:

*filter
...
#************************************************************
# The INPUT CHAIN
# Filters packets destined for this machine
#************************************************************
#------------------------------------------------------------
# send all intput to this machine to the RH-Firewall-1-INPUT
#------------------------------------------------------------
-A INPUT -j RH-Firewall-1-INPUT
#------------------------------------------------------------
# Allow all packets from the public network on connections
# that have already been established, to pass through to 
# the private network
#------------------------------------------------------------
-A FORWARD -i eth1 -m state --state ESTABLISHED,RELATED -j ACCEPT
#------------------------------------------------------------
# Allow all packets from the private network to pass to
# the public network.
------------------------------------------------------------
-A FORWARD -i eth0 -o eth1 -m state --state NEW,ESTABLISHED,RELATED  -j ACCEPT
#-----------------------------------------------------------
# send all other packets to the RH-Firewall-1-INPUT
#-----------------------------------------------------------
-A FORWARD -j RH-Firewall-1-INPUT
....
COMMIT

The Filter queue is responsible for packet filtering. It includes the following built-in chains for filtering input:

The INPUT chain
Handles packets destined for this machine.
The FORWARD chain
Handles packets that are to be passed on to other machines.

Here all packets incomming packets destined for this machine are immediately directed to be handled by another chain called RH-Firewall-1-INPUT. The following rules are then made for the packets that are to be forwarded on by this Grid Appliance:

  • Allow packets from the outside world to pass to the compute nodes but only on already established connections. This is needed because once we allow the VNC Viewer Applet running in the user's web browser (see the changes to the nat queue following), to connect to the VNC Server running on the compute nodes, we have to allow it to continue talking to the VNC Server on the established connection.
  • Allow packets from the compute nodes on the private network to pass through to the outside world. This is so that the VNC Server on the compute node can continue talking to the VNC Viewer Applet running under the user's web browser on the established connection but also so that the user can ssh or scp out from the compute node on a new connection.

All packets that don't meet the conditions defined by those two rules are then sent to RH-Firewall-1-INPUT.

Add the following lines into the nat queue of iptables:

*nat
...
#------------------------------------------------------------------------
# MASQUERADE causes any packet to that is passing through this machine
# destined for somewhere else to appear as if it came from this machine
#------------------------------------------------------------------------
-A POSTROUTING -o eth0 -j MASQUERADE
#------------------------------------------------------------------------
# Do port forwarding
#------------------------------------------------------------------------
-A PREROUTING -i eth1 -p tcp -d 129.1.1.1 --dport 70001 -j DNAT --to 10.0.0.2:5901
-A PREROUTING -i eth1 -p tcp -d 129.1.1.1 --dport 70002 -j DNAT --to 10.0.0.2:5902
...
-A PREROUTING -i eth1 -p tcp -d 129.1.1.1 --dport 70026 -j DNAT --to 10.0.0.3:5901
...
-A PREROUTING -i eth1 -p tcp -d 129.1.1.1 --dport 70100 -J DNAT --to 10.0.0.3:5925
...
COMMIT

The nat queue is reponsible for network address translation. Prerouting is used to change the destination address of a packet and postrouting is used to change the source address of a packet.

Here we make the following changes:

  • For each port in the range 70001 to 70100 we forward the packets coming to this machine (Grid Appliance) to that port, on to the appropriate VNC port on one of the four cluster compute nodes used for interactive jobs. Note that on each compute node, the VNC ports used start with port 5901.
  • We have packets sent to the compute nodes MASQUERADE so that they appear as if they were sent from the Grid Appliance because otherwise the compute nodes might not accept them.

Activate IP Forwarding

As root, add the following line into the /etc/sysctl.conf file.

net/ipv4/ip_forward = 1

Then activate IP forwarding by executing the command:

sysctl -p

SSH

Host-Based Authentication must be used between the Grid Appliance and the cluster nodes. For example, if the address of one of the cluster nodes to be used to run the VNC Server is 10.0.0.2, the Grid Appliance must be able to SSH to 10.0.0.2 as a user without SSH asking for the user's password.

As root on the Grid Appliance:

1. Edit /etc/ssh/shosts.equiv. List, one per line the IP Addresses, either alpha or numeric of both the Grid Appliance and the cluster nodes that will participate in the host-based authentication.

2. Use this file to generate a key by isuing the following command:

  ssh-keyscan -t rsa -f shosts.equiv > ssh_known_hosts2

3. Copy this Grid Appliance key to the cluster node.

4. Copy the cluster node's key to ssh_known_hosts on the Grid Appliance.

5. Edit /etc/ssh/ssh_config and enable HostbasedAuthentication and disable the ssh key prompt for the first time login:

  HostbasedAuthentication yes
  StrictHostKeyChecking no

UGP VncPortMap Database Table

To configure the database in the Grid Portal, login to the Grid Portal Website as Admin. Go to Grid Admin -> Vnc Port Mapping. Add and entry for each route through the Grid Portal to a cluster compute node that you created with iptables. Fill in the form and press the Save button. For example:

Vnc Portal Port:      90001
Appliance Name:       appliance1.ucla.edu
Vnc Appliance Port:   70001
Vnc Local Target:     10.0.0.2
Vnc Local Port:       5901

This will add a line to the VncPortMap database that looks like this:

Vnc Portal Port | Appliance Name      | Vnc Appliance Port | Vnc Local Target | Vnc Local Port 
90001             appliance1.ucla.edu   70001                10.0.0.2           5901 

Add an Application that is to run through VNC to the Applications Database

Each interactive GUI application that will be run through VNC has to be added to the Application table. You have to add an application even if you are already running this application in batch. What is more, the name must be unique. The convention is to name the application:

  applicationName-Cluster-VNC

Here is how Matlab looks when you add it:

Name:            Matlab-Beowulf-VNC
Display Name:    Matlab
Appliance Name:  beowulf2.ats.ucla.edu
Description:     A high-level language and interactive environment for technical computing.
Executable:      /u/local/bin/matlab
Argument:        -desktop
Scheduler Name:  Fork

The Display Name is the name the user's see on the Grid Portal web site. -desktop is specified as an argument because without it Matlab will not open its GUI. The Scheduler Name for interactive applications to be run via VNC must be "Fork".