11.1. General Considerations¶

The general issues to take into consideration when troubleshooting your system are listed below. You should read them carefully before trying to solve more specific problems.

• You should always remember where you are currently located in your terminal. Check it periodically using the pwd, hostname, ifconfig, cat /proc/vz/veinfo commands. One and the same command executed inside a virtual machine or container and on the hardware node can lead to very different results. You can also set up the PS1 environment variable to show the full path in the bash prompt. To do this, add these lines to /root/.bash_profile:

  PS1="[\u@\h \w]$ "
  export PS1
  

• If the hardware node slows down, use vmstat, ps (ps axfw), dmesg, htop (vztop) to find out what is happening, never reboot the machine without investigation. If no thinking helps restore the normal operation, use the Alt+SysRq sequences to dump the memory (showMem) and processes (showPc).

• Do not run any binary or script that belongs to a container directly from the hardware node, for example, do not ever do this:

  # cd /vz/root/99/etc/init.d
  ./httpd status
  

  Any script inside a container could have been changed to whatever the container owner chooses: it could have been trojaned, replaced to something like rm -rf, etc. You can use only prlctl exec/prlctl enter to execute programs inside a container.

• Do not use init scripts on the hardware node. An init script may use killall to stop a service, which means that all similar processes will be killed in all containers. You can check /var/run/Service.pid and kill the correspondent process explicitly.

• You must be able to detect any rootkit inside a container. It is recommended to use the chkrootkit package for detection (you can download the latest version from http://www.chkrootkit.org), or at least run

  # rpm -Va|grep "S.5"
  

  To check if the MD5 sum has changed for any RPM file.

  You can also run nmap, for example:

  # nmap -p 1-65535 192.168.0.1
  Starting nmap V. 2.54BETA22 ( www.insecure.org/nmap/ )
  Interesting ports on  (192.168.0.1):
  (The 65531 ports scanned but not shown below are in
    state: closed)
  Port       State       Service
  21/tcp     open        ftp
  22/tcp     open        ssh
  80/tcp     open        http
  111/tcp    open        sunrpc
  Nmap run completed -- 1 IP address (1 host up) scanned
    in 169 seconds
  

  To check if any ports are open that should normally be closed.

  That could however be a problem to remove a rootkit from a container and make sure it is 100% removed. If you’re not sure, create a new container for that customer and migrate his/her sites and mail there.

• Check the /var/log/ directory on the hardware node to find out what is happening on the system. There are a number of log files that are maintained by the system and Virtuozzo Hybrid Server (the boot.log, messages, etc.), but other services and programs may also put their own log files here depending on your distribution of Linux and the services and applications that you are running. For example, there may be logs associated with running a mail server (the maillog file), automatic tasks (the cron file), and others. However, the first place to look into when you are troubleshooting is the /var/log/messages log file. It contains the boot messages when the system came up as well as other status messages as the system runs. Errors with I/O, networking, and other general system errors are reported in this file. So, we recommend that you read to the messages log file first and then proceed with the other files from the /var/log/ directory.

• Subscribe to bug tracking lists. You should keep track of new public DoS tools or remote exploits for the software and install them into containers or at hardware nodes.

• When using iptables, there is a simple rule for Chains usage to help protect both the hardware node and its containers:

  • Use INPUT, OUTPUT to filter packets that come in/out the hardware node.

  • Use FORWARD to filter packets that are designated for containers