Showing posts with label Linux. Show all posts
Showing posts with label Linux. Show all posts

Tuesday, April 23, 2019

Uncomplicated Firewall with Python

Uncomplicated Firewall with Python

Uncomplicated Firewall, is an interface to iptables that simplifyies the process of configuring a firewall. Iptables is flexible, it can be difficult for beginners to learn how to use it to properly configure a firewall. 

In the example below is a python program that makes it easy allowing and blocking various services by IP address. 

## Description :
## Generate ip-host binding list for a list of nodes, when internal DNS is missing.
## 1. For existing nodes, allow traffic from new nodes
## 2. For new nodes, allow traffic from all nodes
##
## Sample:
## python ./ufw_allow_ip.py --old_ip_list_file /tmp/old_ip_list --new_ip_list_file /tmp/new_ip_list \
## --ssh_username root --ssh_port 22 --ssh_key_file ~/.ssh/id_rsa
##
##-------------------------------------------------------------------
import os, sys
import paramiko
import argparse
# multiple threading for a list of ssh servers
import Queue
import threading
import logging
log_folder = "%s/log" % (os.path.expanduser('~'))
if os.path.exists(log_folder) is False:
os.makedirs(log_folder)
log_file = "%s/%s.log" % (log_folder, os.path.basename(__file__).rstrip('\.py'))
logging.basicConfig(filename=log_file, level=logging.DEBUG, format='%(asctime)s %(message)s')
logging.getLogger().addHandler(logging.StreamHandler())
def get_list_from_file(fname):
l = []
with open(fname,'r') as f:
for row in f:
row = row.strip()
if row.startswith('#') or row == '':
continue
l.append(row)
return l
def ufw_allow_ip_list(server_ip, ip_list, ssh_connect_args):
if len(ip_list) == 0:
print("Skip run ufw update in %s, since ip_list is empty" % (server_ip))
return("OK", "")
[ssh_username, ssh_port, ssh_key_file, key_passphrase] = ssh_connect_args
ssh_command = ""
# TODO: improve this command, by using a library
for ip in ip_list:
ssh_command = "%s && ufw allow from %s" % (ssh_command, ip)
if ssh_command.startswith(" && "):
ssh_command = ssh_command[len(" && "):]
print("Update ufw in %s. ssh_command: %s" % (server_ip, ssh_command))
output = ""
try:
ssh = paramiko.SSHClient()
ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())
key = paramiko.RSAKey.from_private_key_file(ssh_key_file, password=key_passphrase)
ssh.connect(server_ip, username=ssh_username, port=ssh_port, pkey=key)
stdin, stdout, stderr = ssh.exec_command(ssh_command)
output = "\n".join(stdout.readlines())
output = output.rstrip("\n")
print("Command output in %s: %s" % (server_ip, output))
ssh.close()
except:
return ("ERROR", "Unexpected on server: %s error: %s\n" % (server_ip, sys.exc_info()[0]))
return ("OK", output)
###############################################################
if __name__ == '__main__':
# get parameters from users
parser = argparse.ArgumentParser()
parser.add_argument('--old_ip_list_file', required=True, \
help="IP list of current cluster", type=str)
parser.add_argument('--new_ip_list_file', required=True, \
help="IP list of new nodes", type=str)
parser.add_argument('--ssh_username', required=False, default="root", \
help="Which OS user to ssh", type=str)
parser.add_argument('--ssh_port', required=False, default="22", \
help="Which port to connect sshd", type=int)
parser.add_argument('--ssh_key_file', required=False, default="%s/.ssh/id_rsa" % os.path.expanduser('~'), \
help="ssh key file to connect", type=str)
parser.add_argument('--key_passphrase', required=False, default="", \
help="Which OS user to ssh", type=str)
l = parser.parse_args()
ssh_connect_args = [l.ssh_username, l.ssh_port, l.ssh_key_file, l.key_passphrase]
old_ip_list = get_list_from_file(l.old_ip_list_file)
new_ip_list = get_list_from_file(l.new_ip_list_file)
has_error = False
# TODO: speed up this process by multiple threading
for old_ip in old_ip_list:
(status, output) = ufw_allow_ip_list(old_ip, new_ip_list, ssh_connect_args)
if status != "OK":
has_error = True
print("Error in %s. errmsg: %s" % (old_ip, output))
for new_ip in new_ip_list:
(status, output) = ufw_allow_ip_list(new_ip, new_ip_list + old_ip_list, ssh_connect_args)
if status != "OK":
has_error = True
print("Error in %s. errmsg: %s" % (new_ip, output))
if has_error is True:
sys.exit(1)
#!/usr/bin/python


Sunday, October 2, 2016

Quick peek inside Kubernetes Containers.

Quick peek inside Kubernetes Containers

To get a fresh start inside the Container world I have tested a couple of technologies. The new and interresting container managament project that I tested is called Kubernetes. The Wiki definition follows: Kubernetes (commonly referred to as "k8s") is an open source container cluster manager originally designed by Google and donated to the Cloud Native Computing Foundation. It aims to provide a "platform for automating deployment, scaling, and operations of application containers across clusters of hosts".

On my simple home Lab I will install and quickly deploy Kubernetes on a fresh Centos 7 server with minimal desktop features. So let us get started.

First what we need is to disable firewall services.

 systemctl disable firewalld.service
 systemctl stop firewalld.service

After this what we need to install is the network daemon service:

yum -y install ntp
systemctl start ntpd
systemctl enable ntpd

To download the packages need for the Kubernetes cluster manager we need to add a new repository to the Centos defaults. We do this creating a file inside the /etc/yum.repos.d folder called virt7-docker-common-release.repo

This file should contain the following Urls and content for us to be able to download the Kubernetes packages need for the setup of the container manager:

[root@Cent01 yum.repos.d]# cat virt7-docker-common-release.repo
[virt7-docker-common-release]
name=virt7-docker-common-release
baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/
gpgcheck=0

Now are ready to download the basic Kubernetes managament packages and install them:

yum install docker etcd kubernetes 

As the packages are downloaded from the repository they will be automatically installed using the Yellow dog manager for Centos packages. If you need more info or want to see the verbose informatons add the --v switch after the install command.

To restart and enable the services we would need to define a FOR loop:

for SERVICE in docker etcd kube-apiserver kube-controller-manager kube-scheduler kube-proxy kubelet; do 
    systemctl restart $SERVICE
    systemctl enable $SERVICE
done

To make the managament available over the Web GUI (which I find very handy from the beginning) we would need to install the Cockpit manager.

yum install cockpit cockpit-kubernetes
systemctl enable cockpit.socket
systemctl start cockpit.socket

That was for now on installing the packages. The next step would be to start the web GUI using the port 9090. You just need to call this url: https://server_ip:9090

The base image of the Cockpit Kubernetes manager will show us the options needed to create and monitor the container Also the cluster managament is prepared and deployment of Micro services.


Under Tools we can still maintain the Bare metal Centos 7 server using the command line.


And as simpel as that we can find many Docker container images from the repositories and download them. From simple web servers to complex MySQL redundant scenarios. As simple as download-click-run image system is a performance booster.


What I also find interresant is the possibility to build the apps from Manifest files and deploy them as Micro Services. OpenShift supported from RedHat could be also used as a great tool to build the applications on Docker Container technology.


And in a matter of minutes we can have a running cluster based container image managament orchestra. The deploy of manifests will be explained in the next blog where I will research more on Openshift and app buiding.

Stay tuned and follow me and please send your comments.



Sunday, November 30, 2014

Automate Mysql with Puppet

Automate Mysql with Puppet


Puppet, based on my personal experience is a great technology that allows new ways in DevOps environments and automation. With this technology people can code and manage the complete IT infrastructure from a central location, or a within a cloud infrastructure. It has completely changed the way on providing IT services and managing the same. Some cool features like role based access control and activity login allow people to define stable management strategy.

To get started, I created a simple Client / Server based puppet scenario in which we will automate Mysql server implementation in a linux environment. Puppet master server is configured to provide modules, manifests and classes to the agent servers in the complete infrastructure.

After installing the puppet master server the users can use the following folder etc/puppet/manifests  to develop and write some code for deploying the services to the agent servers.

First, on the puppet master server we should download and install the package module for the Mysql server using the following command:

sudo puppet module search mysql

After the successful installation you should verify that the folder mysql exists in the /etc/puppet/modules folder with the source installation of mysql server.


After the validation of folder existence , the source file are located there. Now we have to define the site.pp file inside the /etc/puppet/manifests as a manifest file that will send the code execution of installing the mysql server on an agent server. The content of the file should look like the following:

node 'agent_node01' { include mysql::server }

This line of code defines the agent_node01 will look for the installation of mysql server inside the mysql server module installation on the puppet master server. The next step is to connect to the agent node and start the puppet agent service.

 sudo /etc/init.d/puppet start

After restarting the service I will initiate the agent test command to force the pulling the package and the module from the puppet master.


As we can see the catalog is finished and the Mysql server service is up and running. The allow_virtual parameter can be ignored as that is a deprecation warning by default.

Feel free to test and comment.


Saturday, September 27, 2014

Make linux process invisible with new Centos kernel

Make linux process invisible with new Centos kernel 


Processes carry out tasks within the operating system. A program is a set of machine code instructions and data stored in an executable image on disk and is, as such, a passive entity; a process can be thought of as a computer program in action.

After I have compiled the new version of Centos 3.2 kernel I have decided to test some security features that this version offers. How to check which kernel version you have installed, well easy:

[root@centos01 ~]# uname -r
3.2.48
[root@centos01 ~]#

As many other Linux servers, they run in a multi-user enviroments. That means that every user are using shared hardware and software resources of the server. From a security stand point of view, informations of user/usage processes ownership is not relevant for every user to see it. To prevent these informations to every shared resource we are going to tamper a little bit with the /proc filesystem. So if you have the Centos 3.2+ kernel compiled and installed on your test or production machine you can develope this situation further.
The task is simple, all we have to configure is the /proc file system mount with new security options, so that reading of every process can be delegated only to the owner of the process. The new option that we are going to introduce is hidepid.

We have three options available:

hidepid=0 - anyone can read the /proc/pid files 
hidepid=1 - this option prevents users to access /proc directories , except of their own. Important 
                    background tasks of the server are now prevented to be shown.
hidepid=2 - this option is an addition to the option 1 , with more security, denying everybody the information about the running processes. Now an intruder is not able to list sensitive data.

Before setting this security options we had a normal situation where a local user could read all of the root and system processes informations.



To continue setting this to prevent users information leakage we have to type further commands:

mount -o remount,rw,hidepid=2 /proc

To have the configuration over a rebooted server we have to update the FSTAB file.

vi /etc/fstab

And we have to add the following info to the file:

proc    /proc    proc    defaults,hidepid=2     0     0

Save and update the file.

This is all to it. Log into a stanard user and use the following command to list the processes:

ps -ef

As a standard user you should not be able to see the processes from other users and applications.

Feel free to test and comment.

Saturday, August 16, 2014

Using tcpdump with Linux

Using tcpdump with Linux


I find tcpdump as a very powerfull and useful tool to sniff network traffic from a Linux box. It is independent on the distro you are working and very easy to learn. It`s simplicity is inside the command line shell and can be very useful for remote troubleshooting of server and desktop systems.


It is a built in package that exists in various distros and can be used to capture received and transfered packets over a complete network or only from a host. There are a set of options and switch flags thah can be used with this command.
I will try to demonstrate a couple of them with explanation, the ones that I find useful:
  • -i any listen on any interface that is available on the system
  • -n do not resolve hostnames
  • -c get number of certain packets and then stop (usefull in not getting to much informations)
  • -e get the Ethernet header along with the capture
  • -E decrypt IPSEC traffic with providing a password key
Simple usage of tcpdump for viewing packets can be done with a couple of command line options. Whether you would like to go into the details of the packets or only the basic view it can be displayed on the command line.

tcpdump -nS simple communication of packets inside the network
tcpdump -nnvvS more advanced packet view with more verbosity
tcpdump -nnvvXS a more deeper look into the package with the content details (derrived from the GUI)

We could do now an example of a displaying only two packages TCP with and inside deeper view of the content. This simple command will show us a two packets with their content and headers.

tcpdump -nnvXSs 0 -c2 tcp



A more specific network goal is not to have too many traffic displayed on the shell. This leaves the options that are not needed out of the picture, and makes troubleshooting much easier.

To see the traffic derrived from a particular host we can use the following command:

tcpdump host 192.168.1.100

Another useful command is writting a certain type of traffic inside a text file for later troubleshooting:

tcpdump -s 1514 port 21 -w output.txt

And for the final packet show command in this blog, you can use a simple switch only to filter IPV6 traffic:

tcpdump ip6

This is only a small demonstration of this powerfull tool. More can be read through the MAN pages, or from similar sysadmin books. Feel free to share and comment.


Friday, May 23, 2014

Very Secure FTP on Centos Server

Very Secure FTP on Centos Server


For this blog post I will be using a VSFTPD as a fast, realible and very secure SFTP server for transferring data between client and server sites. One notice FTP is inherently insecure. If you must use FTP, consider securing your FTP connection with SSL/TLS. Otherwise, it is best to use SFTP, a secure alternative to FTP.

I have a basic built of Centos 6.5 server that is updated with the latest kernel and important security packages for this example. Once we have configured and installed the SFTP server, you will need a SFTP client application to test the connection. I often use Filezilla or WinScp as a alternative application.

Login via ssh to your Linux server and use the su command to become root and start the package installation. At the picture below this is a simple first step.


After this issues a simple command to install the VSFTPD package on your linux server:

yum install vsftpd

After this command issued you should have the package installed and as an example on the next picture I have shown here.


Also for a public server, should be a FTP connection availlable if the client has no possibility to use the SFTP protocol with this software. Then we can install also these packages.

yum install ftp

Now a ftp server should be installed and default configured as an service on your Centos machine.


BASIC VSFTPD CONFIGURATION

The default file for the configuration of this service is located under the /etc/vsftpd folder. I will use Nano editor to change the settings here.

nano /etc/vsftpd/vsftpd.conf

The first item you should configure , is the option to disable anoymous login:

anonymous_enable = NO

Next one is to enable local user logins with the command below:

locale_enable = yes

Next very important configuration to uncomment and set is the chroot option. This option will make a possibility for users, only to use their dedicated home folders on the server, and not able to traverse over different folders. This is a good security practice.

chroot_local_user=YES

So now bassically these are the most command and important settings to get the server up and running. You can fine tune the other settings , like change the default port, or certificates and etc. But this is not needed in this blog demontstration of basic secure server setttings.

We should now restart the service and make it a startup one on boot time of the Centos server. We achieve this with two simple commands:

service vsftpd restart
chkconfig vsftpd on


That is all to it. Now we can test our connection with a SFTP or FTP client.


I have used the Filezilla with a SFTP protocol and I have succesfully connected to my server via a secure channel.


Up and ready for receiving traffic. Now the users can enjoy security, performance and stability in your network.

Feel free to comment and suggest more topics.


Sunday, March 2, 2014

Flushing infected mail traffic from Postfix server

Flushing infected mail traffic from Postfix server


Postfix consists of a combination of server programs that run in the background, and client programs that are invoked by user programs or by system administrators.
The Postfix core consists of several dozen server programs that run in the background, each handling one specific aspect of email delivery. Examples are the SMTP server, the scheduler, the address rewriter, and the local delivery server. For damage-control purposes, most server programs run with fixed reduced privileges, and terminate voluntarily after processing a limited number of requests. To conserve system resources, most server programs terminate when they become idle.
Client programs run outside the Postfix core. They interact with Postfix server programs through mail delivery instructions in the user's ~/.forward file, and through small "gate" programs to submit mail or to request queue status information.
Other programs provide administrative support to start or stop Postfix, query status information, manipulate the queue, or to examine or update its configuration files.



If Postfix cannot deliver a message to a recipient it is placed in the deferred queue.  The queue manager will scan the deferred queue to see it if can place mail back into the active queue.  How often this scan occurs is determined by the queue_run_delay.  Postfix will scan the incoming queue at the same time as the deferred queue just to make sure that one does not take all the resources and so each can continue to move messages.

The real question is, What is causing messages to be deferred?  One of the major reasons that messages are deferred is that your server is going to place mail to “unknown recipients” into the deferred queue if they do not have a legitimate user to go to.

First thing that should be done to analyze the mails that are stuck in the queue is typing the mailq command. If you see a lot of mails in the queue shown in the output, than something fishy is going on on you server. Just looking on the mails, IT people should recognize the domain that has the most mails in the queue. When you find out the domain example.com than the next step to do is to run a bash script that will delete only those mails that are from the infected domain.

#!/bin/bash
match="$1"
find /var/spool/postfix/deferred/*/ -type f -exec grep -l $match '{}' \; | xargs -n1 basename | xargs -n1 postsuper -d
find /var/spool/postfix/active/ -type f -exec grep -l $match '{}' \; | xargs -n1 basename | xargs -n1 postsuper -d

This simple scripts are using bash language to find the deferred and active mails from the user keyboard input on the CLI. After that the script is executing the postsuper -d command that is flushing the queue with that specific domain. 
match="$1" is a simple regex that matched text by the first capturing group, in our case a user inputed domain. 

After this the mail queue should be emptied with the infected mails and the server will have some freed up resources. Another faster or simple solution, if the mails are not important at the moment, is to flush the complete queue in the deferred folder.

For this we have a simple command: postsuper -d ALL deferred

Feel free to comment..

Sunday, December 29, 2013

Check status of IMAP server

Check status of IMAP server


The Internet Message Access Protocol (commonly known as IMAP) is an Application Layer Internet protocol that allows an e-mail client to access e-mail on a remote mail server. The current version, IMAP version 4 revision 1 (IMAP4rev1), is defined by RFC 3501. An IMAP server typically listens on well-known port 143.

I had configured a Dovecot server with IMAP status, for many users , so I needed a mechanism to check if the server is responding on client requests during the high traffic. I wanted to do this using a Cron job and a simple script. This script will telnet to the IMAP port on the Linux server and check the status every 60 seconds. This is how often I configured the Cron job, it can be configured on every 5 minutes or so.


Now let us take a look on this simple code I wrote:

#!/bin/bash
#http://itstuffallaround.blogspot.com/
#program to check if connection is possible with Dovecot and log errors and success full connections

if telnet localhost 143 </dev/null 2>&1 | grep -q Escape; then
  echo "Connected Dovecot on $(date)" >> DOVSTATUS.txt
else
  echo "No connection to Dovecot on $(date)"  >> DOVSTATUS.txt
fi


The simple BASH language script is constructed of a single loop, that telnets to port 143 and returns the status of the service to a dovstatus.txt file. 
The parameter /dev/null 2>&1 was very useful to me, because it will disable returning the on screen prompt for action on telnet, and Escape the login sequence because it is not neccessary, it will redirect both the output and the error streams. Even if your program writes to stderr, that output will not be shown. After this rename the file as .sh and add the execute perrmissions on it. Configure it as a cron job and wait for the results in the dovstatus.txt file.

Feel free to code more!

Wednesday, December 4, 2013

Improved Linux DDOS detection program

Improved Linux DDOS detection program


With a lot of help with some friend on the Linux comunity, I have improved the DDOS detection program on Linux systems. This BASH code gives the IT people possibility to fine list what is currently going on at their servers. And what is more important where from.

The code presented in the following blog is not to be used in loops of any sort becuase it would deny the admin resources to log on.

 #Zeljko Milinovic - http://itstuffallaround.blogspot.com/
 #Cached lookup of ddos whois IP sockets
 #v1.1
 #!/bin/bash
 cachefile="$HOME/ddostestercache"
 # return 0 if address is to be filtered from the processed
 filter()
 {
 case "$1" in
 0.* | 127.* | 10.* | 172.1[6-9].* | 172.2[0-9].* | 172.3[0-1].* | 192.168.* | 169.254.*)
 return 0
 ;;
 esac
 return 1
 }
 remote_ips()
 {
 # print only IPv4 addresses
 netstat -tun4 | awk '/:/ {gsub(/:.*/,"",$5);print $5}' | sort -n | uniq -c
 }
 get_country()
 {
 local country=$(sed -nr "s/^$1 (.*)/\1/p" $cachefile 2>/dev/null)
 if [ -z "$country" ];then
 # some queries produce multiple lines so for now use only the first line..
 country=$(whois "$1" | sed -nr 's/^country:[[:space:]]+(.*)/\1/ip' | head -1)
 country=${country:-unknown}
 # cache search result for future use
 echo "$1 $country" >> $cachefile
 fi
 # let's not print the text "unknown" to screen
 [ "$country" = "unknown" ] && unset country
 echo "$country"
 }
 remote_ips | while read count ip;do
 if ! filter $ip;then
 echo "$count $ip $(get_country $ip)"
 fi
 done


And finally the output for the script:

1 50.31.xxx.xxx US
9 98.28.xxx.xxx DK
1 109.12.xxx.xxx BA

As we can see, in the output we have 3 IP addresses, numbered, listed with concurrent connections , and their country origin. This is very useful to detect where from is the attack, and to mitigate fast.

Feel free to code more and comment.

Saturday, November 30, 2013

Linux Security script to determine DDOS origin location

Linux Security script to determine DDOS origin location


In computing, a denial-of-service attack (DoS attack) or distributed denial-of-service attack (DDoS attack) is an attempt to make a machine or network resource unavailable to its intended users. Although the means to carry out, motives for, and targets of a DoS attack may vary, it generally consists of efforts to temporarily or indefinitely interrupt or suspend services of a host connected to the Internet.

On various Nix server setups we are always exposed to the DDOS attacks from various other similar setups or intended use. Often in some cases our server is used as a Botnet machine to exploit resources on other systems.


These attacks can be verified from the shell in a form of many open sockets from one or more IP addresses. Often these open sockets are more than 150 , which is not normal. Many IT people are using a DDOS prevention scripts to ban those IP addresses. I stumbled upon a request from a friend to write a script that will tell us the Country of attack origin. This was always missing in our troubleshooting. 
So I have written a small and useful script, that is a combination of often used Netstat and Whois commands that can be found online. Also similar script code can be found on the internet, and people can adjust the code to their needs. I needed a script that will associate and display the origin of country and the IP socket combination.


Code


#!/bin/bash

{
cat=$( netstat -ntu  |  grep ':'  |  awk '{print $5}'  |  sed 's/::ffff://'  |  cut -f1 -d ':'  |  sort  |  sort -nr | less);

for i in $cat; do

Country=$( whois $i | grep -i Country | awk '{print $2}' );

echo "Land+IP=  $Country $i ";

done;
}


end of code.


To elaborate more on code I will explain the details. I am using the Bash shell scripting, which is very common. This code is using a Netstat command from the classic and tuned ddos Deflate script that is common for fighting DDOS attacks. 

netstat -ntu  |  grep ':'  |  awk '{print $5}'  |  sed 's/::ffff://'  |  cut -f1 -d ':'  |  sort  |  sort -nr | less

This command gives us the output of IP sockets and we print them out using the AWK for text processing. I have attached the sed switch to replace the empty addresses and space with null ffff value. 
The sort and less switches are helpful for sorting and properly displaying the addresses. I have put this into on CAT function and defined this concencated output with a $ sign as a variable.

The variable is further used for a loop that is needed for the WHOIS command which will tel use the Country of origin. Classic for loop is using a i for the increment value. 

Country=$( whois $i | grep -i Country | awk '{print $2}' );

If we use the whois command with the grep function for the Contry it will only display us the Country of origin. So I have used this command with the incremented concencated display in the loop.

Simple enough we get a display of current IP sockets with Country of origin:

Land+IP=  BA 71.222.xxx.xxx
Land+IP=  BA 71.222.xxx.xxx
Land+IP=  BA 71.222.xxx.xxx
Land+IP=  BA 71.222.xxx.xxx
Land+IP=  IT 88.138.xxx.xxx
Land+IP=  IT 88.138.xxx.xxx
Land+IP=  IT 88.138.xxx.xxx
Land+IP=  IT 88.138.xxx.xxx
Land+IP=  BA 61.38.xxx.xxx
Land+IP=  BA 61.38.xxx.xxx
Land+IP=  BA 61.38.xxx.xxx
Land+IP=  BA 61.38.xxx.xxx
Land+IP=  BA 61.38.xxx.xxx

So this output will generate all the Sockets, and if we see many exact same sockets from one Country we can pinpoint the location and origin from the attack. Script can be more fine tuned so everyone is welcome. 

Feel free to code.


Wednesday, November 13, 2013

Compile Source of Apache/MySQL/PHP on a Linux VPS

Compile Source of Apache/MySQL/PHP on a Linux VPS


Linux IT Engineers often use the Debina APT, or RedHat YUM repositories for an quick and easy install of the services on their servers. But , in some cases we often need to test the latest packages. For example the latest version of MySQL is 5.7 and we cannot get it via the apt, we have to manually download and install in on our Virtual Private Server. Then we can configure it to our production enviroment.
I have configured the VPS server with the 12.04 LTS versions. I prefer the LTS version because of the support for the security and long term update. 

First we start with creating the sources folder and downloading the Apache packages that are needed for our web server:

sudo mkdir /usr/src/sources
wget http://httpd.apache.org/dev/dist/httpd-2.4.2.tar.gz
tar xvfz httpd-2.4.2.tar.gz

After downloading the httpd packages and extracting them we can move on further in our process. We now need the APR utilities and the APR package itself. The APR stands for Apache Portable Runtime.

wget http://apache.spinellicreations.com//apr/apr-1.4.8.tar.gz
tar -xzf apr-1.4.8.tar.gz
rm apr-1.4.8.tar.gz
cd apr-1.4.8/
sudo apt-get install make
sudo ./configure
sudo make
sudo make install

Then we need the APR Utils to be configured.

wget http://mirrors.axint.net/apache//apr/apr-util-1.4.1.tar.gz
tar -xvzf apr-util-1.4.1.tar.gz
cd apr-util-1.4.1
./configure --with-apr=/usr/local/apr
make
make install
cd ..

Now we can return to the HTTPD folder to compile and install the Apache:

cd /usr/local/sources/httpd-2.4.2
./configure --enable-file-cache --enable-cache --enable-disk-cache --enable-mem-cache --enable-deflate --enable-expires --enable-headers --enable-usertrack --enable-ssl --enable-cgi --enable-vhost-alias --enable-rewrite --enable-so --with-apr=/usr/local/apr/
make
make install
cd ..

To startup the Web server we will create a soft link to a startup script and copy the startup script to the init.d folder for startup options.

ln -s /usr/local/apache2/bin/apachectl /usr/bin/apachectl
cp /usr/local/apache2/bin/apachectl /etc/init.d
update-rc.d apachectl defaults

Now we can reboot the server and check if it is running. And we can see that the daemon is running.

root@ubsrv1:~# ps aux | grep httpd
root      1063  0.0  0.2     0:00 /usr/local/apache2/bin/httpd -k start
daemon    1065  0.0  0.2     0:00 /usr/local/apache2/bin/httpd -k start
daemon    1066  0.0  0.2 3   0:00 /usr/local/apache2/bin/httpd -k start


Next what we should do is to continue with the PHP support and installation.

cd /usr/src/sources
wget http://us2.php.net/get/php-5.5.5.tar.gz/from/ar2.php.net/mirror
tar xfvz php-5.5.5.tar.gz
cd  php-5.5.5
./configure --prefix=/var/www/ --with-apxs2=/var/apache2/bin/apxs --with-config-file- path=/var/www/php --with-mysql
make 
make install

The --prefix folder depends on the folder where you installed the apache, you can change this to your needs. And the final step is to install the MySQL server.

groupadd mysql
useradd -r -g mysql mysql
cd /usr/src/sources
wget http://dev.mysql.com/get/Downloads/MySQL-5.6/mysql-5.6.14.tar.gz/from/http://cdn.mysql.com/
tar zxvf mysql-5.6.14.tar.gz
ln -s /usr/src/sources/mysql-5.6.14 /usr/local/mysql
cd /usr/local/mysql

chown -R mysql .
chgrp -R mysql .
scripts/mysql_install_db --user=mysql
chown -R root .
chown -R mysql data


Here we have a simply longer procedure. First we create  a user and group. Then download the source files, extract them, create a soft link and run the mysql_install_db script. Add the permissions to the folders and that is all.

Now we can restart the server and everything should work fine. If not check out some cool tutorials on Ubuntu community.


Monday, November 11, 2013

Linux SWAP Partition as twice the RAM size - why ?

Linux SWAP Partition as twice the RAM size - why ?


Linux divides its physical RAM (random access memory) into chucks of memory called pages. Swapping is the process whereby a page of memory is copied to the preconfigured space on the hard disk, called swap space, to free up that page of memory. The combined sizes of the physical memory and the swap space is the amount of virtual memory available.

Swapping is necessary for two important reasons. First, when the system requires more memory than is physically available, the kernel swaps out less used pages and gives memory to the current application (process) that needs the memory immediately. Second, a significant number of the pages used by an application during its startup phase may only be used for initialization and then never used again. The system can swap out those pages and free the memory for other applications or even for the disk cache.

To see a VPS machine with 512 MB of physical RAM and the ratio of SWAP space we can use free -m


The picture shows us that we have total of 490 MB of physical ram and twice the size of the SWAP memory on the hdd (which is not yet used) of 991 MB. As this server has small amount of free memory (only 76MB) I had to investigate further. I have used the HTOP utility to see the real memory consumer.


The process with the ID of 1310 is using some memory resources dynamically. We cannot see the real proccess name, because it is assigned to multiple instances of one application. To investigate further we should use the PID number to see which service is killing the VPS machine.

This can be done with the PMP command :  pmap -x 1310


The output shows us that the SAMBA libraries are attached to the PID. Simply stopping the SAMBA service I will free up some memory.


To get back on the inital question , I will try to have a short explanation of the SWAP size. The memory hirearchy presented to application by the Linux system is arranged in few levels:

  • Processor/CPU registers - bits in size
  • L1 Cache - kbits in size
  • L2 Cache - MBs in size
  • L3 cache - 100 of MBs in size
  • RAM - GBs in size
The data that is used in application loading is moved to RAM, and some of that data need the L3 cache also. The used application data is often moved to the faster L2 and L1 cache memory. As we can see the data is moved from the RAM up to the last CPU register table. In this order we can get that the SWAP data should be between 1.5 and 2 times the Actual RAM is. 
This is the main reason why we should create the SWAP twice the RAM. And applications should not allocate that data to other HDDs, especially large ones, because of the slow I/O operations. If your RAM is free then there is no use of swap partition.

Feel free to comment.