Uncomplicated Firewall with Python

Uncomplicated Firewall with Python

Uncomplicated Firewall, is an interface to iptables that simplifyies the process of configuring a firewall. Iptables is flexible, it can be difficult for beginners to learn how to use it to properly configure a firewall. 

In the example below is a python program that makes it easy allowing and blocking various services by IP address. 

## Description :

## Generate ip-host binding list for a list of nodes, when internal DNS is missing.

## 1. For existing nodes, allow traffic from new nodes

## 2. For new nodes, allow traffic from all nodes

##

## Sample:

## python ./ufw_allow_ip.py --old_ip_list_file /tmp/old_ip_list --new_ip_list_file /tmp/new_ip_list \

## --ssh_username root --ssh_port 22 --ssh_key_file ~/.ssh/id_rsa

##

##-------------------------------------------------------------------

import os, sys

import paramiko

import argparse

# multiple threading for a list of ssh servers

import Queue

import threading

import logging

log_folder = "%s/log" % (os.path.expanduser('~'))

if os.path.exists(log_folder) is False:

os.makedirs(log_folder)

log_file = "%s/%s.log" % (log_folder, os.path.basename(__file__).rstrip('\.py'))

logging.basicConfig(filename=log_file, level=logging.DEBUG, format='%(asctime)s %(message)s')

logging.getLogger().addHandler(logging.StreamHandler())

def get_list_from_file(fname):

l = []

with open(fname,'r') as f:

for row in f:

row = row.strip()

if row.startswith('#') or row == '':

continue

l.append(row)

return l

def ufw_allow_ip_list(server_ip, ip_list, ssh_connect_args):

if len(ip_list) == 0:

print("Skip run ufw update in %s, since ip_list is empty" % (server_ip))

return("OK", "")

[ssh_username, ssh_port, ssh_key_file, key_passphrase] = ssh_connect_args

ssh_command = ""

# TODO: improve this command, by using a library

for ip in ip_list:

ssh_command = "%s && ufw allow from %s" % (ssh_command, ip)

if ssh_command.startswith(" && "):

ssh_command = ssh_command[len(" && "):]

print("Update ufw in %s. ssh_command: %s" % (server_ip, ssh_command))

output = ""

try:

ssh = paramiko.SSHClient()

ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

key = paramiko.RSAKey.from_private_key_file(ssh_key_file, password=key_passphrase)

ssh.connect(server_ip, username=ssh_username, port=ssh_port, pkey=key)

stdin, stdout, stderr = ssh.exec_command(ssh_command)

output = "\n".join(stdout.readlines())

output = output.rstrip("\n")

print("Command output in %s: %s" % (server_ip, output))

ssh.close()

except:

return ("ERROR", "Unexpected on server: %s error: %s\n" % (server_ip, sys.exc_info()[0]))

return ("OK", output)

###############################################################

if __name__ == '__main__':

# get parameters from users

parser = argparse.ArgumentParser()

parser.add_argument('--old_ip_list_file', required=True, \

help="IP list of current cluster", type=str)

parser.add_argument('--new_ip_list_file', required=True, \

help="IP list of new nodes", type=str)

parser.add_argument('--ssh_username', required=False, default="root", \

help="Which OS user to ssh", type=str)

parser.add_argument('--ssh_port', required=False, default="22", \

help="Which port to connect sshd", type=int)

parser.add_argument('--ssh_key_file', required=False, default="%s/.ssh/id_rsa" % os.path.expanduser('~'), \

help="ssh key file to connect", type=str)

parser.add_argument('--key_passphrase', required=False, default="", \

help="Which OS user to ssh", type=str)

l = parser.parse_args()

ssh_connect_args = [l.ssh_username, l.ssh_port, l.ssh_key_file, l.key_passphrase]

old_ip_list = get_list_from_file(l.old_ip_list_file)

new_ip_list = get_list_from_file(l.new_ip_list_file)

has_error = False

# TODO: speed up this process by multiple threading

for old_ip in old_ip_list:

(status, output) = ufw_allow_ip_list(old_ip, new_ip_list, ssh_connect_args)

if status != "OK":

has_error = True

print("Error in %s. errmsg: %s" % (old_ip, output))

for new_ip in new_ip_list:

(status, output) = ufw_allow_ip_list(new_ip, new_ip_list + old_ip_list, ssh_connect_args)

if status != "OK":

has_error = True

print("Error in %s. errmsg: %s" % (new_ip, output))

if has_error is True:

sys.exit(1)

#!/usr/bin/python

Benefits of Blockchain

Benefits of Blockchain

Blockchain’s promise is “the decentralization of trust, enabling value flow without intermediaries”. It allows financial transactions to be verified and cleared without the need for a trusted third party sitting between market participants. Removing intermediaries reduces costs and complexity.

The blockchain’s security and privacy protocols are based on the use of a “cryptographic hash function” — each block (of transactions) in the chain is identified by its own “hash” key. This approach was developed to prevent the “double spending” of  Bitcoins. The complexity of the crypto hash function reduces the blockchain’s susceptibility to fraud.

The distributed ledger approach means that all the members of a financial market (the network) share an identical system of record, rather than each maintaining their own proprietary view of it. This replicated, shared ledger provides consensus, provenance, immutability and finality for the transactions concerned — payments, asset transfers, etc. This shared approach removes the need for reconciliations.

New transactions are only accepted for posting to the distributed ledger (through the creation of new blocks for the chain) once all the computers in the network achieve consensus as to their validity. The

verification of transactions by all network users reduces error rates and queries.

At the heart of blockchain is a new type of distributed database. This provides for the exchange of information in a synchronous and even manner, as well as allows it to be updated constantly, providing near-instant clearing and settlement. The provision of faster settlement means less risk in the financial system and so reduces the capital requirements of market participants.

The new distributed database functionality also allows code to run with the blockchain to modify data (both on and off the chain) automatically. This enables the blockchain to support self-enforcing or “smart” contracts, allowing the automation of a variety of business functions.

Supply chain and trade finance

• Plays directly to many of the blockchain’s principal strengths

• Mathematics is used to achieve trust between the parties to a transaction

- The current role of banks in trade transactionsis principally to accept risk, thereby allowing the

   parties to trust each other.

• Allows shared access for all parties within the supply chain (buyers, suppliers, banks, logistics companies, insurance companies, customs and health authorities, etc.) to the single view of the truth

• A private, permissioned blockchain can restrict access only to those parties involved in the supply

chain concerned • The delivery and use of rich information — i.e., transactions on the chain would include invoice numbers, certificates of origin, bills of lading, bills of exchange, insurance documents, customs documents, health certificates, etc.

• “Smart contract” capabilities enable automated decision making and information handling.

In summary, the focus on blockchain has shifted and is now on understanding the technology and the value it offers, as opposed to some months ago when it was more of a collective huddle and a focus on collaboration and shared intent. However, there is still no clear emergence of where blockchain its in terms of technology, business benefits and application. “We don’t know who will own it and who will benefit so maybe the wrong group within a bank is looking at the benefit?” said one interviewee.

So where next? The banks did not plan for blockchain, but the direction is clear. Few people fully understand blockchain within banks, but everyone is asking powerful questions as to what it is and what it offers. It is not seen as a lethal threat to banking, but it is seen as disruptive new technology they have to understand more fully, and for which they must develop a strategic approach to deployment. As one interviewee said: “I actually don’t understand how we have reached such a stage of involvement in the bank without ownership or control. We need a blockchain czar.”

What is absolutely clear is the recognition that there is a need to separate the hype from the reality and from an executive and strategic perspective, to take a fresh look at what blockchain offers banks, and what banks must do to adopt it.

Source: ACI Worldwide

Application Centric Infrastructure with Python

In application-centric networking, troubleshooting does not mean logging into discrete devices and examining networking state information. If a web application is not performing, you start with the web application. The fact that relevant state information might exist within a router, switch, or even a web server is secondary. You want to gather intelligence on the application itself. That data needs to be collected and correlated. And you aren’t done until the application is up and running as it should.

There are several questions you need to ask: Can you see how applications are talking across your network? For distributed applications, do you have a view of which components are where? Can you see how data is flowing between them?

Here I would like to present a small Python programm that I adapted from the Cisco code repository. 

# Copyright (c) 2015 Cisco Systems #

# All Rights Reserved. 

"""

Simple application that logs on to the APIC and displays all

of the Interfaces.

"""

import sys

import re

import json

import acitoolkit.acitoolkit as aci

def main():

    """

    Main execution routine

    :return: None

    """

    # Take login credentials from the command line if provided

    # Otherwise, take them from your environment variables file ~/.profile

    description = 'Simple application that logs on to the APIC and displays all of the Interfaces.'

    creds = aci.Credentials('apic', description)

    args = creds.get()

    # Login to APIC

    session = aci.Session(args.url, args.login, args.password)

    resp = session.login()

    if not resp.ok:

        print('%% Could not login to APIC')

        sys.exit(0)

    resp = session.get('/api/class/ipv4Addr.json')

    intfs = json.loads(resp.text)['imdata']

    data = {}

    for i in intfs:

        ip = i['ipv4Addr']['attributes']['addr']

        op = i['ipv4Addr']['attributes']['operSt']

        cfg = i['ipv4Addr']['attributes']['operStQual']

        dn = i['ipv4Addr']['attributes']['dn']

        node = dn.split('/')[2]

        intf = re.split(r'\[|\]', dn)[1]

        vrf = re.split(r'/|dom-', dn)[7]

        if vrf not in data.keys():

            data[vrf] = []

        else:

            data[vrf].append((node, intf, ip, cfg, op))

    for k in data.keys():

        header = 'IP Interface Status for VRF "{}"'.format(k)

        print header

        template = "{0:15} {1:10} {2:20} {3:8} {4:10}"

        print(template.format("Node", "Interface", "IP Address ", "Admin Status", "Status"))

        for rec in sorted(data[k]):

            print(template.format(*rec))

if __name__ == '__main__':

main()

With the help of the ACI Toolkit library from Cisco we could define and create many functions and procedures needed to manipulate and programm the ACI Fabric. As an universal data format JSON is used to retrieve data from the APIC controller. The data interested to us are VRFs, IPs, VLANs and so on. In the final function we define a loop that that will search for all of the the values stored inside the dictionary data structure from Python and sort them using a template.

For those wanted to know more about the ACI please checkout the source text from the SDX Central.

Cisco ACI is a tightly coupled policy-driven solution that integrates software and hardware. The hardware for Cisco ACI is based on the Cisco Nexus 9000 family of switches. The software and integration points for ACI include a few components, including Additional Data Center Pod, Data Center Policy Engine, and Non-Directly Attached Virtual and Physical Leaf Switches. While there isn’t an explicit reliance on any specific virtual switch, at this point, policies can only be pushed down to the virtual switches if Cisco’s Application Virtual Switch (AVS) is used, though there has been talk about extending this to Open vSwitch in the near future.

In a leaf-spine ACI fabric, Cisco is provisioning a native Layer 3 IP fabric that supports equal-cost multi-path (ECMP) routing between any two endpoints in the network, but uses overlay protocols, such as virtual extensible local area network (VXLAN) under the covers to allow any workload to exist anywhere in the network. Supporting overlay protocols is what will give the fabric the ability to have machines, either physical or virtual, in the same logical network (Layer 2 domain), even while running Layer 3 routing down to the top of each rack. Cisco ACI supports VLAN, VXLAN, and network virtualization using generic routing encapsulation (NV-GRE), which can be combined and bridged together to create a logical network/domain as needed.

From a management perspective, the central SDN Controller of the ACI solution, the Application Policy Infrastructure Controller (APIC) manages and configures the policy on each of the switches in the ACI fabric. Hardware becomes stateless with Cisco ACI, much like it is with Cisco’s UCS Computing Platform. This means no configuration is tied to the device. The APIC acts as a central repository for all policies and has the ability to rapidly deploy and re-deploy hardware, as needed, by using this stateless computing model.

Feel free to comment.

Quick peek inside Kubernetes Containers.

Quick peek inside Kubernetes Containers

To get a fresh start inside the Container world I have tested a couple of technologies. The new and interresting container managament project that I tested is called Kubernetes. The Wiki definition follows: Kubernetes (commonly referred to as "k8s") is an open source container cluster manager originally designed by Google and donated to the Cloud Native Computing Foundation. It aims to provide a "platform for automating deployment, scaling, and operations of application containers across clusters of hosts".

On my simple home Lab I will install and quickly deploy Kubernetes on a fresh Centos 7 server with minimal desktop features. So let us get started.

First what we need is to disable firewall services.

 systemctl disable firewalld.service

 systemctl stop firewalld.service

After this what we need to install is the network daemon service:

yum -y install ntp

systemctl start ntpd

systemctl enable ntpd

To download the packages need for the Kubernetes cluster manager we need to add a new repository to the Centos defaults. We do this creating a file inside the /etc/yum.repos.d folder called virt7-docker-common-release.repo

This file should contain the following Urls and content for us to be able to download the Kubernetes packages need for the setup of the container manager:

[root@Cent01 yum.repos.d]# cat virt7-docker-common-release.repo

[virt7-docker-common-release]

name=virt7-docker-common-release

baseurl=http://cbs.centos.org/repos/virt7-docker-common-release/x86_64/os/

gpgcheck=0

Now are ready to download the basic Kubernetes managament packages and install them:

yum install docker etcd kubernetes 

As the packages are downloaded from the repository they will be automatically installed using the Yellow dog manager for Centos packages. If you need more info or want to see the verbose informatons add the --v switch after the install command.

To restart and enable the services we would need to define a FOR loop:

for SERVICE in docker etcd kube-apiserver kube-controller-manager kube-scheduler kube-proxy kubelet; do 

    systemctl restart $SERVICE

    systemctl enable $SERVICE

done

To make the managament available over the Web GUI (which I find very handy from the beginning) we would need to install the Cockpit manager.

yum install cockpit cockpit-kubernetes

systemctl enable cockpit.socket

systemctl start cockpit.socket

That was for now on installing the packages. The next step would be to start the web GUI using the port 9090. You just need to call this url: https://server_ip:9090

The base image of the Cockpit Kubernetes manager will show us the options needed to create and monitor the container Also the cluster managament is prepared and deployment of Micro services.

Under Tools we can still maintain the Bare metal Centos 7 server using the command line.

And as simpel as that we can find many Docker container images from the repositories and download them. From simple web servers to complex MySQL redundant scenarios. As simple as download-click-run image system is a performance booster.

What I also find interresant is the possibility to build the apps from Manifest files and deploy them as Micro Services. OpenShift supported from RedHat could be also used as a great tool to build the applications on Docker Container technology.

And in a matter of minutes we can have a running cluster based container image managament orchestra. The deploy of manifests will be explained in the next blog where I will research more on Openshift and app buiding.

Stay tuned and follow me and please send your comments.

Automate Mysql with Puppet

Automate Mysql with Puppet

Puppet, based on my personal experience is a great technology that allows new ways in DevOps environments and automation. With this technology people can code and manage the complete IT infrastructure from a central location, or a within a cloud infrastructure. It has completely changed the way on providing IT services and managing the same. Some cool features like role based access control and activity login allow people to define stable management strategy.

To get started, I created a simple Client / Server based puppet scenario in which we will automate Mysql server implementation in a linux environment. Puppet master server is configured to provide modules, manifests and classes to the agent servers in the complete infrastructure.

After installing the puppet master server the users can use the following folder etc/puppet/manifests  to develop and write some code for deploying the services to the agent servers.

First, on the puppet master server we should download and install the package module for the Mysql server using the following command:

sudo puppet module search mysql

After the successful installation you should verify that the folder mysql exists in the /etc/puppet/modules folder with the source installation of mysql server.

After the validation of folder existence , the source file are located there. Now we have to define the site.pp file inside the /etc/puppet/manifests as a manifest file that will send the code execution of installing the mysql server on an agent server. The content of the file should look like the following:

node 'agent_node01' { include mysql::server }

This line of code defines the agent_node01 will look for the installation of mysql server inside the mysql server module installation on the puppet master server. The next step is to connect to the agent node and start the puppet agent service.

 sudo /etc/init.d/puppet start

After restarting the service I will initiate the agent test command to force the pulling the package and the module from the puppet master.

As we can see the catalog is finished and the Mysql server service is up and running. The allow_virtual parameter can be ignored as that is a deprecation warning by default.

Feel free to test and comment.

Make linux process invisible with new Centos kernel

Make linux process invisible with new Centos kernel 

Processes carry out tasks within the operating system. A program is a set of machine code instructions and data stored in an executable image on disk and is, as such, a passive entity; a process can be thought of as a computer program in action.

After I have compiled the new version of Centos 3.2 kernel I have decided to test some security features that this version offers. How to check which kernel version you have installed, well easy:

[root@centos01 ~]# uname -r

3.2.48

[root@centos01 ~]#

As many other Linux servers, they run in a multi-user enviroments. That means that every user are using shared hardware and software resources of the server. From a security stand point of view, informations of user/usage processes ownership is not relevant for every user to see it. To prevent these informations to every shared resource we are going to tamper a little bit with the /proc filesystem. So if you have the Centos 3.2+ kernel compiled and installed on your test or production machine you can develope this situation further.

The task is simple, all we have to configure is the /proc file system mount with new security options, so that reading of every process can be delegated only to the owner of the process. The new option that we are going to introduce is hidepid.

We have three options available:

hidepid=0 - anyone can read the /proc/pid files 

hidepid=1 - this option prevents users to access /proc directories , except of their own. Important 

                    background tasks of the server are now prevented to be shown.

hidepid=2 - this option is an addition to the option 1 , with more security, denying everybody the information about the running processes. Now an intruder is not able to list sensitive data.

Before setting this security options we had a normal situation where a local user could read all of the root and system processes informations.

To continue setting this to prevent users information leakage we have to type further commands:

mount -o remount,rw,hidepid=2 /proc

To have the configuration over a rebooted server we have to update the FSTAB file.

vi /etc/fstab

And we have to add the following info to the file:

proc    /proc    proc    defaults,hidepid=2     0     0

Save and update the file.

This is all to it. Log into a stanard user and use the following command to list the processes:

ps -ef

As a standard user you should not be able to see the processes from other users and applications.

Feel free to test and comment.

Using tcpdump with Linux

Using tcpdump with Linux

I find tcpdump as a very powerfull and useful tool to sniff network traffic from a Linux box. It is independent on the distro you are working and very easy to learn. It`s simplicity is inside the command line shell and can be very useful for remote troubleshooting of server and desktop systems.

It is a built in package that exists in various distros and can be used to capture received and transfered packets over a complete network or only from a host. There are a set of options and switch flags thah can be used with this command.

I will try to demonstrate a couple of them with explanation, the ones that I find useful:

• -i any listen on any interface that is available on the system
• -n do not resolve hostnames
• -c get number of certain packets and then stop (usefull in not getting to much informations)
• -e get the Ethernet header along with the capture
• -E decrypt IPSEC traffic with providing a password key

Simple usage of tcpdump for viewing packets can be done with a couple of command line options. Whether you would like to go into the details of the packets or only the basic view it can be displayed on the command line.

tcpdump -nS simple communication of packets inside the network

tcpdump -nnvvS more advanced packet view with more verbosity

tcpdump -nnvvXS a more deeper look into the package with the content details (derrived from the GUI)

We could do now an example of a displaying only two packages TCP with and inside deeper view of the content. This simple command will show us a two packets with their content and headers.

tcpdump -nnvXSs 0 -c2 tcp

A more specific network goal is not to have too many traffic displayed on the shell. This leaves the options that are not needed out of the picture, and makes troubleshooting much easier.

To see the traffic derrived from a particular host we can use the following command:

tcpdump host 192.168.1.100

Another useful command is writting a certain type of traffic inside a text file for later troubleshooting:

tcpdump -s 1514 port 21 -w output.txt

And for the final packet show command in this blog, you can use a simple switch only to filter IPV6 traffic:

tcpdump ip6

This is only a small demonstration of this powerfull tool. More can be read through the MAN pages, or from similar sysadmin books. Feel free to share and comment.