Oracle Exadata Smart Scan conditions

Exadata Smart Scan gives a possiblity to the Oracle Database to offload SQL processing to the Exadata Storage Servers. In order to make usage of Smart Scan features of Exadata, the initialization parameter cell_offload_processing must be set to TRUE.

SQL> show parameter cell_offload_pr

cell_offload_processing boolean TRUE

The Smart Scan magic SQL processing happens at the storage tier, instead of the database tier, to improve query performance. It reduces the volume of data sent to the database tier thereby reducing the CPU usage on database nodes.

source: oracle.com

We have 4 basic conditions for Smart Scan to be triggered

- There must be a full scan of an object.

- Oracle’s Direct Path Read mechanis must be used, and path reads are generally used when we read   

  into PGA memory (not the buffer cache).

- Each segment being scanned must be on a Cell disk group (usually with ASSM it is).

- We need the CELL_OFFLOAD_PROCESSING initialization parameter to be enabled.

If we get the STORAGE keyword in the explanation plan, does not guarantee that the offload will occur. Sometimes Exadata Storage Server could decide that Smart Scan will not produce any performance.

 Detect slow PostgreSQL queries, pg_stat vs auto logging overhead

To be able to find and detect slow queries on a RDBMS PostgreSQL is an important thing as the database tends to grow. Optimizing performance and expensive SQL queries is one of the major task of the database system maintainers. These problems can be approached with many ways and external systems, but I have tried to obtain the fast and “keep it simple” way, which I will try to explain in this blog post.

source: stackify

One of the cool PostgreSQL features is the auto_explain module , which provides a means for logging execution plans of slow statements automatically, without having to run EXPLAIN by hand (stated by the documentation). It is also stated in the documentation that there is a price in overhead for that. I would suggest not using the excessive logging from auto_explain on production systems.

The alternative that I am using often is the pg_stat_statements module which can be easily configured on a production or test system.

After that we need to run CREATE EXTENSION pg_stat_statements in our databases, which will allow PostgreSQL to create a view for us.

SELECT * FROM pg_stat_statement

The created view is able to tell us, which query has been executed how often and show us insights about the total runtime of this type of query as well as about the distribution of runtimes for those particular queries.

This allows me to get a quick overview of the CPU percentage being used and a quick overview of the I/O behavior of many types of queries, which could be a great deal of the reason that causes high loads on your production systems.

Oracle Blockchain contributions

Oracle Blockchain Contributions

Oracle has added developer-oriented productivity enhancements, enhanced privacy, confidentiality, and identity management features that are critical to diverse organizations conducting business transactions. New DevOps capabilities make the platform easier to integrate with existing business and IT systems. Additionally, as blockchain becomes an important data store in the enterprise, the platform enables Oracle Autonomous Data Warehouse customers to transparently capture blockchain transaction history and current state data for analytics and to integrate it with other data sources

source: Oracle Blog

I found one of the new cool features to be much welcome in the portfolio:

- Rich history database shadows transaction history into a relational database schema in the Autonomous Data Warehouse or other Oracle databases, which transparently enables analytics integration for interactive dashboards and reports. Here is an example of the use of rich history stored on Oracle's Autonomous Datawarehouse and visualized using Oracle Analytics Cloud.   

One more important thing is the introduction of the Third-party certificate support for registering client organizations on the blockchain network to enable them to use existing certificates issued by trusted third parties. The security was a major topic in the Enterprise world, so did the Oracle what other big developing companies do, introduce the standard methods of security.

To avoid the Lock-In seen in the previous licensing models, the big companies need to contribute to the development of Blockchain services with the same speed as other smaller communities do. This is the only way to hold the sustainabillity of the Blockchain Ecosystem.

Server side request forgery (SSRF) attack

Server side request forgery (SSRF) attack

In a Server-Side Request Forgery (SSRF) attack, the attacker can abuse functionality on the server to read or update internal resources. The attacker can supply or a modify a URL which the code running on the server will read or submit data to, and by carefully selecting the URLs, the attacker may be able to read server configuration such as AWS metadata, connect to internal services like http enabled databases or perform post requests towards internal services which are not intended to be exposed.

source: Acunetix

You can no longer request information from internal systems, but can still make internal API-calls. Imagine the following PHP code (source Detectify):

//getimage.php

$content = file_get_contents($_GET['url']);

file_put_contents(‘image.jpg’, $content);

The above code will fetch data from a URL using PHP’s file_get_contents() function and then save it to the disk. A legitime request would then look like:

GET  /getimage.php?url=https://website.com/images/cat.jpg

And the web application would make a request to https://website.com/images/cat.jpg. This code could be exploited with SSRF. Such an attack could look something like:

GET  /getimage.php?url=http://127.0.0.1/api/v1/getuser/id/1

In this case the vulnerable web application would make a GET request to the internal REST API service, trying to access the /api/v1/getuser/id/1 endpoint. This REST API service is only accessible on the local network, but due to a SSRF vulnerability it was possible for the attacker to make such an internal request and read that response.   Sometimes you can make a request to an external server, and the request itself may contain sensitive headers. One of many examples would be HTTP basic passwords, if a proxy has been used. SSRF can therefore be carried out to both internal and external services.

Due to microservices and serverless platforms, SSRF will probably be a bigger thing in the future. Making internal requests now means that you can interact with other parts of the service, pretending to be the actual service.

Mitigation

The most robust way to avoid Server Side Request Forgery (SSRF) is to whitelist the DNS name or IP address that your application needs to access. If a whitelist approach does not suit you and you must rely on a blacklist, it’s important to validate user input properly. For example, do not allow requests to private (non-routable) IP addresses.

To prevent response data leaking to the attacker, you must ensure that the received response is as expected. Under no circumstances should the raw response body from the request sent by the server be delivered to the client.

If your application only uses HTTP or HTTPS to make requests, allow only these URL schemas. If you disable unused URL schemas, the attacker will be unable to use the web application to make requests using potentially dangerous schemas such as file:///, dict://, ftp:// and gopher://

Uncomplicated Firewall with Python

Uncomplicated Firewall with Python

Uncomplicated Firewall, is an interface to iptables that simplifyies the process of configuring a firewall. Iptables is flexible, it can be difficult for beginners to learn how to use it to properly configure a firewall. 

In the example below is a python program that makes it easy allowing and blocking various services by IP address. 

## Description :

## Generate ip-host binding list for a list of nodes, when internal DNS is missing.

## 1. For existing nodes, allow traffic from new nodes

## 2. For new nodes, allow traffic from all nodes

##

## Sample:

## python ./ufw_allow_ip.py --old_ip_list_file /tmp/old_ip_list --new_ip_list_file /tmp/new_ip_list \

## --ssh_username root --ssh_port 22 --ssh_key_file ~/.ssh/id_rsa

##

##-------------------------------------------------------------------

import os, sys

import paramiko

import argparse

# multiple threading for a list of ssh servers

import Queue

import threading

import logging

log_folder = "%s/log" % (os.path.expanduser('~'))

if os.path.exists(log_folder) is False:

os.makedirs(log_folder)

log_file = "%s/%s.log" % (log_folder, os.path.basename(__file__).rstrip('\.py'))

logging.basicConfig(filename=log_file, level=logging.DEBUG, format='%(asctime)s %(message)s')

logging.getLogger().addHandler(logging.StreamHandler())

def get_list_from_file(fname):

l = []

with open(fname,'r') as f:

for row in f:

row = row.strip()

if row.startswith('#') or row == '':

continue

l.append(row)

return l

def ufw_allow_ip_list(server_ip, ip_list, ssh_connect_args):

if len(ip_list) == 0:

print("Skip run ufw update in %s, since ip_list is empty" % (server_ip))

return("OK", "")

[ssh_username, ssh_port, ssh_key_file, key_passphrase] = ssh_connect_args

ssh_command = ""

# TODO: improve this command, by using a library

for ip in ip_list:

ssh_command = "%s && ufw allow from %s" % (ssh_command, ip)

if ssh_command.startswith(" && "):

ssh_command = ssh_command[len(" && "):]

print("Update ufw in %s. ssh_command: %s" % (server_ip, ssh_command))

output = ""

try:

ssh = paramiko.SSHClient()

ssh.set_missing_host_key_policy(paramiko.AutoAddPolicy())

key = paramiko.RSAKey.from_private_key_file(ssh_key_file, password=key_passphrase)

ssh.connect(server_ip, username=ssh_username, port=ssh_port, pkey=key)

stdin, stdout, stderr = ssh.exec_command(ssh_command)

output = "\n".join(stdout.readlines())

output = output.rstrip("\n")

print("Command output in %s: %s" % (server_ip, output))

ssh.close()

except:

return ("ERROR", "Unexpected on server: %s error: %s\n" % (server_ip, sys.exc_info()[0]))

return ("OK", output)

###############################################################

if __name__ == '__main__':

# get parameters from users

parser = argparse.ArgumentParser()

parser.add_argument('--old_ip_list_file', required=True, \

help="IP list of current cluster", type=str)

parser.add_argument('--new_ip_list_file', required=True, \

help="IP list of new nodes", type=str)

parser.add_argument('--ssh_username', required=False, default="root", \

help="Which OS user to ssh", type=str)

parser.add_argument('--ssh_port', required=False, default="22", \

help="Which port to connect sshd", type=int)

parser.add_argument('--ssh_key_file', required=False, default="%s/.ssh/id_rsa" % os.path.expanduser('~'), \

help="ssh key file to connect", type=str)

parser.add_argument('--key_passphrase', required=False, default="", \

help="Which OS user to ssh", type=str)

l = parser.parse_args()

ssh_connect_args = [l.ssh_username, l.ssh_port, l.ssh_key_file, l.key_passphrase]

old_ip_list = get_list_from_file(l.old_ip_list_file)

new_ip_list = get_list_from_file(l.new_ip_list_file)

has_error = False

# TODO: speed up this process by multiple threading

for old_ip in old_ip_list:

(status, output) = ufw_allow_ip_list(old_ip, new_ip_list, ssh_connect_args)

if status != "OK":

has_error = True

print("Error in %s. errmsg: %s" % (old_ip, output))

for new_ip in new_ip_list:

(status, output) = ufw_allow_ip_list(new_ip, new_ip_list + old_ip_list, ssh_connect_args)

if status != "OK":

has_error = True

print("Error in %s. errmsg: %s" % (new_ip, output))

if has_error is True:

sys.exit(1)

#!/usr/bin/python

Benefits of Blockchain

Benefits of Blockchain

Blockchain’s promise is “the decentralization of trust, enabling value flow without intermediaries”. It allows financial transactions to be verified and cleared without the need for a trusted third party sitting between market participants. Removing intermediaries reduces costs and complexity.

The blockchain’s security and privacy protocols are based on the use of a “cryptographic hash function” — each block (of transactions) in the chain is identified by its own “hash” key. This approach was developed to prevent the “double spending” of  Bitcoins. The complexity of the crypto hash function reduces the blockchain’s susceptibility to fraud.

The distributed ledger approach means that all the members of a financial market (the network) share an identical system of record, rather than each maintaining their own proprietary view of it. This replicated, shared ledger provides consensus, provenance, immutability and finality for the transactions concerned — payments, asset transfers, etc. This shared approach removes the need for reconciliations.

New transactions are only accepted for posting to the distributed ledger (through the creation of new blocks for the chain) once all the computers in the network achieve consensus as to their validity. The

verification of transactions by all network users reduces error rates and queries.

At the heart of blockchain is a new type of distributed database. This provides for the exchange of information in a synchronous and even manner, as well as allows it to be updated constantly, providing near-instant clearing and settlement. The provision of faster settlement means less risk in the financial system and so reduces the capital requirements of market participants.

The new distributed database functionality also allows code to run with the blockchain to modify data (both on and off the chain) automatically. This enables the blockchain to support self-enforcing or “smart” contracts, allowing the automation of a variety of business functions.

Supply chain and trade finance

• Plays directly to many of the blockchain’s principal strengths

• Mathematics is used to achieve trust between the parties to a transaction

- The current role of banks in trade transactionsis principally to accept risk, thereby allowing the

   parties to trust each other.

• Allows shared access for all parties within the supply chain (buyers, suppliers, banks, logistics companies, insurance companies, customs and health authorities, etc.) to the single view of the truth

• A private, permissioned blockchain can restrict access only to those parties involved in the supply

chain concerned • The delivery and use of rich information — i.e., transactions on the chain would include invoice numbers, certificates of origin, bills of lading, bills of exchange, insurance documents, customs documents, health certificates, etc.

• “Smart contract” capabilities enable automated decision making and information handling.

In summary, the focus on blockchain has shifted and is now on understanding the technology and the value it offers, as opposed to some months ago when it was more of a collective huddle and a focus on collaboration and shared intent. However, there is still no clear emergence of where blockchain its in terms of technology, business benefits and application. “We don’t know who will own it and who will benefit so maybe the wrong group within a bank is looking at the benefit?” said one interviewee.

So where next? The banks did not plan for blockchain, but the direction is clear. Few people fully understand blockchain within banks, but everyone is asking powerful questions as to what it is and what it offers. It is not seen as a lethal threat to banking, but it is seen as disruptive new technology they have to understand more fully, and for which they must develop a strategic approach to deployment. As one interviewee said: “I actually don’t understand how we have reached such a stage of involvement in the bank without ownership or control. We need a blockchain czar.”

What is absolutely clear is the recognition that there is a need to separate the hype from the reality and from an executive and strategic perspective, to take a fresh look at what blockchain offers banks, and what banks must do to adopt it.

Source: ACI Worldwide

Application Centric Infrastructure with Python

In application-centric networking, troubleshooting does not mean logging into discrete devices and examining networking state information. If a web application is not performing, you start with the web application. The fact that relevant state information might exist within a router, switch, or even a web server is secondary. You want to gather intelligence on the application itself. That data needs to be collected and correlated. And you aren’t done until the application is up and running as it should.

There are several questions you need to ask: Can you see how applications are talking across your network? For distributed applications, do you have a view of which components are where? Can you see how data is flowing between them?

Here I would like to present a small Python programm that I adapted from the Cisco code repository. 

# Copyright (c) 2015 Cisco Systems #

# All Rights Reserved. 

"""

Simple application that logs on to the APIC and displays all

of the Interfaces.

"""

import sys

import re

import json

import acitoolkit.acitoolkit as aci

def main():

    """

    Main execution routine

    :return: None

    """

    # Take login credentials from the command line if provided

    # Otherwise, take them from your environment variables file ~/.profile

    description = 'Simple application that logs on to the APIC and displays all of the Interfaces.'

    creds = aci.Credentials('apic', description)

    args = creds.get()

    # Login to APIC

    session = aci.Session(args.url, args.login, args.password)

    resp = session.login()

    if not resp.ok:

        print('%% Could not login to APIC')

        sys.exit(0)

    resp = session.get('/api/class/ipv4Addr.json')

    intfs = json.loads(resp.text)['imdata']

    data = {}

    for i in intfs:

        ip = i['ipv4Addr']['attributes']['addr']

        op = i['ipv4Addr']['attributes']['operSt']

        cfg = i['ipv4Addr']['attributes']['operStQual']

        dn = i['ipv4Addr']['attributes']['dn']

        node = dn.split('/')[2]

        intf = re.split(r'\[|\]', dn)[1]

        vrf = re.split(r'/|dom-', dn)[7]

        if vrf not in data.keys():

            data[vrf] = []

        else:

            data[vrf].append((node, intf, ip, cfg, op))

    for k in data.keys():

        header = 'IP Interface Status for VRF "{}"'.format(k)

        print header

        template = "{0:15} {1:10} {2:20} {3:8} {4:10}"

        print(template.format("Node", "Interface", "IP Address ", "Admin Status", "Status"))

        for rec in sorted(data[k]):

            print(template.format(*rec))

if __name__ == '__main__':

main()

With the help of the ACI Toolkit library from Cisco we could define and create many functions and procedures needed to manipulate and programm the ACI Fabric. As an universal data format JSON is used to retrieve data from the APIC controller. The data interested to us are VRFs, IPs, VLANs and so on. In the final function we define a loop that that will search for all of the the values stored inside the dictionary data structure from Python and sort them using a template.

For those wanted to know more about the ACI please checkout the source text from the SDX Central.

Cisco ACI is a tightly coupled policy-driven solution that integrates software and hardware. The hardware for Cisco ACI is based on the Cisco Nexus 9000 family of switches. The software and integration points for ACI include a few components, including Additional Data Center Pod, Data Center Policy Engine, and Non-Directly Attached Virtual and Physical Leaf Switches. While there isn’t an explicit reliance on any specific virtual switch, at this point, policies can only be pushed down to the virtual switches if Cisco’s Application Virtual Switch (AVS) is used, though there has been talk about extending this to Open vSwitch in the near future.

In a leaf-spine ACI fabric, Cisco is provisioning a native Layer 3 IP fabric that supports equal-cost multi-path (ECMP) routing between any two endpoints in the network, but uses overlay protocols, such as virtual extensible local area network (VXLAN) under the covers to allow any workload to exist anywhere in the network. Supporting overlay protocols is what will give the fabric the ability to have machines, either physical or virtual, in the same logical network (Layer 2 domain), even while running Layer 3 routing down to the top of each rack. Cisco ACI supports VLAN, VXLAN, and network virtualization using generic routing encapsulation (NV-GRE), which can be combined and bridged together to create a logical network/domain as needed.

From a management perspective, the central SDN Controller of the ACI solution, the Application Policy Infrastructure Controller (APIC) manages and configures the policy on each of the switches in the ACI fabric. Hardware becomes stateless with Cisco ACI, much like it is with Cisco’s UCS Computing Platform. This means no configuration is tied to the device. The APIC acts as a central repository for all policies and has the ability to rapidly deploy and re-deploy hardware, as needed, by using this stateless computing model.

Feel free to comment.