Showing posts with label Python. Show all posts
Showing posts with label Python. Show all posts

Friday, December 25, 2020

Build your own COVID-19 Data Analytics Leveraging Open Source - Serverless Fn & Autonomous Database

Firstly, a huge thanks to all the frontline workers, healthcare professionals, technology contributions, non-profit organizations and everyone else who are fighting the pandemic everyday amidst the risks involved and more importantly the sacrifices - including the most important thing that can never be retrieved - TIME.

As Oracle Cloud continues to aid COVID-19 vaccine research & trials and also help manage the COVID-19 vaccine distribution program, I came up with a simple (humble) contribution that helps analyze and report COVID-19 data.

Although there are several dashboards and reports readily available on the web, they cater to a specific region, locale & metric. For example, what if? one would like to understand the total COVID-19 cases reported in Belarus on/up-to a specific date sliced by total tests conducted, new cases that were tested positive, hospitalization ratio per million against a median age group?

Data is publicly available but making sense of it is the key. In this article, we will see how we can build our own COVID-19 analytics - leveraging simple & popular open source tools and technology with the help of public cloud (Eg. Autonomous Database in Oracle Cloud - that is auto-managed, auto-tuned & auto-patched) [PS: You can also replace Autonomous DB with MySQL]. What's more? - we can potentially achieve this for free with the Always Free cloud instance.

Let's quickly take a look at the high-level solution architecture;


Each component of this architecture is designed to be loosely coupled and can be replaced or further enhanced for functionality/convenience. For instance, I have leveraged Apache open-source serverless Fn project that natively runs on Oracle Cloud - this can be replaced with a Node.js or java code running on K8s-Docker container. Similarly, the Autonomous Database can be replaced with a MySQL DB 

Let's take a look at the key components of this solution;

1) Source dataset [Courtesy: Our World in Data]

"Our World in Data" offers COVID-19 data in a variety of data formats and most importantly offers daily updates - so we can keep our analytics up-to-date.

In our case, we get the dataset in csv format here

2) Python script deployed on Oracle Fn

I have leveraged the Oracle Cloud Functions (based on Apache Fn) serverless platform to deploy the simple python script to download the COVID-19 dataset into an object storage bucket.

The choice of Oracle Cloud Functions is particularly helpful in this case because I don't have to manage any infrastructure or deal with packaging the docker container and version control them. It lets me focus only on the business logic. Also, it supports a variety of programming languages natively including Python, Go & Java. Most importantly, it has built-in security and offers out of the box support for event notifications & triggers and ability to expose functions as APIs.

Pre-Req: 

Create a dynamic group for Oracle Functions and ensure you have a policy defined in your compartment/tenancy for Oracle Functions the ability to access / read csv in the object storage bucket

Instances that meet the criteria defined by any of these rules will be included in the dynamic group.

ALL {resource.type = 'fnfunc', resource.compartment.id = 'ocid1.compartment.oc1..abcdefgxyz'}

Allow dynamic-group FnDynamicGroup to manage objects in compartment sathya.ag

Let's create an Oracle Functions application;

Oracle Cloud -> Developer Services -> Functions -> Create Application & give it a name

Applications lets us group several Oracle Functions. To create a serverless function;

For quick setup, you can leverage the Cloud Shell under the Getting Started instructions to deploy the following python code. Oracle functions platform packages the code as a docker container, uploads the docker image to the default Oracle Cloud docker registry and automatically deploys it as a serverless function with an invoke endpoint.

import io
import json
import oci
import csv
import requests
import logging
import os
import urllib.request

from fdk import response

def progress_callback(bytes_uploaded):
    print("{} additional bytes uploaded".format(bytes_uploaded))

def handler(ctx, data: io.BytesIO=None):
    logging.getLogger().info("Got incoming request")
    signer = oci.auth.signers.get_resource_principals_signer()
    object_name = bucket_name = namespace = ""
    try:
        cfg = ctx.Config()
        input_bucket = cfg["input-bucket"]
        processed_bucket = cfg["processed-bucket"]
        input_csv = cfg["input-csv"]
        object_name = cfg["object-name"]
    except Exception as e:
        print('Missing function parameters: bucket_name', flush=True)
        raise
    logging.getLogger().info("before calling load data {0} {1} {2}".format(input_bucket,
 input_csv, object_name))
    
    
    logging.getLogger().info("download start!")
    filename, headers = urllib.request.urlretrieve(input_csv, filename="/tmp/covid.csv")
    logging.getLogger().info("download complete!")
    
    load_data(signer, namespace, input_bucket, filename, object_name)
    #move_object(signer, namespace, input_bucket, processed_bucket, object_name)

    return response.Response(
        ctx, 
        response_data=json.dumps({"status": "Success"}),
        headers={"Content-Type": "application/json"}
    )
    
def load_data(signer, namespace, bucket_name, input_csv, object_name):
    logging.getLogger().info("inside load data function {0} {1} {2}".format(signer, 
namespace, bucket_name))
    client = oci.object_storage.ObjectStorageClient(config={}, signer=signer)
    try:
        print("INFO - About to read object {0} from local folder and upload to bucket {1}
...".format(object_name, bucket_name), flush=True)
        namespace = client.get_namespace().data
        
        # Use UploadManager to do multi-part upload of file, with 3 Parallel uploads
        logging.getLogger().info("before calling uploadmanager")
        upload_manager = oci.object_storage.UploadManager(client, 
allow_parallel_uploads=True, parallel_process_count=3)
        response = upload_manager.upload_file(namespace, bucket_name, object_name, 
input_csv, progress_callback=progress_callback)
        logging.getLogger().info("response status {0}".format(response.status))
        if (response.status == 200):
            
            message = "Successfully  uploaded %s in bucket %s." % (object_name, 
bucket_name)
            return True

    except Exception as e:
        logging.getLogger().info("exception message {0}".format(e))
        message = " Image upload Failed  in bucket %s. " % bucket_name
        if "oci" in e.__class__.__module__:
            if hasattr(e, 'message'):
                message = message + e.message
            else:
                message = message + repr(e)
                print(message)
        return False

In the configuration section, provide the key value pairs that can be dynamically processed by Oracle Functions at invoke.

3) COVID Dataset in Object Storage

Let's verify if the COVID-19 dataset is successfully downloaded into our object storage bucket.

4) Loading COVID-19 dataset into Autonomous Database

Since we are leveraging Oracle Cloud Autonomous DB, we can leverage the OOTB cloud SQL package to load data from an external object storage bucket. Another variant of this approach could be to leverage the csv data in object storage as an external table and ADB lets you query on the csv data directly.

Again, the choice for Autonomous Database helps us focus only on loading and querying the dataset and not have to worry about the underlying infrastructure, patching & maintenance. I like to think of Autonomous DB as a "serverless" database.

Execute the DBMS_CLOUD.COPY_DATA procedure to load data from object storage into ADB. [Ensure you have the base table created to hold the COVID-19 dataset]

BEGIN
 DBMS_CLOUD.COPY_DATA(
    table_name =>'COVID',
    credential_name =>'JSON_CRED_AG',
    file_uri_list =>'YOUR OBJECT STORAGE BUCKET URI',
    format => json_object('type' value 'CSV', 'ignoremissingcolumns' value 'true', 
'delimiter' value ',', 'blankasnull' value 'true', 'skipheaders' value '1', 'dateformat' value 'YYYY-MM-DD')
 );
END;
/

5) Analytics Dashboard

Now that we have our data in a database, we can leverage any analytics / reporting engine to make sense of the data and generate dynamic reports. In our case, we leverage Oracle Analytics and/or Oracle Data Visualization.



Tuesday, June 4, 2019

Achieve High Availability (HA) with Oracle Cloud Infrastructure (OCI) - using HAProxy & KeepAlived

Features such as High Availability (HA), Elastic Scaling, Disaster Recovery (DR) are no longer restricted to Tier-1 mission critical applications & services. These are now table-stakes for any enterprise-grade cloud provider and enterprises can leverage these at will - as they are now fundamental building blocks of the cloud platform. Oracle Cloud Infrastructure (OCI) offers several modern state-of-the-art capabilities including - Bare Metal servers for extreme performance & HPC, Availability Domains & Fault-Domain constructs for fault-tolerance, HA & DR, Elastic Scaling of compute & storage independently for future-proofing, RAC for highly-available databases, Regional Subnets for seamless datacenter resiliency & failover, LBaaS (Load Balancer as a Service) for platform-driven fully managed instance failovers etc..

Although there are many different ways to achieve HA on Oracle Cloud Infrastructure, we will look at a more simple/primitive method (just leveraging core OCI capabilities) that leverages open-source technologies.
Note: This could be more elegantly achieved using OCI's native LBaaS PaaS service as well.

However, in certain situations - windows server based apps, container apps, microservice constructs - harproxy (or similar) & keepalived might have semblance to on-premise experience and/or solution preference.

In this article we would walk through detailed step-by-step instructions on how to install, configure and achieve HA & failover using HAProxy & KeepAlived.

This article presumes users have some exposure to HAProxy, KeepAlived concepts as well as cloud constructs such as virtual networking, subnets, private/public IPs, security lists/route tables etc..
A little bit of python & bash shell scripting knowledge would be helpful too.
I not, don't worry - I will try my best to point out / reference documentation as much as possible.

Pre-Requisites:

1) An Oracle Cloud Account / Tenancy. If you don't have one, you can request a trial instance here.
2) A Compartment that would host our HA instances
3) A VCN (Virtual Cloud Network) with at least 1 public subnet
4) Administrator access on the cloud instance to configure Identity, Network Rules, Policies & Dynamic Groups

Spin Up 2 OCI VM Instances:

To start, let's first spin up 2 VMs on OCI.

My VCN looks like below;

Need help setting up a VCN on OCI? Refer here

Note: In this example, we have HA within an AD (Availability Domain) leveraging the "Fault Domain" resilience. However, this can be quickly reconstructed with a "regional subnet" construct for a full "site resiliency".

Within a public subnet, spin up 2 VMs.
In my example I have 2 Angry Bird servers - Terence & Stella.
Both running a standard single core VM with Oracle Enterprise Linux in the same availability domain - but placed strategically in different fault domains for intra-AD HA.

Now, you should see 2 instances up & running within your VCN.

Let's now SSH into our instances.

ssh -i <<private_key>> opc@<<public_ip>>

Note: If you are unable to ssh into your instance. Make sure;
1) The instance is indeed spun up within a public subnet
2) Security List has port 22 enabled (by default this should be there)
3) Ensure you have an internet gateway (IGW) attached to your VCN and route table configured with the route to IGW
4) Finally at the OS level, make sure you open up firewall ports. For a quick test, you can try stopping the firewall on linux OS instances using the command: service firewalld stop

Install your preferred proxy service:

You can choose to install any of your preferred http/https proxy or load balancer service. Some of the most popular ones include - apache httpd, Nginx, HAProxy.

In my example, I used HAProxy.
Install HAProxy on both Terence & Stella VM instances.


sudo su
yum install haproxy

Since we are just going to test the failover / HA configuration, we are not going to actually create any backend sets / services. When the reverse proxy service is called, it will be directed to render a static error page.

Let's create a simple html page under /etc/haproxy/errorfiles/errorpage.http
Replace {ServerName} with appropriate VM names so it helps distinguish the service when it fails over.

HTTP/1.0 503 Service Unavailable
Cache-Control: no-cache
Connection: close
Content-Type: text/html

<html>
  <head>
    <title>503 -  Service Unavailable</title>
  </head>
  <body>
    <div>
          <h2>Hello from {ServerName}</h2>
    </div>
  </body>
</html>

Configure HAProxy /etc/haproxy/haproxy.cfg with the errorfile info and frontend bind on port 80. Remember, we actually don't have any backends configured. But that's okay for our failover test.

defaults
    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except 127.0.0.0/8
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000
    errorfile 503 /etc/haproxy/errorfiles/errorpage.http
#---------------------------------------------------------------------
# main frontend which proxys to the backends
#---------------------------------------------------------------------
frontend hatest
    mode http
    bind *:80
    default_backend             app

#---------------------------------------------------------------------
# static backend for serving up images, stylesheets and such
#---------------------------------------------------------------------
backend static
    balance     roundrobin
    server      static 127.0.0.1:4331 check

#---------------------------------------------------------------------
# round robin balancing between the various backends
#---------------------------------------------------------------------
backend app
    mode http
    balance     roundrobin
    option httpchk HEAD / HTTP/1.1
    server  app1 127.0.0.1:5001 check

Now, let's configure HAProxy to start on VM boot.
chkconfig haproxy on
service haproxy start

Try hitting both server instances with their respective public ip addresses;
http://publicipofinstance

and you should see the appropriate server's 503 error page with welcome messages.
Note:
If you are unable to connect from your browser, check the public subnet security list for port 80, route table for IGW route or the OS firewall could be blocking port 80.

Configure Secondary IP:

In order for instance to failover, we need a reserved public ip that would shuttle over across instances for a seamless HA / failover.

In the OCI Console, go to the primary VM instance - pick any one VM that would serve as the "master" node. Click on "Attached VNICs" under the "Resources" section.
We will now create a secondary IP under the "Primary VNIC".
Click on the "Primary VNIC" > "IP Addresses" > choose "Assign Private IP Address".

In the dialog, enter a private ip that is unused within the VCN/Subnet. In my case, I picked 10.0.0.4.


Under the "Public IP Address" section, choose "Reserved Public IP" and select "Create a New Reserved Public IP". Optionally give it a name.

This would be our reserved public ip - which would move along with our chosen private ip address.

Go back to the primary VNIC of your VM instance and notice you have 2 public IPs (one ephemeral IP that is OCI assigned and another reserved IP) assigned to the instance. This technically means the VM can be accessed with either IPs.


However, we need to make sure the OS config is updated to reflect this.

Quicker option is to execute the following command (however this would not persist on VM reboot).
In my case, the command looks like below;

ip addr add 10.0.0.4/25 dev ens3 label ens3:0
Syntax: ip addr add <address>/<subnet_prefix_len> dev <phys_dev> label <phys_dev>:<addr_seq_num>

To make this change persistent, create an ifcfg file named /etc/sysconfig/network-scripts/ifcfg-<phys_dev>:<addr_seq_num>. To continue with the preceding example, the file name would be /etc/sysconfig/network-scripts/ifcfg-ens3:0, and the contents would be:

DEVICE="ens3:0"
BOOTPROTO=static
IPADDR=10.0.0.4
NETMASK=255.255.255.128
ONBOOT=yes

Note: Only this step has to be performed on both VM instances - as when the private ip moves (along with the reserved ip) to standby instance, the OS must be able to recognize the IP mapping.

To verify this change, try accessing the Terence VM with both IP address via a browser.

Install KeepAlived:

We will leverage KeepAlived to maintain our server pool, monitor the VM instances and shuttle IP address. In our example we would use the VRRP protocol and unicast ip addressing.

Make sure to add the VRRP protocol rule to the subnet security list. This will allow the VM instances to communicate over VRRP.

Let's install keepalived on both VM instances.

sudo su
yum install keepalived


Modify the keepalived config file @ /etc/keepalived/keepalived.conf.
My config file for both Terence (Primary / Master) and Stella (Secondary / Backup) instances look like below;

Note source ip (ip of current server instance), peer ip (ip of backup instance) and state fields. Make sure priority is higher for Master node.

! Configuration File for keepalived

vrrp_script check_haproxy
{
    script "pidof haproxy"
    interval 5
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state MASTER
    interface ens3
    virtual_router_id 50
    priority 101
    unicast_src_ip 10.0.0.2

    unicast_peer
    {
        10.0.0.3
    }
    track_script
    {
        check_haproxy
    }

    notify_master /etc/keepalived/failover.sh
}

! Configuration File for keepalived
vrrp_script check_haproxy
{
    script "pidof haproxy"
    interval 5
    fall 2
    rise 2
}

vrrp_instance VI_1 {
    state BACKUP
    interface ens3
    virtual_router_id 50
    priority 99
    unicast_src_ip 10.0.0.3

    unicast_peer
    {
        10.0.0.2
    }
    track_script
    {
        check_haproxy
    }

    notify_master /etc/keepalived/failover.sh
}

Configure Instance Principals in OCI:

We will leverage the OCI Instance Principals to allow instances within the server pool to manage virtual network connections. This would enable the reserved IP to move across VM instances.

Create a dynamic group and create a matching rule to ensure all VMs within our server pool are added to the group. More details on how to create a dynamic group here.
Now, create a policy to allow the dynamic group to manage virtual network connectivity.

In this case, the policy would look like below;

Allow dynamic-group HAProxyDG to manage virtual-network-family in compartment id ocid1.compartment.oc1..aaaaaaaaaxxxxxxxxxxxxxxxxa

Install Python OCI SDK:

Let's now install python oci sdk on both VM instances.

sudo su
yum install -y python


# Download and install pip
curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py
python get-pip.py

# install python OCI SDK
pip install oci

Let's now start keepalived daemon services on both VMs. You can either use chkconfig to make this service start on boots. There have been several intermittent issues reported with running keepalived as a service.
Another workaround is to start this via command line using;
keepalived -D

Python Script using OCI SDK to Migrate IP from Master VM to Backup VM:

import sys, oci, logging, os

def assign_to_different_vnic(private_ip_id, vnic_id):   

    update_private_ip_details = oci.core.models.UpdatePrivateIpDetails(vnic_id=vnic_id)
    network.update_private_ip(private_ip_id, update_private_ip_details)
    
if __name__ == '__main__':
    signer = oci.auth.signers.InstancePrincipalsSecurityTokenSigner()
    network = oci.core.VirtualNetworkClient(config={}, signer=signer)

    new_vnic_id = sys.argv[1]
    privateip_id = sys.argv[2]
    assign_to_different_vnic(privateip_id, new_vnic_id)

We will now call this python script from a shell script.
Create this shell script under /etc/keepalived/failover.sh
Note: This script will be invoked by the keepalived daemon

#!/bin/bash

logger -s "Floating the private/public VIPs:"
python /home/opc/claimip.py {ocid of vnic} {ocid of private ip} > >(logger -s -t $(basename $0)) 2>&1
logger -s "Private/public VIPs attached to the NEW Master Node!"

Make sure in each of the VM instances, the vnic ocid is set properly. The private ip ocid will remain the same - since it will be the same ip (in our case 10.0.0.4) that will float across VMs.

We are all set. Access the reserved public ip (haproxy service) from a browser;
You should now see "Hello from Terence" - since the ip is assigned to this master node.

Now, try stopping haproxy service: service haproxy stop.
Refreshing the page must now render "Hello from Stella" - as the ip moved over to the backup node.

We now created a HA configuration. For fun, start the haproxy service back on Terence and stop the haproxy service on Stella.