Managing BigAnimal Clusters with BigAnimal’s APIs with Python 3

In case you would like to review details about BigAnimal, you can take a look at the previous blog posting: “Managing BigAnimal Clusters with BigAnimal APIs ”, in which we reviewed and explained what is:  

• DBaaS
• BigAnimal
• BigAnimal Cluster

We also discussed What BigAnimal is, its features and capabilities.

Code

The accompanying code for this posting is available at: https://github.com/EnterpriseDB/blogpostings/tree/main/BigAnimal/python 

Python

Python, is a programming language that is listed as the most popular programming language. It is an interpreted and high-level programming language that was conceived in the late 1980s by Guido van Rossum. 

A few reasons that have contributed to Python’s popularity are:

• Easy to learn and use
• Mature and supportive community
• Hundreds of frameworks and libraries
• Versatile, efficient and reliable

Our goal is to demonstrate the simplicity of calling BigAnimal’s APIs with Python for cluster management.

Cluster Management

Cluster management in any DBaaS is of great importance because there should be multiple methods that allow the ability to: provision, list and destroy clusters. The standardized methods for managing DBaaS clusters usually considered are APIs and CLI. 

How those calls are made should be:

• Simple
• Direct
• Standardized across platforms and programing languages

Pre-Requisites

• Python3 installed and configured
• Previously configured of BigAnimal credentials
• Configured BigAnimal credential with token
• Completed ‘data.json’ for passing all the cluster configuration to the API

Obtaining a Token

A BigAnimal token is created by executing the script ‘get-token.sh’ as shown below:

get-token.sh

After successfully following the instructions displayed on the console successful token  that looks somewhat as below

{
  "access_token": "<yourtoken>",
  "refresh_token": "v1.MZY5PHwcVWeDw82c5C2qkRa5U3Cenp3eD2beagy6DmQCxf7FGU9U9XczyDjWaJmboeoDYwbzgyUBt80lNINMOfk",
  "scope": "openid profile email offline_access",
  "expires_in": 86400,
  "token_type": "Bearer"
}

From the message above, we can tell how long the token will last which is listed in seconds under the “expires_in” setting.

Should there be a need to obtain more tokens the same process can be executed as many times as needed.

Creating a Cluster

Once you have all the prerequisites readily available, we can proceed to edit the “data.json” file with the desired cluster configuration:

{
  "clusterName": "DO-python-1",
  "instanceType": {
    "cpu": 2,
    "id": "azure:Standard_E2s_v3",
    "instanceType": "E2s v3",
    "ram": 16
  },
  "password": "SuperDuper123",
  "pgConfigMap": [
    [
      "max_connections",
      "100"
    ]
  ],
  "postgresType": {
    "dockerImage": "",
    "id": "epas"
  },
  "postgresVersion": "14",
  "privateNetworking": true,
  "provider": {
    "cloudProviderId": "azure"
  },
  "region": {
    "regionId": "westus2"
  },
  "replicas": 3,
  "volumeProperties": "P1",
  "volumeType": {
    "configFieldsList": [
      "secretName",
      "shareName"
    ],
    "displayName": "Azure Premium Storage",
    "id": "azurepremiumstorage",
    "storageClass": "managed-premium"
  },
  "zoneRedundantHa": true
}

The values to be configured should be:

• Cluster Name
• Password
• PostgreSQL Type
• PostgreSQL Version
• Provider
• Region
• Instance Type
• Volume Type
• Volume Properties
• High Availability
• Networking
• Confirmation of cluster creation

The next step is to edit the: “create-cluster.py” file and look for the section where you assign the token, which is on line 7. Should look similar to this:

token = "<yourtoken>"

The code to create a cluster is located inside the “create-cluster.py” file, it looks like this

# Original work by: EnterpriseDB
# https://www.enterprisedb.com/
# Purpose : Provision BigAnimal Clusters
# Author  : Doug Ortiz
# Date    : January 28, 2022
# Version : 1.0
# Copyright (c) 2020 EnterpriseDB
import requests
from requests.structures import CaseInsensitiveDict
import json

# Assign variables values
# Set 'token'
token = "<yourtoken>"

# Headers
headers = CaseInsensitiveDict()
headers["Accept"] = "application/json"
headers["Authorization"] = "Bearer " + token
headers["Content-Type"] = "application/json"

# API URL
url = "https://portal.biganimal.com/api/v1/clusters"

# Perform API POST Request
try:
    response = requests.post(url, headers=headers, data=open('data.json', 'rb'))
    response.raise_for_status()
except requests.exceptions.HTTPError as errHTTP:
    print(errHTTP)
except requests.exceptions.ConnectionError as errorConnection:
    print(errorConnection)
except requests.exceptions.Timeout as errorTimeout:
    print(errorTimeout)
except requests.exceptions.RequestException as errException:
    print(errException)

# Provide status
print(f"Status Code: {response.status_code}, Response: {response.json()}")

The code above: 

• Assigns the token, headers, and url to variables
• Performs a POST request
• Displays the status from the request, and the response data.

A few items to note:

• The token must be assigned where the “token” variable is defined
• Basic error handling can be improved
• The status of the API call is displayed at the end of the code

Creating the cluster is accomplished by executing the python code with the following command:

python3 create-cluster.py

A successful cluster creation will provide a message that looks somewhat as below

Status Code: 202, Response: {'pgId': 'p-q84qsta2gi'}

Listing Clusters

Listing clusters is accomplished by assigning the token in the “get-clusters.py” file along with setting the filter for:

• name
• provider
• pgType
• pgVersion
• sort

The code to list a cluster is located inside the “get-clusters.py” file, it looks like this

# Original work by: EnterpriseDB
# https://www.enterprisedb.com/
# Purpose : List BigAnimal Clusters
# Author  : Doug Ortiz
# Date    : January 28, 2022
# Version : 1.0
# Copyright (c) 2020 EnterpriseDB

import requests
from requests.structures import CaseInsensitiveDict
import json

# Assign variables values
# Set 'token'
token = "<yourtoken>"

# Headers
headers = CaseInsensitiveDict()
headers["Accept"] = "application/json"
headers["Authorization"] = "Bearer " + token
#headers["Content-Type"] = "application/json"

# URL, filter and filteredurl
url = "https://portal.biganimal.com/api/v1/clusters"
# Adjust filter accordingly
filter = "?name=DO-python&provider=azure&pgType=epas&pgVersion=14&sort=%2Bname"
filteredurl = url+filter

# Perform API Get Request
try:
    response = requests.get(filteredurl, headers=headers)
    response.raise_for_status()
    print("Filter: ")
    print(filteredurl)
except requests.exceptions.HTTPError as errHTTP:
    print(errHTTP)
except requests.exceptions.ConnectionError as errorConnection:
    print(errorConnection)
except requests.exceptions.Timeout as errorTimeout:
    print(errorTimeout)
except requests.exceptions.RequestException as errException:
    print(errException)

# Provide status
print("")
print(f"Status Code: {response.status_code}, Response: {response.json()}")

Similar to the previous code file, the code above:

• Assigns the token, headers, url, and filter to variables
• Performs a GET request
• Displays the status from the request, and the response data.

Notice the items below in the code above:

• The token must be assigned where the “token” variable is defined
• The “filter” variable might need some adjusting to fit the parameters of the clusters to list. It is not needed to include: provider, pgType, pgVersion, nor the sort parameters

The execution command to list the clusters available to the current credentials is:

python3 get-clusters.py

Listing clusters results:

Filter: 
https://portal.biganimal.com/api/v1/clusters?name=DO-python&provider=azure&pgType=epas&pgVersion=14&sort=%2Bname

Status Code: 200, Response: [{'name': 'DO-python-1', 'instance': 3, 'currentPrimary': '', 'targetPrimary': 'p-q84qsta2gi-1', 'pvcCount': 0, 'jobCount': 0, 'licenceStatus': '', 'writeService': 'p-q84qsta2gi-rw', 'readService': 'p-q84qsta2gi-r', 'phase': 'Setting up primary', 'phaseReason': 'Creating primary instance p-q84qsta2gi-1-initdb', 'description': '', 'imageName': 'edbclouddp.azurecr.io/enterprisedb/starlight:14.1.0-3-debian-epas@sha256:f5a50306078a48c1af4e6fbdf73e32ece883475d70d29b6323315a8fbb7b2493', 'postgresConfig': '', 'maintainanceWindowNode': '', 'backupDestinationPath': 'https://pgho9cg92c.blob.core.windows.net/p-q84qsta2gi/', 'backupEndpointUrl': '', 'instanceType': {'id': 'azure:Standard_E2s_v3', 'ram': 16, 'cpu': 2, 'instanceType': 'E2s v3', 'description': '', 'familyName': '', 'name': ''}, 'pgId': 'p-q84qsta2gi', 'pgType': {'id': 'epas', 'name': 'EDB Postgres Advanced Server', 'description': '', 'dockerImage': ''}, 'pgVersion': {'versionId': '14', 'versionName': '14', 'description': ''}, 'clusterLocation': {'cloudProvider': {'cloudProviderId': 'azure', 'cloudProviderName': 'Azure', 'description': ''}, 'region': {'regionId': 'westus2', 'regionName': '(US) West US 2', 'description': ''}, 'k8ClusterId': '34cdcbdc-7248-5c15-89e6-d0cd627c8f2a'}, 'privateNetworking': True, 'clusterStorageParameters': {'storageClass': 'managed-premium', 'size': '4 Gi', 'autoscale': False, 'paramsMap': [['secretName', ''], ['shareName', '']], 'volumeTypeId': 
'azurepremiumstorage', 'volumePropertiesId': 'P1'}, 'pgConfigMap': [['max_connections', '100']], 'zoneRedundantHa': False, 'numberOfBackups': 0, 'profileType': '', 'deleteDetails': '', 'replicas': 3, 'storageAccountName': 'pgho9cg92c', 'clusterConnectionInfo': {'serviceName': 'p-q84qsta2gi.private.qsbilba3hlgp1vqr.biganimal.io', 'databaseName': 'edb_admin', 'port': '5432', 'username': 'edb_admin', 'password': '', 'pgpass': '', 'pguri': 'postgresql://edb_admin@p-q84qsta2gi.private.qsbilba3hlgp1vqr.biganimal.io:5432/edb_admin'}, 'createTime': {'seconds': 1643301700, 'nanos': 805427000}, 'backupRetentionPeriod': '30d', 'allowIpRangeMap': [['0.0.0.0/0', 'To allow all access']], 'sourcePgId': '', 'orgId': 'org_QsBILBa3HlGP1VQr', 'instanceTypeId': 'azure:Standard_E2s_v3', 'postgresTypeId': 'epas', 'providerId': 'azure', 'regionId': 'westus2'}]

Deleting Clusters

To delete the BigAnimal clusters, we need to update the token and the pgId in the “delete-cluster.py” code file. 

The code to delete a cluster is located inside the “delete-cluster.py” file, it looks like this

# Original work by: EnterpriseDB
# https://www.enterprisedb.com/
# Purpose : Delete BigAnimal Clusters
# Author  : Doug Ortiz
# Date    : January 28, 2022
# Version : 1.0
# Copyright (c) 2020 EnterpriseDB

import requests
from requests.structures import CaseInsensitiveDict
import json

# Assign variables values
# Set 'token'
token = "<yourtoken>"
headers = CaseInsensitiveDict()
headers["Accept"] = "*/*"
headers["Authorization"] = "Bearer " + token

# Postgres Cluster ID, URL, filter and filteredurl
# pgId must be retrieved or known prior to making the API call
pgId = "<yourbiganimalclusterid>"
url = "https://portal.biganimal.com/api/v1/clusters"
filter = "/" + pgId
filteredurl = url+filter
print("Filter: ")
print(filteredurl)

# Perform API Delete Request
try:
    response = requests.delete(filteredurl, headers=headers)
    response.raise_for_status()
    print("")
    print("Cluster: " + pgId + " deleted successfully!")
except requests.exceptions.HTTPError as errHTTP:
    print(errHTTP)
except requests.exceptions.ConnectionError as errorConnection:
    print(errorConnection)
except requests.exceptions.Timeout as errorTimeout:
    print(errorTimeout)
except requests.exceptions.RequestException as errException:
    print(errException)

In the delete code file, the code above:

• Assigns the token, headers, pgId, url, and filter to variables
• Performs a DELETE request
• Displays the status from the request, and the response data.

Notice the items below in the code above:

• As the other code files, the token must be assigned where the “token” variable is defined
• The BigAnimal Cluster Id must also be assigned

The execution command to delete a cluster is:

python3 delete-cluster.py

A successful cluster deletion will provide a message that looks somewhat as below

Filter: 
https://portal.biganimal.com/api/v1/clusters/p-q84qsta2gi

Cluster: p-q84qsta2gi deleted successfully!

Inspecting how it all works together

Now that we know what each code file accomplishes, we can stop and take a look at the underlying technologies and how all the pieces fall into place.

The top block provides the BigAnimal API services to the bottom block that assembles the json and token into a message that is returned the the API call itself. It is very easy to overlook every step that occurs and while not all inclusive the diagram above does provide a deeper view at what happens behind the scenes.

Conclusion

In this blog we used Python3 and learned how to: 

• Obtain a BigAnimal token
• Create a BigAnimal cluster
• List BigAnimal clusters
• Delete a BigAnimal cluster

See you in the next blog post!

 

Try EDB BigAnimal today!