Venkat$ echo

I have a Rundeck setup running in Linux and DB= mysql. I have all my code/scripts in py. So I thought it will be easy for me to do any maintenance on rundeck if I have a rundeck library in py. This Library will give me easy access to rundeck details that I need for doing my maintenance. USE- CASE The first maintenance I wanted to do on Rundeck is to delete the old executions as the db size was getting bigger and the logs on the machine was also occupying a lot of space. Since I have lots of recurring jobs, I thought it would be better to also have a maintenance job which can delete old executions after a specific no_of_days. Please note that I have lots of library code (like logger, Cquery for mysql_query etc). You will have to replace those with your library function. If you dont have an equivalent library then you need to change the respective area in the code to make this work. If you find any difficulty, please let a comment and I will try to help you. """ Th...

As you all know by default Django offers username and email separately. But if you are like me who wants to use email as the username and want to override the default behavior of django, there are few ways to do it Change the <username> label to 'email' using a custom form. But downside is username field is only 30 char long. Registering a custom user-auth-model, to the existing django-registration process. But downside is you can only do this before the first migration.  There are few modules available which extended username. So you can install that. But the downside is I was not sure if those modules are supported anymore. Building your own auth-model. But this requires lot of time. Making simple tweaks to the existing django-auth-model ( User) and use the existing django-registration ( I'm using django-registration-redux) to take care of everything. There are pros and cons in every model, but l don't want to spend too much time on the login module ...

We were running a Rundeck instance using a File-DB ( not MYSQL) and after a month we found that rundeck webpages were very slow and it was unusable. It was because we had lot of job that runs repeatedly in shorter interval and there were so many execution history. This slowed the whole rundeck web front end. I tried to use the normal bulk delete operation using the API and it was still very slow to get all the execution IDs and then to delete them. So I used a work around to get the Execution IDs from the log-filename instead of using the rundeck API. It was still slow but it reduced the overall time by like 60-70% as getting the IDs was the time consuming task. Below is the code #!/usr/bin/python -tt ''' Rundeck delete execution --------------- change-history --------------- 1.0|21-oct-2015|vsubr|created ''' __version__ = '1.0' import sys import time from datetime import datetime import re from os import listdir from os.path import isfile, join...

Please have a look into the basic options first - http://venkat-echo.blogspot.com.au/2015/12/py-generic-xml-2-relational-data.html 1. If you want to force a specific XML Level into a separate table.  In the above example if you want all toppings details in a separate table and each topping as a seperate row in that able, use the following option #If you want to force a specific XML Level into a separate table. xml2rd.arr_predefined_xmlPath4Tables = ['/items/item/topping'] Result of above parameter 2. If you would like to add/merge/insert different xmlpath to the same table then use the below config. But please note that Merge XML paths as the same DEPTH.  In the above example if you want to merge <batters> and <batterscost> then use this config # to put multiple XMLPath into the same table. But make sure all XMLPATH are at the same level xml2rd.d_common_table_4XmlPaths = { '/items/item/batters':{'table':'common_tab...

Why I need this Generic XML Parser Its often required to load xml data into tables, so that business users can access the XML data and also use those table to write reports.  XMLs I get normally are from 3rd parties and most of the time I dont get the xsd for those XML. Plus we get different XMLs from time to time and if we had hard-code the path to read a specific XML then I had to write code for every XML that I received. So I thought it is better to make a GENERIC XML PARSER which can take any XML file and convert that into RELATIONAL DATA style and write that into CSVs ( which can be used to load the tables) Logic applied for  XML 2 Relational Data conversion Even Level in the XML is a table. ie  items/ item is a table  items/item/baters is a table Can not handle Namespaces All key columns + columns created (by XML2RD) for reference created by with prefix = '_' (eg _xid , _xpath etc) <Element> name becomes the column name. In case of...

I'm doing a python framework which is capable of pushing the data to Geckoboard. Here is a the sample of the code and it handles for "highcharts"(different kinds) and gecko-o-meter. It can be extended to other charts as well. This is just an template/idea. Here is a sample i made #!env python -tt ''' 1) Gecko is used for pushing data to Gecko board ---------------------- Modification History ---------------------- -- 22-Jan-2015| vsubr | created ''' import sys import datetime from collections import OrderedDict import pprint import json import requests import MySQLdb from contextlib import closing import mod.conf.cfg as cfg import mod.lib.logger as logger import mod.conf.settings as settings class Gecko(object): ''' "Task" class takes Gecko id as the input and executes that Gecko-task. ''' #-- use this site to check the colors http://html-color-codes.info/ arr_gecko_colors =[ "#108ec5...

Im currently working on AWS KINESIS and thought I will document all the issue I'm facing When running consumer.py - I got              SEVERE: Failed to start client executable             java.io.IOException: Cannot run program "/bla/bla/bla/my_aws/mySample_kcl_app.py": error=2, No such file or directory SOLUTION: This is what I found you can have absolute path or relative it doesnt matter. it is not necessary to set the PATH if you are using absolute path if you had use the (consumer)sample.py as a template, then fix the shebang (ie.. origina sample.py had #!env python... It is for this the KCL lib threw the error file or directory not found. change this to the actual python path like (#!/usr/bin/env python). Then the file or folder not found error is gone.

PIP is not prepackaged with Python 2.7 and I found that PIP is very helpful to install lots of python modules. So thought why not document the steps to install it. So here it is. Install pip - Straight Forward way  Download the get-pip.py from https://pip.pypa.io/en/latest/installing.html Run this on command line - python get-pip.py If you face any problems like "unable to fetch bla bla..." it is likely that you are having a proxy issue. If so follow the below steps Install pip with Proxy config Find the proxy config- i.e IP and PORT. if you already know it, then use that OR Go to your browser and check the proxy settings ( IE= Tools->Internet Options -> connections Tab -> Lan Settings. chrome= settings-> advanced options -> under Network -> "change proxy settings"). If your setting has "automatic proxy detection" then use the below method to find it. Open command line and type - "ping wpad" and see you can fig...

Click here for other Tableau related Blogs Requirement is to convert data in ESRI Map shapes into Tableau format. Tools Required Quantum GIS -  http://www.qgis.org/en/site/ Python  Download the below Python Code -  https://github.com/venkat-vs-id/python import csv import re import sys #def convertESRI2TableauFormat( p_inputfile, p_outputfile): l_outfileName = 'c:/ESRI_suburb_tab_format.csv' # make changes here.. l_ESRIFileName ='c:/ESRI_suburb.csv' # make changes here.. rdx=0 csv.field_size_limit(sys.maxsize) outfile = open( l_outfileName, 'w', newline='') csvWriter = csv.writer( outfile, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL) with open( l_ESRIFileName, 'r') as csvfile: csvR = csv.reader( csvfile, delimiter=',', quotechar='"') for row in csvR: rdx += 1 ci=0 rowValues = [] rowValues.append(row[1]);# ssc_code rowValues.append(...