CLI tool development in Python

I’m not a Python person. I never developed in Python.

There no particular reason for that, I do not have any complains about it. It is just a question of time, needs.

For all the small scripts I may need, I use bash. And for personal developments 🔗 I used NodeJS.

And then, last week, came an opportunity.

In this small post, I’ll explain you my needs, the reasons why I did not used NodeJS or bash and partial results of my dev (only partial, and I’ll explain why)

The need

In the company I work for, we use Google Apigee 🔗 as API Management platform (not for all your APIs, but, this is another debate).

We have some maintenance tasks to lead on this platform, like cleaning up customer developer setup (that use a certain custom attribute) or replacing TLS certificates in order to expose our services via a more marketing-and-secured domain name.

To prepare these tasks, we could have crawl the console, from pages to pages, etc… but it is a long task, with risk of errors and, I have to admit, a little boring.

So, I decided to develop scripts in order to make it.

The intent is:

browser certain objects (proxies or apps or virtualhosts)
filter some of them according to criterias (presence of this specific custom attribute or…)
from each selected object, fetch additional details
filter again,
etc…

Each verb mentioned (to browse, to fetch, etc..) is equivalent to an API call to Apigee Administration API.

Language choice

Yes, NodeJS definitely could have been used for this. And that could have been my primary choice.

However … I considered one of the strength of NodeJS as a weakness for this particular project: asynchronous capability that brings complexity (and “brain knots"as we could say in French).

My need is to first call an API, parse the result and make several other API calls from it. Even if I have 30 calls to do, there are sequential.

You may say “hey, stupid, you can use promises for that!”. Yes, I could. But …

I just took this opportunity to jump into Python!

Let me be curious and try new things! (and don’t call me stupid please…)😄

`VENV` logic

I pay a lot of attention to dependencies management. I coached dev team about it.

In a previous mission, I complained a lot about a company delivering their software without any decent dependencies management, or even clear list … and let us “discover” the needed packages. Never ever again, it is too painful.

My current workstation setup is already full of Python libraries, installed globally:

$ pip freeze | wc -l
119
$

And I know for sure I do NOT need all of them to do requestsand print()! 😄.

So, I used Virtual Environment 🔗, to create a dedicated workspace.

$ python3 -m venv ./venv

$ source venv/bin/activate

(venv) $

I was able then to install all the really needed.

… and my requirements.txt is now much smaller!

(venv) $ pip freeze | wc -l
6
(venv) $

It’s a first WIN!

Credentials management

Of course, my code MUST NOT have the credentials written somewhere… in the main code, in an aside file. In short, nowhere.

WARNING: OFFENSIVE CONTENT remark to come

My ears bled few days ago when a colleague told me, very seriously:

“I never remember my passwords (including the admin ones), so I wrote them down all in a password.txt file on my desktop”

… I would have prefer this to be a joke … but, unfortunately, it is not…

So, to make it simple, I decided to get this information from environment variables, to be set prior to call the scripts


(venv) $ export APEX_USERNAME="admin@arnaduga.dev"
(venv) $ export APEX_PASSWORD="Password1234"

# OR
(venv) $ APEX_USERNAME="admin@arnaduga.dev" APEX_PASSWORD="Password1234" python3 myscript.py

Note

You can also control what is stored into your history by setting the appropriate option and by inserting a space characetr in front of your command!

bash 🔗 with the $HISTCONTROL variable

ZSH 🔗 with the setopt HIST_IGNORE_SPACE setup

And then, classically, I can retrieve them in my code:

import sys
import os

try:
    username = os.environ['APEX_USERNAME']
    password = os.environ['APEX_PASSWORD']
except Exception as e:
    print("ERROR: APEX_USERNAME and APEX_PASSWORD must be set")
    sys.exit(1)

CLI approach: the great Click library!

There are also 2 arguments I absolutely wanted to keep outside of my code, in order to make it a little bit more generic: organisation and environment.

So, ideally, I wanted to be able to call my script like:

$ python3 myscript.py --env prd --organisation myorg

And, as I’m lazy and I do not appreciate re-inventing the wheel (and often, it will less round than the existing ones…), I had a look on libraries, and more specifically the Click 🔗 one.

It looked to me very efficient, simple enough, and I wanted to give it a try!

Bonus

It comes with utilities, like the progress bar 🔗 for a nice looking script, at no cost (in terms of dev).

Partial results

As I developed the scripts for the company I work for, using company’s hardware on my working time, the code belongs to the company. I cannot share it (even it is not really business related …).

So, I recreated on my personal hardware and time something much more simpler, to illustrate my researches, and you will be able to find the sources on my Gitlab repository.

From the usage point of view, it looks like this:

That’s all, folks!

Arnaud/CLI tool development in Python