Intro
I mostly transitioned from perl to python programming. I resisted for the longest time, but now I would never go back. I realized something. I was never really good at Perl. What I was good at were the regular expressions. So Perl for me was just a framework to write RegExes. But Perl code looks ugly with all those semicolons. Python is neater and therefore more legible at a glance. Python also has more libraries to pick from. Data structures in Perl were just plain ugly. I never mastered the syntax. I think I got it now in Python, which is a huge timesaver – I don’t have to hit the books every time I have a complex data structure.
I will probably find these tips useful and will improve upon them as I find better ways to do things. It’s mostly for my own reference, but maybe someone else will find them handy. I use maybe 5% of python? As I need additional things I’ll throw thm in here.
What is this object?
Say you have an object <obj> and want to know what type it is because you’re a little lost. Do:
print(<obj>.__class__)
Check if this key exists in this dict
if “model” in thisdict:
Remove key from dict
if “model” in thisdict: del thisdict[“model”]
Copy (assign) one dict to another – watch the assignment operator!
Do not use dict2 = dict1! That is accepted, syntactically, but won’t work as you expect because the assignment operator (=) is economical and works by reference. Instead do this:
dict2 = dict1.copy()
It may even be necessary to use deepcopy:
import copy
dict2_complex = copy.deepcopy(dict1_complex)
Multiple assignments in on line
a,b,c = “hi”,23,”there”
Key and value from a single line
for itemid,val in itemvals.items():
Formatting
I guess it is pretty common to use a space-based (not tab) indent of four spaces for each subsequent code block.
Initializing lists and dicts
alist = []
adict = {}
Test for an empty list or empty dict or empty string
if not alist: print(“empty list”)
if not adict: print(“the dict adict is empty”)
astring=””
if not astring: print("the string is empty")
Length of a list or string
len(alist)
len(astring)
Merge two lists together
for elmnt in list2: list1.append(elmnt)
Address first/last element in a list
alist[0] # first element
alist[-1] # last element
First/Last two characters in a string
astring[:2]
astring[-2:]
Conditional operators
if a == b: print(“equals”)
if re.search(r’eq’,a):
do something
elif re.search(r’newstring’,a):
do something else
else:
etc.
Order of evaluation of conditionals and max value of a dictionary
a = {‘hi’:0,’there’:1,’man’:2}
if not a or max(a.values()) < 3: do something
Is the above expression safe to evaluate in the case where the dict a is defined but empty? Answer: yes, it is! Although by itself max(a.values()) would produce an error, in this or conditional, execution, I guess, never reaches that statement because the first statement evaluates as True. Same reasoning applies if the boolean operator is and.
Ternary operator
I don’t think is well-developed in Python and shouldn’t be used (my opinion).
++ operator? Doesn’t exist. += and its ilk does, however.
Absolute Value
abs(a)
Boolean variables + multiple assignment example
a, b=True, False
if a==b: print(“equals”)
if a: print(“a is true”)
Reduce number of lines in the program
for n in range(12): colors[n] = ‘red’
if not mykey in mydict: mydict[mykey] = []
Printing stuff while developing
print(“mydict”,mydict,flush=True)
Python figures it out how to print your object, whatever type it is, which is cool. That flush=True is needed if you want to see our output which you’ve redirected to a file right away! Otherwise it gets buffered.
Reading and writing files – prettyify.py
import requests, json, sys, os
import sys,json
from pathlib import Path
aql_file = sys.argv[1]
aql_path = Path(aql_file)
json_file = str(aql_path.with_suffix('.json'))
# Script path
dir_path = os.path.dirname(os.path.realpath(__file__))
dir_path_files = dir_path + "/files/"
# make ugly json file prettier
# this is kind of a different example, mixed in there
file = sys.argv[1]
f = open(file)
# return json obj as dict
fjson = json.load(f)
nicer = json.dumps(fjson,indent=4)
print(nicer,flush=True)
# back to original example
f = open(dir_path_files + json_file,'w+')
f.write(body)
f.close()
Reading in command-line arguments
Reading in a boolean value
python pgm.py False
So, you could use argparse, but I chose ast. Then I have a line in the script:
import ast overwrite_s = sys.argv[1] # either True of False - whether to overwrite or not overwrite = ast.literal_eval(overwrite_s)
Nota Bene that if you fail to take these steps your argument will be read in as a string, not a boolean!
See Reading and Writing files example.
Parsing command line arguments II
Here is a more versatile and generalized way to parse command line arguments.
import optparse
p = optparse.OptionParser()
p.add_option('-b','--brushWidth',dest='brushWidth',type='float')
p.set_defaults(brushWidth=1.0)
opt, args = p.parse_args()
width = opt.brushWidth
print('brushWidth',width)
print(width.class)
remaining arguments
print(args)
$ python3 tst.py -b 1.2 my_file.png
brushWidth 1.2 ['my_file.png']
Rounding a floating point number to two significant digits
a = round(901/3600,2)
Command line tips
The command line is your friend and should be used for little tests. Moreover, you can print an object without even a print statement.
>>>a =[1,’hi’,3]
>>>a
Going from byte object to string
s_b = b’string’
s = s_b.decode(‘utf-8’)
Test if object is a string
if type(thisobject) == str: print(“It is a string”)
Python as a calculator
I always used python command line as a calculator, even ebfore I knew the language syntax! It’s very handy.
>>> 5 + 6/23
Breaking out of a for loop
Use the continue statement after testing a condition to skip remaining block and continue onto next iteration. Use the break to completely skip out of this loop. Note that break and continue only apply to the innermost loop!
Iterator to get key value pairs out of a dict
>>>a = {‘hi’:’there’,’hi2′:12}
>>>for k,v in a.items():
>>> print(‘key,value’,k,v)
Executing shell commands
import os
os.system(“ls -l”)
But, to capture the output, you can use the subprocess package.
Generate (pseudo-)random numbers
import random
a = random.random()
Accessing environment variables
os.environ[‘ENV_TOKEN’]
Handling glob (wildcards) in your shell command
import glob
for query_results_file in glob.glob(os.path.join(dir_path_files,OSpattern)): print(“query_results_file”,query_results_file)
But, if you want the results in the same order as the shell gives, put a sorted() around that. Otherwise the results come out randomly.
JSON tips
Python is great for reading and writing JSON files.
# Load inventory file
with open(dir_path_files + inventory_file) as inventory_file:
inventory_json = json.load(inventory_file)
sitenoted={'gmtOffset':jdict["gmtOffset"],'timezoneId':jdict["timezoneId"]}# update inventory with custom field Site Notes – put GMT – make sitenoted pretty using json.dumps
sitenote=json.dumps(sitenoted,indent=4)
print("sitenote",sitenote)Convert a string which basically is in json format to a Python data structure
import json txt_d = json.loads(response.text)
Test for null in JSON value
You may see “mykey”:null in your json values. How to test for that?
if my_dict[mykey] == None: continue
Validate a json file
python3 -m json.tool JSON_FILE
RegExes
Although supported in Python, seems kind of ugly. Many RegExes will need to prefaced with r (raw), or else you’ll get yourself into trouble, as in
import re
r'[a-z]{4}.\s*\w(abc|def)’
if re.search(‘EGW-‘,locale): continue
b = re.sub(‘ ‘,’-‘,locale)
b = re.split(r’\s’,’a b c d e f’)
Minimalist URL example
import urllib.request
res = urllib.request.urlopen(‘https://drjohnstechtalk.com/’).read()
Function arguments: are they passed by reference or by value?
This section needs more research and may be inaccurate or simply wrong! By reference for complex objects like a dict (not sure about a list), but by value for a simple object like a Boolean! I got burned by this. I wanted a Boolean to persist across a function call. In the end I simple stuffed it into a dict! And that worked. But python doesn’t use that terminology. But it means you can pass your complex data structure, say a list of dicts of dicts, start appending to the list in your function, and the calling program has access to the additional members.
Print to a string a la sprintf
In python 3.6 and later you have the f-format which is way cool. Stuff between curly braces gets evaluated in place. Say a = 3 and b = ‘man’, then
str = f"first some text mixed with value of a, which is {a} and the text of b, which is {b}"
So no need to paste a string together with awkward combos of strings, plus signs and variables!
Putting the concepts to work: print out n randomly sampled lines from a file
import random,sys
def random_line(fname):
lines = open(fname).read().splitlines()
return random.choice(lines)
file = sys.argv[1]
no_lines = int(sys.argv[2])
for n in range(no_lines):
print(random_line(file))
Excel files
I’ve been using the package openpyxl quite successfully to read and write Excel files but I see that pandas also has built-in functions to read spreadsheets.
Math
numpy seems to be the go-to package.
Can a keyword be a variable?
Yes. Here’s an example.
timeunit = ‘days’
numbr = 3
datetime.now() + timedelta(**{timeunit: numbr})
Python and self-signed certificates, or certificates issued by private CAs
I updated this blog article to help address that: Adding private root CAs in Redhat or SLES or Debian.
Write it with style
Use flake8 to see if your python program conforms to the best practice style. In CentOS I installed flake8 using pip while in Debian linux I installed it using apt-get install flake8.
Does it at least pass the compiler – check syntax without running it
Install pyflakes: pip3 install pyflakes. Then
pyflakes your_script.py
Can I modify a Python script while its running?
Sure. No worries. It is safe to do so.
Print statement prints everything twice
This happens if you unfortunately named your program the same as a module you are importing. In this situation the program imports itself and runs twice. Rename your program something different!
Create virtual environment for portability
I like to call my virtual environment venv.
virtualenv venv
Use this virtual environment
source ./venv/bin/activate
List all the packages in this virtual environment
Good portable development style would have you install the minimal set of packages in your virtual environment and then build a requirements.txt file:
pip3 freeze > requirements.txt
Leave this virtual environment
deactivate
Conclusion
I’ve written down some of my favorite tips for using python effectively.