How to get "i" th bit of a number in Python

 Background

In various problem-solving techniques, we often need to find the i'th bit of a number. For eg., lets say we have the number 5 , it's binary notation is 101. If I want to get 0th bit it is 1, 1st bit is 0 and 2nd bit is again 1. In this post, we will see how to get the ith bit for a given number in Python.

Use of getting i'th bit

Before we see how to get ith bit let's try to understand a problem-solving use case where we can use this logic. Consider the following problem statement:

Given an array of integers, "data" , find all possible subsets of data, including the empty set.

The question is basically asking us to find all possible subsets of a given array of integers.

For e.g, consider input as [2, 5, 7] the possible combinations are

• i=0: {}
• i=1: {2}
• i=2: {5}
• i=3: {2,5}
• i=4: {7}
• i=5: {2,7}
• i=6: {5,7}
• i=7: {2,5,7}

The total combinations possible for a given list is 2^N where N is the length of the array. In the above case since N=3 (length of input array) the expected combinations are 2^3 = 8 which is exactly what you see above.

Now to compute these we iterate from i=0 to 2^N-1 which in this case will be from 0 to 7.

For each of these numbers, we can find the binary notation and then check if the binary bit is 1 for a given index and include it in the combination.

For eg., let's take i=3, binary notation is 110 which means we should consider numbers from 1st and 2nd position of the original array. Since the original array was [2,5,7] and we are considering 1st and 2nd position arrays the subset is {2,5} which is what you see in above combination list as well.

For all "i" values subsets computed are shown below:

Now let's see the Python code to get the ith bit for a number.

NOTE: In python for computing 2^N we do 2 ** N, In Java you would do Math.pow(2, N).

How to get "i" th bit of a number in Python

We will test the same example we took above when num =3 and we want to get all bits of it to decide which position data from the original array to include in the current subset. Python code for the above is as follows:

def get_ith_bit(num, i):
    # Shift the operand specified number of bits to the left
    temp = (1 << i)
    temp = temp & num
    if temp == 0:
        return 0
    return 1

print(get_ith_bit(3, 0))
print(get_ith_bit(3, 1))
print(get_ith_bit(3, 2))

Output is:

1

1

0

which is 110 in binary as we saw before as well.

Complete code

The complete code for the problem statement is as follows:

PS: Given an array of integers, "data" , find all possible subsets of data, including the empty set.

def get_ith_bit(num, i):
    # Shift the operand specified number of bits to the left
    temp = (1 << i)
    temp = temp & num
    if temp == 0:
        return 0
    return 1

def find_all_subsets(data):
    subsets = []

    if not data:
        return [[]]
    else:
        combinations = 2 ** len(data)
        for i in range(0, combinations):
            subset = set()
            for j in range(0, len(data)):
                if get_ith_bit(i, j) == 1 and data[j] not in subset:
                    subset.add(data[j])

            if i == 0:
                subsets.append([])
            else:
                subsets.append(list(subset))
    return subsets

print(find_all_subsets([2,5,7]))

Output:

[[], [2], [5], [2, 5], [7], [2, 7], [5, 7], [2, 5, 7]]

Related links

• Slicing strings in Python (OSFG)
• Different ways to iterate over a list in Python (OSFG)
• Working with dictionaries in Python (OSFG)
• Working with heaps in Python (OSFG)

Working with heaps in Python

 Background

As we all know a heap is a complete binary tree that satisfies the heap property:

• Min heap: The value of children is greater than or equal to the parent node. This means the root node of a min heap stores the lowest value or minimum value data.
• Max heap: The value of children is smaller than or equal to the parent. This means that the root node of a max heap stores the highest value or the maximum value data.

Heaps are used to implement priority queues for various use cases. See the below diagrams on what constitutes a valid and invalid heap (Credit: GeeksForGeeks)

Valid min heaps

Invalid min heaps

Valid max heaps

Invalid max heaps

It is an interesting read on how to insert in a heap, how to remove from the heap, how to heapify an array, how to use an array to represent a heap (The above diagrams show binary tree representation), how to use heaps for implementing priority queue etc. 

The time complexity for various heap operations is as follows:

Working with heaps in Python

Now that we brushed upon heap data structure above let's get to the original purpose of this post, which s to understand how to work with heaps in Python.

In Java, you would usually use PriorityQueue implementation to work with heaps. In Python we use python’s inbuilt library named heapq.

Look at the following code:

import heapq

data = [3, 5, 9, 14, 4, 24, 2]
print(f"Original data: {data}")
heapq.heapify(data) # Create min heap
print(f"After heapify: {data}")
heapq.heappush(data, 1)
print(f"After pushing to heap: {data}")
heapq.heappop(data)
print(f"After poping from heap: {data}")

It Prints:

Original data: [3, 5, 9, 14, 4, 24, 2]
After heapify: [2, 4, 3, 14, 5, 24, 9]
After pushing to heap: [1, 2, 3, 4, 5, 24, 9, 14]|
After poping from heap: [2, 4, 3, 14, 5, 24, 9]

As you can see we initially has a simple list called data which we then converted into a min heap (default heap behavior) using heapify method (O(N) time complexity). We then pushed an element to the min heap (O(Log(N)) time complexity) and then poped one which remoes the element from the root - minimum element in this case of min heap (Also has time complexity O(Log(N)))

NOTE: If you want max heap implementation you can just negate the data and push it in the heap.

NOTE: Heap elements can be tuples. This is useful for assigning comparison values (such as task priorities) alongside the main record being tracked. 1st element in the tuple is use for the comparison/priority.

See below code of how we can add tuple in the heap:

import heapq

data = []
heapq.heappush(data, (3, "China"))
heapq.heappush(data, (1, "India"))
heapq.heappush(data, (2, "USA"))
print(data)

It prints:

[(1, 'India'), (3, 'China'), (2, 'USA')]

Related Links

• HeapSort in Java (OSFG)
• Sorting Techniques (OSFG)
• Heap data structure (GeeksForGeeks)
• Learning to Love Heaps (Medium)
• Heap implementation as Priority queue in Java (SO)
• heapq python library(python docs)

Working with dictionaries in Python

 Background

In this post, we will see how we can use dictionaries and specifically how we can enable a custom class in Python to be used as a dict key. I am originally from a Java background and have written posts before on how HashMap/HashTable works in Java (See Related links section towards the end of this post). 

If someone is new

• Dictionaries are data structures that store data of format key: value pairs
• Data stored in dict is unique (does not allow duplicate keys), mutable (can edit), and ordered

NOTE: As of Python version 3.7, dictionaries are ordered. In Python 3.6 and earlier, dictionaries are unordered.

Dictionaries in Python

You can initialize a dictionary in Python using {} or dict keyword

• my_dict = {}
• my_dict_1 = {"Name": "Aniket", "Country": "India"}
• my_dict_2 = dict(name="Aniket")

You can print these and see the dictionary. It will print

{}

{'Name': 'Aniket', 'Country': 'India'}

{'name': 'Aniket'}

A normal dict like above is unordered which means the order in which you insert data in dict is not maintained, when you iterate over items in dict it may give you a different order. This is the same as HashMap in Java. However, similar to LinkedHashMap we have OrderedDict in Python (Please note that as of Python version 3.7, dictionaries are ordered. In Python 3.6 and earlier, dictionaries are unordered).

You can also merge one map to another using the update method. Remember key's of a dict are unique and similar to HashMap if you try to insert a key that is already present in the dict then it is going to override the key data with the new value (See below code).

my_dict = {}
my_dict["name"] = "Aniket"
my_dict["Country"] = "India"
for key in my_dict:
    print(f"{key} : {my_dict[key]}")
print("--------------")
others_dict = {}
others_dict["name"] = "John"
others_dict["State"] = "NYC"
for key in others_dict:
    print(f"{key} : {others_dict[key]}")
print("--------------")
my_dict.update(others_dict)
for key in my_dict:
    print(f"{key} : {my_dict[key]}")

Output is:

name : Aniket

Country : India

--------------

name : John

State : NYC

--------------

name : John

Country : India

State : NYC

See how name key value was overridden by new dict key, Country key was not overridden so it stated the same and finally a new key called State was added.

Using a custom class as a key in dict

In Java for a new class, you would typically override equals and hashcode method to make it work, in Python we have to do something similar as well.

Let's create an Employee class with just name and age, try to create objects out of it, push to dict and print it.

class Employee:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __str__(self):
        return f"Employee with name: {self.name} & age: {self.age}"


my_dict = {}
e1 = Employee("Aniket", 30)
e2 = Employee("Aniket", 30)
my_dict[e1] = 1
my_dict[e2] = 2
for key, value in my_dict.items():
    print(f"{key} : {value}")

Output is:

Employee with name: Aniket & age: 30 : 1

Employee with name: Aniket & age: 30 : 2

As you see from output it took the same key as different keys in dict and added separate entries. Since the employee (uniquely defined by name and age) is same we want just one entry in dict corresponding to a unique name and age. In this case it should have overridden value from 1 to 2 and just printed single entry with value 2. Let's see how we do that.

For the above to work similarly to Java (where we override equals and hashcode), in Python we override the two methods below

• __eq__()
• __hasg__()

So your code will now look like

class Employee:
    def __init__(self, name, age):
        self.name = name
        self.age = age

    def __str__(self):
        return f"Employee with name: {self.name} & age: {self.age}"

    def __eq__(self, other):
        return (self.name, self.age) == (other.name, other.age)

    def __hash__(self):
        return hash((self.name, self.age))

my_dict = {}
e1 = Employee("Aniket", 30)
e2 = Employee("Aniket", 30)
my_dict[e1] = 1
my_dict[e2] = 2
for key, value in my_dict.items():
    print(f"{key} : {value}")

This now prints:

Employee with name: Aniket & age: 30 : 2

 which is in line with our expectations.

See the below diagram for all methods supported by Python dict

Related Links

• Understanding how hashing (HashMap and HashSet) work in Java (OSFG)
• Differences between HashMap and Hashtable in Java? (OSFG)
• OrderedDict in Python (GeeksForGeeks)
• Object of custom type as dictionary key(SO)

Different ways to iterate over a list in Python

 Background

In this post, we will see the different ways to iterate over a list in Python.

Different ways to iterate over a list in Python

Using for loop

A simple way to iterate over a list is to use a for loop.

  
names = ["A", "B", "C"]
for name in names:
    print(name)

This prints

A

B

C

Using for loop with range

You can also use a for loop with range if you want to access the list with it's index

  
names = ["A", "B", "C"]
for name in names:
    print(name)

This also prints

A

B

C

Using enumerate

If you want index and value both then you can use enumerate as follows

  
names = ["A", "B", "C"]
for idx, name in enumerate(names):
    print(f"{name} at index {idx}")

This prints

A at index 0

B at index 1

C at index 2

Using while loop

You can also use a while loop for iterating over a list as follows

  
names = ["A", "B", "C"]
i=0
while i<len(names):
    print(names[i])
    i = i + 1

This prints:

A

B

C

List Comprehension

List comprehension is a more concise way to iterate over a list.

  
names = ["A", "B", "C"]
[print(name) for name in names]

This prints:

A

B

C

If you print above list it will print [None, None, None] as you are not creating any new element for storing in new list on iterating the original list

NOTE: This creates a new list and is not a recommended way to iterate over a list. You can use this if you have a usecase to create a new list from existing one with filtering or modifying the original list.

Related Links

• Slicing strings in Python (OSFG)
• Stack and Queues in Python (OSFG)
• Working with classes in Python(OSFG)
• Static methods vs class methods in Python(OSFG)

Slicing strings in Python

 Background

Working with string is very common in any programming language and what comes in handy in Python is slicing. In this post, we will try to understand how slicing works for strings in Python.

Slicing in Python

The syntax for slicing is : Object [start:stop:step] where

• "start" specifies the starting index of a slice 
• "stop" specifies the ending element of a slice (not included)
• "step" specifies the number of elements to jump/step on between start and stop.

Slicing can be done with negative indexes as well and if start/stop is negative that the indexes are counted from the end backward (See diagram below to understand the indexes - positive & negative).

See a few examples below

  
text = "ABCDE"
print(text[0:3])
print(text[0:3:2])
print(text[-4:-1])
print(text[-4:-1:2])

This prints:

ABC

AC

BCD

BD

You can also have the step as a negative number which will essentially consider elements backwards (reversing the data structure). See the following examples (more details on using this to reverse data structure are added in a later section).

  
text = "ABCDE"
print(text[3:0:-1])
print(text[3:0:-2])
print(text[::-1])
print(text[-1:-5:-1])

This prints:

DCB

DB

EDCBA

EDCB

NOTE: text[::] or text[:] is going to create a copy of the existing data structure.

Reversing Elements of Data Structure

You can use a negative step to reverse the elements of the following data structures

  
# list
l = ["a", "b", "c"]
print(l)
print(l[::-1])

# string
s = "abc"
print(s)
print(s[::-1])

# tuple
t = ("a", "b", "c")
print(t)
print(t[::-1])

This prints:

['a', 'b', 'c']

['c', 'b', 'a']

abc

cba

('a', 'b', 'c')

('c', 'b', 'a')

Cheat Sheet

• a[start:stop]  # items start through stop-1
• a[start:]      # items start through the rest of the array
• a[:stop]       # items from the beginning through stop-1
• a[:]           # a copy of the whole array

We can also pass step to the slicing which determines how many steps to jump for slicing

• a[start:stop:step] # start through not past stop, by step

Step can also be a negative number

• a[::-1]    # all items in the array, reversed
• a[1::-1]   # the first two items, reversed
• a[:-3:-1]  # the last two items, reversed
• a[-3::-1]  # everything except the last two items, reversed

Related Links

• How Slicing works in Python (SO)

Understanding descriptors in Python

 Background

As you know, Python does not have a concept of private variables or getters/setters. In one of the previous posts, we saw the use of property to achieve something similar. In this post, we will examine the concept underlying the property functionality, called descriptors. Using the descriptors is the pythonic way to handle attributes of a class. Descriptors are the mechanisms behind properties, methods, static methods, class methods, and super().

Understanding descriptors

Before we see any let's take an example to see why we need descriptors. Consider a simple Employee class below:

class Employee:
    def __init__(self, name):
        self.name = name


emp = Employee("Aniket")
print(emp.name)

For simplicity, it just has one instance variable called "name". In real life you would like to have some check when you set the name of Employee to something - like you might want to ensure it at least has one character, it can have a maximum of 10 chars, etc. We saw how to do this via properties in the last post, in this post we will see how to use descriptors which are the basis of property as well.

class Name:
class Name:
    def __set__(self, instance, value):
        print("Invoking __set__ on Name")
        if not isinstance(instance, Employee):
            raise ValueError("Name descriptor is to be used with Employee instance only")
        if len(value) < 1 or len(value) > 10:
            raise ValueError("Name cannot be less than 1 char or more than 10 char")
        instance._name = value

    def __get__(self, instance, owner):
        print("Invoking __get__ on Name")
        if not isinstance(instance, Employee):
            raise ValueError("Name descriptor is to be used with Employee instance only")
        return instance._name


class Employee:
    name = Name()

    def __init__(self, name):
        self._name = None
        self.name = name


emp = Employee("")
print(emp.name)


emp = Employee("Aniket")
print(emp.name)
emp.name = "Abhijit"
print(emp.name)

The above code defines a descriptor called "Name" and uses it to manage attributes for the Employee class instead. Above prints:

Invoking __set__ on Name

Invoking __get__ on Name

Aniket

Invoking __set__ on Name

Invoking __get__ on Name

Abhijit

You can play around passing names in the constructor as 

• "Aniket" - Works fine. print Aniket
• "" - Fails & prints ValueError: Name cannot be less than 1 char or more than 10 char
• "Aniket Thakur" & prints ValueError: Name cannot be less than 1 char or more than 10 char

See how we now have more granular control over the Name attribute of the Employee class. That's the power of descriptors. 

Notice how emp.name = "Abhijit" works. Normally it would have set name attribute of Employee class to a string "Abhijit" but since in this case it is a descriptor it called __set__ dunder / magic method of the corresponding descriptor class.

Descriptor protocol

A class will be descriptor if it has one of the following methods:

• __get__(self, obj, type=None) -> value

• __set__(self, obj, value) -> None

• __delete__(self, obj) -> None

Define any of these methods and an object is considered a descriptor and can override default behavior upon being looked up as an attribute.

• If an object defines __set__() or __delete__() , it is considered a data descriptor. 
• Descriptors that only define __get__() are called non-data descriptors

NOTE: Data descriptors always override instance dictionaries.

The example we saw above was of a data descriptor. Consider following example

class Name:

    def __get__(self, instance, owner):
        print("Invoking __get__ on Name")
        if not isinstance(instance, Employee):
            raise ValueError("Name descriptor is to be used with Employee instance only")
        return instance._name


class Employee:
    name = Name()

    def __init__(self, name):
        self._name = name


emp = Employee("Aniket")
print(emp.name)
emp.name = "Abhijit"
print(emp.name)

Here we do not have a __set__ method and consequently, it is not a data descriptor hence we can override it with instance dictionaries. Above prints

Invoking __get__ on Name

Aniket

Abhijit

Notice how it involved __get__ exactly once and when we set it to "Abhijit" it actually replaced the name from a data descriptor instance to a normal string stored in the instance dictionary.

One last thing is that as you see above a class is data descriptor if it has __set__ or __delete__ so if you do not have __set__ but just have delete then it is still a data descriptor and you cannot override in the instance dictionary.

class Name:

    def __get__(self, instance, owner):
        print("Invoking __get__ on Name")
        if not isinstance(instance, Employee):
            raise ValueError("Name descriptor is to be used with Employee instance only")
        return instance._name

    def __delete__(self, instance):
        del instance._name


class Employee:
    name = Name()

    def __init__(self, name):
        self._name = name


emp = Employee("Aniket")
print(emp.name)
emp.name = "Abhijit"
print(emp.name)

The above code will fail by printing

Invoking __get__ on Name

Aniket

Traceback (most recent call last):

  File "/Users/aniketthakur/PycharmProjects/HelloWorld/descriptors.py", line 29, in <module>

    emp.name = "Abhijit"

AttributeError: __set__

and that is because it could not find a __set_ method.

Related Links

• Working with classes in Python (OSFG)

Stack and Queues in Python

 Background

In the last few posts, we saw how to work with classes in Python and how it can be used in an object-oriented way. In this post, I will explain how you use a stack and queue implementation in Python. This is needed if you are doing problem-solving questions using Python or in general using Python for coding.

Remember:

• Stack:  "Last in First out"
• Queue:  "First in First out"

Using stack in Python 

Stack works on logic "Last in First out".Stack is pretty straightforward. You can use a standard Python list as your stack. See below the Python code

stack = []
stack.append("Aniket")
stack.append("Abhijit")
stack.append("Awantika")
print(stack)
print("Pushing Anvi to Stack")
stack.append("Anvi")
print(stack)
print(f"Popped {stack.pop()}")
print(stack)
print(f"Popped {stack.pop()}")
print(stack)

This prints the following:

['Aniket', 'Abhijit', 'Awantika']
['Aniket', 'Abhijit', 'Awantika', 'Anvi']
Popped Anvi
['Aniket', 'Abhijit', 'Awantika']
Popped Awantika
['Aniket', 'Abhijit']

As you can see initial values in the stack were "Aniket", "Abhijit" & "Awantika". Then we pushed "Anvi" to the stack. Then we popped which gave us "Anvi", then we popped again and we got "Awantika". Remember stack is "Last in First out" and the above behavior is in line with it.

NOTE: The list works better at the end of the list, so append and pop operations happen in O(1) time complexity. So if you need a stack implementation go ahead and use a list. You could also use dequeue implementation for a stack with the same O(1) complexity - It has append and pop methods similar to queue. (That's not the case for the queue - see below).

Using Queue in Python

The queue is "First in First out". You could use list implementation in Python for queue as well but remember the note from above - The list works better at the end of the list. It does not work well with the start of the list, so if you try to remove the element from the beginning of the list, the list needs to be shifted back one place making the time complexity O(N).

You should use dequeue implementation from the collections module for O(1) push/pop operations in the queue. I will show both use cases below. 

Using a list for queue implementation (Not recommended)

See below the Python code:

queue = []
queue.append("Aniket")
queue.append("Abhijit")
queue.append("Awantika")
print(queue)
print("Pushing Anvi to Queue")
queue.append("Anvi")
print(queue)
print(f"Popped {queue.pop(0)}")
print(queue)
print(f"Popped {queue.pop(0)}")
print(queue)

The output in this case is:

['Aniket', 'Abhijit', 'Awantika']
Pushing Anvi to Queue
['Aniket', 'Abhijit', 'Awantika', 'Anvi']
Popped Aniket
['Abhijit', 'Awantika', 'Anvi']
Popped Abhijit
['Awantika', 'Anvi']

As you can see this time it popped "Aniket" & "Abhijit" which are first in the list (Remember - "First in First out").

Using dequeue for queue implementation (recommended)

A deque is a generalization of the stack and a queue (It is short for "double-ended queue"). Deques support thread-safe, memory-efficient appends and pops from either side of the deque with approximately the same O(1) performance in either direction. (See documentation for more details)

NOTE: Though list objects support similar operations, they are optimized for fast fixed-length operations and incur O(n) memory movement costs for pop(0) and insert(0, v) operations which change both the size and position of the underlying data representation.

See below the Python code:

from collections import  deque

queue = deque([])
queue.append("Aniket")
queue.append("Abhijit")
queue.append("Awantika")
print(queue)
print("Pushing Anvi to Queue")
queue.append("Anvi")
print(queue)
print(f"Popped {queue.popleft()}")
print(queue)
print(f"Popped {queue.popleft()}")
print(queue)

Output is the same as that of above (as using list for queue) but internally the operation is O(1) in case of dequeue unlike O(N) in case of list.

['Aniket', 'Abhijit', 'Awantika']
Pushing Anvi to Queue
['Aniket', 'Abhijit', 'Awantika', 'Anvi']
Popped Aniket
['Abhijit', 'Awantika', 'Anvi']
Popped Abhijit
['Awantika', 'Anvi']

Add in the comments if you have any questions.

Related Links

• Working with classes in Python (OSFG)
• Static methods vs class methods in Python (OSFG)
• Decorators in Python (OSFG)
• Python dequeue official documentation

Static methods vs class methods in Python

 Background

In the last post, we saw how to work with classes in Python. We saw private methods, instance variables, class variables, etc. In this post, we are going to explore static and class methods in Python. If you are from a Java background you would use class methods or static methods interchangeably but that is not the case with Python. 

Class methods in Python

• Class methods are your methods that are at class level (they belong to the class & not individual instances).
• They will get the reference to the class as the implicit first argument (similar to how instance methods get a reference to self)
• Class methods have access to class variables /state and can modify them

Consider the following  example:

class Employee:

    BASE_INCOME = 10000

    def __init__(self, name :str) -> None:
        self.name = name

    @classmethod
    def get_base_income_with_cls_method(cls) -> int:
        return cls.BASE_INCOME

    @classmethod
    def set_base_income_with_cls_method(cls, income: int) -> None:
        cls.BASE_INCOME = income


e = Employee("Aniket")
print(Employee.BASE_INCOME)
print(Employee.get_base_income_with_cls_method())
print(e.get_base_income_with_cls_method())
Employee.set_base_income_with_cls_method(100)
print(Employee.BASE_INCOME)
print(e.get_base_income_with_cls_method())
e.set_base_income_with_cls_method(10)
print(Employee.BASE_INCOME)
print(e.get_base_income_with_cls_method())

It prints:

10000

10000

10000

100

100

10

10

Notice

• You declare a method as a class method using @classmethod decorator
• The class method automatically gets an implicit argument cls which is a reference to the class.
• BASE_INCOME is defined outside __init__ and hence is a class variable.
• We have defined two class methods get_base_income_with_cls_method and set_base_income_with_cls_method which get and set that class variable (You actually do not need this and can do it directly with class reference E.g., Employee.BASE_INCOME=X etc. but have added to show how the class method has access to class variables via cls - 1st implicit argument).
• Changes made to the state by class methods are reflected across all instances of the class.

This is what you would know as a static method in Java.

Now let's see what a static method in Python looks like:

Static methods in Python

• A static method does not get an implicit first argument (as class methods did). 
• Similar to class methods a static method is also bound to the class (and not the object of the class) however this method can’t access or modify the class state. 
• It is simply present in a class because it makes sense for the method to be present in class (rather than at individual instance level, for eg., some utility methods).
• You can create a static method using @staticmethod decorator.

Consider the following example:

class Employee:

    def __init__(self, name: str) -> None:
        self.name = name

    @staticmethod
    def is_same_name(name1: str, name2: str) -> bool:
        return name1 == name2


print(Employee.is_same_name("Aniket", "Abhijit"))
print(Employee.is_same_name("Aniket", "Aniket"))

It prints:

False

True

As you can see these can be used for simple utility methods.

When to use class methods and static methods?

• Class methods are generally used to create factory methods. Factory methods return class objects depending on the usecase.
• Static methods are generally used to create utility functions.

Related Links

• Working with classes in Python (OSFG)

Working with classes in Python

 Background

In the last few posts, we saw how Python works, what are closures, decorators, etc. In this post, I am going to explain how to write a class in Python and work with it. As you might already be aware though Python is used as a scripting language we can use it as an Object-oriented programming language.

Writing classes in Python

Let's start with a simple class and how to use it.

class Employee:
    def __init__(self, name: str, age: int) -> None:
        self.name = name
        self.age = age

    def __str__(self) -> str:
        return f"Employee with name {self.name} and age {self.age}"

e = Employee("Aniket", 32)
print(e)

The above code prints: Employee with name Aniket and age 32

Let's try to understand what the above code does.

• First, we have defined a simple class Employee using a keyword class
• Then we have defined two magic methods (We will see what magic methods are later, for now just imagine these are special methods that are created for some specific intended functionality, wherever you see a method starting with double underscores you know it is a magic method)
• __init__ : __init__() method in Python is used to initialize objects of a class. It is also called a constructor. 
• __str__: The __str__() method returns a human-readable, or informal, string representation of an object. This method is called by the built-in print() , str() , and format() functions. If you don't define a __str__() method for a class, then the built-in object implementation calls the __repr__() method instead.
• Notice how the constructor takes 3 arguments
• self: Every method of a class that is an instance method (& not a class/static method - more on this later) will always have self as 1st argument. It is a reference to the instance created itself (e instance in the case of above).
• name & age: These are 2 new parameters that init takes which means when we want to create an instance of Emploee we need to pass name and age. These are our class instance variables.
• Also, notice how we have provided hints for the type of arguments (E.g., the name being of str type) and return type of method (-> None). These are optional but good to have. These are called annotations and are used only as hints (Nothing is actually enforced at run time)
• Lastly, we have just created an instance of Employee and printed it. Notice unlike Java you don't need to use a new keyword here. Since we defined our own implementation of __str__ it printed the same when we printed the object (Else it would have printed something like <__main__.Employee object at 0x000001CB20653050>).

Class variables and private methods

• Unlike Java, we do not define the type of variables (It's dynamic). 
• If you have declared variables inside __init__ using self then those are instance variables (specific to the instance you created).
• If you declare variables outside __init__ then those belong to the class (are called class variables) and can be accessed as ClassName.variable.
• Similarly, there are no access modifiers like Java
• If you want a method or variable to be private just start it with a single underscore. 
• E.g., _MY_PRIVATE_ID = "ABC"
• Please note this is more of a convention and nothing is enforced at run time (similar to the hints annotations I explained above). If you see such variables/methods then you should use caution before you use them as public variables/methods.

Let's look at an example to understand class variables and private methods.

class Employee:
    _BASE_INCOME = 10000

    def __init__(self, name: str, age: int) -> None:
        self.name = name
        self.age = age
        self.income = None

    def _calculate_income(self) -> int:
        return Employee._BASE_INCOME * self.age

    def get_income(self) -> int:
        return self._calculate_income()


e = Employee("Aniket", 32)
print(Employee._BASE_INCOME)
print(e._BASE_INCOME)
print(e.get_income())
print(e._calculate_income())

Above prints:
10000
10000
320000
320000

A couple of points to note here

• _BASE_INCOME since it is defined outside __init__ is a class variable. It can be accessed using class - Employee._BASE_INCOME. As you can see this starts with an underscore and is a private variable.
• _calculate_income is a private method defined to calculate income based on _BASE_INCOME and age. There is a public method get_inome exposed that uses the private method to compute the actual income.
• As you can see even though both variables and methods mentioned above are private you can access them. That is because as I mentioned before python does not enforce these, it's the developers' responsibility to use these with caution.

Now the natural question is how do we add validations to ensure that the instance variables are of the correct type. Let's see that next.

Using Properties

If you have worked on Java before you know that you can have getters and setter for instance variables. In Python, you can do the same using properties. See the below code for example:

class Employee:
    _MAX_AGE = 150

    def __init__(self, age: int) -> None:
        self._age = None
        self.age = age

    @property
    def age(self) -> int:
        return self._age

    @age.setter
    def age(self, value: int) -> None:
        if value > Employee._MAX_AGE:
            raise ValueError(f"Age cannot be more that {Employee._MAX_AGE}, Passed: {value}")
        self._age = value


e = Employee(200)

This prints:

Traceback (most recent call last):
...
  File "C:\Users\Computer\Documents\python\test.py", line 15, in age
    raise ValueError(f"Age cannot be more that {Employee._MAX_AGE}, Passed: {value}")
ValueError: Age cannot be more that 150, Passed: 200

Process finished with exit code 1

Notice

• How we are using a corresponding private variable(_name) to track the actual value stored internally and using the name as an interface to be used publically.
• When you try to create an Employee instance by passing age as 200 it actually calls the setter method which checks it is more than 200 and throws a ValueError.

Lastly, let's see a compact way to write the classes using dataclass decorator.

Using the dataclass decorator

You can use the dataclass decorator to write a class compactly. See the following example.

from dataclasses import dataclass, field


@dataclass
class Employee:
    name: str
    age: int
    BASE_INCOME: int = field(init=False, repr=False)


e = Employee("Aniket", 33)
print(e)

Above prints: Employee(name='Aniket', age=33)

Note here

• Notice the import of dataclass
• Once you decorate your class with @dataclass you get a few things for free
• It automatically creates instance variables with variables defined with annotation (name and age in the above case)
• You do not need to define an __init__ method for this.
• If you do not provide annotation hints then it automatically considers it as class variables.
• If you want to use hints annotations and still want it to be a class variable then you can initialize it with the field method by passing init and repr arguments as False. Eg., in the above case BASE_INCOME is not a class variable and will not be used in init or repr mafic methods.

Related Links

• Installing Python & related tools on Windows (OSFG)
• How Python works? (OSFG)
• Decorators in Python (OSFG)
• Understanding Python closures (OSFG)

Installing Python & related tools on Windows

Background

In this post, we will see how we can install Python & related tools on the Windows Operating System. 

Installing Python

Let's start with installing Python itself. For this, you can go to the Python official website and download the Windows installer (link). The version at the time I am writing this is Python 3.12.2. You can download the installer based on your Windows architecture. I have Windows 64-bit, so I downloaded the corresponding installer.

You can download and execute the installer. It will install Python and add it to your PATH which will let you execute the Python command on the cmd console or powershell terminal. You can check the Python version by following the command

• "py --version"

You can also start the py console by just typing "py" in the console.

You can also create a python file with the extension ".py", add python code, and execute it with the command

• py filename.py

Installing other tools

It is important we install a few other tools to work efficiently with Python.

Pycharm

Let's start with installing IDE. For any programming language, it's important we install an IDE to improve the convenience of use & productivity. I prefer using Pycharm but you could use other alternatives like Eclipse.

You can install the community version of Pycharm from their official site (link). You can scroll down to download and install the community version which is free to use. You can use inbuilt tools like create python project/file, run code, etc. to build and run your code.

Jupyter Notebook

Next, I would recommend installing Jupiter Notebook. You can use Anaconda distribution to use it (Just install the Anaconda installer and you can use a notebook from it). You can download the installer from here. You can use it as mentioned here. It helps execute code snippets for quick testing.

Recommended links

• How to Install IPython Jupyter Notebook on Ubuntu (OSFG)
• How Python work? (OSFG)

Decorators in Python

Background

In the last post, we saw what are closures in Python. In this post, we will see what are decorators in Python. Closures are heavily used by Decorators. Let's see how. (You can also check my youtube video).

 Decorators in Python

As we know already everything in Python is an Object. A decorator is an object that is used to modify a function or a class. The function decorator takes a reference to the decorated function and returns a new function. It will internally call the actual decorated/referenced function.

You can also have a class as a decorator instead of a function. When we decorate a method with a class, that function becomes an instance method of that class.

In either case, the original code is not changed. 

Let us say we have a function that makes a remote API call and we want to monitor the start and end time for that API. You have a method as follows:

def make_api_call(*params):
    # Simulate API call
    time.sleep(3)
    print("Done")

I have just added a sleep of 3 seconds to simulate the actual API call. So now if we want to monitor the start and end time of this API, we can modify this function to print time before and after the API call, but it is not correct to modify existing functions. This is where the decorator comes into the picture.

from datetime import datetime
import time

def monitor_performance(func):
    
    def wrapper_func(*params):
        print("Making API call with params {} at {}".format(params, datetime.now()))
        func(params)
        print("Finishing API call with params {} at {}".format(params, datetime.now()))
              
    return wrapper_func

@monitor_performance
def make_api_call(*params):
    # Simulate API call
    time.sleep(3)
    print("Done")
              
make_api_call("param1", "param2")

When you execute this code it prints the following output.
Making API call with params ('param1', 'param2') at 2020-05-24 22:31:19.480543
Done
Finishing API call with params ('param1', 'param2') at 2020-05-24 22:31:22.484090

You can notice how we decorated our actual method make_api_call with a custom decorator monitor_performance. Also, you must have noticed our decorator function used a closure - another internal method called wrapper_func to actually monitor start and end time.

As I mentioned before the decorator can be used to modify the actual method and internally calls the actual method. In this case, before we call the actual method we print start and end time.

You would have also noticed that the parameters passed to make_api_call are automatically passed to our wrapper function as we are returning this wrapper function from the decorator. Also, notice how we have declared decorator for our function using '@' notation.

Using class instead of function for a decorator

The same code can be used with a class as a decorator. As we already know everything in Python is an object and it is callable if it defines the __call__() method.

from datetime import datetime
import time

class monitor_performace:
    def __init__(self, actual_func):
        self.actual_func = actual_func
    
    def __call__(self, *params):
        print("Making API call with params {} at {}".format(params, datetime.now()))
        self.actual_func(params)
        print("Finishing API call with params {} at {}".format(params, datetime.now()))
        

@monitor_performace
def make_api_call(*params):
    # Simulate API call
    time.sleep(3)
    print("Done")
              
make_api_call("param1", "param2")

It prints similar output as before:
Making API call with params ('param1', 'param2') at 2020-05-24 22:39:22.895176
Done
Finishing API call with params ('param1', 'param2') at 2020-05-24 22:39:25.896923

Notes:

• You can also decorate a class with another class
• You can chain decorated as well. Each decorated function/class will be called serially.

Related Links

• Understanding python closure(OSFG)

Understanding python closure

Background

In the last couple of posts, we saw how we can pass multiple arguments in Python functions and what are generators in Python. Next, I want to explain what decorators are in python. Decorator is a very powerful feature of python but in order to explain it, we need to understand closure in Python. You must have heard of Javascript closures, these are kind of similar. We will see this in detail now

Understanding python closure

Python closures are related to nested functions. Consider the following example:

def print_func(text):
    text_to_print = text
    
    def printer():
        print(text_to_print)
        
    return printer

print_function_reference = print_func("Hello World!")
print(print_function_reference)
print_function_reference()

What happens when you execute the above piece of code? It prints:
<function print_func.<locals>.printer at 0x7f21b16d1950>
Hello World!

So what's happening here?
We have defined a function called print_func which takes in a string argument which we like to print. Then this method returns a reference new method called printer() and when you invoke this method(reference) you see your value is printed.

But wait a second? I am good with the part where I get a reference of printer method as seen in output but when I invoke it how does it get the value of text_to_print? It does not seem to be in printer methods scope.
>> This is exactly what we call closure.

A couple of other pointers before we go to the definition of closure:

• printer() function is called a nested function
• A nested function has read-only access to variables defined in the outer scope. 'text_to_print' in this case.
• Such variables are called "non-local" variables.
• Everything in python is an object with set of attributes. Yes, even a function. They all have some common attributes like "__doc__".

So, Closure is a function object that is used to access variables from enclosing scope, even though they are not present in the memory(out of scope). Eg. 'text_to_print' in this case.

They are used to invoke functions not in scope. Eg. printer() in the above case. The scope for printer() function is inside print_func() function yet we can invoke it from outside.

NOTE: You can try deleting print_func() and then invoke print_function_reference(). It will still work, even though its closing function is deleted.

When and Why to use Closures?

As you can see closure helps with data hiding. You don't need to define global variables. That's exactly the primary use case of closures.

They are used in callbacks and decorators. We will have a detailed post on decorators (stay tuned!).

Typically when you have a few methods you can go with closure. However, if you have multiple methods you are better off with a class.

You can also see closure contents as follows:

Related Links

• How to pass variable arguments using *args and **kwargs in Python?(OSFG)
• Difference between yield and return in python(OSFG)
• Hide variable from console access - Javascript Closures and self executing functions
• Youtube video

Difference between yield and return in python

Background

If you have been using Python you would have come across return and yield statements. Both are used to return something from a function. Though both of them are used to return something from the function both serve different purposes. In this post, I will try to explain what those are.

I use Jupyter notebook for running python code snippets. To see how you can install it refer to my earlier post (link in the Related Sections at the end of this post).

Understanding return in Python

Let's try to understand returns first as that's the simplest. The return keyword is used is a similar fashion in multiple programming languages. It is used to return a value to its caller.

A function can have multiple return statements but only the 1st one encountered during its execution flow will be executed and the corresponding value will be returned to the called. No code after the return statement will be executed (dead code) - the method with return statement exists. The return statement is generally the last statement in a function but it could be before as well if we want to skip subsequent code based on a certain condition. 

Consider the following example:

def is_even(num):
    if num is None:
        return False
    elif num%2 == 0:
        return True
    else:
        return False
    
print(is_even(2))
print(is_even(3))
print(is_even(None))

The output is:
True
False
False

As you can see the function is_even has multiple return statements and exactly one will get executed for a single execution call based on the conditions. You could also rewrite it as :

def is_even(num):
    is_even=True
    if num is None or num%2 != 0:
        is_even = False
    return is_even
    
print(is_even(2))
print(is_even(3))
print(is_even(None))

This is a slightly better version than the previous one. we just have one return statement and less code.

Understanding yield in Python

Yield statement also returns the value to the caller but it retains the state. When you call the function again it resumes where it left off (last yield statement). This enabled code to generate a series of data when needed without generating all at once.

Such structures that generate a series of data are called generators in python and yield keywords are used to create generators. Whenever generators need to produce a value they do it using the yield keyword. Consider the following example:

def get_even():
    i=0
    while True:
        i = i + 2
        yield i
    
test = get_even()
print(next(test))
print(next(test))
print(next(test))

This outputs:
2
4
6

As you can see this is a forever going generator. It will keep on generating the sequence. If you do something like below, it will go in an infinite loop:

def get_even():
    i=0
    while True:
        i = i + 2
        yield i
   
test = get_even()
for evennum in test:
    print(evennum)

This outputs an infinite series of even numbers starting from 2. You will have to manually kill the process as it goes into an infinite loop.

If there are limited field statements in the function, the generator will run that many times. If you try to run a generator after that it will raise an Exception.

def get_even():
    i=0
    i = i + 2
    yield i
    i = i + 2
    yield i 

test = get_even()
for evennum in test:
    print(evennum)
next(test)

Output:
2
4
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-40-ef2477e0fee5> in <module>
     10 for evennum in test:
     11     print(evennum)
---> 12 next(test)

StopIteration: 

You can have return statements in a generator as well which simply signifies the end of that generator. No yield statement after the return statement will be respected. Consider the following example:

def get_even():
    i=0
    i = i + 2
    yield i
    return
    i = i + 2
    yield i

    
test = get_even()
for evennum in test:
    print(evennum)
next(test)

This will print:

2
---------------------------------------------------------------------------
StopIteration                             Traceback (most recent call last)
<ipython-input-41-554cedfc9db7> in <module>
     11 for evennum in test:
     12     print(evennum)
---> 13 next(test)

StopIteration: 


As you can see it encountered 1st yield and returned back 2. In the next call, it saw a return statement and terminated the generator. So, calling it again is going to raise an Exception as we saw above. Having return and yield statement was not allowed in python2 but it's allowed in python3 and it simply means the end of the generator.

Summarizing the difference

• Return statements terminate the execution of the method and return the value to the caller whereas yield returns the value to the caller but retains the current state so that the same method can be called again. 
• The yield keyword is used in Python generators. A generator is a special function that returns a generator object to the caller instead of the actual value. 
• Generators save memory as the new value is generated only when the generator is called. If the same thing you had to do with the return, you would need to compute the series, store all in an array and return the array once.
• As generators with the yield keyword resume the execution all local variables are retained whereas in case of return they are destroyed.

Related Links

• How to Install IPython Jupyter Notebook on Ubuntu(OSFG)
• How to pass variable arguments using *args and **kwargs in Python?(OSFG)

How to pass variable arguments using *args and **kwargs in Python?

Background

If you are working with python you must have come across following notations:

• *args
• **kwargs

args and kwargs are just argument names. It can be replaced by any other variable name, but the important part is the syntax and how it is used. If you have come across this you would also know they are used to pass the variable number of arguments. In this post, I will try to explain how they work with some examples.

Note: If you have not installed Jupyter notebook for python, please refer to my earlier blog post: How to Install IPython Jupyter Notebook on Ubuntu

How to pass variable arguments using *args and **kwargs in Python?

Let's take cases of *args and **kwargs one at a time and then we will see some combined examples.

Understanding *args

• *args is used to take a variable number of non-keyworded arguments that are not your formal arguments. 
• arguments passed in *args become iterable. Think of this as a list.

We will understand "non-keyworded" meaning better when we go to **kwargs but for now, let's try to focus on *args.

Consider the following example:

def foo(param1, *param2):
    print("In foo:")
    print(param1)
    print(param2)

And now if you pass:

foo(1,2,3,4)

You will get the output:

In foo: 1 (2, 3, 4)

As you can see argument 1 got mapped to param1 (your formal argument) and the rest for mapped to *param2 (*param2 is your *args. As I mentioned before variable name does not matter). 

You can pass any number of params after 1 and they will be part of param2.

You can even iterate over param2 to print all variables.

def foo(param1, *param2):
    print("In foo:")
    print(param1)
    for no in param2:
        print(no)

Output:

In foo:

1

2

3

4

Understanding *kwargs

• *kwargs is used to take a variable number of keyworded arguments that are not your formal arguments. When I say keyword it means that you pass arguments by providing a name to that variable
• Think of this as a dictionary of variable name and value you passed as arguments to the function.

Consider the following example:

def bar(param1, **param2):
    print("In bar:")
    print(param1)
    print(param2)

And if you pass bar(1,a=2,b=3,c=4) it will output
In bar:
1
{'a': 2, 'b': 3, 'c': 4}

1 which is your formal parameter maps to param1 and rest named parameters go as dict in param2.
Obviously, you cannot pass bar(1, 2, 3, a=2,b=3,c=4)
as it does not know what to do with 2,3,4


Hopefully, now you understand what keyworded arguments are. They are basically named parameters passed in the function call.

You can also iterate it as a dictionary

def bar(param1, **param2):
    print("In bar:")
    print(param1)
    for key, value in param2.items():
        print("{}:{}".format(key,value))

Output:
In bar:
1
a:2
b:3
c:4

Your functions would actually have both *args and **kwargs. So let's try to see a combined example

def foobar(param0, *param1, **param2):
    print("In foobar:")
    print(param0)
    print(param1)
    print(param2)

And now if you call this as  foobar(1,2,3,4,a=1)
you will get the following output:
In foobar:
1
(2, 3, 4)
{'a': 1}

Again 1 is your formal parameter and maps to param0
2,3,4 are your non-keyword params that get mapped to param1
and a=1 is your keyword param that gets mapped to param2

• Note the order is important. *args should always come before **kwargs. 
• Also, there cannot be positional arguments after  **kwargs
• Also, you cannot do something like foobar(1,2,3,4,a=1,5) as it will not know how to map 5.

Hopefully, this clarifies differences between *args and **kwargs. You can play around more in the Jupyter notebook or python terminal if you have installed it (See the link below if you haven't)

Related Links

• How to Install IPython Jupyter Notebook on Ubuntu(OSFG)