Returning value from a thread in python

Background

In this post we will see how python functions can be executed in a new thread and also how can we return some value from it.

NOTE : Python is pass by value similar to java. But for objects it is pass by value of the reference (again just like Java).

Returning value from a thread in python

Create a file called test.py and add the following code to it - 

import threading

print("Starting Test Python Modules");

def threadtarget(array):
    array.append("Hello World!")

array=[]
t = threading.Thread(target=threadtarget, args=(array,))
t.start()
t.join()
print("Value received from thread : " + array[0])
    
print("Terminating Test Python Module");

Explanation :  We are have a simple function threadtarget that accepts a list in it's argument and adds a String called "Hello World!" to it. In our script we are creating a new thread to run the method threadtarget and finally read the value of list we pass to confirm value added by thread to the list is retained.

Execute the script with -

• python test.py

This is captured in below screenshot

 Related Links

• Shutdown hook in Python(OSFG) 
• Capturing python output in a file along with the stdout 

Capturing python output in a file along with the stdout

Background

In last couple of posts we saw how python works and how can we add a shutdown hook in python. Whenever we execute any python script it's output gets logged to the console from which you execute your script. In this post we will see how can we log this output to a file while retaining what gets printed on your console.

Capturing python output in a file along with the stdout

create a file test.py and add following code to it

import atexit
import sys
import datetime
import os

class Logger(object):
    def __init__(self, filename="output.txt"):
        # Initialize the logfile
        self.terminal = sys.stdout
        self.log = open(filename, "a")

        # Write the command and arguments if any
        self.log.write("CMD: " + str(sys.argv) + "\n")
    def __del__(self):
        self.log.flush()
        del self.log
    def write(self, message):
        self.terminal.write(message)
        self.log.write(message)
    def isatty(self):
        return True
    def flush(self):
        self.log.flush()

now = datetime.datetime.utcnow()
if not os.path.exists(os.getcwd() + os.sep + "logs"):
    os.makedirs(os.getcwd() + os.sep + "logs")
logFile = os.getcwd() + os.sep + "logs" + os.sep +'log.txt'
sys.stdout = Logger(logFile)
del now

print("Starting Test Python Modules");

def testmethod():
    print("From test method")

atexit.register(testmethod)    
    
print("Terminating Test Python Module");

Explanation : What above cod does? We are defining a custom class called Logger and defining methods that a stdout object defines. In this new custom class and methods we retain output to console and also add a output to a file whose path can be provided in the objects constructor while instantiating. Finally we say sys.sysout points to this new Logger object. Lets try this out. Run this script

• python test.py

You should see a directory called logs getting created in current directory and it will have a file called log.txt with your script output. All of this is captured in below screenshot.

Related Links

• Shutdown hook in Java(OSFG)   
• How Python works? (OSFG)

How Python works?

Background

Since I have covered some of python topics already I thought it would be a nice time to understand how Python actually worked. There is s common misconception that python is a purely interpreted language. We will see how it actually works now.

How Python works?

Python like Java is first compiled into byte codes which is the interpreted by Python VM by running an interpreter.

1. When a module is imported for the first time, or when the source is more recent than the current compiled file, a .pyc file containing the compiled code will usually be created in the same directory as the .py file. When you run the program next time, Python uses this file to skip the compilation step.
2. Running a script is not considered an import and no .pyc will be created. For example, if you have a script file abc.py that imports another module xyz.py, when you run abc, xyz.pyc will be created since xyz is imported, but no abc.pyc file will be created since abc.py isn’t being imported.
3. You can manually compile your source file using compileall module. You can just say python -m compileall -l test.py and that should create your compiled .pyc file under a directory called __pycache__ under your current directory. Thereafter you can directly run this .pyc file using python test.cpython-34.pyc  (All of this is captured below in the screenshot)

As I mentioned before the imported modules are automatically compiled for further use. Lets see how that works out.

In my current folder I have two py files

1. test.py and
2. testlib.py

testlib.py has a method called helloworld which simply prints "Hello World!" to the console.

It's contents are as follows

test.py

import testlib

print("Test Python Module")
testlib.helloworld()

testlib.py

def helloworld():
    print("Hello World!")

Now lets run our test.py and see the results. You will see the output and also see a __pycache__ directory getting create under current directory with compile testlib.pyc file. All of this is captured is screenshot below

Note : Only difference is imported modules are compiled to pyc files automatically based on timestamps where as your source files are compiled on the fly. To compile your source files you can use compileall module.

Compiled files with bytecodes are faster to run by python interpreter. Generally .pyc files are the once you ship.

Note :  " It's worth noting that while running a compiled script has a faster startup time (as it doesn't need to be compiled), it doesn't run any faster. "

Given above note will help you understand why modules are compiled and not the main source file. Byte compiled modules simply help in faster startup time.

Related Links

1. Shutdown hook in Python(OSFG) 
2. Using Fabric Python command-line tool(OSFG) 
3. Difference between functions and methods. (OSFG)
4. If Python is interpreted, what are .pyc files? (SO)
5. How to avoid .pyc files? (SO)
6. Difference between Compiler, Interpreter and Assembler (OSFG)
7. compileall python module documentation

Adding shutdown hook in python using atexit module

Background

A shutdown hook is basically a code snippet that allows programmers to run some code at program termination. I had discussed shutdown hook in Java some time back. In this post we will see shutdown hooks in python.

Adding shutdown hook in python using atexit module

You can use the atexit module in python to register shutdown hooks.  See the following code

import atexit

print("Starting Test Python Module");

def testmethod():
    print("From test method")

atexit.register(testmethod)    
    
print("Terminating Test Python Module");

Save it in a file called test.py and simply run it using

• python test.py

And this outputs :

As you can notice it prints "From test method" at the end by executing registered method  testmethod.


Note : 

1. Previously you could do the same by importing sys module and then using sys.exitfunc = testmethod but  sys.exitfunc is deprecated since python 2.4 and is removed since python 3.0.
2. The atexit module defines a single function to register cleanup functions. Functions thus registered are automatically executed upon normal interpreter termination. 
3. atexit runs these functions in the reverse order in which they were registered; if you register A, B, and C, at interpreter termination time they will be run in the order C, B, A.
4. The functions registered via this module are not called when the program is killed by a signal not handled by Python, when a Python fatal internal error is detected, or when os._exit() is called.

Related Links

• Shutdown hook in Java(OSFG)
• Using Fabric Python command-line tool(OSFG)
• Difference between functions and methods. (OSFG)
• atexit documenttaion

Using Fabric Python command-line tool

What is Fabric?

You can visit the fabric site . It gives the most apt description of what fabric is.

Fabric is a Python (2.5-2.7) library and command-line tool for streamlining the use of SSH for application deployment or systems administration tasks.

More specifically, Fabric is:

• A tool that lets you execute arbitrary Python functions via the command line
• A library of subroutines (built on top of a lower-level library) to make executing shell commands over SSH easy and Pythonic.

Prerequisites

1. Assuming you have fair knowledge of python basics. If not please go over the basics in the tutorial - The Python Tutorial
2. Also you mus have python installed. If not refer to the guide -  The Hitchhiker’s Guide to Python!
1. Installing Python on Mac OS 
2. Installing Python on Windows
3. Installing Python on Linux 

Note : Fabric is supported only by Python versions (2.5-2.7).

Installing Fabric

 Before you install Fabric you need to install modules needed to install other modules. For that you can either install - 

1. pip or
2. setuptools

I am going to use pip to install fabric as it is the latest one. To install Fabric use the following command -

 pip install fabric

and your fabric should be installed.  You can see various related installation ways here. You can see the installed module in  

• <Python installation directory>\Lib\site-packages .

For me it is located in 

• C:\Python34\Lib\site-packages\fabric

You can see the fab. exe file in

• C:\Python34\Scripts

which we have already added in the PATH environment variable [This is the same place where you have pip and easy_install executable]

Getting Started

Lets start with our usual "Hello World!"  example. Create a file called fabfile.py and add following content in it - 

def helloworld():
        """This is a test hello world fab method"""
        print("Hello World!")

To see list of available fabric commands do  - 

• fab -l

For above file you should see something like - 

[athakur@localhost athakur]$ fab -l
Available commands:

    helloworld  This is a test hello world fab method

Next it's time to execute the command itself. Execute

• fab helloworld

and you should see Hello World! printed on the console.

Note :  Any method that starts with an underscore (_) is a private method and cannot be used directly as fab command. It will even not be listed in fab -l command.

Consider following code -

def helloworld():
        """This is a test hello world fab method"""
        print("Hello World")
        _private_helloworld()

def _private_helloworld():
        """ This is private method and should not be listed with fab -l"""
        print("Private hello World")

Execute fab -l  and you should again see the same output as above i.e you should not see _private_helloworld method listed there.

Executing fab helloworld should print following now -

[athakur@localhost athakur]$ fab helloworld
Hello World
Private hello World
Done.

Note : The fab tool simply imports your fabfile and executes the function or functions you instruct it to. There’s nothing magic about it – anything you can do in a normal Python script can be done in a fabfile!

Fabfile discovery

Fabric is capable of loading Python modules (e.g. fabfile.py) or packages (e.g. a fabfile/ directory containing an __init__.py). By default, it looks for something named (to Python’s import machinery) fabfile - so either fabfile/ or fabfile.py.

The fabfile discovery algorithm searches in the invoking user’s current working directory or any parent directories. Thus, it is oriented around “project” use, where one keeps e.g. a fabfile.py at the root of a source code tree. Such a fabfile will then be discovered no matter where in the tree the user invokes fab.

(More Details) 

Fabric Tasks with arguments

Put the following  content in fabfile.py

def helloworld(name="Aniket"):
        """This is a test hello world fab method. Syntax : fab helloworld:name=<your name>"""
        print("Hello World from %s!" %name)
        _private_helloworld()

def _private_helloworld():
        """ This is private method and should not be listed with fab -l"""
        print("Private hello World")

and run 

• fab helloworld

You should get

[athakur@localhost athakur]$ fab helloworld
Hello World from Aniket!
Private hello World
Done.

Or you can execute by giving name argument as

• fab helloworld:name="John" or simply
• fab helloworld:"John"

and you should get output as

[athakur@localhost athakur]$ fab helloworld:"John"
Hello World from John!
Private hello World
Done.

Running fabric tasks on other machines

Replace your fabfile with following contents :

from fabric.api import run

def host_name():
        run('uname -a')

and then run it. You should br prompted for hostname and then password for the user you are already logged in as.

[athakur@localhost athakur]$ fab host_name
No hosts found. Please specify (single) host string for connection: localhost
[localhost] run: uname -a
[localhost] Login password for 'athakur':
[localhost] out: Linux localhost 2.6.32-504.8.1.el6.x86_64 #1 SMP Wed Jan 28 21:11:36 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

Done.
Disconnecting from localhost... done.

Troubleshooting

In Windows while installing Fabric you may get following error - 

building 'Crypto.Random.OSRNG.winrandom' extension
warning: GMP or MPIR library not found; Not building Crypto.PublicKey._fastmath
error: Microsoft Visual C++ 10.0 is required (Unable to find vcvarsall.bat).

If you get this you will have to install Microsoft Visual C++ compiler. See Stack overflow answers in the 1st link of related Sections. This is windows specific issue. If you are using Linux this should not occur.

Or You can install MinGW (Minimalist GNU for Windows) as I did. You need to install msys package under MinGW

 and add following entries in your PATH env variable.

• C:\MinGW\bin
• C:\MinGW\msys\1.0\bin [This is where you will find chmod executable]

Then run your command from normal windows command prompt.

Related Links

• error: Unable to find vcvarsall.bat (SO)
• How do I install PyCrypto on Windows?(SO)
• How To Use Fabric To Automate Administration Tasks And Deployments
• Using Fabric for SSH

Difference between functions and methods.

Background

Let me be honest here. I always thought they are the same and used them interchangeably but as it turns out they are not. Each has it's significance or scope.  I stumbled on this question today when I started with python again after a long time. I simply googled "methods in python" and the results on different pages  were using different terminology. And that's when it struck me that methods and functions are indeed different and we should be careful to use each in proper context. I will tell what to use it context of few programming languages but lets start with the understanding process.

I believe programming can be fun.  Above is my 1st meme so please bear with me :)

Difference between Functions and Methods

 Ever tried to explain anyone what a method or function is ? You may have read some definition back in college. Anyway lets try to explain that first.

So a function has a name (called as function name). There it has the data that is passed to it that the function can operate on (called arguments). There arguments are generally provided in parenthesis "()" adjacent to the function name. And lastly there is method body that has the logic to execute when this code module is called. If you ask about a method you may probably answer the same way. 

But there is a difference. 

• Functions are standalone where as method are associated with an Object. When you call a method it is in scope of that object which has it's own blueprint (called class).  
• Another difference is that all arguments in function are explicit where as method had one implicit argument and that is reference to the object that the method belongs to. For example methods in Java can use this  in the method body or in python you can use self.

Just think of "Methods" as object oriented way of calling "Functions.

Note :  I can understand if you argue back saying what about static methods in Java? They are not associated with any instance and we definitely cannot use this in static methods. So are they still methods? Well answer is Yes. You will still call them methods. As I said "Methods" are object oriented way of calling "Functions". I know this is a lame argument but that is what it is. Do let me know in comments if you have some other view point. Would love to discuss this :)

Example in Python

I will give example in python due to which this all started for me. We can have both functions and methods in python. Hope it helps to understand this better.

Functions in python :

def test(name, place="India") :
    """ This is a test function"""
    print("Hi " + name + "! Welcome to " + place)


test("Aniket", "USA")
test("John")

Methods in python : 

class TestClass:
    def test(self, name, place="India"):
        """ This is a test method """
        print("Hi " + name + "! Welcome to " + place)

testObj = TestClass()
testObj.test("Aniket", "USA")
testObj.test("John")
 

And you should see output as (in both cases) : 

General Terminology

• For Java, there are only methods.
• For C, there are only functions.
• For C++ or Python it would depend on whether or not you're in a class.

Note : I know I have labelled this post as "programming concept" but it is more of a terminology use. There were many instance when I answered on Stack overflow using these words interchangeably and ended up corrected by someone and each time I am like "What.... how does it matter?". Well it does .. specially when you are in a circle of programmers/developers. So better late than never :)  you know now!

Related Links

• Difference between a method and a function(SO)
• Python Functions
• What is a “method” in Python?(SO)
• Functional programming(Wiki)