Unit Testing

What is unit testing?

Unit testing simply verifies that individual units of code (mostly functions) work as expected. Usually you write the test cases yourself, but some can be automatically generated.

The unit testing should be done as often as possible. The most obvious benefit is knowing down the road that when a change is made, no other individual units of code were affected by it if they all pass the tests.

Example of testing Student class

Here I’m going to make a very simple example to show how it works.

First, build the Student class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

class Student():
    def __init__(self, name, score):
        self.name = name
        self.score = score
        self.grade = None

    def to_grade(self):
        if self.score >= 60:
            self.grade = 'Pass'
        else:
            self.grade = 'Fail'

    def get_grade(self):
        if self.grade:
            return self.grade
        else:
            self.to_grade()
            return self.grade

Then, build the unit testing script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

import unittest
from Student import Student


class TestStudent(unittest.TestCase):
    def test_above_60(self):
        s1 = Student('bart', 60)
        self.assertEqual(s1.get_grade(), 'Pass')

    def test_below_60(self):
        s1 = Student('bart', 50)
        self.assertEqual(s1.get_grade(), 'Fail')

    def test_invalid(self):
        pass


if __name__ == "__main__":
    unittest.main()

Execution

Execute the script to make testing:

1

python test_student.py

to make the outut more verbose, add the -v into the command.

1

python test_student.py -v

Reference

1. blog of liaoxuefeng: https://www.liaoxuefeng.com/wiki/1016959663602400/1017604210683936
2. doc of Python 3.8: https://docs.python.org/3/library/unittest.html

Decorators

What is decorators?

It accepts the functions and add the decorator-like function into the original functions to achieve certain functions.

Examples:

Apply the function of measuring execution time on each function.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22

# -*- coding: utf-8 -*-
import time

# --exe_time
def exe_time(func):
	def new_func(*args, **args2):
		t0 = time.time()
		print "@%s, {%s} start" % (time.strftime("%X", time.localtime()), func.__name__)
		back = func(*args, **args2)
		print "@%s, {%s} end" % (time.strftime("%X", time.localtime()), func.__name__)
		print "@%.3fs taken for {%s}" % (time.time() - t0, func.__name__)
		return back
	return new_func
# --end of exe_time

@exe_time
def foo():
	for i in xrange(10000000):
		pass

if __name__ == "__main__":
	foo()

See more details in the blog of liaoxuefeng.

Reference

1. liaoxuefeng: Decorators https://www.liaoxuefeng.com/wiki/1016959663602400/1017451662295584
2. Python: 使用装饰器“@”取得函数执行时间 https://oldj.net/blog/2010/05/22/decorator-get-execution-time

Map, Filter and Reduce

Map

Map applies a function to all the items in an input_list. Given an example:

1
2

items = [1, 2, 3, 4, 5]
squared = list(map(lambda x: x**2, items))

The use blueprint is

1

map(function_to_apply, list_of_inputs)

Note that the function could be list of functions.

Filter

As the name suggests, filter returns the list of elements for which a function returns true. Given an example:

1
2
3
4
5

number_list = range(-5, 5)
less_than_zero = list(filter(lambda x: x < 0, number_list))
print(less_than_zero)

# Output: [-5, -4, -3, -2, -1]

Reduce

Reduce is a really useful function for performing some computation on a list and returning the result. It applies a rolling computation to sequential pairs of values in a list. For example, if you wanted to compute the product of a list of integers.

So the normal way you might go about doing this task in python is using a basic for loop:

1
2
3
4
5
6

product = 1
list = [1, 2, 3, 4]
for num in list:
    product = product * num

# product = 24

Now let’s try it with reduce:

1
2
3
4
5
6

from functools import reduce
product = reduce((lambda x, y: x * y), [1, 2, 3, 4])

# Output: 24
# map and filter are both in built-in module. 
# The REDUCE need to be imported.

Reference

1. Map, Filter and Reduce https://book.pythontips.com/en/latest/map_filter.html

stdin is used for all interactive input, including calls to input().
stdout is used for the output of print() and expression statements and for the prompts of input()

For sys.stdin, you can use

• sys.stdin.readline()
• sys.stdin.readlines()

For more details, read the reference articles.

Reference

1. Chapter 20 - The sys Module https://python101.pythonlibrary.org/chapter20_sys.html

   (this is a good website for python learner)

2. Python Concepts/Console Input https://en.wikiversity.org/wiki/Python_Concepts/Console_Input

Common Usage of os.path module

This article records some very common usages of os.path module in Python.

Since there are few examples in the docs of os.path module, I added examples for all part in this article.

os.path.expanduser()

Expand ~ and ~user constructions. If user or $HOME is unknown, do nothing.

1
2
3
4
5
6
7

path = "~/file.txt"
full_path = os.path.expanduser(path)
# return: '/Users/shuo/file.txt'

os.environ["HOME"] = "/home / GeeksForGeeks"
full_path = os.path.expanduser(path)
# return: '/home / GeeksForGeeks/file.txt'

os.path.join()

Join one or more path components intelligently.

1
2
3
4

>>> os.path.join('a','b','c')
'a/b/c'
>>> os.path.join('a','b','c','')
'a/b/c/'

os.path.split()

Split the pathname path into a pair, (head, tail) where tail is the last pathname component and head is everything leading up to that.

• The tail part will never contain a slash;
  • if path ends in a slash, tail will be empty.
  • If there is no slash in path, head will be empty.
  • If path is empty, both head and tail are empty.

    1 2 3 4 5 6 7

    >>> test_s="/home/sa2y18/mydocuments" >>> os.path.split(test_s) ('/home/sa2y18', 'mydocuments') >>> test_s=test_s+'/' >>> os.path.split(test_s) ('/home/sa2y18/mydocuments', '')

Reference

1. https://docs.python.org/2/library/os.path.html
2. my gist

In Python you need to give access to a file by opening it.

open() in usual case

If you don’t use with statement, you should write like this:

1
2
3
4
5
6
7

file = open("welcome.txt")

data = file.read()

print data

file.close()  # It's important to close the file when you're done with it

file.close() is important, especially in the writing mode. If you don’t close(), the writing action can’t really take effect.

use with in Python IO

1
2
3
4
5
6
7
8
9
10

with open("welcome.txt") as file: # Use file to refer to the file object

   data = file.read()

   do something with data
   
# open in write mode 
with open('output.txt', 'w') as file:  # Use file to refer to the file object

    file.write('Hi there!')

Notice, that we didn’t have to write file.close() in the case of using with open() as xx. That will automatically be called.

Reference

1. With statement in Python : https://www.pythonforbeginners.com/files/with-statement-in-python

What is torch.no_grad() for?

When using Pytorch, if we want to do testing after the training, the model.eval() switches the mode into eval mode. The batchnorm and dropout layers would not work on that case and make sure the values pass through the network.

When I read more code, I found that the torch.no_grad() is also widely used in Pytorch project. What is this for?

According to the discussion on https://discuss.pytorch.org/t/model-eval-vs-with-torch-no-grad/19615.

The model.eval() and with torch.no_grad() two functions have different goals:

• model.eval() will notify all your layers that you are in eval mode, that way, batchnorm or dropout layers will work in eval mode instead of training mode.
• torch.no_grad() impacts the autograd engine and deactivate it. It will reduce memory usage and speed up computations but you won’t be able to backprop (which you don’t want in an eval script).

— answered by @albanD

Example of torch.no_grad()

• SPATIAL TRANSFORMER NETWORKS TUTORIAL:
  https://pytorch.org/tutorials/intermediate/spatial_transformer_tutorial.html

  1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19

  def test(): with torch.no_grad(): model.eval() test_loss = 0 correct = 0 for data, target in test_loader: data, target = data.to(device), target.to(device) output = model(data) # sum up batch loss test_loss += F.nll_loss(output, target, size_average=False).item() # get the index of the max log-probability pred = output.max(1, keepdim=True)[1] correct += pred.eq(target.view_as(pred)).sum().item() test_loss /= len(test_loader.dataset) print('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n' .format(test_loss, correct, len(test_loader.dataset), 100. * correct / len(test_loader.dataset)))

I also reproduce this code on my github repo: https://github.com/ShuoGH/deepLearningAl

Using torch.no_grad() would accelerate the computation of neural network.

Reference

1. Pytorch Forum: ‘model.eval()’ vs ‘with torch.no_grad()’: https://discuss.pytorch.org/t/model-eval-vs-with-torch-no-grad/19615

Install the Miniconda environment on Remote Server

Anaconda installation doesn’t need to ask for the admin permission. So it’s a good choice used on server. I would show how to install on the server in the following.

Download the Miniconda package

1
2
3
4
5

# use `curl` to download the package from miniconda. In own computer of MacOS.
curl -O "https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh"

# use `wget` to download the package, in the server
wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh -O miniconda_package

Install on the Server

1
2
3
4
5

# give the execute permission of the file
chmod +x /path/to/yourscript.sh

# run. `./` is important, since it points to the current path
./yourscript.sh

Activate the Environment

1
2
3
4

source Miniconda/bin/activate 

# deactivate the environment, just type in the terminal
deactivate

Then you have the environment for your project, you can just use pip install xx to install the modules you need.

Reference

1. Reddit: what advantages of using anaconda

Intro of argparse

The argparse module makes it easy to write user-friendly command-line interfaces. The program defines what arguments it requires, and argparse will figure out how to parse those out of sys.argv(see the following part for the details of this).

You can’t keep using Jupyter notebook all the time. The scripts files and a whole projects needs to be created for our task. The argparse is the communication tools which let developers to make changes to the script from the terminal and don’t need to go into the code and change the parameters.

example

1
2
3
4
5
6
7
8
9
10
11

import argparse

parser = argparse.ArgumentParser(description='Process some integers.')
parser.add_argument('integers', metavar='N', type=int, nargs='+',
                    help='an integer for the accumulator')
parser.add_argument('--sum', dest='accumulate', action='store_const',
                    const=sum, default=max,
                    help='sum the integers (default: find the max)')

args = parser.parse_args()
print(args.accumulate(args.integers))

This is a program that takes a list of integers and produce either the sum or the max.

Assume the name of file is prog.py, and execute the file:

1
2
3
4
5

python prog.py 1 2 3 4
# output: 4

python prog.py 1 2 3 4 --sum
# output: 10

See the details from docs of argparse.

Else

sys.argv

During using the argparse module, sys.argv is important since you may need it in the main()function.

Code is sometimes like this:

1
2
3
4
5
6
7
8
9
10

import argparse
import sys 

def main(argv):
    parser = argparse.ArgumentParser()
    # parser.add_argument()
    # Do some argument parsing

if __name__ == '__main__':
    main(sys.argv)

sys.argv is the list of commandline arguments passed to the Python program. It represents all the items that come along via the command line input.

For the content of argv, the first argument is always the script name, and it is also counted in the number of arguments. So even if you don’t pass any argument into your script, the sys.argv variable always contains at least one element and it’s the script name.

Example

Execute this python script, assume it is named test.py.

1
2
3
4

import sys
print "This is the name of the script: ", sys.argv[0]
print "Number of arguments: ", len(sys.argv)
print "The arguments are: " , str(sys.argv)

1

python test.py

The output should be:

1
2
3

This is the name of the script:  sysargv.py
Number of arguments in:  1
The arguments are:  ['test.py']

Reference

1. python docs: argparse
2. How to use sys.argv in Python ](https://www.pythonforbeginners.com/system/python-sys-argv)
3. https://appdividend.com/2019/01/22/python-sys-argv-tutorial-command-line-arguments-example/

Counting the files in a directory.

When I was doing my summer project, I downloaded the datasets into the server and unzipped the zip file. Since it has a large amount of files in the dir, I want to count the number of files just through the terminal.

First try

1

ls -1 | wc -l

Use this command after getting into the dir, it can count the number of files in the current dir. Note: the first 1 is ONE, and the second l is the l for large.

Result: it seems that I can’t perform well, since the speed is slow and need to spend a lot of time to get the result( I didn’t wait for that long to get the final result).

Note:

wc is great when I search this command.

1
2

wc -l < file.txt
wc -w file2.txt

The first line of code can count the number of lines in a text file.

The second line of code can count the words in the text file.

Improvement to get my result

Since I have the nohup.out, I can count the line number to know how many files I have uncompressed.

The uncompressing process is a really long journey…for my project

Reference

1. Bash Prompt HOWTO: Counting Files in the Current Directory