Comprehensive Python - Advanced Topics

87 flashcards · Programming Languages · by claudehsu

1. Given a Python object. class Dog: def __init__(self, name, rating): self.name = name self.rating = rating dog = Dog("Liyu", 5) print(dog) If we simply print it, we get something like below instead of the properties of the object. <__main__.Dog object at 0x000001CFD44D9E50> How to get a more meaningful string representation of the object without modifying the definition of the class, so that we know the states of the object? Note In some cases, your solution many not work. Think about the except…

   There are two approaches <obj>.__dict__ vars(<obj>) print(dog.__dict__) print(vars(dog)) With above code, we get {'name': 'Liyu', 'rating': 5} {'name': 'Liyu', 'rating': 5} Note `vars` actually returns the underlying `__dict__`. However, underscore methods are for internal usage, so `vars` built-in function is a better choice `__slots__` tuple defines the legitimate attributes of an object. If `__slots__` is defined, `__dict__` will not be created, so in that case calling `vars` built-in will c…

2. Write a basic class definition structure. Hint For our convenience, we can define an 'empty' body

   class Dog: #body pass To define a class without particular body, we can use `pass` . Now let's try the class liyu = Dog() print(liyu) Note we do NOT need to use `new` keyword to create the instance Execute above code snippet, we get the result <__main__.Dog object at 0x000002144AD89CD0>

3. Once an instance of a class is created, can I add attributes to that instance on the fly? If we can, is it a good practice to do so for initialization?

   Python is a dynamically typed language, meaning the type is checked only at runtime, so after the instance is created, we can add additional attributes to it on the fly. class Dog: pass liyu = Dog() liyu.name = "Liyu" print(liyu.name) However, this is NOT a good practice. Especially for initialization, we can define the `__init__` method. class Dog: def __init__(self, name): self.name = name liyu = Dog("Liyu") print(liyu.name) One design for additional/ad-hoc attributes is to define the specifi…

4. We can create an 'empty' class like below >>> class Dog: ... pass ... >>> dir(Dog) ['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getstate__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__'] Although we does NOT provide any methods, some undersco…

   Both `__new__` and `__init__` are related to instantiation. Below is the difference `__new__` a static method To invoke a class method, we use form `<class>.<class_method>(<params>)` and the <params> don't include `cls`. To invoke `__new__`, we need to provide `cls`, like `<class>.__new__(<class>, <params>)` called as the 1st step to create a new object instance returns any object (including subclass) `__init__` an instance method called as the 2nd step to initialize the object's attributes ret…

5. Given below code snippet class Dog: def bark(): print("woof") dog = Dog() dog.bark() When executing the code, why do we get below error message? Traceback (most recent call last): File "C:\Sandbox\QuickPython\lab.py", line 6, in <module> dog.bark() TypeError: Dog.bark() takes 0 positional arguments but 1 was given How to fix the code?

   That's a common mistake made by people who have other oo programming experience with languages like Java, C#, and etc. In Python, with above code snippet, when we do `dog.bark()`, we are actually call `Dog.bark` method with passing `dog` instance to it. However, per above code snippet, the `Dog.bark` method takes no argument, so we got the error message. To fix the issue, simply put one parameter representing the passed in object in the parameter list of `Dog.bark` method. class Dog: def bark(s…

6. In Python, an instance method invocation is like `instance.method(arg1, arg2,...)`. Analyze how the 'method' is identified/located.

   Below rules are applied in order when Python tries to locate the method to call via an instance look for the method name in the instance namespace, since the method may get overridden in the namespace of the instance look up the class definition of the instance to look for the method of the desired name(including the instance method and the static method. Those methods with the same name cannot co-exist and in the class definition the last one trumps the others) look for the method according to…

7. In Python, what is a class variable and how to define it?

   A class variable a variable associated with a class, not an instance of a class accessible by all instances of the class To define a class variable, simply define it inside a class and outside of a method class Circle: pi = 3.1416 def __init__(self, radius): self.radius = radius def area(self): return self.radius*self.radius*Circle.pi c1 = Circle(2) print(c1.area()) Circle.pi = 0 # change class variable c2 = Circle(2) print(c2.area()) # get zero Note `pi` is defined as a class variable To acces…

8. Given below code snippet class Circle: pi = 3.1416 def __init__(self, radius): self.radius = radius def area(self): return self.radius*self.radius*Circle.pi You may feel on the last line, using hardcoded class name `Circle` is not that good. Is there a way to avoid hardcoding?

   Yes, we can leverage the `__class__` attribute of the instance. In short, `__class__` attribute represents the corresponding class of the instance. So the last line of the code snippet can be modified as return self.radius*self.radius*self.__class__.pi `self` here actually is not different from normal instance reference, so we can do c = Circle(2) print(c.__class__.pi)

9. Given below Python code snippet class Circle: pi = 3.1416 def __init__(self, radius): self.radius = radius c1 = Circle(1) print(c1.pi) c1.pi = 3.14 print(c1.pi) c2 = Circle(2) print(c2.pi) Is above code working? If it is, what's the execution result? Note we try accessing `pi` via the instance `c1`, although `pi` is a class variable

   There is no syntactical error and the code works well. The core here is to understand when accessing the attribute of a instance, if the attribute name cannot be found in the instance namespace, Python will look up on the class level(class variable). If there is a class variable of the same name, the value of that class variable will be returned. In our code snippet, after `c1.pi=3.14`, an instance variable is attached to the `c1` instance, and from then on Python can find the instance variable…

10. Python provides below two decorators @staticmethod @classmethod Given below code snippet class MyClass: @staticmethod def foo(): print("foo static method") @classmethod def bar(cls): print("bar class method") MyClass.foo() MyClass.bar() The two annotations seem similar to each other. What's the difference?

    Static methods and class methods are similar, because both of them define methods that are not bound to instances of a class, but to class. However there are still differences Access to class variable static method: no access to class variables, unless we hardcode that via class name class method: has access to class variables via the first parameter, which is conventionally named as `cls` Method signature static method: as normal function class method: the first parameter is the corresponding…

11. Given below code snippet class MyClass: def foo(): print("This is foo") mc = MyClass() MyClass.foo() Q Is `foo` an instance method? Is above code working?

    `foo` is not defined as a static method or class method. In fact, `foo` is really an instance method that we cannot invoke with instance. That sounds weird. An instance method but we cannot invoke with an instance? The right way to invoke that sort of method is to use the unbound form like `MyClass.foo()`, because we does not define `foo` to have `self` as its first parameter, as normal instance methods. But how can we prove `foo` is not a static method, since the abovementioned unbound form is…

12. Given below code snippet class MyClass: @staticmethod def foo(): print("foo static method") @classmethod def bar(cls): print("bar class method") class MyDerivedClass(MyClass): @staticmethod def dfoo(): foo() @classmethod def dbar(cls): bar() MyDerivedClass.dfoo() MyDerivedClass.dbar() Q: Is there any defect in above code snippet? If there is, how to fix the defects? If there isn't, what's the execution result?

    Yes, there are defects in the code snippet. The core is to realize below two points `foo` is defined as a static method in super class `bar` is defined as a class method in super class For both static method and class method, when invoking them, we need to access the method via class, such as `<class>.<static_method>` or `<class>.<class_method>`. Without `<class>`, if we call the function `<fn>` directly, we are actually NOT calling static method, or class method, or instance method, but a func…

13. Given below code snippet class MyClass: @staticmethod def foo(): fn() print("foo done") @classmethod def bar(cls): fn() print("bar done") def fn(): print("fn instance method done") def fn(): print("fn function done") MyClass.foo() MyClass.bar() MyClass.fn() Q: Does above code snippet work? If it works, what's the execution result? If it doesn't, how to fix the code snippet?

    It works. The core here is to figure out which `fn` gets invoked. `fn` called inside of `foo` and `bar` is not bound to an instance or class, so it actually means the `fn` function defined in the current scope, and in this case the module level, instead of the `fn` instance method The `fn` instance method is defined without `self`, which is not normal but still valid. It's known as unbound form instance method. The last line seems static method call or class call, but actually, that's a call to…

14. How to show the principle of Zen of Python by Tim Peters? In the interactive Python shell, issue below command import this

15. Given below base class class Shape: def __init__(self, x, y): self.x = x self.y = y `x` and `y` represent the position in the coordinate system. Now define below two derived classes Square: it needs a variable to hold the side length of the square Circle: it needs a variable to hold the radius of the circle

    class Shape: def __init__(self, x, y): self.x = x self.y = y class Square(Shape): def __init__(self, side=1, x=0, y=0): super().__init__(x, y) self.side = side class Circle(Shape): def __init__(self, radius=1, x=0, y=0): super().__init__(x, y) self.radius = radius s = Square(5) c = Circle(3) print(f"s: {vars(s)}") print(f"c: {vars(c)}") Note `super()` returns a proxy object (temporary object of the superclass) that allows us to access methods of the base class, while still letting us pass the '…

16. 'Virtual' is a more traditional OOP term, and it means if a method cannot be found in the current class, the list of superclass is searched (superclass, superclass of superclass, so on and so forth) for the method, and the first one found will be used. Does Python use that 'virtual' method approach? If Python does, think about the following case. We attempt to call an instance method, but we cannot find it in the class definition. Instead we have below methods with the same name as the desired…

    Yes, Python uses the 'virtual' method approach. If a method cannot be found in the 'current' class, the instance method with the same name in the superclass will be used. class Robot: def greet(self): print("This is Robot greeting") class HumanoidRobot(Robot): pass r = Robot() r.greet() # invoke the one defined in `Robot` The confusing case is when the desired instance method does NOT exist in the current class, but both static method and superclass instance method with the desired name exist.…

17. Class attribute is a different idea from instance variable in Python. Q: Can class attributes be inherited? If they can, is it possible to 'override' the class attributes? Analyze above questions with below code snippet class Robot: desc = "Robot Desc" class HumanoidRobot(Robot): pass arale = HumanoidRobot() print(f"Arale: {arale.desc}") Robot.desc = "Robot Desc Modified" print(f"After Robot being modified | Arale: {arale.desc}") HumanoidRobot.desc = "HumanoidRobot Desc" print(f"After HumanoidR…

    Class attributes can be inherited. Keep in mind Class attributes can be accessed by all instances of the class. Once a class attribute cannot be found, Python will look up in the superclass, which means inheritance of class attributes is supported. Python is dynamically typed language, so we can add class attributes on the fly. `<class>.<attribute> = <new_val>` can be used to set the new value for the class attribute. However `<instance>.<attribute> = <new-val>` actually sets the value for the…

18. Given below code snippet class Robot: desc = "Robot Desc" def greet(): return "Robot is greeting" r = Robot() print(r.desc) print(r.greet()) What's the execution result?

    The first `print` statement works well, but the second does not. `r.desc` in the first `print` statements is actually a class attribute instead of an instance variable `r.greet()` indicates that the intention is an invocation to an instance method. The `greet` method defined on the third line is an instance method, but it has no `self` in the parameter list, which leads to the fact that the method can only be invoked with unbound form like `Robot.greet()`. The invocation form looks very similar…

19. Given below code snippet class Robot: @classmethod def greet(cls): print(f"Class Robot greet: {cls}") @staticmethod def greet(): print("Static Robot greet") Robot.greet() What's the execution result? Note we define a class method and a static method with the same name. What if we add an unbound instance method `greet` to class `Robot`?

    The execution result is "Static Robot greet". In above code, the class method is defined first with name `greet`. Later a static method with the same name `greet` is defined, and that 'overrides' the method bound to name 'greet' on the class level. This is similar to re-assigning a new value to a variable. We can try moving the class method definition below static method definition as below class Robot: @staticmethod def greet(): print("Staic Robot greet") @classmethod def greet(cls): print(f"C…

20. Can class level methods, aka static method and class method be inherited? class Robot: @staticmethod def rsm(): print("Staic Method in Robot") @classmethod def rcm(cls): print(f"Class Method in Robot: {cls}") class HumanoidRobot(Robot): pass HumanidRobot.rsm() HumanidRobot.rcm() In above code snippet, what's the execution result? Error or executed successfully?

    Class level methods, like static methods and class methods can be inherited. The code snippet can run smoothly. PS > python lab.py Staic Method in Robot Class Method in Robot: <class '__main__.HumanoidRobot'>

21. Will it be conflict if we define instance method and static method with the same name in a class? For example class Robot: @staticmethod def greet(): print("Robot static method") def greet(self): print("Robot instance method") r = Robot() r.greet() Robot.greet() Wht's the execution result?

    Yes and No. Although we define two methods with the same name `greet`, the latter one actually overrides the previous one. So, in short, conflict, but 'resolved' automatically by overriding the previous one with the latter one. In above case, it works as if we only have the instance method defined in `Robot`. As a result, the 2nd last line `r.greet()` works well. However the last line is actually the unbound form of calling the instance method. The instance method requires one parameter `self`,…

22. We can use the instance reference to access a class attribute. Can we use the instance reference to access class level methods? static methods class methods

    Yes, we can use instance reference to access class level methods like static methods and class methods. class Robot: @classmethod def rcm(cls): print("Robot class method") @staticmethod def rsm(): print("Robot static method") r = Robot() r.rcm() r.rsm() However, notice that it's not recommended to do so, since it looks confusing.

23. Given below code base class Dog: pass d = Dog() Now add the same functionality to the object `d` in two different ways `bark` instance method: add via class `d.__class__` `bark2` instance method: add via the instance `d` directly

    Add `bark` instance method via class `d.__class__` d.__class__.bark = lambda self: print("bark wroof") d.bark() d2 = Dog() d2.bark() Note: as instance method added to the class, the `self` reference is needed. It's just like how we define an instance method in class `Dog`. Add `bark2` instance method via the instance `d` directly d.bark2 = lambda: print("bark2 wroof") d.bark2() d2 = Dog() d2.bark2() # cause error Note: To add the method via the instance directly, the `self` reference is not nee…

24. How to define a private member of a class in Python? What's the visibility of a private member?

    How to define a private member of a class in Python? Use double underscore. Notice, any method or instance variable whose name begins, but doesn't end with a double underscore __ is private; anything else isn't private. What's the visibility of a private member? It cannot be seen outside the methods of the class. The access control is per class bases, just as Java. class Dog: def __init__(self, name, rating): self.name = name self.__rating = rating def sniff(self, other): print(f"{self.name} is…

25. Does Python follows a strict access control on private members of a class?

    No, Python does NOT do that. Technically, we can still visit the private members via some tricks. Under the hood, the privacy is implemented by mangling the name of private variables and private methods. To be more specific, the private members are named as `_<class_name>.<member_name>`. Given below class class Dog: def __init__(self, name, rating): self.name = name self.__rating = rating # private instance vairable __rating Now let's create an object of `Dog`, and inspect it in REPL >>> from l…

26. There seems no getter and setter methods in Python. Is that true?

    Not exactly true. Python's approach is to utilize property, which is similar to getter and setter. class BalanceAccount: def __init__(self): self._amount_in_cents = 0 @property def amount(self): return f"${self._amount_in_cents/100:.2f}" @amount.setter def amount(self, new_val): self._amount_in_cents = new_val*100 # store in cents ba = BalanceAccount() ba.amount = 5.23 print(f"The current amount is: {ba.amount}") ba.amount = 523 print(f"The current amount is: {ba.amount}") Note `@property` make…

27. Is it required in Python to define and call a destructor to ensure the memory is freed?

    No, it's NOT required. Python provides automatic memory management through a reference-counting mechanism. You almost never need to define a destructor. Don't get confused with deallocating an external resource explicitly, such as closing a file, or db connection. In such as situation, the best practice is to leverage a context manager.

28. Regarding multiple inheritance, there are programming languages like C++ that support multiple inheritance, while there are other languages like Java that don't support it(let's forget about the multiple inheritance on Java interface for now :-p). How about Python? Does it support multiple inheritance?

    Yes, Python support multiple inheritance. For the notorious diamond inheritance problem, Python use the MRO approach(Method Resource Order) to determine which method to invoke. The MRO approach follows a depth-first search logic to locate the method. Below is an example. Suppose we have have below class hierarchy According to MRO, `foo` on `D` will be invoked if we invoke `foo` on an instance of `A`. class D: def foo(self): print("foo of D") class C: def foo(self): print("foo of C") class B(D):…

29. Given below class hierarchy Below code snippet is created according to above diagram. class D: def foo(self): print("foo of D") class C: def foo(self): print("foo of C") class B(D): pass class A(B, C): pass According to MRO, when calling `foo` on an instance of `A`, the `foo` method on D will be invoked. How to change it to call `foo` method of `C` regardless of MRO?

    We need to explicitly define `foo` on A, and in that method, we can use unbound format to invoke the `foo` method of the desired class. class D: def foo(self): print("foo of D") class C: def foo(self): print("foo of C") class B(D): pass class A(B, C): def foo(self): C.foo(self) # use unbound format a = A() a.foo() As shown above, we can use the unbound format to call the method of the desired class so that we don't need to follow the MRO.

30. Given below code snippet class Robot: def __init__(self, name): self.name = name @staticmethod def greet(): print(f"Greeting from Robot[{name}]") # call instance variable name r = Robot("Jarvis") r.greet() Is there any defect in above code snippet?

    Yes, there are several defects. Instance variable cannot be visited in the static method. In addition, if we'd like to visit instance variable `name`, we should explicitly indicate the instance reference, like `self.name`, instead of simply putting `name` there. `greet` is a statice method. Although we can visit it using instance reference, like `r.greet()`, that looks very confusing. In addition, if somewhere else in the code, we set `greet` method to `r` on the fly, just like 'hijacking', we…

31. Write examples respectively to illustrate how to use the method invocation form `<class>.<method>.(<param.)` for class method static method instance method

    Class method class Dog: @classmethod def bark(cls): print("Class method bark") d = Dog() Dog.bark() Static method class Dog: @staticmethod def bark(): print("Static method bark") d = Dog() Dog.bark() Instance method class Dog: def bark(self): print("Instance method bark") d = Dog() Dog.bark(d)

32. How to test whether an instance `r` is of class type `Robot` Use the built-in function isinstance(r, Robot) Use the underscore attribute r.__class__ is Robot In addition, think about which approach is a a better one.

    Note: Relying on underscore attribute is usually considered as code smell, so using the built-in function is a better approach

33. How to check the MRO of a class? For example, given below class hierarchy, print the MRO info of class `D`. class A: pass class B: pass class C(A): pass class D(C, B): pass print(D.mro()) alternatively, we can use the dunder attribute print(D.__mro__)

34. In inheritance cases, `super` can be used to invoke the methods defined in the superclass. Usually we do NOT pass any argument to `super`, but technically, `super` can accept two parameters `super(type, instance_or_type2)` type instance_or_type2 Explain how to use the parameters with `super`.

    In short type: define the starting point on the MRO to search the desired method to call instance_or_type2: define the MRO In an inheritance structure as below class A: def foo(self): print("foo in A") class AA(A): def foo(self): print("foo in AA") class AAA(AA): def foo(self): print("invoke foo from AAA") super().foo() # foo in AA AAA().foo() `super().foo()` in `AAA` calls the `foo` method defined in `AA`. In this case `super()` actually means `super(AAA, self)`. In detail `AAA`: starting from…

35. In the form of `super(type, instance_or_type2)`, is there any limitation on the second parameter?

    Yes, there is limitation. The core is that the second parameter needs to provide effective MRO to cooperate with `type` For `super(type, instance)`, the `instance` must be an instance of `type`. If `instance` is not, then `type` will not be in the MRO of `instance`, and we do not have the effective MRO to locate the type of poxy object according to `type' For `super(type, type2)`, the `type2` must be the sub-type of `type`. If `type2` is not a sub-type of `type, similar to above bullet, we will…

36. Given below code snippet of a multiple inheritance case class Robot: pass class HumanoidRobot(Robot): def move(self): print("Move as a human") class CaterpillarTrack: def move(self): print("Move via caterpillar track") class WallERobot(HumanoidRobot, CaterpillarTrack): def move(self): print("WallE Robot is moving...") super(Robot, self).move() WallERobot().move() What's the execution result? Hint print(WallERobot.__mro__) to check the MRO

    The execution result is WallE Robot is moving... Move via caterpillar track The `move` method defined in `CaterpillarTrack` class is invoked. The `WallERobot.__mro__` actually provides us some clue for why that happened. (<class '__main__.WallERobot'>, <class '__main__.HumanoidRobot'>, <class '__main__.Robot'>, <class '__main__.CaterpillarTrack'>, <class 'object'>) As we can see, after `Robot`, in the MRO, it is `CaterpillarTrack`. Although `CaterpillarTrack` is not a superclass of `Robot`, per…

37. Suppose we have a text file `test.txt` containing below contents hello hello hahah lo hello Write a program utilizing regex to count the lines containing the word `hello`

    import re regexp = re.compile("hello") count = 0 with open("test.txt") as f: lines = f.readlines() for line in lines: if regexp.search(line): count += 1 print(f"There are {count} line(s) containing the word 'hello'") # exec in python interpreter >>> exec(open("lab.py").read()) There are 3 line(s) containing the word 'hello'

38. Given below code snippet >>> import re >>> regex = re.compile("hello", re.IGNORECASE) >>> regex.search("Hello world!") >>> regex.search("jenis") What `search` method returns in different scenarios?

    The return value depends on if the regex can find the desired strings found: a `re.Match` object, via which we can do further operations like extracting the string not found: None

39. Suppose we'd like to search a string "\ten", meaning a backslash followed by 't', 'e' and 'n'. We use below regex regexp = re.compile("\\ten") Is there any defect? If there is, how to fix it?

    The functions that compile regular expressions, understand that certain character sequences in strings, such as `\n`, `\t`, and etc. In our implementation, "\\ten" will be translated to "\ten" due to escape sequence in Python string, and then "\ten" will be passed to `re.compile` function. The function understand that "\t" means a tab, so the regex matches to the string a tab followed by "e" and "n". To fix the issue, we have two options `regexp = re.compile("\\\\ten")`: "\\ten" will be passed…

40. What's the difference between below two functions? re.search re.match

    re.search a more "flexible" function, which returns the first matching occurrence, searching the entire string >>> re.compile("world").search("hello world") <re.Match object; span=(6, 11), match='world'> re.match a more "strict" function, it's like re.search, but it only searches from the beginning of the string >>> re.compile("hello").match("hello world") <re.Match object; span=(0, 5), match='hello'> >>> re.compile("world").match("hello world") >>> # No matching

41. Given below text string 123-4567-9876 How to extract all the consecutive numbers? In above case 123 4567 9876

    import re s = "123-4567-9876" regex = re.compile(r"\d{1,}") matches = regex.findall(s) print(matches) Note The returned `matches` in above code is a list. An alternative approach is to use `re.finditer` import re s = "123-4567-9876" regex = re.compile(r"\d{1,}") matches = regex.finditer(s) # returns iterator of re.Match objs for match in matches: print(match.group(0)) # group(0) means the entire group Note `re.finditer` returns an iterator of re.Match objects To extract strings from each match,…

42. Given below regex pattern ([a-zA-Z]+) ([a-zA-Z]+) (\d+-\d+-\d+) and text string containing first name, last name and phone number John Doe 123-456-7890 With below code, we extract the personal info. import re person_info = "John Doe 123-456-7890" regex = re.compile(r"([a-zA-Z]+) ([a-zA-Z]+) (\d+-\d+-\d+)") match = regex.search(person_info) print(f"first name: {match.group(1)}") print(f"last name: {match.group(2)}") print(f"phone number: {match.group(3)}") However, using numbers with `group` see…

    To improve the readability, we can provide names to groups in regex. import re person_info = "John Doe 123-456-7890" regex = re.compile(r"(?P<first_name>[a-zA-Z]+) (?P<last_name>[a-zA-Z]+) (?P<phone_number>\d+-\d+-\d+)") match = regex.search(person_info) print(f"first name: {match.group('first_name')}") print(f"last name: {match.group('last_name')}") print(f"phone number: {match.group('phone_number')}") Note The `?P<group_name>` can be used to name the group However, note the trade-off is it ma…

43. Given below text string s = "If the the problem is textual, use the the re module" How to correct "the the" to "the"? Hint: use `re` to do string replacement

    import re s = "If the the problem is textual, use the the re module" regex = re.compile(r"the the") corrected_s = regex.sub("the", s) print(corrected_s) Note The `re.sub` function replaces all the occurrence of the desired string.

44. Given below text string s = "the quick fox jumps over the lazy dog" Change the text string to below 'weird' form. ThE QuicK FoX JumpS OveR ThE LazY DoG Hint use `re.sub`

    The core is to understand `re.sub` accepts a function to do the substitution. import re s = "the quick fox jumps over the lazy dog" def border(match_obj): head = match_obj.group(1).upper() tail = match_obj.group(2).upper() return f"{head}{match_obj.group(0)[1:-1]}{tail}" regex = re.compile(r"([a-zA-Z])[a-zA-Z]+([a-zA-z])") new_s = regex.sub(border, s) print(new_s) Note apparently we may have other implementations, and the above one is a contrived one demonstrating the use of `re.sub` with a fun…

45. Analyze below language characteristic of Python statically typed or dynamically typed language? strongly typed or weakly typed language?

    Python is dynamically typed language, meaning types are determined at runtime strongly type language, meaning no dangerous implicit type coercion will be done >>> "a" + 1 Traceback (most recent call last): File "<stdin>", line 1, in <module> TypeError: can only concatenate str (not "int") to str

46. What does built-in function `type` return?

    The `type` function returns an instance of the class named `type`, just as in Java an instance of 'Class' type represents a class. >>> type(5) <class 'int'> >>> type((1,)) <class 'tuple'> >>> type(type(5)) <class 'type'>

47. eIn Python program, how to check if a variable 'holds' the value of a certain type. For example >>> dog = "Liyu" How to test whether the variable dog points to a string object?

    We can simply get the corresponding type instance of the variable, and compare it with the type `str`, which stands for the string type. >>> type(dog) == str True

48. Given below code snippet building a hierarchy of types class Vehicle: pass class Machine: pass class Robot(Machine, Vehicle): pass class HumanoidRobot(Robot): pass arale = HumanoidRobot() How to check whether `arale` is of type `Machine`? Can we simply check like `type(arale)`? How to get all the base classes of `arale`?

    How to check `arale` is of type `Machine`? Can we simply check like `type(arale)`? We cannot directly use `type(arale)` to check if the instance is of type `Machine`, because it returns the most specific type, instead of the most general type. >>> type(arale) <class 'lab.HumanoidRobot'> To check whether the instance is of type `Machine`, we can use the `isinstance` build-in function. >>> isinstance(arale, Machine) True How to get all the base classes of `arale`? The core is to understand that `…

49. How to check whether a type is a super type of another? For example, given below two types Robot HumanoidRobot How to check whether Robot is a base(super) type of HumanoidRobot?

    Use the built-in function `issubclass`, we can test whether a given class is a subclass of another. In above case, testing whether Robot is a base type of HumanoidRobot meaning to test whether HumanoidRobot is a subtype of Robot. >>> issubclass(HumanoidRobot, Robot) True # note a class is treated as a subclass of itself >>> issubclass(Robot, Robot) True

50. What does duck type mean? Explain with an example

    "If it walks like a duck and quacks like a duck, it probably is a duck." It's Python's way of determining whether an object is the required type for an operation, focusing on the object's interface, rather than its type. If an operation needs an iterator, for example, the object used doesn't need to be a subclass of any particular iterator or of any iterator at all. All that matters is that the object used as an iterator is able to yield a series of objects in the expected way. In Python, you d…

51. There are special methods in Python that are usually not directly invoked, but called automatically by Python in response to a demand made on an instance of that class. `__str__` is one of those special methods. Write an example and explain the use of `__str__`.

    `__str__` method is used when Python requires a user-readable string representation, for example with `print`. Below is an example class Robot: def __init__(self, name): self.name = name r = Robot("Arale") print(r) Without `__srt__` method to return meaningful strings, we get <__main__.Robot object at 0x0000022FA7589D30> Now add `__str__` method and try again class Robot: def __init__(self, name): self.name = name def __str__(self): return f"Robot[{self.name}]" This time we get the desired stri…

52. Given below code snippet class StrOfLines: def __init__(self, lines): if type(lines) != str: raise ValueError("lines must be string") else: self.lines = lines def __getitem__(self, index): if self.lines: contents = self.lines.split("\n") self.lines = "\n".join(contents[1:]) return contents[0] else: raise IndexError("Empty String") lines = StrOfLines("ab\ncdef\n123\n4567") for line in lines: print(line) The code, especially the `for' loop, seems working correctly, but we still have two questions…

    According to `for` loop, we would have eventually encountered `IndexError`, but we did not. Why? Behind the sense, `obj[i]` is transferrd to `obj.__getitem__[i]`, so `IndexError` eventually happens. However the exception happens inside of a `for` loop, and in Python `for` catches `IndexError` and handles them by exiting the loop. As a result, we don't get an error. Note if the exception is not raised in a `for` loop, we will get the issue. lines = StrOfLines("ab\ncdef") print(lines[9]) print(li…

53. To write a self defined class which behaves like a list may involve defining many Dunder methods(special methods which have the form __xx__. Double underscore methods is abbreviated as Dunder methods). For example __add__ __mul__ __getitem__ ... Is there any approach to simply above job?

    Yes, we can define our type to derive from `list`, so that we can delegate our version of dunder methods to the ones of list type. For instance class StrOfLines(list): def __init__(self, lines_str): if type(lines_str) != str: raise ValueError("Expect lines of string to be passed in") else: self._lines = lines_str.split("\n") super().__init__(self._lines) Note The `__init__` on the last line is critical here. We pass the list containing lines of strings to the `__init__` of `list` type. The dele…

54. Is it recommended to define and use special methods all the time?

    No, that's not a good idea. You may define several types, and for some types, for example, they may not behave like a sequence, so you don't define dunder methods for them. Your client programmers will have to think for a while why some of your classes support things like subscription operation, while some do not. A worse case can be your class supports part of those dunder methods, and that will cause more confusion. Some general guidelines are to use special methods in either of two situation…

55. The `import` of Python has two form absolute import relative import When using relative import, what is the import relative to?

    The relative import form is relative to `__name__` of the module.

56. Some useful modules in the standard library fileinput module: iterate over lines from multiple input streams tempfile module: generate temporary files and directories linecache module: grain random access to text lines pickle, shelve modules: enable object serialization and persistence zlib module: for archive and compressions time module: time access and conversions logging module: logging facility of Python getpass module: portable password input threading module: higher-level threading inter…

57. Some useful modules in the standard library socket, ssl modules: low-leve networking interface and SSL wrapper for socket objects emal module: email and MIM handling package json module: JSON encoder and decoder mimetypes module: map filenames to MIME types html.parser, html.entities modules: parse HTML and XHTML urllib.request, urllib.parse modules: open and parse URLs socketserver module: framework for network servers http.server module: HTTP servers

58. Below is the command to use pip to install a library called `requests` python -m pip install requests

59. Suppose the library `requests` has been installed in the system. How to upgrade its version?

    The core is to use `--upgrade` option with `pip install` python -m pip install --upgrade requests

60. How to install the `requests` with a specific version via pip? For example install the version 2.11.1 install the version that is after 2.9

    install the version 2.11.1 python -m pip install requests==2.11.1 install the version that is after 2.9 python -m pip install requests>=2.9

61. You may see below pip install python -m pip install --user requests What does the option `--user` mean?

    It means to install the library in the user's home directory, so that it's still possible to install the library if the user doesn't have the privilege to install the libraries on the system level Python instance.

62. What is a virtual environment, and how do we utilize it?

    A virtual environment in python is a self-contained directory structure that contains Python interpreter Libraries With that said, we can have different versions of Python interpreter or libraries for different applications. We can use `venv` library to work with virtual environments. To utilize a virtual environment, we need to do two steps create the virtual environment activate it To create the virtual environment python -m venv my-env To activate the newly created virtual environment .\my-e…

63. Once a virtual environment is activated, when we use `pip` standalone command to install the libraries, where those libraries will be installed, the virtual environment or the system Python instance?

    The libraries will be installed to the virtual environment only.

64. Suppose in our system, we have Python version 3.11. How to create a virtual environment with Python version 3.9?

    First, we need to install the Python version 3.9 in our operating system, which means we will have multiple versions of Python installed in OS. Then when create the virtual environment, we need to explicitly indicate to use Python interpreter version 3.9 like below python -m venv my-env39 --python=/usr/bin/python3.9

65. Below code snippet contains a simple class definition class Dog: def __init__(self, name, rating): self.name = name self.rating = rating d = Dog("Liyu", 4.5) print(vars(d)) Update the `__init__` method to use type hinting/type annotation.

    Below is the code using type hinting class Dog: def __init__(self, name: str, rating: int) -> None: self.name = name self.rating = rating d = Dog("Liyu", 5) print(vars(d))

66. Given below code snippet class Robot: def __init__(self, name): self.name = name def clean(self): print(f"Robot {self.name} is cleaning.") class HumanoidRobot(Robot): def __init__(self, name, height): super().__init__(name) self.height = height def clean(self): print(f"Humanoid Robot[{self.name}|{self.height}] is cleaning") arale = HumanoidRobot("Arale", 139) How to invoke the `clean` method defined in `Robot` class with the instance `arale`?

    We can use the `super` function to get the proxy object of type `Robot`, and then pass `arale` to the corresponding `clean` method of that proxy object. super(HumanoidRobot, HumanoidRobot).clean(arale) According to MRO, `super(HumanoidRobot, HumanoidRobot)` returns the proxy object of type `Robot.

67. Suppose we are dealing with a situation that a daily feed of product data is sent to us in below three files item_info.txt item_attributes.txt related_item.txt For better file management, we decide to archive the processed files to directories representing the corresponding date. The design is get the date string of today create the directory with the date string move the files to the newly created directory for archiving. The design looks like below Implement above design.

    import pathlib import datetime DATA_FEED_DIR = "C:/Sandbox/DataFeed" ARCHIVE_BASE_DIR = f"{DATA_FEED_DIR}/Archive" data_feed_dir_obj = pathlib.Path(DATA_FEED_DIR) date_str = datetime.date.today().isoformat() archive_dir_obj = pathlib.Path(ARCHIVE_BASE_DIR).joinpath(date_str) if not archive_dir_obj.exists(): archive_dir_obj.mkdir(parents=True) data_feed_files_obj = data_feed_dir_obj.glob("*.txt") for data_feed_file_obj in data_feed_files_obj: data_feed_file_name = data_feed_file_obj.name archive…

68. Suppose we are dealing with a situation that a daily feed of product data is sent to us in below three files item_info.txt item_attributes.txt related_item.txt For better file management, we decide to archive the processed files in a zip file representing the corresponding date. The design looks like below working/ archive/ 20240910.zip Implement above design

    import pathlib import datetime import zipfile DATA_FEED_DIR = "C:/Sandbox/DataFeed" ARCHIVE_BASE_DIR = f"{DATA_FEED_DIR}/Archive" date_str = datetime.date.today().isoformat() archive_zip_dir_obj = pathlib.Path(ARCHIVE_BASE_DIR) if not archive_zip_dir_obj.exists(): archive_zip_dir_obj.mkdir(parents=True) archive_zip_obj_path = archive_zip_dir_obj.joinpath(f"{date_str}.zip") with zipfile.PyZipFile(archive_zip_obj_path, "w") as archive_zip_obj: data_feed_dir_obj = pathlib.Path(DATA_FEED_DIR) data_…

69. Suppose we are dealing with a situation that a daily feed of product data is sent to us in below three files item_info.txt item_attributes.txt related_item.txt For better file management, we decide to archive the processed files in a zip file representing the corresponding date. The design looks like below working/ archive/ 20240910.zip Now we need another script to delete some old archives. The design is for archives longer than 30 days, we only keep the archive files generated on Tuesday.

    import pathlib import datetime ARCHIVE_BASE_DIR = f"/Sandbox/DataFeed/Archive" archive_files_obj = pathlib.Path(ARCHIVE_BASE_DIR).glob("*.zip") today_obj = datetime.date.today() for archive_file_obj in archive_files_obj: archive_file_date = datetime.date.fromisoformat(archive_file_obj.stem) if archive_file_date + datetime.timedelta(days=30) < today_obj \ and archive_file_date.weekday() != 1: archive_file_obj.unlink() print("Done")

70. With below code >>> open('test.txt', 'wb').write(bytes([65, 66, 67, 255, 192,193])) We write some garbled words into the text file If we read the file as below using UTF-8 encoding/decoding >>> open('test.txt', encoding='utf-8').read() will we get error? If we get error, how to handle the situation? If we don't get error, what is printed?

    We will get error because those three bytes are not normal UTF-8 encoded bytes. >>> open('test.txt', encoding='utf-8').read() Traceback (most recent call last): File "<stdin>", line 1, in <module> File "<frozen codecs>", line 322, in decode UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 3: invalid start byte To handle the encoding error, we can set `errors` argument of `open` built-in function with one of below options. `ignore`: to skip the character causing the error `re…

71. Give below quoted contents lines = '''Name, Times, Cost, "Month Day, Year" Liyu, 50, "$1,000", "Sep 22, 2029" Jenis, 30, "$1,500", "May 5, 2030"'''.split("\n") Now let's use `csv` module to read above contents as comma separated contents >>> list(csv.reader(lines)) [['Name', ' Times', ' Cost', ' "Month Day', ' Year"'], ['Liyu', ' 50', ' "$1', '000"', ' "Sep 22', ' 2029"'], ['Jenis', ' 30', ' "$1', '500"', ' "May 5', ' 2030"']] We got the issue that values like "$1,000" and "Sep 22, 2029" are in…

    The issue is how `csv.reader` recognizes quotes. For example given below content Name, Times, Cost, "Month Day, Year" Focus on `Cost, "Month` part. There is a space in between the command and double quote. That space is treated as the first character of the content, so the following double quote will be treated part of the content as well, instead of the "quote". As a result, it eases the human readability, but not intuitive for computer to process. To fix the issue, we can specify the argument…

72. Given below contents with header line header_line = "Name|Rating|Times" line1 = "Liyu|5|80" line2 = "Jenis|4|50" line3 = "vikcy|3.5|8" lines = [header_line, line1, line2, line3] How to get a list of dictionaries like below? [{'Name': 'Liyu', 'Rating': '5', 'Times': '80'}, {'Name': 'Jenis', 'Rating': '4', 'Times': '50'}, {'Name': 'vikcy', 'Rating': '3.5', 'Times': '8'}] Hint Use `csv` module

    >>> list(csv.DictReader(lines, delimiter="|")) [{'Name': 'Liyu', 'Rating': '5', 'Times': '80'}, {'Name': 'Jenis', 'Rating': '4', 'Times': '50'}, {'Name': 'vikcy', 'Rating': '3.5', 'Times': '8'}] Note `csv.DictReader` returns `OrderedDicts`, so the fields stay in their original order.

73. Using `csv` module, how to write below data into a csv file? data = [ ["Liyu", 5, 100], ["Vicky", 3.5, 10], ["Jenis", 4, 30] ] Suppose the file path is "/sandbox/temp/output.csv"

    >>> csv.writer(open("/sandbox/temp/output.csv", "wt", newline="")).writerows(data) Note `newline=""` means to use `\n` as the line separator.

74. Given below data in dictionary format data = [ {"Name":"Liyu", "Rating":5, "Times": 100}, {"Name":"Vicky", "Rating":3.5, "Times": 10}, {"Name":"Jenis", "Rating":4, "Times": 30} ] Using `csv` module, how to write above data into a csv file with fields = ["Name", "Rating", "Times"] ?

    >>> dict_writer = csv.DictWriter(open("/sandbox/temp/output_dict.csv", "wt", newline=""), fieldnames=fields) >>> dict_writer.writeheader() 19 >>> dict_writer.writerows(data) >>> del dict_writer Note we need to call `writeheader` method first because usually the header line is the first line of the csv file.

75. Using `ftplib` from the standard library, we can interact with FTP server conveniently. >>> import ftplib # create object representing ftp >>> myftp = ftplib.FTP('127.0.0.1') >>> myftp.login('liyu', 'dog1') '230 Login successful.' # list items on ftp under the remote working directory >>> myftp.nlst() ['hello.txt', 'Robust_Python.epub']

76. For security reasons, it's common to use SFTP. A community developed library `paramiko` can help use deal with SFTP. Suppose below configuration has been done to the SFTP server it disables password login and uses ssh keys it allows the users to use SFTP commands only it configures the SFTP jail for the users >>> import paramiko >>> transport = paramiko.Transport(("127.0.0.1",22)) >>> private_key = paramiko.RSAKey.from_private_key_file("c:/users/naomi/.ssh/id_rsa") >>> transport.connect(usernam…

77. To simply retrieve a resource, such as a text file, according to a URL, we can utilize requests library >>> import requests >>> response = requests.get("http://www.metoffice.gov.uk/pub/data/weather/uk/climate/stationdata/heathrowdata.txt") >>> print(response.text) Heathrow (London Airport) Location 507800E 176700N, Lat 51.479 Lon -0.449, 25m amsl ...

    Usually, we'd write the response text to a file for later processing.

78. Below programs invokes the GET method on endpoint 'https://pokeapi.co/api/v2/location/1' >>> import requests >>> response = requests.get("https://pokeapi.co/api/v2/location/1") >>> response.text '{"areas":[{"name":"canalave-city-area","url":"https://pokeapi.co/api/v2/location-area/1/"}],"...'

79. It's quite common that RESTful API returns a JSON string to the caller. >>> response = requests.get("https://pokeapi.co/api/v2/location/1") >>> response.text '{"areas":[{"name":"... To transform JSON string representation to dictionary object representation, we do >>> import json >>> json.loads(response.text) What's the shortcut to get the dictionary object based on `response`? >>> response.json()

80. What's the difference between below JSON serialization methods json.dump json.dumps Demonstrate with examples respectively.

    Both methods 'translates' JSON object from dictionary representation to str representation for serialization purpose. `s` in word `dumps` means str. In other words, `json.dumps` translates the dictionary to str representation directly and we get a str `json.dump` output the str representation to a file object. `write` method of the file object will be utilized to output the JSON str to the file json.dumps >>> import json >>> json_dict = {"name":"Liyu", "rating":4.5, "measure":[72,48,76]} >>> js…

81. XML format is also widely used in API calls. xmltodict library can be used to transform XML str representation to dict representation and vice versa. To translate XML str representation to dictionary object representation, use method xmltodict.parse(xml_str_contents) To translate XML dictionary object representation to str represetnation, use method xmltodict.unparse(xml_dict_obj)

82. For web scraping/crawling, we can utilize library beautifulsoap. To install the library, we do PS > pip install bs4 >>> import pathlib >>> import bs4 >>> html_contents = pathlib.Path().joinpath("/","sandbox","temp","test.html").read_text() >>> bs = bs4.BeautifulSoup(html_contents, "html.parser") # get all the a tag >>> a_tag_list = bs("a") # get the text node of the first a tag >>> a_tag_list[0].text # get all the tags which use 'special' class style >>> special_class_list = bs.select(".special…

83. Create a small script to finish below tasks(steps) in regrading to storing data in a sqlite3 database using `sqlite3` package in the standard library connect to the sqlite3 database(file name is `sample.db`) create a table called dog with below columns id: integer and primary key name: text to store name of the dog rating: integer to store the rating of the person insert 2 records of dog check the insertion result by retrieving the inserted records Hint `conn.row_factory = sqlite3.Row` can help…

    import sqlite3 conn = sqlite3.connect("sample.db") conn.row_factory = sqlite3.Row cursor = conn.cursor() create_table_sql = "create table dogs(id integer primary key, name text, rating integer)" cursor.execute(create_table_sql) cursor.execute("insert into dogs(name, rating) values(:dogName, :dogRating)", {"dogName":"Liyu", "dogRating": 8}) cursor.execute("insert into dogs(name, rating) values(?, ?)", ("Jenis", 7)) conn.commit() rs = cursor.execute("select * from dogs") all_dogs = rs.fetchall()…

84. ORM has pros and cons. Think about why sometimes we may consider to use ORM?

    Here is one important reason Different SQL dbs have implemented SQL in subtly differently ways. Using ORM can help us write protable code interacting with different relational dbs.

85. Finish below tasks(steps) with SQLAlchemy connect to a SQLite db named `sampelCore.db` create a table named `dog` with below columns id: integer as primary key name: string to store the name of the dog rating: integer to store the rating score of the dog insert a record to the table check the insertion result update that record check the update result Hint Issue `pip install sqlalchemy` to install the library if it's not installed For this mem card, use "Core" approach

    from sqlalchemy import create_engine, select, MetaData, Table, Column, Integer, String from sqlalchemy.orm import sessionmaker db_file_path = "sampleCore.db" engine = create_engine(f"sqlite:///{db_file_path}") metadata = MetaData() dogs = Table('dog', metadata, Column('id', Integer, primary_key=True), Column('name', String), Column('rating', Integer) ) Session = sessionmaker(bind=engine) session = Session() metadata.create_all(engine) dogs_insert = dogs.insert().values(name="Liyu", rating=8) pr…

86. Finish below tasks(steps) with SQLAlchemy connect to a SQLite db named `sampelORM.db` create a table named `dog` with below columns id: integer as primary key name: string to store the name of the dog rating: integer to store the rating score of the dog insert a record to the table check the insertion result update that record check the update result Hint Issue `pip install sqlalchemy` to install the library if it's not installed For this mem card, use "ORM" approach

    from sqlalchemy import create_engine, Column, Integer, String from sqlalchemy.orm import sessionmaker, declarative_base db_file_path = "sampleORM.db" engine = create_engine(f"sqlite:///{db_file_path}") Session = sessionmaker(bind=engine) session = Session() Base = declarative_base() class Dog(Base): __tablename__ = "dog" id = Column(Integer, primary_key=True) name = Column(String) rating = Column(Integer) Base.metadata.create_all(engine) dog_liyu = Dog(name="Liyu", rating=8) session.add(dog_liy…

87. Given below code snippet class Dog: def __init__(self, name): self.name = name def bark(self, msg): print(f"The dog[{self.name}] is barking: {self.msg}") def foo(self): bark() Can above code snippet execute smoothly or there are defects?

    There is a defect on the last line. Calling `bark` instance method like `bark()` is not correct. With that form, `bark` will be treated as a normal function defined in the current module scope, instead of an instance method. To correct the code class Dog: def __init__(self, name): self.name = name def bark(self, msg): print(f"The dog[{self.name}] is barking: {self.msg}") def foo(self): self.bark() Simply refere to `bark` with `self`