Pages

Search This Blog

March 05, 2021

Software complexity due to dependencies

In this write-up we shall have a quick look at why handling dependencies properly  is important for any software system. This is just a distillation of what I have learned from different resources trying to learn ways of managing software complexity. The ideas discussed have been around long and I have just tried to write down my understanding and put together a quick guide on how to make use of them. 

Ideas of Software and System are used interchangeably as the difference does not matter for this write-up.

What is the most important singular problem in Software Design?

Coping with change. Why should software be changed? Simply because it is inevitable. In other words, since software is how-to knowledge of things, and the why-s and what-s of a software change for it to be useful in ever changing world, the how-s also should change or improve.

Why is change difficult in a system? 

Change is difficult and expensive when the system is Rigid, Fragile, Immobile and Viscous1.

Rigidity is when a change done in one part of the system has ripple effects in other parts and so every change becomes prohibitively tedious and expensive. This inability to make any adjustments to the system in an easy way gives rigid appearance to the system.

Fragility is closely related to Rigidity in that every change to the system breaks something else potentially unrelated even, to break.

Immobility is when no component in isolation can be reused. If pulling a component to be used elsewhere brings with it a load of unrelated dependencies, the components in the system are immobile and there is no reusability.

Viscosity happens when it is easy and straightforward to do a hacky solution in the system than doing the right solution. This causes the hacky solutions to pile up to an extent that the entire system is hacky and design erodes to a point of uselessness. 

Why would a system become rigid, fragile, immobile and viscous?

Because of dependencies among objects in the system and how those dependencies are handled. 

Dependencies

Say there are 2 objects A and B and A is dependent on B. Any object has reasons to change in its lifetime, say it gets new features, bugs in it get fixed or refactored. If Object A has to change because Object B changed due to reasons stated above, this is tight coupling and the more of such couplings in the system, it is more difficult to change that system. 

So what are the techniques to design software so that objects in it stay open to extension and closed for modification while also are less rigid, fragile, immobile and viscous. There are many and we look here at 2 most important and related ideas.

Single Responsibility and Dependency Injection

Single Responsibility means that when we try to  explain what a method does, if there are conjunctions AND or OR in the explanation, the method does more than one thing in increasingly bad effects. Eg: This method downloads a file AND parses the file AND stores the parsed contents in a database. 

Dependency Injection means that when an object has a dependency hard coded in it - like (pseudo-code)

class A {

A() {

B b = new B();

}

}

In code above, there is no way to change B for a DifferentB or ImprovedB. This class is closed for extension and modification only can help achieve desired changes. If instead, it was written like below - 

Class A {

A(BInterface bObj) {

b = bObj;

}

BInterface b;

}

The benefits are

  • bObj can take any of B, DifferentB or ImprovedB which means it is perfectly open for extension.

  • Since object A is dependent on interface of B, with a right level of abstraction at the interface, object A will not have to change ever when the implementation details of B change.

  • And most importantly, bObj can be mocked for testing 

When combined with Single Responsibility Principle, Dependency Injection helps achieve a system in which the components are reusable, changes can be made without breaking the system inadvertently.

Musing: The idea of Dependency Injection seems to be parallel to higher order functions recommended by Functional Programming paradigm, where, as we go higher up in the level of abstraction, the higher level structures work with lower level structures with a contract at function signature. Eg: map(fn, list) can work with any function and list, provided the function has a contract that it accepts a value and returns a value. Dependency Injection seems to recommend this in a lateral direction where the abstraction doesn’t move necessarily upwards.


Unit Testing

What are the effects of dependencies of code that gets tested to the code that actually tests them i.e. from the object to corresponding tests. It is the same effect in that if the tests are too tightly coupled with details of the object being tested, every time the object implementation details change the tests break making unit testing a bottleneck for improving the system Sandi Metz2 recommends following tricks to handle this - 

  • Test only the public interface of the object taking into account whether the methods getting tested are Query (no side effects) or Command (with side effects) 

  • Do not test internal (private) methods as they are by definition implementation details and testing them only increases the coupling of tests and the object.

  • Test outgoing calls from an object at the API level i.e. just to ensure that the call is made and not the effects of the call are achieved. Eg: if an object X calls object Y that launches a rocket, testing object X should cover only the fact that it calls Y and not that a rocket was in fact launched. That should be covered in unit tests of Y.

  • Be mindful that Stubs and Mocks are different 

  • Ensure that tests using Mocks take care of “API drift” 

Ideally, none of the tests should break if the interface of the object doesn’t change. If there are tests that break when the implementation details change, delete them. 


Footnotes:

1 - http://www.cvc.uab.es/shared/teach/a21291/temes/object_oriented_design/materials_adicionals/principles_and_patterns.pdf

2 - https://www.youtube.com/watch?v=URSWYvyc42M


October 23, 2019

n-th percentile latency for fan-out requests

Reference: https://ai.google/research/pubs/pub40801

Problem: Say there is a server with 99th (n = 99) percentile latency  1 seconds, when a root server needs to serve requests consulting N such servers, what percentage of requests will have latency  1 second.

Solution: 1 - pow(99/100, N) 

Explanation: 

What we need to estimate is joint probability when at least one of the requests will have latency  1. Flipping the definition, subtracting joint probability of all the requests getting sub-second latency from 1 gives us the required value. 

Since fan-out requests can be assumed to be independent events, the joint probability of all the requests getting sub-second latency is (0.99 x 0.99 ... N times). 

Example: For N = 100, 1 - pow(0.99, 100) = ~0.63. It means, about 63% of requests will have latency  1 second, when a root server hits 100 servers with 99th percentile latency of  1 second. 

December 02, 2018

வேரறுத்த புயலால்...

இயற்கைக்கு என்னதான் தாகமோ
எங்கள் வாழ்வாதரத்தின்மீது...
யானைகட்டி சோறுடைத்த
சோழமண்டலம் - இன்று
கஞ்சிக்கு வழியில்லாமல் ...
அன்னாடங்காய்சிகளாய்...

எங்கள் தலைமுறையின்
எதிர்காலம், சேமிப்புகளெல்லாம்
புதைக்க இடமில்லாமல்
சிதறிகிடக்கின்றன...
கூரைவீடோ , ஓட்டுவீடோ
பாகுபாடில்லாமல்
கௌரவமா வாழ்ந்த எங்களை
நிர்வாணமாக்கி விட்டது..
ஒருவர் முகத்தை ஒருவர்
ஆறுதலாய் பார்த்து
நடைபிணமாய் அலைகிறோம்..
தொலைபேசி விசாரிப்புகளெல்லாம்
எப்படி இருக்கிறீர்கள்(சாகாமல்)
என்று கேட்பதுபோலுள்ளது...

பிள்ளைக்கு “பிள்ளையாய்” வளர்த்த
தென்னை மரங்கள் சாய்ந்த
பொழுதும் ,
 சாய மறுத்த பிள்ளைகள்
தாலியை இழந்த
தங்கையை ஞாவகப்படுத்துகிறது..

இந்த ஊடகங்களால்
டெல்டா விவசாயிகளின்
வெறுங்கைகளுக்கு
முழம் போட்டேன்ன பயன்..
அதனால்
சின்மயி(ர்)களுக்கு
சிக்கலேடுத்துக்கொண்டிருக்கிறது..
இவர்களின் வாதம்
உள்ளே புகமுடியவில்லையென்று
நித்தியானந்தாவின்
படுக்கையறைக்குள் புகுந்து
வேசித்தனம் பண்ணமுடிகிறது??..,.

சென்னைக்கு ஒன்றென்ரால்
இந்தியாவே அலறுகிறது..
ஊடகங்களும் உறங்க மறுக்கிறது...
(வருமானம் அப்படி....)
சென்னை மட்டும்தான்
தமிழ்தேசமென்றால்
இத்தனை அமைச்சர்கள்
எதற்கு???
இத்தனை தொகுதிகள்
எதற்கு???
எங்களுக்கும் அரசு
பின்(னா)ளில் அறிவிக்கும்
நிவா”ரணமாக” ...

ஹெலிகாப்டரில் பறந்து
பார்வையிட்ட எங்கள்
மகாராஜா ...
தேர்தல் நேரமென்றால்
ஹெலிகாப்டரில்
பறந்துகொண்டே
ஓட்டு கேட்பாரா???

அரசியல்வாதிகளின் லட்சனத்தை
எதேதோ புயலில் ஆரம்பித்து
கஜா புயல்வரை பார்த்தாயிற்று
இன்னுமா இந்த நாய்களுக்கு....

அரசாங்கத்து வேலைகூட வேண்டாமென்ற
திமிர் பிடித்த ரத்தம் இன்னும்
ஓடிக்கொண்டுதான் இருக்கிறது...
வருவோம்..மீண்டும்
மீண்டு..வருவோம் ...
திமிரோடு..
எங்கள் மண் காக்க..
தலைமுறை காக்க..
எம்மக்களை காக்க..
அரசியலறிந்து நாட்டை காக்க...

ப(பொ)ணந்திண்னி அரசியல்வாதிகளையும்,,
கைகூலி ஊடகங்களையும்,,
மறுதலித்து
அரசியல் தேவையறிந்து..
நாங்கள் வேண்டுவது
ஆட்சி மாற்றமல்ல
அரசியல் மாற்றம் ...
ஜனநாயக மாற்றம்..
                         இவண்,
                 —-ஏகலைவன்—
                              23/11/18
                              ஆம்பலாப்பட்டு..

September 05, 2018

Some nuggets of wisdom

Source: http://muratbuffalo.blogspot.com/2012/05/my-advice-to-2012-class.html

1. Knowing a thing and knowing its name are not same - Feynman (not exact paraphrase tho'). In other words, you won't learn much by just following how it is done rather by actually doing it. Doing the thing first hand, making mistakes and working through helps you to internalize the material.

2. Expertise is a result of years and years of practice. Not innately knowing something.

3. Achieving mastery on a topic, in a short period of time, is not an end but rather a beginning for years of apprenticeship towards perfection in the craft.

August 30, 2018

Insertion and Quick Sort - No explanation.

import sys
import random

def insertion_sort(a):
    """ Returns the sorted version of 'a'
    """

    for i, _ in enumerate(a):
        for j in range(i):
            if a[i] < a[j]:
                a[i], a[j] = a[j], a[i]

    return a
            
def quick_sort(a):
    """ Returns the sorted version of 'a'

    Quick sort - find a pivot element and return
        quick-sorted elements less than pivot +
        pivot +
        quick-sorted elements greater than pivot.
    *** Following implementation is not optimal in space ***
    """

    if len(a) <= 0:
        return a

    # use first element as pivot - not so superior, but doesn't matter.
    pivot = a[0]
    left = [x for x in a[1:] if x <= pivot]
    right = [x for x in a[1:] if x > pivot]
    
    return quick_sort(left) + [pivot] + quick_sort(right)

def quick_sort_helper(a, _min, _max):

    if _max <= _min:
        return a

    # pick a random pivot element
    _piv = partition(a, _min, _max)
    quick_sort_helper(a, _min, _piv-1)
    quick_sort_helper(a, _piv+1, _max)
    return a

def partition(a, _min, _max):
    """
    1. Put the pivot in its position
    2. Partition the array in place so that all elements less than pivot
    are arranged to left of it and vice-versa
    
    """
    _piv = random.randint(_min, _max)
    pivot = a[_piv]
    
    i = _min+1 # pivot will be put in first position
    j = _max

    a[_piv], a[_min] = a[_min], a[_piv]
    while True:
        while (i <= j) and (a[i] <= pivot):
            i += 1
        while (j >= i) and (a[j] > pivot):
            j -= 1
        if i < j:
            a[i], a[j] = a[j], a[i]
        else:
            break

    a[_min], a[j] = a[j], a[_min]
    return j

def quick_sort_opt(a):
    """ Returns the sorted version of 'a'
    Space efficient.
    """

    if len(a) <= 0:
        return a

    return quick_sort_helper(a, 0, len(a)-1)

if __name__ == "__main__":
    print(insertion_sort([4, 3, 2, 1]))
    print(insertion_sort([10,3,6,1,2,-4,7,23,22,21]))
    print(insertion_sort(list('INSERTIONSORT')))
    print(quick_sort([4, 3, 2, 1]))
    print(quick_sort(list('insertionsort')))
    print(partition([4,3,2,1], 0, 3))
    print(partition([6, 8, 6, 7, 2, 3], 0, 5))
    print(quick_sort_opt([4, 3, 2, 1]))
    print(quick_sort_opt(list('insertionsort')))
    print(quick_sort_opt([6, 8, 6, 7, 2, 3]))
    print(quick_sort_opt([2]*8))
    print("In main - Hello world!")

January 23, 2016

Ubuntu hangs during boot after upgrade to 14.04

After I upgraded my laptop to Ubuntu 14.04, first the system failed to start because GRUB couldn't start for whatever reason. It needed "boot repair" to fix the problem.

After the above issue was fixed, the system started booting but was hung in the splash screen. Ctrl + Alt + F helped to look at the boot messages and I noticed that it was stuck after outputting "Restoring resolver state". This thread pointed me to a boot level setting to add "acpi=force" and then this helped boot to continue.

Next, the boot sequence again stalled after outputting "Starting CUPS printing spooler/server". After some googling around, I found that I could open a tty using Ctrl + Alt + F2. Opened a terminal and tried /etc/init.d/lightdm restart as pointed out in this thread.

It brought up the login screen. I then learnt that I should specify full path of lightdm i.e. /usr/sbin/lightdm in /etc/X11/default-window-manager for this to work properly because lightdb doesn't land in /usr/bin/lightdm always (Reference: this).

Also, in one of the above steps I added a boot option "nomodeset" along with "acpi=force" but I am not sure if that helped or not.

Update: I guess the above "nomodeset" option caused Ubuntu to not load the video driver. This resulted in horrendously high CPU usage by "compiz" process. Removing that option caused the CPU usage to drop to normal.