May 24, 2015

Teachers at St. Xavier's Calcutta - circa 1970

Before these names fade out of memory ...

Front row : Mr Subramanium, Mr Tony D'Abrew, Fr Bouche, Fr Wavreil, Fr Desbrulais, Mr Verma, Mr Dey (Goba)
Second row :Mr Nelson, X, Mr Sajal Banerjee, Mr Kamalendu Chaudhury[?], Mr Engineer, Mr Abraham[?], Mr Leslie Davey, Mr Mishra, Mr A K Samajpati, Mr A P Sarkar, Mr Carlyle Rosario, Mr Sushil Sarkar[?], Fr Maliyekal[?], Mr Nemai Sengupta[?], Mr Lobo, Mr Les D'Gama, X,X, Dr Magno Correa
Back Row : Mr Redden, Mr T Vianna, Mr Gomes[?] Mr Rai[?], Mr Ganga Singh, Mr Balai Banerjee[?], Mr Chittaranjan Roy, Mr Tripathi, Fr L Hous, Mr Pinto, Mr Brown, Mr Gass

original picture with tags available at Les D'Gama's facebook timeline

May 19, 2015

Technology, Management & Systems : The Holy Trinity ?

India has a strange fascination for engineers and MBAs. Everybody wants to become one, or preferably both. So is the case with systems, or as they say in India, the IT sector. But this fascination is not because of any natural aptitude for these disciplines but simply because they help one to get a job in an otherwise dismal economy. This is unfortunate, because if we step back for a moment and think through  issues that haunt this country, it would seem that our salvation lies in leveraging this holy trinity to dig us out of the hole that we find ourselves in!

Let us consider a few exemplary scenarios.

Till the 1990s, telephones in India were a disgrace. While landline technology was readily available and widely used all across the developed world, we were still at the mercy of the corrupt and inefficient P&T department that ensured that very few of us had access to one. This changed dramatically with the arrival of cell-phones that  bypassed the constraints of the local loop and managed to put a phone in every hand. But just as we were about to take off, the evil empire struck back with the 2G scam that again put us back by several decades in 3G and 4G services.
If we go back further, to the 1960s, we would see that India was facing a massive shortage of food and waiting for PL480 handouts from the United States to fight famines. Then again it was a burst of technology -- food technology, in the form of better fertilizers and high yielding crops -- that saved the day. We managed to stave off starvation but once again the theft in the public distribution system and the mismanagement of the supply chain -- as visible in images of grains rotting in godowns --   brought back the spectre of Kalahandi and haunts us even today.

In fact, India lost the plot much earlier, in the middle ages when we failed to board the Renaissance bus. While the Mughals were celebrating the high noon of their culture with the Taj Mahal and Urdu shayaries, Europe was passing us by with the structured and rational thinking of Galileo, Rousseau and Newton. Administrative systems like the modern nation state, with its civil and military services, economic systems like joint stock companies, banks, insurance agencies and educational systems built around universities that granted formal degrees never really got off the ground in India in the same way that they did in Europe. Not until the British brought them here.

Systems, in this context, are a metaphor for the rational approach to address social and commercial problems -- free of divine directives or bureaucratic whimsy. Such systems result in the development of superior technology and in the efficient usage of natural and social resources. This in turn reduces the  degree of social conflict, increases the physical living standards and empower societies with the luxury of confidence. Such systematic societies can then confront, conquer and convert societies that are irrational, unstructured or unsystematic -- as we have seen in the case of Europeans conquering large parts of America, Asia and all of Australia.

Computer “systems”, that appeared much later, draw their inspiration, and of course the moniker systems, from the same structured, rational,  or systematic approach that they bring to bear on solving problems and are a specific example of a more general approach. Can such systems free the future of India from the confines of its past and its present?

A little introspection will show us that we are an inherently irrational in our thoughts and chaotic, if not anarchic, in our deeds. In our irrationality, we could have still lived with our little obsessions with gods, “godmen” and superstitions but the real problem is when the irrationality seeps into our secular and political structures. We scream and protest against corruption but are the first when it comes to boasting about how we could bribe our way through the system and -- this is far worse -- a majority of us would not hesitate to seek bribes and favours whenever we are in a position to do so. This is particularly true for anyone with any kind of discretionary power within the government -- whether it is a peon, who will not let you meet the officer, or the officer himself who can sign a paper and make your life a little easier.

Corruption is a characteristic that is woven into the warp and woof of the Indian administration and yet we have the unedifying spectacle hundreds and thousands of irrational souls sitting on dharna at Jantar Mantar and asking for a Lokpal -- as if one man can do what a whole zoo full of institutions like the CAG, CVC, CBI could not achieve! We refuse to believe, to quote a Bengali maxim, that the evil spirit is in the very mustard that is being used for the exorcism!

Our irrationality extends to our naive belief in the democratic process where we cannot see through the chicanery of the promises made by the candidates. We believe that it is correct to vote for the person who promises illegal and untenable benefits for me and my caste. This natural irrationality is extended and reinforced by our love for anarchy. We believe that burning buses and other public property, calling bandhs and blockading roads is a natural and justified way in which our “leaders” or elected representatives can help us towards a better life. This irrationality is not the exclusive preserve of the uneducated and we see even people with college education passionately espousing economic theories that have been discarded in the dustbins of history. Robust, rational pragmatism is hard to come by in this country.

We could go on with similar anecdotes but the point is that as a nation we are incapable of governing ourselves. In state after state and in every Lok Sabha election we have voted for, elected and given to ourselves the terrible governments that we, as an irrational and anarchic people, justly deserve. But does this mean that it is time for the British Parliament to repeal the Indian Independence Act of 1947? Fortunately, there could be a better solution  based on the trinity of technology, management and systems.

Expecting government officials to shun corruption or the electorate to vote rationally is like expecting Kolkata to have a weather like Darjeeling. There may be a few exceptions here and there but by and large, very, very unlikely. We need to work with these two chips on our shoulders. Bending, breaking and abusing the system is a leitmotif of India. Individual Indians may be sane and rational but in a mass and as a collective, they will never be. This the foundational principle on which the governance of India needs to be based.

Since people are at the heart of the problem we need to minimise their role in discretionary decisions and, to the extent possible, from the delivery process. Cell phones succeeded where land line phones failed because they did not need an army of corrupt, anarchic people to maintain the thousands of lines running across the country. The towers are unmanned and the central switches need a few competent people. This is a perfect example of a technology trumping the accumulated hubris of centuries and is the model that we must try to emulate in other areas as well.

This though, is easier said than done!

The technology is never the issue but to implement it against the wishes of people, who see this as an infringement of their fundamental right to be corrupt or anarchic, is the real challenge. This is where smart management techniques come in very handy. The key is to use the carrot and the stick to cajole, convince, convert, confuse or coerce everyone so that they have no option but to be yoked to structured, technology-enabled systems. Individual brilliance and creativity is great and diversity is something wonderful to celebrate but, if cars do not stop at traffic lights but only when people people block the road, then society collapses into the kind of anarchy that India is familiar with. Net-net we need to design systems that will bring technology and management techniques into the governance process in a manner that minimizes the need for people in the governance process.

Is this possible at all ? To a large extent, yes.

Since data should be the basis of any rational decision, our systems must forcibly collect data and place it in the public domain. Next a clear set of algorithms, or rules, must be put in place so that the data itself drives decisions -- say, for example, approvals for or limits on expenditure, the quantum of taxes due -- in a way where humans have only a supervisory role. Finally the data, the process of arriving at a decision and the decision itself must be automatically visible to the public. This is a generic template for transparent governance. A simple example would be a Wikimapia style map showing physical locations of NREGA projects along with time-stamped, GPS encoded pictures shot before and after the project is executed -- without which no further funds will be released to the panchayat in question. Three previous articles in these columns have shown how similar systems can indeed be designed to help expedite justice in courts, facilitate elections and track corruption at the operational level.

The design and implementation of such systems would of course eliminate a lot of redundant but hugely lucrative positions in the administration and so would be stoutly resisted by an army of the most corrupt. This can be overcome only if the elected leadership has the political will and the administrative wherewithal to place a few honest and technophile administrators at key decision making posts in government. This is the only, and minimal, ask if we want to see technology enabled rationality in the governance of this country.

The tyranny of a Singapore style benevolent dictatorship may pose too big a risk for a big, multicultural country like India but the tyranny of systems developed and deployed by a few smart and well meaning people employed by an elected government is the answer to India’s perennial problems.

This article originally appeared in Swarajyamag -- the Magazine that reads India right

May 03, 2015

Maps of India : DIY with R and GADM data

Displaying spatial data on maps is always interesting but most Visualisation tools do not offer facilities to create maps of India, especially at the state and lower levels. In this post, we will show how such maps can be made.

The base data for such maps, the "polygons" that define the country, the states, the districts and even the talukas ( or sub-divisions) is available from an organisation called Global Administrative Areas or Country level files for almost all countries are available in a variety of formats including R and these are at three different levels. For India, these files can be downloaded as IND_admN.RData where R = 1,2,3. These will form the raw data from which we will create our maps.

Unfortunately, the GADM files represent a truncated Kashmir. How I wish that the Government of India and the National Atlas and Thematic Mapping Organisation would publish similar files for us. Anyway, we work with what we readily have ...

Working with R, we will need two R packages :

# Load required libraries

Assuming that the downloaded RData file is located in the R working directory, the following code will generate a basic India showing the states

# simple map of India with states drawn
# unfortunately, Kashmir will get truncated
spplot(ind1, "NAME_1", scales=list(draw=T), colorkey=F, main="India")
Now suppose there is some data ( economic, demographic or whatever ...) and we wish to colour each state with a colour that represents this data. We simulate this scenario by assigning a random number ( between 0 and 1) to each state and then defining the RGB colour of this region with a very simple function that converts the data into a colour value. [ This idea borrowed from gis.stackexchange ]

# map of India with states coloured with an arbitrary fake data
ind1$NAME_1 = as.factor(ind1$NAME_1)
ind1$ = runif(length(ind1$NAME_1))
spplot(ind1,"NAME_1",  col.regions=rgb(0,ind1$,0), colorkey=T, main="Indian States")

Now let us draw the map of any one state. First check the spelling of each state by listing the states:

and then executing these commands :
# map of West Bengal ( or any other state )
wb1 = (ind1[ind1$NAME_1=="West Bengal",])
spplot(wb1,"NAME_1", col.regions=rgb(0,0,1), main = "West Bengal, India",scales=list(draw=T), colorkey =F)

# map of Karnataka ( or any other state )
kt1 = (ind1[ind1$NAME_1=="Karnataka",])
spplot(kt1,"NAME_1", col.regions=rgb(0,1,0), main = "Karnataka, India",scales=list(draw=T), colorkey =F)

If we want to get and map district level data then we need to use the level 2 data as follows :

# load level 2 india data downloaded from
ind2 = gadm

and then plot the various districts as

# plotting districts of a State, in this case West Bengal
wb2 = (ind2[ind2$NAME_1=="West Bengal",])
spplot(wb2,"NAME_1", main = "West Bengal Districts", colorkey =F)

To identify each district with a beautiful colour we can use the following commands :
# colouring the districts with rainbow of colours
wb2$NAME_2 = as.factor(wb2$NAME_2)
col = rainbow(length(levels(wb2$NAME_2)))
spplot(wb2,"NAME_2",  col.regions=col, colorkey=T)

As in the case of the states, we can assume that each district has some (economic or demographic) data and we wish to colour the districts according to the intensity of this data, then we can use the following code :

# colouring the districts with some simulated, fake data
wb2$NAME_2 = as.factor(wb2$NAME_2)
wb2$ = runif(length(wb2$NAME_1)) 
spplot(wb2,"NAME_2",  col.regions=rgb(0,wb2$, 0), colorkey=T)

But we can be even more clever by allocating certain shades of colour to certain ranges of data as with this code, adapted from this website

# colouring the districts with range of colours
col_no = as.factor(as.numeric(cut(wb2$, c(0,0.2,0.4,0.6,0.8,1))))
levels(col_no) = c("<20%", "20-40%", "40-60%","60-80%", ">80%")
wb2$col_no = col_no
myPalette = brewer.pal(5,"Greens")
spplot(wb2, "col_no", col=grey(.9), col.regions=myPalette, main="District Wise Data")

To move to the district, sub-division ( or taluk) level we need to use the level three data file :

# load level 3 india data downloaded from
ind3 = gadm

# extracting data for West Bengal
wb3 = (ind3[ind3$NAME_1=="West Bengal",])

and then plot the subdivision or taluk level map as follows :

#plotting districts and sub-divisions / taluk
wb3$NAME_3 = as.factor(wb3$NAME_3)
col = rainbow(length(levels(wb3$NAME_3)))
spplot(wb3,"NAME_3", main = "Taluk, District - West Bengal", colorkey=T,col.regions=col,scales=list(draw=T))

Now let us get a map of the district - North 24 Parganas. Make sure that the name is spelt correctly.

# get map for "North 24 Parganas District"
wb3 = (ind3[ind3$NAME_1=="West Bengal",])
n24pgns3 = (wb3[wb3$NAME_2=="North 24 Parganas",])
spplot(n24pgns3,"NAME_3", colorkey =F, scales=list(draw=T), main = "24 Pgns (N) West Bengal")

and within North 24 Parganas district, we can go down to the Basirhat Subdivision ( Taluk) and draw the map as follows: 

# now draw the map of Basirhat subdivision
# recreate North 24 Parganas data
n24pgns3 = (wb3[wb3$NAME_2=="North 24 Parganas",])
basirhat3 = (n24pgns3[n24pgns3$NAME_3=="Basirhat",])
spplot(basirhat3,"NAME_3", colorkey =F, scales=list(draw=T), main = "Basirhat,24 Pgns (N) West Bengal")

This is the highest resolution ( or lowest administrative division ) that we can go with data from gadm. However even within a map,  one "zoom" into and enlarge an area by specifying the latitude and longitudes of a zoom box as shown here.

# zoomed in data
wb2 = (ind2[ind2$NAME_1=="West Bengal",])
wb2$NAME_2 = as.factor(wb2$NAME_2)
col = rainbow(length(levels(wb2$NAME_2)))
spplot(wb2,"NAME_2",  col.regions=col,scales=list(draw=T),ylim=c(23.5,25),xlim=c(87,89), colorkey=T)

With this it should be possible to draw any map of India. For more comprehensive examples of such maps, please see this page.

new PostScript : The full code for creating this maps as well as additional information on how to place text and markers on these maps is available on my specialist visualisation blog.