January 02, 2018

Astrology - an application of Data Science

Every morning, when the sun rises, birds wake up and start chirping. We are not surprised because we see a very clear connection between a cause, the increase in ambient light, and the effect it has in waking up the birds. But when the sun rises, the Ispat Express moves out from Howrah station as well. Even in this case we are not surprised even though there is no obvious connection between the ambient light and behaviour of the engine. We explain the first phenomenon through causation and the second through correlation. Perhaps if we look very hard we might find a causative mechanism that explains why the Ispat Express departs at the crack of dawn and not at any other time, but frankly that is of little interest to the passenger. We are happy with the correlation, we are not really bothered about the causation. That is the crux of this article where we explore astrology from the perspective of data science and machine learning.

Traditional scientists would of course hesitate to view astrology as a science because they are trained to interpret phenomena through the prism of cause and effect. Naive astrologers also fall into the trap of trying to justify astrology by invoking gravitational, electromagnetic and even undiscovered, unheard of “rays” that emanate from astronomical bodies and influence human destiny. Both are wrong because they are searching for causation whereas the underlying principle is one of correlation. Astrology is interested in predicting an outcome based on some correlation -- an event, say an accident, happens not because a certain planet was at a particular location and caused the event but because it has been historically observed that the presence of a planet at a particular position is associated or strongly correlated with the occurrence of that specific event.

Is correlation an acceptable way to analyse situations and predict outcomes? Certainly, if we consider ...

The world of data science, of which artificial intelligence (AI) is a specific example, is largely based on the study of correlations. Most well known AI systems -- for example those that control autonomous, self-driving cars, recognise people and objects in photographs, react to voice commands, play board games like GO or perform other tasks that eerily mimic human behaviour -- are based on the technology of Artificial Neural Networks (ANN). In all such cases, the system is presented with a set, or pattern, of input data, that in the case of face recognition, would be the colour and brightness of light at every point of an image. If the system has been “trained” adequately, it should predict the identity of the person in the image even though this particular image has never been encountered before. A similar ANN software, if “trained” differently, and presented with an appropriate set of input data, should also be able to predict the best possible move in a competitive board game, the possible motion of a pedestrian in the vicinity of an autonomous car, or the intention and ability of a bank customer to repay a loan. In all such cases,  “training” consists of feeding the system with thousands of pieces of historical data and identifying which of the corresponding predictions were correct and which were not. Initially, most predictions will be erroneous but after a few thousand errors, the system will “learn” or get “trained” and from then on, the number of correct predictions would improve dramatically.

What happens inside the system, that is almost like a black  box, is not very clear. Some parameters, also known as weights, are assigned numeric values, but it is not possible to explain why specific values were chosen. All that is known is that changing these values leads to a change in the accuracy of the prediction. Once the values have been set correctly -- the system has been “trained” -- the subsequent predictions are almost always correct. This situation, where the system works well but we cannot explain why it works well,  is described in “The Dark Secret at the Heart of AI” [ MIT Technology Review, April 2017, see ]. This article describes a new kind of autonomous car from NVidia that is not explicitly programmed to drive through traffic.  Instead, it learns driving by observing the environment and recording the actions of a human driver who is navigating through the environment.

“Getting a car to drive this way was an impressive feat. But ... it isn’t completely clear how the car makes its decisions. Information from the vehicle’s sensors goes straight into a huge network of artificial neurons that process the data and then deliver the commands required to operate the steering wheel, the brakes, and other systems. But .. if .. it did something unexpected ... it might be difficult to find out why. The system is so complicated that even the engineers who designed it [cannot] isolate the reason for any single action. And ... there is no obvious way to design such a system so that it could always explain why it did what it did.”

This inability to explain the reason for a behaviour is not because the system has supernatural or magical properties. This is a purely deterministic machine, that follows the laws of cause and effect at the micro level. But the situation is so complex -- in terms of the the number of micro-operations and the way they are related to each other --  that the connection between cause and eventual effect is impossible to establish at the macro level. But this does not detract from the fact that these and many other similar systems are being successfully used in situations where there is a need to predict outcomes based on (a) a set, or pattern, of current input data  and (b) historical, or “training”, data that maps thousands of  similar patterns observed earlier  to corresponding outcomes that have actually occurred.

Since this approach is “scientifically” acceptable in the case of AI research, we can extend the same to astrology! But before we do so, there are two difficult challenges to overcome.

The first challenge is the question whether the world is indeed deterministic, and hence predictable?  -- are all events pre-determined? -- or do events happen because of free will or pure chance? This profound question cannot be answered in this short article. So we will seek an acceptable answer that skirts the quicksands of philosophy and yet helps us move ahead without being facetious.

We accept that at the micro level the world is indeed completely deterministic but not so at the macro level. For example, a man dies in a plane crash because he boarded the aircraft to attend a meeting that was called by his boss and the mechanic who repaired the aircraft engine did not tighten a nut on an engine because he was thinking about his wife. The meeting was called by the boss because {of some reason} and the mechanic was thinking of his wife because { of some other reason } and this chain of events or reasons can stretch backward through millions of interrelated links that hide the eventual effect from the possible causes. Conceptually, this is no different from the behaviour of ANN systems, where despite the existence of deterministic causality at the micro level, we are not in a position to explain the full chain of causality.

A simple demonstration of such a mechanism is the 2003 Honda advertisement ( https://www.youtube.com/watch?v=_ve4M4UsJQo ) that shows how interrelated the world could be. A tiny gear rolls across the table, pushes a lever, that in turn causes some liquid to spill, that in turn causes a table to tilt, that in turn causes a spring to snap … and so on and on … until a Honda Car rolls out of the store. This video demonstrates how precisely Honda technology works and so one can see how a small change in any of the actions will result in the final car not moving. But unless this was explicitly shown, one would never know how the movement of a small ball or the flow of a liquid can in any way affect the final outcome. A similar video is available in the Facebook Japanese Community page ( http://bit.ly/2eUXAks ) that shows an even more complicated piece of mechanism where apparently unrelated movements of many objects finally end with a desired result.

To the profound question of whether the world is deterministic, a simple answer would be that while the world must indeed be deterministic, it may not be possible to establish a clear and unambiguous cause-and-effect linkage between two events that are widely separated in space and time.

The second and more difficult challenge is to explain how the position of astronomical bodies could in any way be a part of this huge chain of inexplicable causality. This is where we completely discard causality as the operating principle and instead, introduce correlations into our description of astrology.

The universe as we know it is a long and complex chain, or web, of interconnected events, that have a location in space and time. From this universal set of all events, a certain subset is related to human beings and of which an even smaller sub-subset is related to a specific individual. So the life of every individual can be represented by a unique set of events, many of which he shares with others. Indian astrology considers  ten astronomical objects, namely, the Sun, the Moon, five actual planets, the Lagna or ascendant horizon and Rahu and Ketu, the two points in space where the lunar orbit intersects the ecliptic plane.  For every person, the presence of these ten notional “planets” at specific locations on the Zodiac at the moment of birth are also events that are a part of the huge collection of events that define the life of the person.

Each event in a person’s life, including these ten planetary events, can be interpreted as a piece of data. Some of this data is binary - the event happened or did not happen; others can be interpreted as ordinal -- for example, favourable, neutral or unfavourable; categorical -- for example career in medicine, law etc; or even numeric -- for example, age at marriage or at death. With this interpretation, the life of a person becomes, what mathematicans refers to as, a vector in a multi-dimensional space, with the number of dimensions being equal to the number of events used to represent his life. Pictorially, every person can now be viewed as a point in multi-dimensional space and a group of persons looks like a random scattering of dots in this multi-dimensional space.

But are these points scattered randomly? Or do they form clumps and clusters?

Data science tools -- that is software programs based on algorithms like K-Means, K-Nearest-Neighbour, Decision Trees, Logistic Regression and of course the very popular ANN --  are quite efficient in detecting and isolating clusters by identifying points that are “similar” to others. This is routinely used in recommendation systems used by websites like Amazon and Netflix to identify individuals who have similar preferences -- individuals in a cluster would tend to display similar buying preferences -- and show them advertisements of products that they are likely to buy. In fact these recommendation systems try to predict who is likely to buy what.

For example, consider the attached figure, based on representative, simulated data, that shows the kinds of movies that people prefer. If the data of many people are plotted as points in  three-dimensional space where the three dimensions are represented by gender (shown as colour, blue or red), preference for action movies ( position on Y axis) and preference for romantic comedies (position on X axis), then there will be two clear clusters of points along with a few outliers. From the diagram, or actually the data, we can clearly see that men prefer action movies and women prefer romantic comedies.

Can we explain why men prefer action movies and women prefer romcoms? No, the diagram offers no explanations, and we do not care. But we do know that there is a clear correlation between gender and movie preferences. So by knowing a preferences of movies we could, in principle, predict whether a person is a man or a woman. Obviously with more data and more dimensions our prediction would be more accurate but there is nothing magical or supernatural about this prediction that is based on data and data alone. This ability to predict events without being concerned about the cause of the event is the basis of all forms of astrology.

But of all the events that define a person’s life, why is it that these ten planetary positions are the ones that considered in our search for correlations with other life events?

In principle, many mathematical techniques can be used to cluster people defined through their hundreds of dimensions. In practice this becomes difficult when the number of dimension becomes very large. Data scientists call this “The Curse of Dimensionality” where the number of dimensions is high compared to the number of data points available. Fortunately, there exist sophisticated mathematical tools, like Principal Component Analysis (PCA) and Factor Analysis (FA) that help identify key or important dimensions and allow the scientist to ignore others. Not everybody needs to understand the complex mathematics of PCA and FA but everyone uses it to reduce the number of dimensions to a manageable number.

Dimensions are also reduced through mathematical projections. Civil and mechanical engineers regularly project a three dimensional structure, say a building or a machine component, into two dimensional images called plan and elevation. This process can be generalised by projecting an n-dimensional structure into an m-dimensional surface where m is less than n. Representing high dimensional data pictorially and making it visible or understandable to a lay person is difficult, but for mathematicians, this is a very routine matter because these are standard mathematical operations.

From the thousands of potential dimensions that could be used to describe a person’s life, astrology chooses a smaller subset --  the positions of the ten astronomical bodies, or “planets”, on the various charts -- and this is used to identify clusters. Why do we choose the dimensions that are based on planetary positions at birth? The most likely reason would be that, thanks to astronomy that is closely associated with astrology, these are dimensions that are guaranteed to have numerical values that can be accurately determined if the date, time and place of birth is known. Other dimensions, associated with event, like omens, may not have valid data for everyone. If we choose these planetary dimensions -- and of course their derivatives like, aspects, conjuncts, combinations, Navamsha positions etc -- we are assured of a standardised structure that can be used to create a lower dimension surface, a  projection, of a person’s life that is otherwise defined as a point in a space of far higher dimensionality. So the choice of the planetary positions as key variables is an exercise in reduction of dimensionality -- a process that is, again, quite common in data science.

Once we plot the lives of individuals on this lower, ten-dimensional surface, then the search for clusters becomes a tractable problem. For example, mangaliks -- those with Mars  at the 4th, 8th or 12th position from the Ascendant or the Moon -- are likely to have marital problems unless they are married to fellow mangaliks. More specific events require far more complex patterns to be identified. For example, the birth of a child is associated with a fairly complex pattern in the charts of both parents that involves the dashas of planets associated with the Ascendant, the 5th and 9th houses and of Jupiter. The specific birth event is associated with another complex pattern that involves Jupiter, Saturn, Mars and Moon casting their aspects on the 5th and 9th house. Similarly, if a person is in the medical profession, it is very likely that the chart will show an afflicted Moon, a strong Jupiter associated with Ketu and Sun and an association of the 5th Lord with the Lords of 6th, 8th  and 12th houses. These are examples of positive correlations, where the patterns and events happen together, but there are also cases of negative correlations, the “limiting” patterns, that do not occur with certain events and seem to inhibit, rather than cause these events.

Such complex patterns are not easy to spot by manual means and the failure to do so leads to errors in prediction. Competent astrologers, like good chess and card players, are adept at spotting patterns and the inablity to do so is the reason why many astrologers end up with wrong predictions. However, the usage of data analytics tools, particularly software tools, can eliminate the possibility of missing patterns and improve accuracy of predictions.

Can we explain why these specific patterns are associated with specific events? No, and we do not need to. This is similar to the correlation between the gender and movie preferences that we have encountered earlier. Since we observe a correlations between patterns in the horoscope and events like marital bliss, childbirth and professions, we can use the existence of these correlations to  predict the occurrence of events on the basis of the existence of the patterns in the chart. As in the earlier case, there is nothing magical or supernatural about these predictions -- it is based on data and data alone. This ability to predict events without being concerned about the cause of the event is the basis of all forms of astrology.

These examples are taken from the area of natal astrology where the date and time of the person’s birth is the key variable but there are other types of astrology as well. For example, in horary astrology it is the date and time when a question is being asked and in mundane astrology or “Medini Jyotish”, it is the date and time of the creation or Independence of a country that is used to predict it’s destiny. In all such cases, the position of the planets at a point in time are the principal dimensions on which prediction models are built.

We have now overcome two major hurdles in our attempt to show the similarity between the predictive powers of astrology and data science. First, we have addressed the question of determinism and second, the importance or necessity of planetary positions. Let us now look at the next big hurdle -- the availability of data, or the lack thereof!

Data science begins with large sets of training data that has been collected in the past. It then uses mathematical techniques to deduce the rules of correlation that associate patterns of data with corresponding outcomes. In astrology, the history of this training data is lost in the mists of antiquity but the rules have survived. Are these rules perfect? Are they always correct? Most of them are, but possibly some are not,  but that is equally true for data science as well. Errors and wrong predictions are quite common in  data science but these can only be reduced if we have more and better data.

Physics as we know it today got a head-start over life sciences and social sciences because it was data driven. Tycho Brahe was the first European scientist who collected and collated a huge amount of data on the movement of planets. This data was subsequently used, first by Johannes Kepler to identify the correlations between duration and size of planetary orbits and then by  Isaac Newton who formulated the laws of gravitation. As an aside, it is interesting to note that Newton never really explained the cause of attraction, why bodies attract, but only identified the circumstances under which the attraction was observed. Eventually it was left to Albert Einstein to explain gravitation as arising out of the curvature of space-time but that is another story.

Like Kepler and Newton in the world of European physics, India has had sages like Bhrigu, Parashara and Jaimini who identified the laws of correlation that we have inherited today as the principles of astrology. But the the training data that might have been used to deduce these laws is not available any more. We can only speculate that Indian astronomers like Aryabhatta might have collected data like Tycho Brahe did, but unfortunately, there is no hard evidence for this. It has also been suggested that sages like Bhrigu had divine insight and rules of astrology were revealed to them through a direct and transcendent vision similar to the knowledge of the Vedas. We can respectfully consider such possibilities, but the exploration of such divine origins is beyond the scope of the current article. In fact it is best to keep religion and faith away from this discussion because astrology is not restricted to people of Indic origin or beliefs but is something that is found across many cultures in Asia and Europe. In fact, Tajik jyotish, one of the systems of Indian astrology -- like Parashari, Jaimini and others -- is supposed to have originated in the Middle East.

Data science validates its predictive models by testing them against a part of the historical data but astrology cannot validate its predictive models because there is no historical data to validate its models with. This is why classical scientists are reluctant to accept astrological predictions while being quite comfortable with the predictive abilities of data science. This can be addressed by creating a pool of training and test data from the personal information of thousands of real people. We need data on the  date, time and place of birth followed by data on key life events like dates of college admission, marriage, birth of children, entry into job, change of job, change of residence, major illness, loss of job, loss in business, death and illness of parents, spouse, siblings or children plus qualitative data like profession, level of wealth, happiness, spirituality, place of residence, relationship with spouse, siblings, parents children and so on. Coincidentally, a similar data collection is already happening every day in the world of internet and social media! Companies like Google, Facebook, Amazon, Uber and many others are already collecting personal data, often at the cost of privacy. Many of these companies have made it their business to collect personal data and then either use it to build their own predictive models or sell it to those who do. With a little bit of imagination, a similar data collection initiative could devise mechanisms that will incentivise people to voluntarily contribute data about events in their life, in return, perhaps, for a detailed astrological reading or other benefits.

Once sufficient data is collected it must be codified and stored with modern database technology so that it can be accessed from computer programs. Standard relational database technology can be used but since the data is of high dimensionality with many missing values, sparse matrix techniques or columnar databases can be used for efficient storage. Once the data is accessible to programs, then the same data science tools that are used by companies like Google to build predictive tools for, say, voice or image recognition,  can be used to either validate traditional predictive models of astrology or create a new rules of prediction. In fact, these new models may contradict traditional models and generate more accurate predictions. Going further, once we associate astrology with data science, then it can be included in the curriculum in universities. This will not only result in more structured research and better models but will also drive hundreds of fraudulent astrologers -- who sell fake remedies to gullible people, out of business. Astrology, like data science, can make predictions but cannot change destiny. There are some who claim that it could -- but that is another debate that is beyond the scope of this article.

Isaac Newton had said that he could see further because he was standing on the shoulders of giants. Astrology today is based on traditions of scholarship whose origins are long lost in the swirling mists of myths, legends and history. This traditional wisdom is an excellent foundation, a starting point, on which we could build the edifice of a new science of astrology that will be powered by tools and techniques that are commonly used in the corporate world today. Pattern recognition and machine learning techniques can be used to build predictive models with data  about key life events of thousands of interested volunteers connected through social media platforms. Such models will help identify natal planetary patterns, recorded at birth, and their degree of correlation with subsequent life events. This will either confirm, or contradict, ancient astrological aphorisms or even lead to new clues about what could lie in the womb of futurity.

As a data driven science, astrology must move from the Age of Parashara to the Age of Google.

Acknowledgment : The astrological patterns alluded to in this article have been identified by eminent astrologer K N Rao and explained to the author by his IIT Kharagpur batch mate, Raghuram Hebbar, consultant engineer and amateur astrologer extraordinaire. Any misunderstanding or misrepresentation of astrological rules is obviously the fault of the author.

This article originally appeared in Swarajya, the magazine that read India right

December 28, 2017

The fourth R : Computer Programming in Schools

The three ‘R’s -- Reading, wRiting and aRithmetic -- are the three fundamental skills that are compulsory in the curriculum of all primary schools. Should another R --  computer progRamming --  be also included as the fourth fundamental skill?  Computer programming -- as in Java or C++ -- is already being taught as an optional subject in many schools in classes  IX and X as  and Computer Science is generally available as the “fourth” subject in classes XI and XII. Then why do we need to make it compulsory and burden our already stressed out kids with one more subject?

Before we address this question, we must first stop viewing computers as just another career option, a vocational skill, on par with, say, medicine, accounting, music or sports.  Instead, we need to see it as an enabling tool that impacts all aspects of life. Programming teaches students to think logically, solve problems in a systematic manner and stimulates creativity. Eventually it will become a prerequisite to be able to share our world with artificial intelligence and even new forms of life! In this article, we explain why this is so.

A quick Google search on “why should schools teach computer programming” will throw up many pages that support the premise of this article and we will base our arguments on some of them. But before that, the very first step in our quest, namely searching in Google, reveals how entangled we are with computers. From the phone, through banks, e-commerce, GPS controlled app-cabs all the way to cars, aircraft and hospitals, nothing moves without a computer and each and every computer is controlled with a program. So at the very least, being able to write  computer programs ensures a degree of relevance in the employment market. But however powerful that may be as a reason to learn coding, the real reasons are significantly different and far more powerful.

In the past, managers would have secretaries who would, on their behalf, manage their “paperwork” consisting of physical “files” and postal mail. But today it is almost entirely a do-it-yourself scenario and people, including the most senior managers, have learnt how to perform these tasks themselves. A manager who needs a secretary today, to dictate memos to or to type, send and receive letters, would be a misfit in the corporate hierarchy. Replace the word “secretary” with an “IT professional” and the same would continue to be true for many corporate tasks. The ability to build a company website, to set up a system to accept customer feedback or carry out market research, to register members for an event and many other similar tasks, which in the past would have needed a staff from the IT department, are being done, or have to be done, by “non-computer” professionals. Just as knowledge of MS-Office and GMail have made secretaries almost obsolete, so has a whole range of free and easy-to-use tools, like Blogger and Google Docs, have made it almost essential for “non-computer” people to be able do what was earlier being done earlier by the “computer-people”. Similarly, students in schools need to learn computer programming not just to master Java and C++  but to facilitate their understanding of traditional subjects like history, geography, physics and chemistry. A certain fluency with computers is essential to explore new topics, through Google Maps and Youtube and software based science laboratories. A computer, like the English language, is no more an end by itself but a means, or medium, to reach out to and touch a vast realm of knowledge. Clearly, computers hold the keys to the new kingdom that is ruled by knowledge and information.

But does one need to learn computer programming to be able to use computers? Does one need to learn computer programming to be able to, say, use Blogger to create a website?  Or use Google Docs to create a workflow that routes an expense reimbursement request through an authoriser to the cashier who makes an authorised payment? Many of these tasks can be accomplished without writing a line of code as is the case when we use MS-Word to create a report.  But there is no doubt that having a fluency with coding gives a tremendous advantage, and more importantly the confidence, to be able to configure and implement these apparently codeless systems.

Programming, teaches one how to break up a big task into a set of smaller tasks, define the sequence in which these smaller tasks need to be done, or, under which circumstances some of these tasks need to be replaced by other similar tasks ( IF - THEN - ELSE ) and how many times each of these tasks need to be performed ( in a LOOP ) to achieve the final goal. The ability to think in such a structured manner is the key to the successful execution of any complex activity. This includes domestic chores, like cooking, where a recipe is actually a non-digital program, through corporate tasks, like manufacturing goods in a factory, to society wide programs like implementing Swachh Bharat or GST. Hence teaching a student how to write a computer program will in fact equip her with a crucial set of skills that is essential for the successful execution of a whole range human activities. This is where computer programming becomes as essential a skill as language and mathematics. Just as language and maths skills are useful even if the student does not become an author or an accountant, computer programming is a skill that remains important and useful even if the student has no intention of becoming a professional programmer.

How easy is it to learn computer programming? Many kids pursue a liberal arts education perhaps because they are so inclined but also possibly because they are mortally afraid of science and mathematics. Would it not be an unnecessary overload if computer programming is imposed on them? Not really. While the rules of  languages like Java and Python, so beloved of professional programmers, may deter those with a disposition towards the liberal arts, there exist many languages that use graphical metaphors to accurately represent complex programming constructs in a way that almost any child will be able to use with a ease and pleasure. Scratch, created at MIT Lifelong Kindergarten Lab in 2003 and subsequently adapted by Google and released as Blockly, is one such language. Components of this language are represented by coloured blocks that students have to collect, arrange and “stick” together, to create programs that perform desired tasks like leading a butterfly to a flower while avoiding a cat. Hidden in all these playful constructs are hard core programming concepts that can be revealed, if the child so desires, as actual Python or JavaScript code that will run on a real computer. Blockly is available at many portals like Code.org where it is used to create delightful modules that teach computer programming through self-driven exercises. These modules can be used not just by students of different ages but also by adults who have never written programs.

But the greatest value of teaching computer programming is the sense of empowerment that it fosters in students by allowing them to actually create something tangible. While many programs perform the same task or look similar, each program is in fact an original creation of the person who has coded it and there is no joy as overpowering as that that is felt when a program executes and does what the programmer wanted it to do. One of the unfortunate drawbacks of India’s school education is its emphasis on rote learning, on memorising, and its utter inability to incentivise students to create something original. Programming allows every student to virtually create something original and with a little effort, this can be hooked to or extended into the physical world.  For example, products like Mindstorm from Lego allow students to build and control robots that move while others allow the creation of apps that run on phones and tablets. As an aside, products like Twine allow those who wish to be authors, journalists or movie makers to create structured storyboards that help them put together storylines and presentations that do not follow a linear structure.

Creativity, whether strong or weak, is inherent in every student and teaching computer programming from a young age is perhaps the best, or easiest, way to nurture this precious asset. By being able to create something unique, each and every student gets the confidence that she too can build the kind of toys and gadgets that she sees in the market and in people’s hands. Sixteen year old Kavya Koparapu, originally from India and currently studying in a high school in the US, has built a lens-and-app contraption that helps diagnose critical eye diseases. Stories like this can emerge from India only when our own kids get the confidence to be able to build products from an early age.

Can schools in India afford to teach computer programming? Let alone teachers, do we have the infrastructure? When schools do not have books and blackboards should they even think of acquiring computers where students will learn programming? Good questions. For private schools this is a non-issue because most of them have infrastructure far in excess of what is required to teach programming but for government schools, this could be a dilemma. This is a different problem and the answer lies elsewhere. The real problem in government schools is not the lack of money but it’s rampant misuse. If this can be curtailed and if some of the sops that are handed out during elections can be diverted towards schools and not stolen, then there will be no shortage of computers anywhere. Management of money, not money itself, is what our schools lack.

The evolution of natural languages has shown that the ability to communicate is an essential driver of the evolutionary ascent of man. Without being able to communicate, the solo ape could never have come together to form flocks and packs which eventually became clans, tribes, nations and finally the civil society that we see today.  But in the last thirty years there has been the far more rapid evolution of a parallel intelligence in digital computers. This began with machines beating humans in chess and it now threatens to replace humans in many business and industrial activities. The social impact of the rapid rise of this artificial intelligence [see author’s article in Swarajya, March 2017] is still not very clear but there is no doubt that it represents a new kind of species, a non-biological, perhaps non-physical species, with which traditional biological species will have to share space in the future. Also, the rise of genetic engineering and biohacking [see author’s article in Swarajya, April 2017] is leading to the creation of man-machine hybrid cyborgs.  Machines with artificial intelligence are rapidly acquiring natural human language capabilities so as to be able to work better with humans. In some cases, as in the case of the famous Facebook example, “they” are learning to create their own languages. In this context, it is equally important for human beings to be as just adept with computer languages so as not to lose the advantage when interacting with this digital species.

In the past, people would learn foreign languages to benefit from commercial and cultural ties with foreign nationals with whom they share the world. In the future, people will need to learn the language of this new digital species with which they will share an ecosystem that consists of both man and machine. That is the real reason why we need to teach computer programming in schools, not just because it is a skill that helps one get a job in the IT industry.

This article was originally published in Swarajya, the magazine that reads India right.

November 02, 2017

Managing GST efficiently with Blockchain and GSTCoin (⇞)

Bitcoin is the rage! With an unbelievable appreciation in prices, many people want to invest in this red-hot cryptocurrency but whether the current valuation is a bubble that will burst is an open question. What is indisputable however is the immense versatility of the blockchain technology that is used in bitcoin -- and all cryptocurrencies. GST too is the rage, or rather the cause of rage because of the inefficiency of the software implementation. This article explains how a decentralised blockchain application could solve GST problems. But what is bitcoin?

image borrowed from financeminutes
A bitcoin is a unit of value, like an equity share of a company, that can be owned and transferred. It resides in an account in a ledger, like a dematerialised share in a demat account with NSDL. The account number, the public key of the account, is known to all and so anyone can send or deposit demat shares into this account. However to sell or transfer shares out of this account, the anonymous account holder must use a password, a private key that only he knows, to create and publish an outbound transaction pointing to another account identified by its public key. In cryptocurrency jargon, an account is called a wallet and the ledger is called the blockchain. A wallet is defined by a {public-key : private-key} pair consisting of two very large numbers that have special cryptographic properties. But what is really novel is that the blockchain ledger, the record of all transactions, is not maintained by or at any one institution, like the NSDL for equity shares, but jointly by all participants in the network. Everyone has a copy of the blockchain-ledger that has a record of all coin transfers and so everyone can both verify and confirm each transaction before they accept it in their own copy. An invalid transaction can pass into the blockchain-ledger if and only if, it is accepted by more than 50% of the network and this has never happened since 2009.

Verification means that a payment transaction is valid -- the total value of all inbound or credit transactions to a wallet less value of all outbound or debit transactions is more than or equal to the current outbound debit transaction. Confirmation means that there is no double spend and the same set of unspent, inbound, credit transactions (“UTXO” or unspent transactions outputs) are not being used to create more than one outbound payment transactions. Since everyone has access to all transactions, anyone can perform the verification and confirmation. But because this needs a lot of computational power, this task is voluntarily done by a group of miners. As a reward for doing this public service, miners who run a full node blockchain software on a powerful computer, are rewarded with newly created coins that are added to their wallet when they have verified and confirmed a new block of transactions -- that is then added to the blockchain. However the reward is not given to any miner who performs the verification and confirmation but to the one specific miner who, in addition to the verification and confirmation, also solves a difficult mathematical puzzle first.

The payment, or output  transaction that deposits a newly created coin into a successful miner’s wallet is called a coinbase transaction. It is different from all other transactions because it is not backed by any previous input transaction. Hence the analogy of mining, as if a coin was dug out of the ground and not received from anyone else, whereas all other coins would have to be received from someone before they can be sent to someone else. However a better analogy would be to view bitcoin, as sweat equity that is given, in lieu of salary, to the accountants in a bank for checking and approving all transactions. The brilliance of “Satoshi Nakamoto”, who designed bitcoin, was in equating the sweat equity of the bank to the assets that are managed by the bank and initiating a self-sustaining network that is working flawlessly since 2009. The magic mathematics of cryptography ensures that this decentralised autonomous organisation (DAO) runs without any formal management and yet has achieved a market capitalisation of over US$ 70 billion.

But why should these “sweat equity” shares of a non-existent bank, the actual bitcoins, be each worth thousands of dollars today? Many people, who are not miners, buy these coins from the miners for investment or payment purposes and this demand is pushing up the market price.  Bitcoin can be purchased at many cryptocurrency exchanges with fiat currency like US$ or INR ₹. After KYC compliance, these exchanges will convert fiat currency into cryptocurrency and vice versa at market driven prices. Bitcoin is both a currency that is extremely useful as a payment mechanism because transfers are simple, fast and anonymous and is also a commodity that has appreciated in value and hence worth investing in.

How can GST use this technology?

Bitcoin is the most popular and valuable coin but there are many others. The software to create similar coins is freely available and anyone can do so. But these alternate coins do not have any value in the bitcoin network because transactions with these coins lack the cryptographic signatures of past owners that are necessary for verification and confirmation. However these coins can have value in other networks where they are recognised and traded.

Let us consider a scenario where where we have 4 companies identified as A, B, C and D. C buys ₹1000 worth of goods from A and ₹ 2000 worth of goods from B. Assuming GST @ 18%, C pays ₹ 1180 to A and ₹ 2360 to B. D buys ₹ 5000 worth of goods from C and pays ₹  5900 to C. Under GST rules, A, B, C are required to deposit ₹ 180, ₹ 360 and ₹ 900 respectively to the GST Authority (GSTA). However C can claim ₹ 540 as GST Input Tax Credit and deposit only ₹ 360 provided GSTA has evidence, that A and B have made the corresponding deposits on behalf of C.

The idea is simple but the process is very complex and laborious. There are millions and millions of GST transactions and all this data accumulates in the central GSTA database. GSTA needs to run a  matching exercise within this huge and ever expanding pool of data and identify the two deposits from A and B that benefit C. This is a big bottleneck, a possible single point of failure and so cause of delay before C gets credit for ₹540.

Instead, let the GSTA define a new cryptocurrency called GSTCoin identified by the symbol ⇞ and through an Initial Coin Offering (ICO), release, say,  ⇞100 crore into its own GSTCoin wallet. To deposit GST, one has to purchase GSTCoins of equivalent value.  GSTCoins (⇞) can be purchased at one of many GST Exchanges (GSTX) operated by, say, banks. Every GST registered company will need a GSTCoin wallet whose public key is, for example, the GST registration number. This GSTCoin wallet will be hosted by the same GSTX where the GST registered company becomes a member and this will hold balances in two currencies namely, Rupee (₹ ) and GSTCoin (⇞). As in any currency exchange, the fiat currency Rupee (₹ ) may be deposited from and withdrawn into a traditional bank account using NEFT or credit card. GSTA too will have a wallet in each GSTX.

A GSTX would differ from a normal cryptocurrency exchange in two ways. First, members can only trade with GSTA, not with other members. When a member deposits GST, he actually buys GSTCoins(⇞) from GSTA, using either his Rupee (₹ ) balance or his GSTCoin (⇞) balance. The second key difference is that the purchased GSTCoins (⇞) are not delivered to the purchaser’s, wallet but to a public key associated with the wallet of the beneficiary that may be on a different GSTX. This is quite easily done in any cryptocurrency network. If the identity of the beneficiary for whom GST being paid is not known, or if the beneficiary does not have GST registration, then the GSTCoins are delivered to a special terminus wallet of the GSTA.

All transactions are recorded in the GSTCoin blockchain ledger and a full copy of which is maintained by all those who runs a full node on the GSTCoin network. Any organisation with an interest in the GST network, including large GST payers, may run these full nodes.

In our case, A and B will first deposit Rupees into their respective GSTX wallets and then buy GSTCoins worth ⇞180 and ⇞360, respectively from GSTA using their Rupee balances.  These will be delivered to the wallet of C which will now show a credit balance of ⇞540. Since GSTCoins could not have been transferred from the GSTA wallet to C’s wallet without a corresponding GST payment in either Rupee or GSTCoin, there is no further need for any matching. The GSTCoins in a beneficiary’s wallet must have been received against some GST deposit somewhere. Hence credit is automatic and instantaneous. For the GST deposit that C needs to make on his sales to D, he will purchase a total of ⇞ 900 GSTCoins through two transactions :  a ⇞540 transaction from his GSTCoin balance and a ₹360 transaction from his Rupee balance. This fulfills his obligation to deposit ₹ 900 as GST while actually spending only ₹360. In case D is registered for GST, the ⇞900 GSTCoins that have been purchased by C will go to D’s wallet. If D is not registered, ⇞900 GSTCoins will be delivered to GSTA’s own terminus wallet where it will be accounted for as nett GST received.

As in any cryptocurrency network, all transactions will be verified and confirmed by miners who operate full nodes and who, on finding a block of valid transactions, will be rewarded with a certain number of GSTCoins through a coinbase transaction. Like any other GSTCoin, these too can be used for GST deposits.

How is this blockchain based architecture any different from the current architecture of the GST software application that runs on a centralised database? Can we not modify this centralised software to make automatic NEFT or UPI payments to beneficiaries when GST is received? Functionally the two architectures are no different. But by physically distributing the processing requirements across multiple GSTXs we eliminate bottlenecks caused by the limitations in the computing power of a single central machine. Such a distributed system is inherently scalable and will not get choked by high transaction volumes. So delays associated with uploading data and transferring credit will be significantly reduced. Moreover, as a distributed system it will be almost immune to any form of hacking or disruption at any single point of failure. Finally, there is total transparency because the blockchain database is immutable and visible to the network of full node operators.

One challenge is privacy. Since all transactions are visible to everyone, all GST deposits and hence corresponding purchase prices are in the public domain. Fortunately all cryptocurrency wallets allow the creation and management of any number of public keys and it would not be difficult for the depositor’s wallet to pick up a new public key from the beneficiary’s wallet before initiating a GSTCoin purchase transaction, and use that to deliver the GSTCoins. This will make all blockchain transactions anonymous while leaving a clear audit trail within the wallet of the depositor that can be used in case of any subsequent disputes. In fact the wallet itself could automatically generate the GST return on a daily basis.

Another challenge is that GST consists of Central, State and Interstate GST and dues in one category cannot be offset with credits in the other. This problem can also be easily addressed by creating three different GSTCoins, say of three “colours”. The wallets will become a little more complex, but the same underlying principles can be used to meet the requirements of three different, but similar, taxes.

With the RBI apparently exploring the possibility of its own cryptocurrency, the GSTCoin could be a good way to test Blockchain technology as well as improving the efficiency of the current GST system.

This article has been written with help from Bhartendu Tilak, a Chartered Accountant and a colleague of the author. Any inaccuracy or inconsistency is of course the responsibility of the author.

This article originally appeared in Swarajya, the magazine that reads India right

October 28, 2017

Google Sheets with SQL

The Google App Suite ( or G-Suite) of products is a powerful and free alternative to the traditional Office Suite and includes among others Docs, Sheets and Slides. Google Sheets offers full spreadsheet functionality and in this post and sample, we show how data stored in Google Sheets spreadsheets can be accessed with SQL. This post assumes that the reader is familar with standard spreadsheet terms like sheet, range and functions.

Google Script

The primary programming language used here is Google App Script, a javascript look alike that can be used to enhance and extend G-Suite products. Google script can also be interleaved with traditional HTML / CSS / JavaScript to create an elegant user interface as is shown in this example.
A Google Script project sits in the Google Drive just like any other Sheet, Doc or Slides and consists of code in a code.gs file along with html files for the interface. Google Script functionality can be built into other Google products but in this example, we build a web-app that can be called independently with its own URL or can be embedded into an iframe :

So how does one start with this new technology?

The developers guide to web apps is a good place to start and build a traditional Hello World application that creates and serves an HTML file. Then things turned a little difficult because it is not directly possible to make pure SQL calls from Google Script to Google Sheets.

A quick search through and post on Stackoverflow threw up two possible ways of addressing this problem and both these two approaches have been demonstrated in this application. However there were more challenges to overcome! How to navigate from one HTML page to another, how to store global variables that will carry data across functions and pages and how to embed the app inside an iframe. After sorting out all this the web app was up and running but it was looking rather clumsy and so there was a need for some cute CSS for the input form and the output report. But all this CSS had made the code almost unreadable and so there was a need to remove the CSS and Javascript code into external files and have them included in the HTML.

Before the app can be used it is nice to keep the following points in mind:
  1. For the googleViz route, the underlying spreadsheet must have public access, though this does not seem to be necessary for the query formula route
  2. For gViz, the name of the worksheet is adequate but for the query route the worksheet name followed by the cell range is necessary
  3. The spreadsheet document needs to be identified by the document ID that is a part of the URL that leads to the sheet. However the URL to the sheet can also be used by calling a different function that was tried but then commented out.
The source code for this web app -- with all script code, html, javascript and CSS -- is available in G-drive. You would need to login with your Gmail id to inspect it and use it if you like this approach.

Some known Issues :
  1. The worksheet names should not have spaces
  2. Sometimes, the last displayed report shows up instead of the new one. Reloading the page, fixes the problem 
  3. Multi-table joins are of course not supported! That is where you need a real RDBMS

October 04, 2017

Quantum Computers

My presentation on Quantum Computers at the Cypher 2017 conference in Bangalore on 21 September 2017

more information about Quantum Computers is available in an earlier blog post.

October 02, 2017

A Central Cyber Defence Authority for Digital India

The weekend of 8th - 10th July 2017 was a little different from most other weekends. First, late on Friday night, the Airtel network in the NCR region went down because of a data corruption in one of their critical computers. Then, and the exact time is not known, someone hacked, or illegally accessed, the Jio customer database, retrieved confidential identity data about customers, including phone and Aadhaar numbers, and published the same on a public website. Finally, and again the exact time is not known, something unpleasant happened to the National Stock Exchange computers so that when the market opened on Monday morning, nobody could trade for almost the entire day.

It is possible that these three events were independent, random events but as Goldfinger says, in Ian Fleming’s eponymous novel, “Once is happenstance. Twice is coincidence. The third time it's enemy action." So let us not have any delusions about incompetence or equipment malfunction. This was a cyber attack.

Who could the enemy be?

Obviously we do not know, as yet, but consider three more facts. First, this was when India was in a tense stand-off in the Sikkim sector of the Indo-Tibetan border where the Indian army had a significant situational advantage. Second, China has publicly warned India that 2017 will not be the same as 1962. Finally, in 2014, five officers of the the Chinese People’s Liberation Army Unit 61398, operating out a 12 storey building in the outskirts of Shanghai, were indicted by a US Federal grand jury “on charges of theft of confidential business information and intellectual property from U.S. commercial firms and of planting malware on their computers” leading to a tense stand-off between the United States and China over state-backed cyber espionage.

While it is true that no bullets, shells or rockets have been fired across the Himalayan border, at least not till this article was written, could it be that something else is being fired?

Most of us civilians, living far away from the LoC or International borders have not seen guns being fired in anger but thanks to television and social media coverage we have a fair idea of what happens there. But what does a cyber attack look like?   

Consider these two screenshots from two well known cyber security companies that show live cyber attacks in real time :

Watch live cyber attacks at : http://map.norsecorp.com/#/

Watch live cyber attacks at : https://cybermap.kaspersky.com/

This is only a small subset of actual cyber criminal activities that security companies can track and have chosen to make public -- like an excerpt from the register of FIRs that is maintained in every police thana in India. As in real life, most crimes are neither recorded nor publicized unless they reach epidemic or pandemic proportions like the WannaCry virus that disabled thousands of computers by encrypting critical data.

Now that we know how pervasive and ubiquitous cyber attacks are, what should we be doing to counter them? Some of us use anti-virus and anti-malware software on our personal machines and many technology savvy companies use firewalls to protect their internal networks -- that connect both users’ personal machines and company servers containing operational databases -- from hostile external access. But is this adequate?

While technology exists to stop almost every kind of cyber attack, not all end users have the knowledge, the ability and most importantly the determination to use it effectively. Consider a small or medium size company that uses a billing or financial accounting software. In the past, these would be on stand alone machines and hence inherently safe because it was “air gapped” -- or physically disconnected -- from the big, bad external world. But with more and more bills, invoices and money receipts being exchanged over mail this is no longer possible. So is the case with electronic filing of various tax returns and GST in particular. It is now impossible for any useful computer to be isolated from the internet and hence  be safe from hostile attacks from anyone, anywhere in the world. Are the computers that form the backbone of our central and state governments safe? Unfortunately, the answer is NO. So what if “non-state” hackers shut down the computers that control Power Grid’s electricity distribution network in India as was the case of the National Stock Exchange? The damage would be worse than a bomb exploding  in Howrah Station!

The challenge is less about ability and more about the attitude towards security. We know that our homes, offices and factories face threats from thieves and robbers but do we all learn martial arts and purchase guns? No, we hire security guards or outsource the security to specialised security agencies who have the expertise to handle thugs and thieves. Can our software programmers and IT staff not protect our computer systems? In principle they can, and in many companies they do keep hackers at bay but most software programmers, have expertise in a completely different area -- meeting customer and business requirements in an efficient and economical manner. Security for them is more often than not an afterthought, not the core competence. On the other hand, the durwan at the gate does not care two hoots about how and what is being produced in the factory but only knows that neither should anything go out or nor should anyone enter the premises without an approval from an authorised person. That security mindset lacking in most of our IT installations.

Which is why we have the police in towns, the CISF in factories and airports, the RPF at railway stations, the BSF and the ITBP on the borders and of course the Army as specialist agencies of the state whose only job is to ensure the security of our citizens, our factories, our infrastructure and hence of the country itself. Where is equivalent agency that guards our cyber assets? Critical machines in the GST network, the bank ATM network, the telephone network, computers that control the generation and distribution of power, computers that store Aadhaar and voter information are at the moment being guarded, if at all, by people who know little about cyber security and certainly do not have the “police” mindset that anticipates crime and thwarts threats. CERT-IN, the Indian Computer Emergency Response Team, under the Ministry of Electronics and Information is merely a technical body, not a security agency, whose responsibility is limited to collecting and disseminating information on threats and offering advice to anyone who chooses to listen. They do have the mandate to intervene during or, as is usually the case, after an attack but do not have the executive or operational responsibility to actually to prevent attacks, as is the case of the CISF or the BSF. The so called “cyber cells” of the metropolitan police are hardly any better -- all that they can do is track down mischief makers who put up politically inconvenient Facebook posts.

Going forward what we need is to separate the operational roles from security roles. Just as the security of an industrial plant is not the responsibility of the production manager but instead, is handled by a separate security department, so should be the case of security for our government installations. Those who operate IT systems should not have the additional responsibility of ensuring their security. This is not because local IT staff may not be competent enough, but because we need a consistent and comprehensive security stance at all possible threat points. It is not enough for some installations to be secure. Since all systems are interconnected, a breach anywhere is a threat everywhere and that is why we need consistent security everywhere. Hence the cyber security team should not be a part of the  local IT management but should be a part of a central organisation, the Central Cyber Defence Authority, CCDA -- analogous to CISF or BSF --  reporting directly into the security establishment in the Home Ministry.

In fact, CCDA should be an organisation on par with any other central security agency like CISF, CRPF, BSF, ITBP and like them should be headed by a person from a police, or crime prevention, background with a rank equivalent to that of the head of existing central forces. While CCDA should be responsible for government and public assets, private companies, unless they create their own separate cyber-security organisations, could outsource their cyber-security requirements to professional security companies, for whom this will be an additional line of business above and beyond their their normal fire and crime prevention services.

But while our security establishments, the Army, police, CISF, CRPF etc, may have the psychological mindset, the security stance, to anticipate criminal behaviour and prevent crime,  they would not have the technical skills to do so. Cybersecurity is not a part of the curriculum either at the Indian Military Academy or the National Police Academy and it is unlikely that it will ever be so. Even if some basic training is imparted, it will never have the technical depth required to defeat the sophisticated hacker. However the Manhattan Project, to build the atom bomb, was run by the US Army Corp of Engineers under General Leslie Groves but he had the best nuclear scientists like Robert Oppenheimer and Nobel Laureates like Richard Feynman working for him. So should be the case of the CCDA -- led by people from a police background, with an aptitude in computers and an interest in cyber security,  but staffed with people who have the deep technical knowledge, recruited laterally, or on lien,  from the IT industry.

Just as the CISF reports to the Home Ministry but is deployed in airports that report to the Aviation Ministry, the CCDA should report to the Home Ministry but should be deployed across all computer installations in all government departments, power generation and distribution companies and other critical utilities like roads, railways, telecom, ATMs. In these deployments, CCDA should be THE executive body, not be an advisory one and should have both the responsibility and the authority to ensure security. For example, it should be CCDA technicians who should have passwords for the firewall servers -- that protect government computers on, for example, the GST network  or the power transmission network -- and should be responsible for  configuring the security settings on the same.  This will be analogous to the CISF -- not DGCA, AAI or airline staff -- being the custodian of the door keys, frisking passengers and operating the X-ray scanners at the airport.

In fact, CCDA, like the Army, should also acquire offensive, or “Strike”, capabilities in addition to its professed defensive, or “Holding”, capabilities. Building offensive capabilities is a good way to test its own defenses and sometimes, offense is often the best form of defence!  

But unlike other central forces, the CCDA need not physically relocate its expert staff to distant locations even when it is deployed to protect dispersed digital assets. Just as the attacker can attack from anywhere in the world, so too can the defender protect from one or two central locations because all activity -- both offensive and defensive -- can and will be carried out over the same networks.

The HBO network was recently hacked by people who demand a multi-million dollar ransom in untraceable bitcoins to refrain from leaking episodes of the billion dollar Game of Thrones serial. What would happen if someone were to hold the Government of India to ransom with a similar hack? Just as we need to have the BSF jawan with his INSAS rifle at the LoC or the CISF jawan with his X-ray scanner at airports, we also need the CCDA jawan -- or in this case, the CCDA technician -- with his “hardened” firewall to stand guard on the digital assets that are connected to the web. The arrival of nuclear technology in the battlefield, led India to set up the Nuclear Command Authority. With the emergence of Digital India, we need the CCDA to protect the core digital assets that are critical for safety and security of the country.

Originally published in Swarajya, the magazine that reads India right!

About This Blog

  © Blogger template 'External' by Ourblogtemplates.com 2008

Back to TOP