Astrology - an application of Data Science

Every morning, when the sun rises, birds wake up and start chirping. We are not surprised because we see a very clear connection between a cause, the increase in ambient light, and the effect it has in waking up the birds. But when the sun rises, the Ispat Express moves out from Howrah station as well. Even in this case we are not surprised even though there is no obvious connection between the ambient light and behaviour of the engine. We explain the first phenomenon through causation and the second through correlation. Perhaps if we look very hard we might find a causative mechanism that explains why the Ispat Express departs at the crack of dawn and not at any other time, but frankly that is of little interest to the passenger. We are happy with the correlation, we are not really bothered about the causation. That is the crux of this article where we explore astrology from the perspective of data science and machine learning.

Traditional scientists would of course hesitate to view astrology as a science because they are trained to interpret phenomena through the prism of cause and effect. Naive astrologers also fall into the trap of trying to justify astrology by invoking gravitational, electromagnetic and even undiscovered, unheard of “rays” that emanate from astronomical bodies and influence human destiny. Both are wrong because they are searching for causation whereas the underlying principle is one of correlation. Astrology is interested in predicting an outcome based on some correlation -- an event, say an accident, happens not because a certain planet was at a particular location and caused the event but because it has been historically observed that the presence of a planet at a particular position is associated or strongly correlated with the occurrence of that specific event.

Is correlation an acceptable way to analyse situations and predict outcomes? Certainly, if we consider ...

The world of data science, of which artificial intelligence (AI) is a specific example, is largely based on the study of correlations. Most well known AI systems -- for example those that control autonomous, self-driving cars, recognise people and objects in photographs, react to voice commands, play board games like GO or perform other tasks that eerily mimic human behaviour -- are based on the technology of Artificial Neural Networks (ANN). In all such cases, the system is presented with a set, or pattern, of input data, that in the case of face recognition, would be the colour and brightness of light at every point of an image. If the system has been “trained” adequately, it should predict the identity of the person in the image even though this particular image has never been encountered before. A similar ANN software, if “trained” differently, and presented with an appropriate set of input data, should also be able to predict the best possible move in a competitive board game, the possible motion of a pedestrian in the vicinity of an autonomous car, or the intention and ability of a bank customer to repay a loan. In all such cases,  “training” consists of feeding the system with thousands of pieces of historical data and identifying which of the corresponding predictions were correct and which were not. Initially, most predictions will be erroneous but after a few thousand errors, the system will “learn” or get “trained” and from then on, the number of correct predictions would improve dramatically.

What happens inside the system, that is almost like a black  box, is not very clear. Some parameters, also known as weights, are assigned numeric values, but it is not possible to explain why specific values were chosen. All that is known is that changing these values leads to a change in the accuracy of the prediction. Once the values have been set correctly -- the system has been “trained” -- the subsequent predictions are almost always correct. This situation, where the system works well but we cannot explain why it works well,  is described in “The Dark Secret at the Heart of AI” [ MIT Technology Review, April 2017, see ]. This article describes a new kind of autonomous car from NVidia that is not explicitly programmed to drive through traffic.  Instead, it learns driving by observing the environment and recording the actions of a human driver who is navigating through the environment.

“Getting a car to drive this way was an impressive feat. But ... it isn’t completely clear how the car makes its decisions. Information from the vehicle’s sensors goes straight into a huge network of artificial neurons that process the data and then deliver the commands required to operate the steering wheel, the brakes, and other systems. But .. if .. it did something unexpected ... it might be difficult to find out why. The system is so complicated that even the engineers who designed it [cannot] isolate the reason for any single action. And ... there is no obvious way to design such a system so that it could always explain why it did what it did.”

This inability to explain the reason for a behaviour is not because the system has supernatural or magical properties. This is a purely deterministic machine, that follows the laws of cause and effect at the micro level. But the situation is so complex -- in terms of the the number of micro-operations and the way they are related to each other --  that the connection between cause and eventual effect is impossible to establish at the macro level. But this does not detract from the fact that these and many other similar systems are being successfully used in situations where there is a need to predict outcomes based on (a) a set, or pattern, of current input data  and (b) historical, or “training”, data that maps thousands of  similar patterns observed earlier  to corresponding outcomes that have actually occurred.

Since this approach is “scientifically” acceptable in the case of AI research, we can extend the same to astrology! But before we do so, there are two difficult challenges to overcome.

The first challenge is the question whether the world is indeed deterministic, and hence predictable?  -- are all events pre-determined? -- or do events happen because of free will or pure chance? This profound question cannot be answered in this short article. So we will seek an acceptable answer that skirts the quicksands of philosophy and yet helps us move ahead without being facetious.

We accept that at the micro level the world is indeed completely deterministic but not so at the macro level. For example, a man dies in a plane crash because he boarded the aircraft to attend a meeting that was called by his boss and the mechanic who repaired the aircraft engine did not tighten a nut on an engine because he was thinking about his wife. The meeting was called by the boss because {of some reason} and the mechanic was thinking of his wife because { of some other reason } and this chain of events or reasons can stretch backward through millions of interrelated links that hide the eventual effect from the possible causes. Conceptually, this is no different from the behaviour of ANN systems, where despite the existence of deterministic causality at the micro level, we are not in a position to explain the full chain of causality.

A simple demonstration of such a mechanism is the 2003 Honda advertisement ( https://www.youtube.com/watch?v=_ve4M4UsJQo ) that shows how interrelated the world could be. A tiny gear rolls across the table, pushes a lever, that in turn causes some liquid to spill, that in turn causes a table to tilt, that in turn causes a spring to snap … and so on and on … until a Honda Car rolls out of the store. This video demonstrates how precisely Honda technology works and so one can see how a small change in any of the actions will result in the final car not moving. But unless this was explicitly shown, one would never know how the movement of a small ball or the flow of a liquid can in any way affect the final outcome. A similar video is available in the Facebook Japanese Community page ( http://bit.ly/2eUXAks ) that shows an even more complicated piece of mechanism where apparently unrelated movements of many objects finally end with a desired result.

To the profound question of whether the world is deterministic, a simple answer would be that while the world must indeed be deterministic, it may not be possible to establish a clear and unambiguous cause-and-effect linkage between two events that are widely separated in space and time.

The second and more difficult challenge is to explain how the position of astronomical bodies could in any way be a part of this huge chain of inexplicable causality. This is where we completely discard causality as the operating principle and instead, introduce correlations into our description of astrology.

The universe as we know it is a long and complex chain, or web, of interconnected events, that have a location in space and time. From this universal set of all events, a certain subset is related to human beings and of which an even smaller sub-subset is related to a specific individual. So the life of every individual can be represented by a unique set of events, many of which he shares with others. Indian astrology considers  ten astronomical objects, namely, the Sun, the Moon, five actual planets, the Lagna or ascendant horizon and Rahu and Ketu, the two points in space where the lunar orbit intersects the ecliptic plane.  For every person, the presence of these ten notional “planets” at specific locations on the Zodiac at the moment of birth are also events that are a part of the huge collection of events that define the life of the person.

Each event in a person’s life, including these ten planetary events, can be interpreted as a piece of data. Some of this data is binary - the event happened or did not happen; others can be interpreted as ordinal -- for example, favourable, neutral or unfavourable; categorical -- for example career in medicine, law etc; or even numeric -- for example, age at marriage or at death. With this interpretation, the life of a person becomes, what mathematicans refers to as, a vector in a multi-dimensional space, with the number of dimensions being equal to the number of events used to represent his life. Pictorially, every person can now be viewed as a point in multi-dimensional space and a group of persons looks like a random scattering of dots in this multi-dimensional space.

But are these points scattered randomly? Or do they form clumps and clusters?

Data science tools -- that is software programs based on algorithms like K-Means, K-Nearest-Neighbour, Decision Trees, Logistic Regression and of course the very popular ANN --  are quite efficient in detecting and isolating clusters by identifying points that are “similar” to others. This is routinely used in recommendation systems used by websites like Amazon and Netflix to identify individuals who have similar preferences -- individuals in a cluster would tend to display similar buying preferences -- and show them advertisements of products that they are likely to buy. In fact these recommendation systems try to predict who is likely to buy what.

For example, consider the attached figure, based on representative, simulated data, that shows the kinds of movies that people prefer. If the data of many people are plotted as points in  three-dimensional space where the three dimensions are represented by gender (shown as colour, blue or red), preference for action movies ( position on Y axis) and preference for romantic comedies (position on X axis), then there will be two clear clusters of points along with a few outliers. From the diagram, or actually the data, we can clearly see that men prefer action movies and women prefer romantic comedies.

Can we explain why men prefer action movies and women prefer romcoms? No, the diagram offers no explanations, and we do not care. But we do know that there is a clear correlation between gender and movie preferences. So by knowing a preferences of movies we could, in principle, predict whether a person is a man or a woman. Obviously with more data and more dimensions our prediction would be more accurate but there is nothing magical or supernatural about this prediction that is based on data and data alone. This ability to predict events without being concerned about the cause of the event is the basis of all forms of astrology.

But of all the events that define a person’s life, why is it that these ten planetary positions are the ones that considered in our search for correlations with other life events?

In principle, many mathematical techniques can be used to cluster people defined through their hundreds of dimensions. In practice this becomes difficult when the number of dimension becomes very large. Data scientists call this “The Curse of Dimensionality” where the number of dimensions is high compared to the number of data points available. Fortunately, there exist sophisticated mathematical tools, like Principal Component Analysis (PCA) and Factor Analysis (FA) that help identify key or important dimensions and allow the scientist to ignore others. Not everybody needs to understand the complex mathematics of PCA and FA but everyone uses it to reduce the number of dimensions to a manageable number.

Dimensions are also reduced through mathematical projections. Civil and mechanical engineers regularly project a three dimensional structure, say a building or a machine component, into two dimensional images called plan and elevation. This process can be generalised by projecting an n-dimensional structure into an m-dimensional surface where m is less than n. Representing high dimensional data pictorially and making it visible or understandable to a lay person is difficult, but for mathematicians, this is a very routine matter because these are standard mathematical operations.

From the thousands of potential dimensions that could be used to describe a person’s life, astrology chooses a smaller subset --  the positions of the ten astronomical bodies, or “planets”, on the various charts -- and this is used to identify clusters. Why do we choose the dimensions that are based on planetary positions at birth? The most likely reason would be that, thanks to astronomy that is closely associated with astrology, these are dimensions that are guaranteed to have numerical values that can be accurately determined if the date, time and place of birth is known. Other dimensions, associated with event, like omens, may not have valid data for everyone. If we choose these planetary dimensions -- and of course their derivatives like, aspects, conjuncts, combinations, Navamsha positions etc -- we are assured of a standardised structure that can be used to create a lower dimension surface, a  projection, of a person’s life that is otherwise defined as a point in a space of far higher dimensionality. So the choice of the planetary positions as key variables is an exercise in reduction of dimensionality -- a process that is, again, quite common in data science.

Once we plot the lives of individuals on this lower, ten-dimensional surface, then the search for clusters becomes a tractable problem. For example, mangaliks -- those with Mars  at the 4th, 8th or 12th position from the Ascendant or the Moon -- are likely to have marital problems unless they are married to fellow mangaliks. More specific events require far more complex patterns to be identified. For example, the birth of a child is associated with a fairly complex pattern in the charts of both parents that involves the dashas of planets associated with the Ascendant, the 5th and 9th houses and of Jupiter. The specific birth event is associated with another complex pattern that involves Jupiter, Saturn, Mars and Moon casting their aspects on the 5th and 9th house. Similarly, if a person is in the medical profession, it is very likely that the chart will show an afflicted Moon, a strong Jupiter associated with Ketu and Sun and an association of the 5th Lord with the Lords of 6th, 8th  and 12th houses. These are examples of positive correlations, where the patterns and events happen together, but there are also cases of negative correlations, the “limiting” patterns, that do not occur with certain events and seem to inhibit, rather than cause these events.

Such complex patterns are not easy to spot by manual means and the failure to do so leads to errors in prediction. Competent astrologers, like good chess and card players, are adept at spotting patterns and the inablity to do so is the reason why many astrologers end up with wrong predictions. However, the usage of data analytics tools, particularly software tools, can eliminate the possibility of missing patterns and improve accuracy of predictions.

Can we explain why these specific patterns are associated with specific events? No, and we do not need to. This is similar to the correlation between the gender and movie preferences that we have encountered earlier. Since we observe a correlations between patterns in the horoscope and events like marital bliss, childbirth and professions, we can use the existence of these correlations to  predict the occurrence of events on the basis of the existence of the patterns in the chart. As in the earlier case, there is nothing magical or supernatural about these predictions -- it is based on data and data alone. This ability to predict events without being concerned about the cause of the event is the basis of all forms of astrology.

These examples are taken from the area of natal astrology where the date and time of the person’s birth is the key variable but there are other types of astrology as well. For example, in horary astrology it is the date and time when a question is being asked and in mundane astrology or “Medini Jyotish”, it is the date and time of the creation or Independence of a country that is used to predict it’s destiny. In all such cases, the position of the planets at a point in time are the principal dimensions on which prediction models are built.

We have now overcome two major hurdles in our attempt to show the similarity between the predictive powers of astrology and data science. First, we have addressed the question of determinism and second, the importance or necessity of planetary positions. Let us now look at the next big hurdle -- the availability of data, or the lack thereof!

Data science begins with large sets of training data that has been collected in the past. It then uses mathematical techniques to deduce the rules of correlation that associate patterns of data with corresponding outcomes. In astrology, the history of this training data is lost in the mists of antiquity but the rules have survived. Are these rules perfect? Are they always correct? Most of them are, but possibly some are not,  but that is equally true for data science as well. Errors and wrong predictions are quite common in  data science but these can only be reduced if we have more and better data.

Physics as we know it today got a head-start over life sciences and social sciences because it was data driven. Tycho Brahe was the first European scientist who collected and collated a huge amount of data on the movement of planets. This data was subsequently used, first by Johannes Kepler to identify the correlations between duration and size of planetary orbits and then by  Isaac Newton who formulated the laws of gravitation. As an aside, it is interesting to note that Newton never really explained the cause of attraction, why bodies attract, but only identified the circumstances under which the attraction was observed. Eventually it was left to Albert Einstein to explain gravitation as arising out of the curvature of space-time but that is another story.

Like Kepler and Newton in the world of European physics, India has had sages like Bhrigu, Parashara and Jaimini who identified the laws of correlation that we have inherited today as the principles of astrology. But the the training data that might have been used to deduce these laws is not available any more. We can only speculate that Indian astronomers like Aryabhatta might have collected data like Tycho Brahe did, but unfortunately, there is no hard evidence for this. It has also been suggested that sages like Bhrigu had divine insight and rules of astrology were revealed to them through a direct and transcendent vision similar to the knowledge of the Vedas. We can respectfully consider such possibilities, but the exploration of such divine origins is beyond the scope of the current article. In fact it is best to keep religion and faith away from this discussion because astrology is not restricted to people of Indic origin or beliefs but is something that is found across many cultures in Asia and Europe. In fact, Tajik jyotish, one of the systems of Indian astrology -- like Parashari, Jaimini and others -- is supposed to have originated in the Middle East.

Data science validates its predictive models by testing them against a part of the historical data but astrology cannot validate its predictive models because there is no historical data to validate its models with. This is why classical scientists are reluctant to accept astrological predictions while being quite comfortable with the predictive abilities of data science. This can be addressed by creating a pool of training and test data from the personal information of thousands of real people. We need data on the  date, time and place of birth followed by data on key life events like dates of college admission, marriage, birth of children, entry into job, change of job, change of residence, major illness, loss of job, loss in business, death and illness of parents, spouse, siblings or children plus qualitative data like profession, level of wealth, happiness, spirituality, place of residence, relationship with spouse, siblings, parents children and so on. Coincidentally, a similar data collection is already happening every day in the world of internet and social media! Companies like Google, Facebook, Amazon, Uber and many others are already collecting personal data, often at the cost of privacy. Many of these companies have made it their business to collect personal data and then either use it to build their own predictive models or sell it to those who do. With a little bit of imagination, a similar data collection initiative could devise mechanisms that will incentivise people to voluntarily contribute data about events in their life, in return, perhaps, for a detailed astrological reading or other benefits.

Once sufficient data is collected it must be codified and stored with modern database technology so that it can be accessed from computer programs. Standard relational database technology can be used but since the data is of high dimensionality with many missing values, sparse matrix techniques or columnar databases can be used for efficient storage. Once the data is accessible to programs, then the same data science tools that are used by companies like Google to build predictive tools for, say, voice or image recognition,  can be used to either validate traditional predictive models of astrology or create a new rules of prediction. In fact, these new models may contradict traditional models and generate more accurate predictions. Going further, once we associate astrology with data science, then it can be included in the curriculum in universities. This will not only result in more structured research and better models but will also drive hundreds of fraudulent astrologers -- who sell fake remedies to gullible people, out of business. Astrology, like data science, can make predictions but cannot change destiny. There are some who claim that it could -- but that is another debate that is beyond the scope of this article.

Isaac Newton had said that he could see further because he was standing on the shoulders of giants. Astrology today is based on traditions of scholarship whose origins are long lost in the swirling mists of myths, legends and history. This traditional wisdom is an excellent foundation, a starting point, on which we could build the edifice of a new science of astrology that will be powered by tools and techniques that are commonly used in the corporate world today. Pattern recognition and machine learning techniques can be used to build predictive models with data  about key life events of thousands of interested volunteers connected through social media platforms. Such models will help identify natal planetary patterns, recorded at birth, and their degree of correlation with subsequent life events. This will either confirm, or contradict, ancient astrological aphorisms or even lead to new clues about what could lie in the womb of futurity.

As a data driven science, astrology must move from the Age of Parashara to the Age of Google.

---

Acknowledgment : The astrological patterns alluded to in this article have been identified by eminent astrologer K N Rao and explained to the author by his IIT Kharagpur batch mate, Raghuram Hebbar, consultant engineer and amateur astrologer extraordinaire. Any misunderstanding or misrepresentation of astrological rules is obviously the fault of the author.

---

This article originally appeared in Swarajya, the magazine that read India right