blog@Yantrajaal

Ever since I started my career in software and IT as a database administrator, I have had a great fascination, if not love, for the SQL language. I also know that there are many in the software fraternity (and sorority) who share my comfort with using SQL. Even after moving into the current enthusiasm, or should I call it a fad, with data analytics, I see that there exists many who prefer the ease and convenience of SQL even in big data. For example, Hadoop had to hide the complexity of its Map-Reduce magic behind HIVE. Spark offers SparkSQL. In an earlier post, I had shown how SQL can be used for scoring Data Mining models developed with R through the PMML route. In this post, we show how K-means clustering, a very common and widely used data mining activity can be done with SQL, using a technique adopted from Joni Salonen's blog post . The following codes were executed in MySQL. First we create the tables : drop table if exists km_data; -- contains initial data, on...

As a former DBA I love the fact that SQL has a life of its own and the way it is still used after it was first conceived of in the 1980s as the natural way to query Relational Database Management Systems. A vast amount of data in the world is still available in SQL compliant RDBMS tables and today when business analytics and data science seems to be overshadowing SQL, I was delighted to find that SQL can still play a very important role in the implementation of complex data mining applications. This post explains how this evergreen tool is still very, very relevant in data mining. Many data mining tools like Rattle , RapidMiner are used to create "models" for Classification / Decision Trees and Heirarchical Clustering  but then the models have to put into production by using them to score large datasets. This is where SQL can play a very powerful role. The models created in the data mining tools need to be exported as PMML documents and then converted to SQL using any of...

image from Responsemagazine.com A retail store tracks the volume of sale for each stock-keeping-unit (SKU) that the store deals with. Given the sales for days 1 through 5, is it possible to predict the sales on days 6 through 10 ? Common sense dictates that sales will remain constant and the average sales per day for the first 5 days will be the same as the average sales per day for the next 5 days. However this may not always be the case, if there is a rising or falling trend. If there is a strongly rising trend, caused by a some strong promotional activity, then the assumption of constant sale will lead to a stock-out and loss of potential business. Similarly, if there is a strongly falling trend, then a similar assumption will lead to accumulation of dead stock and hence a loss related to excessive inventory. Instead of days, the same analysis can be done on the basis of weeks, fortnights or even month. Net-net given the sales over 5 periods of time, it is useful to be able to ...

Everybody wants to become a data scientist but given the huge number of tools and techniques that are available no one really seems to know where to start and how to go about acquiring the right skills. image taken from  http://www.rosebt.com In the process of teaching data science to students at the Praxis Business School I have realised that there is a huge amount of information that is freely available on the web and thanks to Google, one can find this information very easily. If one is motivated enough, one can learn data science on his own. In an earlier post I have given a very broad Introduction to Data Science but in this post, that I plan to update every now and then, I have tried to assemble a collection of tutorials, grouped by topic that will help anyone get a good grip on this subject by actually doing something practical. R : Statistical Programming Language Forget about SAS, SPSS and other proprietary languages that want to extort money from you and simply...

Unless you are Rip Van Winkle who has just woken up from long sleep, it is very unlikely that you would be unaware of how social media frenzy has been and is sweeping through the ecosystem. Facebook, Twitter and Youtube is the new trinity that everyone is interested in, not the Brahma-Vishnu-Maheshwar or the Father-Son-HolyGhost that our ancestor talked about. While usage and consumption of social media is what most of us are interested in, social scientists and market research professionals have a deeper interest in this amazing phenomenon. In the past, such research was carried out through traditional tools like questionnaires but given the vast quantity of data and the speed with which it changes we need newer and better tools. As a part of the curriculum of the MBA program at the Vinod Gupta School of Management, IIT Kharagpur students were required to explore and present newer tools -- that are legitimately available at no cost -- for extracting and analysing social media da...

The Doors of Perception is an iconic book by Aldous Huxley where, while exploring  the hallucinogenic properties of certain shrubs and herbs, he came to the unusual conclusion that any cognitive sentient has instant and automatic access to all possible knowledge in the universe. However the neural faculties that are expected to process this information and render them useful in a socio-cognitive context are in danger of being overwhelmed by the sheer mass of data -- as occassionally evident in what we refer to as 'madness' -- and so are protected and isolated from the same by the mind that works as a gate or valve that reduces the quantum of information that the brain is eventually exposed to. This mind or gate or valve is what Huxley refers to as the Doors of Perception : that can be opened wider by the use of mental techniques ( Yoga ?) or the use of hallucinogenic drugs -- to reveal a greater amount of "significance" or meaning to the sterile ream...

In an earlier post on I had tried to explain why in the study of astrology it makes more sense to observe, track and validate correlations between events in the celestial sphere and that in personal life. An analysis of correlations is a standard tool or technique in statistics and so it makes sense to apply this technique in astrology as well. However, there are certain challenges in this approach because the format in which astrological charts are created or stored is not amenable to the kind of statistical analyses that we are familar with. To overcome this limitation and I have come out with a new way to store and retrieve horoscope charts using sparse matrices and relational database management systems. You can read about this mathematical technique in my paper Statistical Testing of Hypotheses in Astrology . An unexpected byproduct of this effort was my introduction to the Scribd service that allows you to convert any form of document OpenOffice, MS-Office, PDF and what not int...

With Pakistan succumbing to the Taliban and the US blindly stumbling around in an economic graveyard, we in India must start looking out for ways to defend our way of life from the misery that is engulfing the neighbourhood -- but the question is what and how ? No political activist in any party -- whether the Congress, the BJP or any of the satraps now jostling for an opportunity to loot the country after the elections -- has any clear idea on how to ensure the security of life and limb and so it is up us to try to figure out a way forward. When India was threatened by food security in the sixties and seventies, we used technology to pull us out of the mire. Despite the adverse impact on the environment, the massive use of fertilisers and irrigation techniques -- the Green Revolution -- helped us evade the immediate crisis of mass starvation and allowed us to live and fight another day ! Today it is physical security from terrorism that is emerging as a threat as big as the food secur...