Data Mining on Social Media Platforms

Unless you are Rip Van Winkle who has just woken up from long sleep, it is very unlikely that you would be unaware of how social media frenzy has been and is sweeping through the ecosystem. Facebook, Twitter and Youtube is the new trinity that everyone is interested in, not the Brahma-Vishnu-Maheshwar or the Father-Son-HolyGhost that our ancestor talked about.

While usage and consumption of social media is what most of us are interested in, social scientists and market research professionals have a deeper interest in this amazing phenomenon. In the past, such research was carried out through traditional tools like questionnaires but given the vast quantity of data and the speed with which it changes we need newer and better tools. As a part of the curriculum of the MBA program at the Vinod Gupta School of Management, IIT Kharagpur students were required to explore and present newer tools -- that are legitimately available at no cost -- for extracting and analysing social media data. In this post, we see a sample of their work -- a list of free, open source tools that will allow anyone to access data from various social media platforms and analyse them for information and insights.

NodeXL is very simple tool based on Microsoft Excel that will allow you to extract a whole lot of data from most social media platforms. The following presentation has an overview of how it an be used.

Node XL - features and demo 

View more PowerPoint from Mayank Mohan

Social Media data is all about the study of graphs along with their nodes and edges and some more information the nomenclature can be found on this link in addition their home page at http://nodexl.codeplex.com/

 

Gephi ( pronounced jee-phai) is another very useful tool that can extract data from various social media and make useful analysis of the same. This is explained further in this tutorial. Tutorial - Gephi
 The next interesting tool that we came across was Impure and this explained in the following presentation

Impure data analytics & visualization tool

View more PowerPoint from Shubham Gupta

This needs a little more coding effort but the results are pretty impressive. More information is available in this tutorial and on their website.

Social media is the biggest battlefield ( the Kurukshtra) between Google and Facebook ( who are the Pandavas ? who are the Kauravas ?) and but for us who dabble in social media, the intersection between the two is a rich source of data.

The best way to access the treasure trove of data inside Facebook is to use the Graph API explorer. If you go into your own Facebook page and search for this you will land up on the Graph API page and you can obtain your authorisation token. Using this token to extract and analye data is not easy and this where the power of Google Refine becomes evident.

Google refine tutotial

View more PowerPoint from Vijaya Prabhu

You can also see this Youtube video to understand how to use these two powerful tools together. Two tutorials are available at http://mpvp4u.blogspot.in/2012/04/google-refine-tutorial.html and at http://gaarora.blogspot.in/2012/04/google-refinevgsom.html

Last by not the least, let us look at Twittersave

This is a simple tool developed by Mr Amod Gupta of VGSOM that allows on to quickly extract some data from Twitter.

The philosophy that we try to imbibe in our students at VGSOM is that they should be creators -- not just consumers -- of knowledge and these assignments are designed to help them to go beyond text books and push the frontiers of what is possible in today's techno economic environment. These are some examples of what we do at VGSOM. Keep reading this blog for more insights.

Tax Free Bonds to fund Education Infrastructure

Tax free bonds have always been very popular in India. In the good old days of the distant past we had RBI “Relief Bonds” that offered up to 10% tax free income but it was never quite sure for whose relief the bonds were for -- perhaps for the Indian tax payer! That was then. Now we have similar bonds floated by  IIFCL and NHAI, albeit at far lower rates that vary between 6.5% and 8.3% but even these have garnered Rs 30,000 crores in FY 11-12. Carried away by this appetite for tax free bonds, the finance minister has proposed to float similar bonds to the tune of Rs 60,000 crores in the next fiscal.

But where could or should this money be used ? The NHAI has of course been building roads but I think there is another piece of critical infrastructure that can be funded through this route -- education !

Why ? Because education is the mother of all infrastructure that this country needs very badly..

For a variety of reasons not directly attributable to government policy or purposeful planning, India is “blessed” with a burgeoning population of young people that we flaunt in front of an ageing China, Western Europe and North America. But between this young population and the hypothetical demographic dividend that the nation expects from them lies a deep moat of a non-existent educational infrastructure ! If our youngsters cannot be taught and trained in a manner makes them productive the demographic dividend will disappear into the blue and we will be left instead with the demographic liability of a sullen, disillusioned population that will take to urban and rural terrorism under the guise of Maoism and related labels.

But why can we not build enough schools and colleges to accommodate the educational aspirations of our youth and their worried parents ?

To begin with the Government has neither the money nor the project management skills -- let us admit it, our sarkari babus do not have the drive, energy, enthusiasm  -- to create the lakhs of school and college seats that the nation needs urgently. Should it even try, it will get bogged down in corruption and redtape and all that will result is another scam followed  by more howls of anguish in Parliament.

But what is worse is that even private entrepreneurs are debarred, by policy,  from filling this gap. How ? Because our policy makers have allowed only not-for-profit organisations to build schools and colleges. Prima facie this seems to be a noble idea but the devil lies in the detail. Even if there were a few public spirited individuals or groups of individuals who are keen to build good schools and colleges they will be caught in a policy quagmire that is possible only in India.

Consider the following : AICTE regulations require that the society running a college must own the premises. Now unless the entrepreneur has a rich uncle ( or more unsavoury sources of money ) how will they acquire the land and construct the premises ? Will any sensible bank give a loan to a  not-for-profit society that wants to start a college ? Very unlikely unless the society has political connections !

Net-net it is not possible for a normal, honest, entrepreneurial educationist to start a college. So how do we see so many private engineering, medical and management schools sprouting up all around the country. Because they have all been created with shady money or by people with strong political connections and are designed not to educate our youth but to siphon off the money from their parents wallets. There could be a few exceptions like BITS Pilani, created by the Birlas but that is a manifestation of the rich-uncle syndrome and not likely to be replicated on a large scale.

There could be many ways to break this logjam. First our obsession with not-for-profit education can be overcome. Second the stranglehold of the AICTE and other unsavoury educational regulators can be weakened. But these are major policy matters that would call for an extensive debate. Instead, a simpler solution would be to create a National Educational Funding Agency (NEFA) on the lines of the NHAI and IIFCL.

With the blessings of the Finance Ministry, NEFA could float tax free bonds with coupon rates and tenors similar to the other tax free bonds. Funds collected through these bonds could be deployed as long term loans advanced exclusively to new and existing not-for-profit societies that have been approved by the AICTE or other regulatory bodies. This will easily fund the capital expenditure necessary to create new infrastructure in the form of classrooms, hostels, laboratories and playgrounds that are essential for institutions as per regulatory norms.

With ready access to legitimate, low cost capital, entrepreneurs would be free to direct their energy to create good quality educational institutions where the running costs -- revenue expenditure and loan service -- can easily be met from tuition fees. Most parents today are already paying a lot of money for private tuition and coaching classes and would be happy to transfer a portion of this expenditure towards fees of schools and colleges set up by honest and competent educational entrepreneurs.

Tax free  bonds to fund private educational infrastructure could be a neat way to transfer private funds to private entrepreneurs to build national infrastructure that would be under the supervision and control of a national regulator -- could anything be more win-win than this ?  But is there a catch ? Who or what could derail this idea ? And why would they ?

Education in India is a highly regulated market run under a ruthless license-and-permit raj. Most private educational institutes are owned and operated by politicians or their crony capitalist friends to fleece the hapless parent and earn enormous profits right under the nose of our hypocritical not-for-profit education policy. These individuals and their lobbying agencies will work hard to ensure that policies that hit their cosy oligopoly are nipped in the bud.

Can the nation free itself from such self-tied knots ? And live up the aspirations of its youth ? Let us wait for the answer that lies in the womb of the futurity and in the sagacity of the Finance Minister.
............................................................................................................................
image taken from http://www.faadooengineers.com -- a source of information about engineering education

Sweating Assets to Expedite Justice

If justice delayed is justice denied then the citizens of India have had no justice for many years. We all know that court cases in India drag on for years while litigants grow old and die or settle out of court. For those who are entangled in criminal cases the situation is worse because more often than not a significant part of their lives are wasted behind the bars of overcrowded jails. One estimate of the size of the problem says that in 2009 there were 30 million court cases pending all over the country. What is worse is that there is no solution in sight !

Or is it ?

Let us first redefine the judiciary as a mechanism to deliver judicial services where judges, who are paid by the Indian tax payer, are service providers and the litigants are the customers or consumers. In fact these customers pay for the services either directly through court fees or indirectly through the taxes. The moment we redefine the litigant as a consumer then he is eligible for protection under the Consumer Protection Act that should translate into some kind of a service level guarantee. Today, if my case has been dragging for the past three years there is no way  thatI can ask the court “Why ?” and the court has no obligation to answer me. This is intolerable. The sixth Fundamental Right granted under the Constitution guarantees me the Right to Constitutional Remedies for the enforcement of the other rights and could be redefined to explicitly allow the citizen to question the courts about the delay or deficiency of judicial services.

Next, we need to create a metric for judicial efficiency because what we do not measure, we cannot control. In creating this metric let us begin with a focus on the judges because it is the judge who has the maximum say in the matter of adjournments and dates. Unfortunately many judges believe that they are above all accountability and take many liberties with the system. To check this we need two simple metrics : one for quantity and one for quality. First every judge must be given a target number  of cases to be disposed of --  monthly, quarterly and annually. Inability to meet these targets must be reflected in an annual appraisal process and there could be significant financial incentives for judges to meet and exceed targets.

But this urgency to meet targets might cause judges to simply hear and clear cases without due care and concern about the merits. Hence we need a countervailing quality metric that would keep track of the fate of the judgements passed by a judge in the higher appellate courts. The quality metric could be taken as the ratio of judgments that are upheld to judgements that are overturned on appeal. If a judge in his urgency to meet quantity targets passes bad judgments that are routinely overturned in higher courts then the financial incentive that would become due to him would be reduced drastically.

This approach of “sweating our assets” or making our judges work harder to achieve our goals is good up to a point. After that we need invest in more assets -- have more judges. Adding judges is easy -- there are enough good lawyers in this country who can be interviewed and appointed to the various courts -- and the costs associated with them, salary and incentives, would be an investment that would yield generous returns in terms of quick justice.

But the trouble is where would they sit ? and hold court ? Doubling ( or even tripling ) the number of judges would need double or triple the number of court rooms. That means more expense but what is worse is the time that will be spent on floating tenders, assigning contracts, litigation on award of unfair contracts, land acquisition, construction and so on is tremendous. The judicial transformation project will become a gigantic civil construction project much to the delight of civil contractors but of little use to the litigant.

Instead, let us have the courts operate in two or three shifts ? Most production companies use their productive assets across the entire 24 hour cycle by having three sets of workers and there is no reason why our judicial system cannot work in three shifts as well. In any case no judge is ever present in court premises for more than 8 hours -- actually far less -- and so accommodating multiple sets of judges on the same premises would not be a problem. Having the courts operate at night may be an alien concept for the Indian judicial system but there have been suggestion and requests in this regard for quite some time and no one has yet come out with any reason why it is not possible.

There can be a night shift allowance for judges and there can be a rotation of shifts so that one judge does not have to work every night -- these are matters of detail that have been adequately addressed in thousands of companies in the country. What is even better is that there will be an appropriate utilisation of the bloated and unproductive non-judicial staff in the courts. Today there is an army of peons and clerks who do little work but cannot be fired because they are the highly protected government employees. But deploying them on rotating shifts should be easier and will lead to better utilisation. Once again we are “sweating our assets” -- the physical infrastructure and the support system -- to derive more benefit from what we already have.

This two-pronged approach should lead to a significant increase in the throughput of court cases through the system. But for this approach to work, our judges must accept that they are not divine creatures, to he held in awe and reverence by the litigants, dispensing nebulous justice  and operating beyond the pale of accountability. Judges must remember that they are nothing more than service providers paid by the public to pass decisions that are enforced by the executive and like any other service provider they must provide service level guarantees on the services that they provide.

This is a mindset issue and it can easily be resolved if it has the blessings of the senior judiciary.
......................................................................................................................................................
(image taken from http://escapefromindia.wordpress.com )