October 04, 2017

Quantum Computers

My presentation on Quantum Computers at the Cypher 2017 conference in Bangalore on 21 September 2017

more information about Quantum Computers is available in an earlier blog post.

October 02, 2017

A Central Cyber Defence Authority for Digital India

The weekend of 8th - 10th July 2017 was a little different from most other weekends. First, late on Friday night, the Airtel network in the NCR region went down because of a data corruption in one of their critical computers. Then, and the exact time is not known, someone hacked, or illegally accessed, the Jio customer database, retrieved confidential identity data about customers, including phone and Aadhaar numbers, and published the same on a public website. Finally, and again the exact time is not known, something unpleasant happened to the National Stock Exchange computers so that when the market opened on Monday morning, nobody could trade for almost the entire day.

It is possible that these three events were independent, random events but as Goldfinger says, in Ian Fleming’s eponymous novel, “Once is happenstance. Twice is coincidence. The third time it's enemy action." So let us not have any delusions about incompetence or equipment malfunction. This was a cyber attack.

Who could the enemy be?

Obviously we do not know, as yet, but consider three more facts. First, this was when India was in a tense stand-off in the Sikkim sector of the Indo-Tibetan border where the Indian army had a significant situational advantage. Second, China has publicly warned India that 2017 will not be the same as 1962. Finally, in 2014, five officers of the the Chinese People’s Liberation Army Unit 61398, operating out a 12 storey building in the outskirts of Shanghai, were indicted by a US Federal grand jury “on charges of theft of confidential business information and intellectual property from U.S. commercial firms and of planting malware on their computers” leading to a tense stand-off between the United States and China over state-backed cyber espionage.

While it is true that no bullets, shells or rockets have been fired across the Himalayan border, at least not till this article was written, could it be that something else is being fired?

Most of us civilians, living far away from the LoC or International borders have not seen guns being fired in anger but thanks to television and social media coverage we have a fair idea of what happens there. But what does a cyber attack look like?   

Consider these two screenshots from two well known cyber security companies that show live cyber attacks in real time :

Watch live cyber attacks at : http://map.norsecorp.com/#/

Watch live cyber attacks at : https://cybermap.kaspersky.com/

This is only a small subset of actual cyber criminal activities that security companies can track and have chosen to make public -- like an excerpt from the register of FIRs that is maintained in every police thana in India. As in real life, most crimes are neither recorded nor publicized unless they reach epidemic or pandemic proportions like the WannaCry virus that disabled thousands of computers by encrypting critical data.

Now that we know how pervasive and ubiquitous cyber attacks are, what should we be doing to counter them? Some of us use anti-virus and anti-malware software on our personal machines and many technology savvy companies use firewalls to protect their internal networks -- that connect both users’ personal machines and company servers containing operational databases -- from hostile external access. But is this adequate?

While technology exists to stop almost every kind of cyber attack, not all end users have the knowledge, the ability and most importantly the determination to use it effectively. Consider a small or medium size company that uses a billing or financial accounting software. In the past, these would be on stand alone machines and hence inherently safe because it was “air gapped” -- or physically disconnected -- from the big, bad external world. But with more and more bills, invoices and money receipts being exchanged over mail this is no longer possible. So is the case with electronic filing of various tax returns and GST in particular. It is now impossible for any useful computer to be isolated from the internet and hence  be safe from hostile attacks from anyone, anywhere in the world. Are the computers that form the backbone of our central and state governments safe? Unfortunately, the answer is NO. So what if “non-state” hackers shut down the computers that control Power Grid’s electricity distribution network in India as was the case of the National Stock Exchange? The damage would be worse than a bomb exploding  in Howrah Station!

The challenge is less about ability and more about the attitude towards security. We know that our homes, offices and factories face threats from thieves and robbers but do we all learn martial arts and purchase guns? No, we hire security guards or outsource the security to specialised security agencies who have the expertise to handle thugs and thieves. Can our software programmers and IT staff not protect our computer systems? In principle they can, and in many companies they do keep hackers at bay but most software programmers, have expertise in a completely different area -- meeting customer and business requirements in an efficient and economical manner. Security for them is more often than not an afterthought, not the core competence. On the other hand, the durwan at the gate does not care two hoots about how and what is being produced in the factory but only knows that neither should anything go out or nor should anyone enter the premises without an approval from an authorised person. That security mindset lacking in most of our IT installations.

Which is why we have the police in towns, the CISF in factories and airports, the RPF at railway stations, the BSF and the ITBP on the borders and of course the Army as specialist agencies of the state whose only job is to ensure the security of our citizens, our factories, our infrastructure and hence of the country itself. Where is equivalent agency that guards our cyber assets? Critical machines in the GST network, the bank ATM network, the telephone network, computers that control the generation and distribution of power, computers that store Aadhaar and voter information are at the moment being guarded, if at all, by people who know little about cyber security and certainly do not have the “police” mindset that anticipates crime and thwarts threats. CERT-IN, the Indian Computer Emergency Response Team, under the Ministry of Electronics and Information is merely a technical body, not a security agency, whose responsibility is limited to collecting and disseminating information on threats and offering advice to anyone who chooses to listen. They do have the mandate to intervene during or, as is usually the case, after an attack but do not have the executive or operational responsibility to actually to prevent attacks, as is the case of the CISF or the BSF. The so called “cyber cells” of the metropolitan police are hardly any better -- all that they can do is track down mischief makers who put up politically inconvenient Facebook posts.

Going forward what we need is to separate the operational roles from security roles. Just as the security of an industrial plant is not the responsibility of the production manager but instead, is handled by a separate security department, so should be the case of security for our government installations. Those who operate IT systems should not have the additional responsibility of ensuring their security. This is not because local IT staff may not be competent enough, but because we need a consistent and comprehensive security stance at all possible threat points. It is not enough for some installations to be secure. Since all systems are interconnected, a breach anywhere is a threat everywhere and that is why we need consistent security everywhere. Hence the cyber security team should not be a part of the  local IT management but should be a part of a central organisation, the Central Cyber Defence Authority, CCDA -- analogous to CISF or BSF --  reporting directly into the security establishment in the Home Ministry.

In fact, CCDA should be an organisation on par with any other central security agency like CISF, CRPF, BSF, ITBP and like them should be headed by a person from a police, or crime prevention, background with a rank equivalent to that of the head of existing central forces. While CCDA should be responsible for government and public assets, private companies, unless they create their own separate cyber-security organisations, could outsource their cyber-security requirements to professional security companies, for whom this will be an additional line of business above and beyond their their normal fire and crime prevention services.

But while our security establishments, the Army, police, CISF, CRPF etc, may have the psychological mindset, the security stance, to anticipate criminal behaviour and prevent crime,  they would not have the technical skills to do so. Cybersecurity is not a part of the curriculum either at the Indian Military Academy or the National Police Academy and it is unlikely that it will ever be so. Even if some basic training is imparted, it will never have the technical depth required to defeat the sophisticated hacker. However the Manhattan Project, to build the atom bomb, was run by the US Army Corp of Engineers under General Leslie Groves but he had the best nuclear scientists like Robert Oppenheimer and Nobel Laureates like Richard Feynman working for him. So should be the case of the CCDA -- led by people from a police background, with an aptitude in computers and an interest in cyber security,  but staffed with people who have the deep technical knowledge, recruited laterally, or on lien,  from the IT industry.

Just as the CISF reports to the Home Ministry but is deployed in airports that report to the Aviation Ministry, the CCDA should report to the Home Ministry but should be deployed across all computer installations in all government departments, power generation and distribution companies and other critical utilities like roads, railways, telecom, ATMs. In these deployments, CCDA should be THE executive body, not be an advisory one and should have both the responsibility and the authority to ensure security. For example, it should be CCDA technicians who should have passwords for the firewall servers -- that protect government computers on, for example, the GST network  or the power transmission network -- and should be responsible for  configuring the security settings on the same.  This will be analogous to the CISF -- not DGCA, AAI or airline staff -- being the custodian of the door keys, frisking passengers and operating the X-ray scanners at the airport.

In fact, CCDA, like the Army, should also acquire offensive, or “Strike”, capabilities in addition to its professed defensive, or “Holding”, capabilities. Building offensive capabilities is a good way to test its own defenses and sometimes, offense is often the best form of defence!  

But unlike other central forces, the CCDA need not physically relocate its expert staff to distant locations even when it is deployed to protect dispersed digital assets. Just as the attacker can attack from anywhere in the world, so too can the defender protect from one or two central locations because all activity -- both offensive and defensive -- can and will be carried out over the same networks.

The HBO network was recently hacked by people who demand a multi-million dollar ransom in untraceable bitcoins to refrain from leaking episodes of the billion dollar Game of Thrones serial. What would happen if someone were to hold the Government of India to ransom with a similar hack? Just as we need to have the BSF jawan with his INSAS rifle at the LoC or the CISF jawan with his X-ray scanner at airports, we also need the CCDA jawan -- or in this case, the CCDA technician -- with his “hardened” firewall to stand guard on the digital assets that are connected to the web. The arrival of nuclear technology in the battlefield, led India to set up the Nuclear Command Authority. With the emergence of Digital India, we need the CCDA to protect the core digital assets that are critical for safety and security of the country.

Originally published in Swarajya, the magazine that reads India right!

September 03, 2017

The IT Professional & the Zone of Comfort

Rapid advances in machine learning and the spectacular success of artificial intelligence software in, say, self-driving cars, voice recognition and chatbots for customer service, is sending shivers of anxiety through IT employees. The havoc that robots and automation technology has played with the jobs of blue collared workers on the shop floor, is now travelling upward into white collar offices and not a day passes without a new report about automation eliminating jobs. In India, the IT sector -- that includes actual software developers, application maintenance staff, tech-support personnel and BPO call centre operators -- seems to be particularly vulnerable and it is no secret that a sense of doom and gloom hangs over the cubicles and around coffee machines in large and small IT companies. To make matters worse, some companies have started to shed mid-level people managers, who have stopped writing code for years,  and even senior managers who give poor returns of billability on their bloated salaries. The last straw on the back of the vanishing optimism is the reduction in campus hiring of bus-loads of low quality engineers from the hundreds of engineering colleges that have mushroomed on the promise of the Y2K inspired IT revolution. How much of this gloomy scenario is true and what can be done to bring the sunshine back? Obviously there is no quick fix but let us explore the terrain to seek a way out of these difficult times.

image borrowed from Financial Express
TCS, the biggest IT company in India was founded in 1968 and its 3.75 lakh employees generate a revenue of $18 billion while Microsoft, founded in 1975 pulls in $86 billion with 1.2 lakh employees. These, and similar, statistics has been used, ad nauseum, to pontificate that India must move up the value chain from TCS style services to Microsoft style products. But why has this not happened despite being talked about for years? One reason of course is the distance from the customer. Before the advent of the world wide web, a software builder in India would be so far removed from both the technology and the customer for the technology that it would have been impossible for him to create anything relevant. Hence the great divide between the smaller, H1B fuelled, onsite gang and their poor cousins in the offshore team. But even this results in only more services but no products. However with the internet bridging the gap this should not have been an issue -- except that it still is!

It is said that the eclectic ecosystem of Silicon Valley, with it’s simmering cauldron of technology evangelists, dreamers, brilliant programmers, venture capitalists along with legal and infrastructural support, is a fertile bed where innovative products sprout like weeds. Then how come Skype, that defines web based video conferencing was developed in Estonia? a country of 1.3 million people that most of us may not be able to locate on a map. Similarly, AVG, one of the most popular anti-virus products,  was developed in Czechoslovakia, a country with only 16 million people. But India, with over 100 million people of whom 3 million are software professionals is yet to come out with any such software product that has global acceptance and recognition. Do Indians not know how to write programs? That is highly unlikely, given the size of the IT sector in India, but what is surely missing is the ability to complete the full cycle of identifying requirements, architecting the design, securing funding, coding, building the product and eventually managing and monetising intellectual property rights. Instead, what our professionals know and do best is to receive instructions from an overseas client and code to their specifications.

This inability to go beyond meticulously following instructions and that too at a price point so beloved of our overseas clients, is the root cause of the insecurity created by the arrival of AI.  This technology is best geared to target tasks that are reasonably well defined and needs to be done repetitively. This means that in the spectrum of IT services, call centre operations and tech-support jobs are the most vulnerable. Neither is application maintenance any safer because fault diagnostics and repair is something that AI can do pretty well. What is most safe is new application, or product, development -- though even here, there are rapid application development tools that reduce the effort, and people required -- and that is where Indian IT is on its weakest wicket.

One reason why we in India are unable to come up with new products is that as a people, we are perhaps very comfortable in our respective Zones of Comfort. Our reverence for what is old, established and running is phenomenal and we are very reluctant to try out anything new. Consider the transition from MS Office, with which all of us are comfortable, to cloud based free products like Google Docs, Sheets and Slides. Despite the fact that web connectivity is as ubiquitous as electricity for our IT folks, and that the Google products meets all the requirements for 95% of IT professionals, they will almost inevitably begin with MS Office whenever they want to create a new document. Why? Zone of Comfort! This inability to try out something new inhibits our mid-level managers from “dirtying their hands” with any new technology. In fact, for many of our managers, trying out technology is considered infra-dig! Most of them prefer “management” tasks like allocation of people, attending client conference calls, preparing schedules, recording and tracking issues in minutes of meeting and so on because all that this needs is comfort with email and MS Office. In fact many managers overtly claim that is beneath their dignity to touch code -- something left for the new hires -- when the covert reality is that it is beyond their ability to do so.

In fact, this reluctance to actually “do something new” is a part of a larger tendency of being involved with consumption and avoiding creation. We would rather read a webpage than actually write a blog. It is an even greater effort for us to write a book when print-on-demand services are available for anyone who wants to publish on his own. The genesis of this mindset of consumption can perhaps be traced back to the path that our kids take from class rooms in schools to the desk at the IT company. Given the historical scarcity of jobs and the lure of campus placements, there is a mad rush for engineering entrance examinations because only those who can crack exams get selected for engineering and then placed in IT and even non-IT companies. The creative types, who are misfits in the rigid constraints of coaching classes, are automatically excluded not just from our engineering colleges but subsequently from the corporate sector. But the exam-crackers, most of whom have been successfully hammered by coaching institutes to abandon their originality and conform to patterns required by entrance examinations, enter the sector, rise through the ranks and in a pernicious cycle, recruit more and more conformists like them thus perpetuating the scarcity of creativity and innovation in our IT companies.

But all that is history. It is easy to say that we must change the system but that is neither something that will happen very soon nor will it benefit anyone in the IT industry today. What should one do to stay employable and relevant?

First, stop blaming the system, the nation or your company and take charge of your life. Light a fire under your seat and move out of your Zone of Comfort. Install an RSS reader in your browser and instead of reading client mail and following company gossip, keep an eye on RSS feeds from Slashdot, TechCrunch and Wired for latest technology trends -- say, machine learning or cybersecurity. Create blogs, contribute on discussion forums. Google and locate technology tutorials. Invest time and money -- lots of time and a little money, because not everything is free -- to acquire new skills. Skip that latest smartphone and instead, buy a personal laptop to install new, experimental software that your employer’s security policy bars on company machines. Write code, build proofs-of-concept, purchase hosting services to make these applications public and highlight these in your Linkedin profile. Go beyond the laptop, get a Raspberry Pi or an Arduino, connect it to a smartphone or even a drone -- available online in India -- to create something that people can touch and play with. Obviously things will not work as easily as they do in office projects but Stackoverflow is always there to help one go past the bleeding edge. Finally, get your kids out of coaching classes and encourage them to join you in exploring new technology! Move from the cool comfort of consumption to the caustic crucible of creation.

AI  is certainly a threat to all those who stay within their Zone of Comfort.  But technology offers infinite possibilities for those who choose to stay relevant through this fourth revolution -- agricultural, industrial, digital and now cognitive -- in human society.

This article originally appeared in Swarajya, the magazine that reads India right

August 10, 2017

Facebook : How it meddles with your mind

Facebook is the mythical 800-lb gorilla in the media world that, as the original joke goes, “sits down wherever it wants to”. With 1.2 billion pairs of eyeballs eyeing it every day, it has an audience greater than any American, European or Asian TV news network, newspaper or online news portal. This immense reach also makes it the most effective medium of entertainment. In societies where it has crossed a critical threshold of penetration, it has become the most potent mobilising force in politics and all this eventually translates into Facebook being one of the  most valuable companies in the world.
image borrowed from https://mymuddledmind.blog/

We know that information is power. We also know that power corrupts and absolute power corrupts absolutely. Should we be wary of Facebook? Consider the following ...

In the Foundation series of iconic science fiction novels by Isaac Asimov, we have  the villain, a mutant psychopath called the Mule, using popular musical concerts as a mechanism, a medium, to transmit subliminal messages to an unsuspecting audience, that demoralizes the population and breaks its resistance to the Mule’s political hegemony.  On December 17, 1997, in a chilling realisation of this fictional scenario, many news channels, including the New York Times and CNN, reported from Tokyo, that “The bright flashing lights of a popular TV cartoon became a serious matter Tuesday evening, when they triggered seizures in hundreds of Japanese children. In a national survey, the Tokyo fire department found that at least 618 children had suffered convulsions, vomiting, irritated eyes, and other symptoms after watching "Pokemon."”

Can a mass media platform be used to meddle with or influence, human minds, en masse?

As an early adopter and ardent evangelist of social media, I had always thought that platforms like Facebook and Twitter were an excellent replacement for television and newspapers as channels for current news and diverse views. But after getting drawn into a series of unintentional and inconclusive spats and flame wars with strangers with whom I have little in common and which left both sides as unconvinced about the other’s point of view as ever, I am sceptical. Was the price I was paying for using these “free” channels far too high in terms of the collaterals of irritation and anger generated in an otherwise placid and cheerful person like me? Was this my fault? Was I not savvy enough to handle this new media just as an earlier generation is psychologically uncomfortable with shopping at Flipkart or using an Android smartphone. How did the evangelist in me morph into a social media luddite, ranting against a technology? Was it just me? Or is this feeling universal?

In a peer reviewed paper published in the Harvard Business Review in April 2017, Holly Shakya and Nicholas Christakis has established what I had recently come to believe, namely, that “The More You Use Facebook, the Worse You Feel”! This is paradoxical because social interaction is a necessary and healthy part of human existence and many studies have shown that people thrive when they have strong, positive relationships with others. But when real world, physical relationships are replaced by digital and virtual relationships, the situation changes. The authors measured well-being -- through self reported life satisfaction, mental and physical health and body-mass index -- and Facebook usage -- through the number of likes, posts and clicks on links -- from three waves of data of 5208 users over two years, and came to the conclusion that overall well-being was negatively associated with Facebook usage, with the results being particularly strong for mental health. Moreover, the study also showed that the decline in well-being is strongly tied to the quantity of Facebook usage and not just the quality of interactions as it was believed to be in the past.

While the authors offer no explanation for this negative association of well-being with Facebook usage, it is not difficult to see why this is so if we consider what shows up on your newsfeed. Depending on the number of posts that your friends, and pages that you have liked, have shared there would be approximately 2000+ items that Facebook could show you but since this  leads to an uncomfortable information overload, the actual number shown is possibly as low as 200. This selection or curation is not performed by any human editor but by an artificial intelligence (AI) program that is designed to maximise benefits for Facebook. Since it is in Facebook’s interest to stimulate conversations, it’s AI will obviously select items that would provoke a user to react -- just as in a zoo, visitors throw stones at the animals instead of allowing them to rest in peace. Hence, while placid and informative items will not be totally ignored, there will always be a slight bias towards items that will provoke a reaction. For example, a Hindutva follower -- and Facebook knows our preferences to the last detail -- will be shown more items on minority appeasement, knowing fully well that is more likely to trigger a torrid response, and a subsequent equally torrid counter response,  than pictures of flowers and birds. Of course this bias is neither obvious nor in-your-face. You will still see the usual quota of bland, feel-good quotes and pictures of friends holidaying in Goa or Singapore. Which is fine, except that you just might feel a tad disappointed that you are stuck in messy Mumbai instead of being in Goa which in another reason for feeling a bit sore with yourself! Since nobody posts about their problems, this too leads to the depressing belief that everyone except you is happy.

In fact, playing and tampering with Facebook users’ emotions and deliberately trying to modify it is the subject of a very controversial paper - “Experimental evidence of massive-scale emotional contagion through social networks”, published in the June 2014 issue of the Proceedings of the National Academy of Science USA, by members of the data science team of Facebook. For the purpose of this paper, the Facebook team deliberately introduced a certain bias in the nature of items included in the Facebook user’s newsfeed and observed the impact on their subsequent behaviour. To quote the authors, “In an experiment with people who use Facebook, we test whether emotional contagion occurs outside of in-person interaction between individuals by reducing the amount of emotional content in the News Feed. When positive expressions were reduced, people produced fewer positive posts and more negative posts; when negative expressions were reduced, the opposite pattern occurred. These results indicate that emotions expressed by others on Facebook influence our own emotions, constituting experimental evidence for massive-scale contagion via social networks.”

This paper was criticised for violating basic ethical principles of psychology research because no consent was sought from the subjects whose emotions were being tampered with. That does not detract from the fundamental premise that Facebook has the ability to modify the emotions of its users and has done so in the past.  In fact, what is even more disturbing is that Facebook now has the technology to use  webcams and smartphone cameras to track emotions in real-time by detecting, decoding facial expressions as we read posts! While there is no evidence of any deliberate evil intent as yet, the fact that it’s AI based news selection service can detect and tamper with the emotions of users is a big red flag because, as noted earlier, Facebook touches more people than any newspaper, television channel or news portal and so has the ability to mould the emotions of a significant part of the global population.

While Facebook has been targeted for being a channel or firehose for fake and unstantiated news, the real danger lies in its ability to tamper with our emotions and, as reported in the HBR paper, make all of us feel angry, frustrated, jealous and upset with the world around us. Can we do anything to mitigate this unfortunate state of affairs? At a personal level, one could reduce the amount of time spent on the platform but since Facebook is an addiction like tobacco or alcohol with similar withdrawal symptoms, this may not be a feasible solution for everyone.

What users could ask for instead, is greater transparency in the algorithm, the procedure, used to determine what they see or don’t. If I want to see posts about birds and flowers, I must not be shown pictures of stone-pelters in Kashmir. In fact, such a process does exist, because you can indicate the kinds of posts that you want to see less of, but a more direct method should go a long way to restore the sense of choice that we have in newspapers and TV to read or ignore specific items of news and views

Social media is here to stay and Facebook, with its unassailable reach and immense clout, is something that -- like the monsoon rain -- we have to learn to live with. However knowing the danger that it poses and working on ways to reduce its impact is something that needs urgent action.

This article originally appeared in Swarajya, the magazine that reads India right.

July 27, 2017

OLAP Data Cube with SQL

As an erstwhile DBA, a long time user and a great admirer of the SQL language -- that has stood the test of time for the last 30 years -- I have always sought to use SQL in many useful ways. In an earlier post, I had shown how SQL can be used to solve a classic data science problem, namely Clustering, using the K-Means algorithm and today, I demonstrate how SQL can be used to process OLAP data cubes and generate the popular cross-tabs table.

Data cubes, or OLAP cubes, are a way to store historic data using the dimensional model, as opposed to the relational or 3rd normal form model. These data cubes can be "sliced" and "diced" to reveal data relevant to particular dimensions. Because of the immense popularity and ubiquity of relational databases, like Oracle and MySQL, data in the dimensional model is routinely stored in relational tables and retrieved -- by slicing and dicing the cube -- using standard SQL constructs like the WHERE clause. This is called Relational OLAP or ROLAP.

Data cubes are very popular because they allow multidimensional data to be collapsed to any two dimensions and shown as a "CrossTab" -- and human beings can comfortably visualise only two dimensions on a page or a screen. Unfortunately, creating CrossTabs is not very easy with normal SQL and that is why there exist a genre of specialist products -- Multidimensional OLAP or MOLAP -- that allow users to create CrossTabs by "rotating" the data cube as necessary.

Microsoft SQL-Server, a RDBMS product, has a proprietary construct called CUBE that allows this feature but this is not available in most RDBMS products and certainly not in MySQL, the free and open-source product that is the most widely used RDBMS on the planet.

The following slide deck shows how MySQL can be used to "rotate" an OLAP data cube and generate CrossTabs for any cube of dimension 3 or higher

(please view the slide deck in full screen mode)
We also show how a "pivot" table, so beloved of Excel users can also be generated using MySQL and hence by extension in any RDBMS.

But why would anyone wish to use SQL or MySQL to build and work with data cubes when MOLAP tools are available?
  1. First, SQL is easily understood and widely used by a vast majority of IT professionals
  2. Second, MySQL is a free and open-source product that is used in almost every web application
  3. Third, SQL is supported in a multi-machine, clustered environments like Hadoop/Hive and Spark and so this technique can be used -- at least in principle -- to support data cubes built with ultra large data sets.
Unless one wants the bells and whistles that come along with most MOLAP products, MySQL is good enough for almost any OLAP activity and can be scaled up with Hive / Spark for very large data.

Acknowledgement : The technique demonstrated in this post has been adopted from information provided at http://www.artfulsoftware.com/infotree/qrytip.php?id=78

June 30, 2017

Quantum Computers

Quantum mechanics is a subject that has the strange property of simultaneously being logically rigorous and yet completely counterintuitive. So much so, that even a towering intellect like Einstein could never bring himself to accept its principles even though products based on the same exist all around us. The earliest oddity, identified by Schrodinger, one of the founders of quantum mechanics is about a hypothetical cat that is neither dead nor alive until someone actually observes it. A similar oddity is that of quantum entanglement, where the behaviour of one particle is instantly affected by the behaviour of another particle, however distant it may be -- an example of “spooky” action-at-a-distance. Explaining these phenomena is beyond the scope and temerity of this article and so the reader would have to accept them here in good, almost religious, faith and carry on with the belief that such phenomenon has been observed and explained by scientists under the most rigorous experimental circumstances.

Image borrowed from Quanta Magazine
Any programmable digital computer that we use, the desktop,the smartphone or the ones at Google, is based on a finite state machine (FSM). It can, at any instant of time, be in one of a large, but finite, number of well defined states. The state of a FSM is defined by the value stored in each of its memory locations and we know that these can either be 0 or 1. So an FSM with, say, 16 bits of memory could in principle be in any one of 2^16 states. Any instruction to the FSM changes the value of one or more bits and and the FSM moves to a different state. An FSM along with the ability to read binary input, from an infinite tape, and write back on the same tape, is the Turing machine that is the theoretical basis of any modern computer.

The fundamental principle of computer science is that the world is computable, meaning that any logically decidable problem can be represented and solved on a Turing machine and hence by extension on some, possibly very powerful, digital computer. This is the basis of our immense belief in computer technology that powers everything from smartphones to artificial intelligence. But even as long back as 1982, Richard Feynman had questioned this principle because he realised that Turing / FSM based computers could not solve the problem of simulating the movement of multiple particles whereas nature was doing it all the time! Did the quantum mechanical behaviour of nature mean that nature had a computing device that was inherently superior to the Turing machines built by classical computer technology? This is where the concept of a quantum computer was born.

A computer, is a state-machine where it’s state is defined by the collective states of each of its memory locations. In a classical computer, each memory location, or bit, can either be 0 or 1 certainly not both, but in a quantum computer it can be both 0 and 1 simultaneously -- very much like Schrodinger’s cat that was dead and alive at the same time!  This is where the going gets really rough for anyone who has spent a lifetime in classical computer science because this is something that is completely counter-intuitive. A memory location, a bit, is a transistor, or switch, made of silicon that is either ON or OFF. How can it be both? Turns out, that if you keep aside computer science and open your books on quantum mechanics, it is indeed possible that a body can be in two states at the same time based on the well established principle of quantum superposition. Now if we go back to our 16 bit classical computer with its 2^16 states and replace it with a quantum computer with 16 quantum bits, or qubits, of memory we have a machine that can be in 2^16 states simultaneously. If that is not mind-bending enough, all these 2^16 states will collapse into any one of the states as soon as we try to observe it. It is almost as if nature is playing a game with us, pretending to be classical whereas it is actually quantum.

But why are we obsessed with this counter-intuitive phenomenon? Will it have a drastic improvement on existing digital computer technology? Not really. Your spreadsheet, email, YouTube, eCommerce, smartphone will hardly change but two things could. First, current cybersecurity systems, that are based on our inability to decompose integers into their prime factors in a reasonable amount of time, could be ripped apart by quantum computers, leaving all passwords vulnerable to hackers. Second artificial intelligence could be taken to altogether and unbelievable levels of sophistication. So quantum computers will soon have a very important role to play -- but how far away are we from real, practical systems?

The biggest challenge is the construction of the physical memory locations and the complexity of the engineering problem is evident from the following : A modern IBM classical computer chip has anything between 2 and 7 billion transistors each of which can be ON or OFF. The corresponding IBM quantum computer chip, that powers the IBM Quantum Experience machine, has only 5, yes just 5, qubits of memory that can be in quantum superposition of ON and OFF. Why so? First, the memory locations have to be cooled to near zero Kelvin to exhibit their quantum superposition behaviour and if the cryogenic challenge was not enough, the second challenge is even bigger. Unlike the memory locations of classical computers whose state can be determined by sensing the presence or absence of an electrical voltage, the multiple, superimposed quantum states collapse as soon as any effort is made to observe them. This is as if a room has a house of cards that collapse as soon as the door is opened by the observer and the observer has to figure out what the house looked like by observing the disposition of the cards on the floor! Since the qubits can never be accessed directly, as in a classical computer with read and write statements, they can only be “influenced” indirectly.

To put things in perspective, ENIAC, one of the world’s first, 1st generation, vacuum tube based classical computer had 20 memory units, or accumulators, in 1945, and a 2nd generation, transistor-based computer from the University of Manchester had only 200 transistors in 1955. Since then we have moved through 3rd generation integrated chips and the current 4th generation of microprocessors have scaled up to billions of transistors thanks to the inexorable pressure of Moore’s Law. If we remember that even with its 20 memory units, ENIAC was used to solve problems in weather forecasting, atomic energy calculations, wind tunnel design, the current 5 qubit IBM machine does not look as hopeless, or helpless, as it seems to be.

But actually things are a little better off. D-Wave a Canadian company that has been building quantum computers since 1999  have come out with a 128 qubit machine in 2010, a 512 qubit machine in 2012 and 1000 qubit machine in 2015. Initially there were some doubts about whether these were quantum machines at all but after these machines were actually installed and used first by Lockheed Martin at the University of Southern California and later at the Quantum AI Lab of NASA Ames Research Centre by a team from Google, these doubts have receded to a large extent. But even though some doubts persist, there is enough evidence of quantum behaviour or at least great promise that these doubts will be removed soon. In early 2017, D-wave announced the sale of their first, commercial available $15 million 2000-qubit machine to cyber-security firm, Temporal Defence Systems.

IBM’s 5-qubit Quantum Experience is positioned as general purpose computer. It could be used for any computational task but would be efficient only if the program was designed to use quantum properties -- a colour TV is useful only if the broadcast is in colour. Very few programs can do this today but Shor’s algorithm, used to crack passwords, is definitely one such. D-Wave systems on the other hand are designed to solve one class of problems that minimise the weighted sum of large number of interrelated, or entangled, variables. This may sound restrictive but the reason why everyone from Google to Temporal is interested is because this class of problems is similar to the ones that occur in artificial neural networks that lie at the heart of systems based on machine learning.

Spectacular progress in machine learning with artificial neural networks using classical computers itself, is rapidly closing the gap between biological and nonbiological intelligence or even between carbon and silicon “life-forms”. With the advent of quantum computers one more crucial barrier between the natural world and it’s man-made, artificial model could break down -- as could the increasingly thin line that delineates man from machine. Will this drag man down to the level of machines? Or will these machines push man up towards his eventual union, or Yoga, with the transcendent omniscience that some refer to as God or Brahman?

This article originally appeared in Swarajya -- The magazine that reads India right!

June 03, 2017

Order, Stability or Chaos?

Global, national and local societies face many threats. We are threatened by enemies -- internal and external -- who want to destroy our way of life. We are plagued with environmental degradation as we quickly try to ramp up the economy and improve our living standards. Finally our own social systems are in tatters because efforts to mitigate the effects of the first two reasons are stymied by venal corruption and a cynical disregard for the rule of law. In fact the last reason is perhaps the most over arching reason, because it leads to the other two.

image from 5rhythms
We have solutions to most of our problems. Technology solutions are available to grow more food, generate more energy, combat disease and check crime. There are public structures like hospitals, schools, municipal, state and central governments, the legislature, each having its own set of rules and procedures, to guide and govern matters. There are commercial structures, like corporates, cooperatives and professional networks that transform natural and human resources into disposable surplus that can be used for material pleasure. Then there are clubs, non-profits and political parties that lubricates the gears and facilitates the work of the public and private structures. Finally, we have a whole set of checks and balances, like police, the courts of law, and institutions that recursively keep checks on the checks and balances, like Vigilance Department, the CBI and the LokPal to ensure that everyone does what they should. So in principle, if everything were to work like clockwork, there should not be any unresolved problems on the planet.

But obviously this is absurd. Unlike the precise determinism of classical mechanics, the social mechanism that governs society is based on the non-deterministic behaviour of human beings. No two persons are alike and so no two will respond to a situation in an identical manner. One may be afraid to break the law even if there is a benefit but another may be willing to do so. So there is an element of randomness that permeates society and it is this randomness that is key determinant of social outcomes.

Randomness leads the environment from order to disorder. Physics equates disorder with entropy and the Second Law of Thermodynamics states that entropy of a closed system can only increase over time. In fact the direction of the “arrow of time” is often determined by the level of entropy between two states of the system. Information theory also associates entropy with randomness. Uncertain, random events are associated with high information content and hence high entropy. Certain events, like the daily sunrise, that have a probability of 1, are associated with zero entropy, as are impossible events like a horse giving birth to a dog, that have a probability of 0. But entropy is high when there is uncertainty and unpredictability as in the outcome of a toss of a fair coin, the results of an election or a war.

Increase in entropy, in randomness, in unpredictability, leads to chaos that can be analysed in terms of Chaos Theory. Chaos is the inevitable outcome of any adaptive, dynamic and complex system which is exactly what human society is. Chaos is unpredictability in the face of apparent determinism -- and as Edward Lorenz puts it so elegantly, Chaos is when the present determines the future, but the approximate present does not approximately determine the future. What this means is that a slight change in initial conditions -- a crow flapping its wings in Calcutta -- can cause a major upheaval far away -- a tornado in Texas. Mapped to human society, it means that social uncertainty caused by the erratic, unpredictable behaviour of a even a small group of people can cause ripples and upheavals across the world.

Chaos theory allows for strange attractors, or periodic repetitions of somewhat predictable outcomes, which is why human society settles into equilibria that gives us a sense of stability.  But given its colossal complexity even one incident, like 9/11, can tip it into a new, possibly more uncomfortable and anarchic equilibrium. Complexity is in fact impossible to manage in large organisations which is why we have the eventual collapse of centrally governed empires -- the Kaurava, the Pharaonic, the Roman, the Mauryan, the Holy Roman, the Ottoman, the Mughal, the British, the Soviet and finally the European Union. We can only hope that India will not join this list. Well governed human societies are based on the rule of law and order and it is this order that is under threat from the Second Law of thermodynamics and Chaos Theory. While we all crave for order, the reason why we rarely attain it is because the laws of the universe inexorably push us towards disorder and anarchy.

But will entropy always increase? Not really. In a small closed system -- as in a school, a company, a factory, a state like Singapore, or perhaps a human colony on Mars -- it is possible to reduce the local entropy within the system and impose perfect order, but this needs one of two prerequisites. Either we need an external agency imposing order from outside -- a non-popular dictatorship -- or there has to exist a mechanism of self-organisation, that resolves contradictions and guides the system towards greater order. A small school or factory is an example of the first while well governed US cities that are cleaner and more habitable than anarchic municipalities in India is an example of the second.

But even in a small society, that is somehow isolated from the random anarchy of the global environment, the ability to self regulate is not guaranteed. Self regulation is actually an outcome of enlightened self-interest that seeks to create the proverbial win-win situation that benefits all at the cost of none. But this is not easy. To understand why, consider the Prisoner’s Dilemma, a special case of a mathematical oddity called Nash Equilibrium that is a part of Game Theory.

Consider two persons who have been arrested for a murder but the police do not have any clinching evidence, with which they can ensure a conviction. So both prisoners are offered a plea-bargain offer. If any one turns approver and betrays the other, then the betrayer will be let off but the other will serve twenty years in jail. If both turn approver, then both serve ten years in jail. But if both cooperate and neither betrays the other, then the police will imprison them for a year on a lesser crime. Unfortunately, neither do the prisoners have any knowledge of what the other prisoner will do and nor do they trust each other. Ideally neither should betray the other, because this will ensure light punishment for both which is the best solution. But in reality, given the uncertainty, neither will trust the other, both will betray each other and so ensure ten year hardship for both. A classic lose-lose scenario.

This scenario is reflected in many real life situations like women wearing makeup to look more elegant, athletes using steroids to enhance performance, over-exploitation of resources like fishes or minerals, countries spending money on arms and ammunitions, countries refusing restrictions on environmental pollutants that hamper economic growth, advertisers spending money to push competing products or bidders at an auction being afflicted with the winner’s curse. In India, aggressive drivers break traffic rules to squeeze past others and in the process create  massive traffic jam whereas everyone could reach home earlier by waiting and obeying traffic rules.

If only people would cooperate with each other, the world will be a better place but the inexorable laws of Game Theory says that this will never happen. If all political parties were to cooperate on matters of national interest, like implementing labour reforms or fighting Islamic terror, many of the social and economic problems that bedevil India can be quickly eliminated but as in the case of the Prisoner’s Dilemma, each political party thinks that cooperating with the other means sealing one’s own electoral fate and facilitating a landslide victory for the other.

Human society is in a bind. The Second Law and Chaos Theory pushes us towards anarchy while Game Theory prevents us from self-organising. So we are forced to reconcile ourselves to a chaotic future. Given the inevitability of chaos in complex systems, our only hope for stability and order would be to have smaller, simpler systems that are easier to manage. Small states, municipalities, panchayats and even gated communities, where the number of players, or variables, is small and where complexity is manageable, have a far better chance of avoiding anarchy.  Going forward, as complex social and security challenges -- both international and now more often intra-national -- overwhelm the world, a loosely-coupled federation of small, self-sustainable, technology enabled, well-managed, elitist communities or “smart-cities”, spread across the Earth and nearby planets, may be the only way towards a reasonably stable future.

The Prisoner’s Dilemma and the inability of people to collaborate for the common good may be a persistent roadblock on the path to global peace with prosperity.

This article first appeared in Swarajya - the magazine that reads India right

About This Blog

  © Blogger template 'External' by Ourblogtemplates.com 2008

Back to TOP