blog@Yantrajaal

Last June, I had the good fortune to visit Pangong Lake, in Ladakh, on the Indo-Tibetan border -- shown in the movie 3 Idiots -- and spent a cold and windswept night in a tent at Spangmik. The place is awesome. A huge blue-green mass of salty water -- a remnant of the seas that got trapped between India and Tibet when the former rammed into the latter and created the Himalayan range -- is surrounded by immense snowy peaks that turn golden at sunrise. But what was even more impressive was the night sky with its hundreds of stars, that we in the cities have forgotten about. Thanks to the Global Positioning System (GPS) and the Google Sky app on my smartphone, I could, for the first time in many, many years see and, more importantly, identify so many stars and constellations, including, Polaris - one of the most important stars used by ancient mariners to determine their position. Then it struck me -- in the past, stars were used to determine location but today the GPS location from my i...

A key challenge in building the "Internet of Things" is to be able to connect a device to a computer over the internet and to use as simple and lightweight an infrastructure as possible. In this post we demonstrate how a public XMPP chat server can be used to transmit data and commands from one device to another using a chat client at one end and a python "bot" sitting on the other end. We will demonstrate the ability to INSERT data into an SQLite  database, SELECT records from the same, play a variety of .wav files and execute any system commands on a "central" machine from any distant machine that supports an XMPP chat client. Before starting on this exercise, we searched the web for prior activity in this area and we came across this webpage  that suggests a similar approach for controlling devices over the internet, but the strategy explained here is simpler to code and implement. We looked for a list of public XMPP chat servers and selected ...

As a former DBA I love the fact that SQL has a life of its own and the way it is still used after it was first conceived of in the 1980s as the natural way to query Relational Database Management Systems. A vast amount of data in the world is still available in SQL compliant RDBMS tables and today when business analytics and data science seems to be overshadowing SQL, I was delighted to find that SQL can still play a very important role in the implementation of complex data mining applications. This post explains how this evergreen tool is still very, very relevant in data mining. Many data mining tools like Rattle , RapidMiner are used to create "models" for Classification / Decision Trees and Heirarchical Clustering  but then the models have to put into production by using them to score large datasets. This is where SQL can play a very powerful role. The models created in the data mining tools need to be exported as PMML documents and then converted to SQL using any of...

Continuing with my story of Yantrajaal and the Internet of Things .... A key challenge in building the "Internet of Things" is to be able to connect a device to a computer over the internet and to use as simple and lightweight an infrastructure as possible. In this post we demonstrate how a public XMPP chat server can be used to transmit data and commands from one device to another using a chat client at one end and a python "bot" sitting on the other end. We will demonstrate the ability to INSERT data into an SQLite  database, SELECT records from the same, play a variety of .wav files and execute any system commands on a "central" machine from any distant machine that supports an XMPP chat client. Before starting on this exercise, we searched the web for prior activity in this area and we came across this webpage  that suggests a similar approach for controlling devices over the internet, but the strategy explained here is simpler to code and im...

Privacy, or the Right to Privacy, has recently entered the public debate in India because the government has claimed in the Supreme Court that this right is not fundamental to the life and liberty that is otherwise guaranteed in the Constitution. But before we weigh in, in this debate, let us consider that even otherwise, how much privacy do modern, technology enabled individuals actually enjoy today? All of us use Google directly - for search, for mail, for watching videos, possibly for cloud storage. We also use Google Maps when we use Uber, Zomato and similar location based apps on our Android phones that use its Global Positioning System (GPS) features. Facebook is our preferred way to keep in touch with family, friends and acquaintances with whom we share updates and pictures and respond to their updates with comments of our own. There are many other services that we use but let us stick to these two that are most ubiquitous and are immensely popular -- and of course, the ici...

In June 1999, when this domain YANTRAJAAL was created I had envisioned and written that : The World Wide Web has outgrown its initial concept of a mere network of computers and is being seen today more as a way of life. As it thrives and grows, this web is encompassing and pulling into itself a greater and greater diversity of devices – set top boxes, smart television sets, terrestrial and cellular phones, palmtops, network computers and very soon common household gadgets like refrigerators and microwave ovens. And along with these gadgets, our entire lifestyle is being dragged into this great web or Jaal. YantraJaal – is an eZine that reflects this new reality. In Sanskrit, “Yantra” is an artifice or a device and “Jaal” is the net or the web. YantraJaal thus, represents the web of connected devices.  Today, the Internet of Things or IOT is finally becoming a reality but most of us are still grappling in the dark on how to move beyond the simple web servers and the world of b...

Can Digital India be launched on the back of a private social media network ? E-Governance is a concept that has fascinated bureaucrats, academics and the IT industry in India for many years but the term is ill defined. At one end of the spectrum we have government departments that put up web portals with static, mostly obsolete, data while at the other end we have useful applications for, say, passports and income tax. This great disparity in sophistication and utility is because each such application is the result of an independent initiative and reflects the vision of the owner and the competence of the vendor who was awarded the tender on the L1 ( lowest cost) basis. In the corporate sector this is referred to as the “Thousand Island” scenario  -- with “islands” of automation separated by gaps of inconsistent data -- and the common solution that is offered is based on an “ERP” like SAP that can tie together all parts of the organisation with one coherent software. Unfortunat...

Distance learning using the internet -- that allows students anywhere in the country to learn from the best teachers --  is old news! We have heard of Massively Open Online Courseware (MOOCs) popularised by Coursera and Udemy, applauded the good work done by the Khan Academy and proudly talk about the gigabytes of, rather boring, videos loaded into Youtube by IIT professors under the NPTEL program. But none of all this has had any impact on the critical skill gap that separates students who pass out of India’s colleges from the jobs that await them in a booming economy. Institutions like IITs, IIMs,  NITs, Presidency and others, may be doing well, but there are another 600 degree granting institutions with more than 35,000 affiliated colleges that have lost the plot completely. Most of these have inadequate infrastructure, teachers who are barely competent or rarely in the classroom, outdated syllabi and an academic atmosphere vitiated by student politics and yet it ...

The challenge of combining the statistical power of R and the "Big Data" capabilities of Hadoop is something that has always fascinated me. Over a year ago, I had finally broken free from from the stupidity of the WordCount ( and various other counting ) programs and tried to solve a real like retail problem with linear regression using R and Hadoop. This is documented in my blog post Forecasting Retail Sales -- Linear Regression with R and Hadoop . In this case however I had used the Hadoop streaming API to call to separate R programs. Subsequently I had come across the Hortonworks HDP platform that dramatically simplified the process of installing and running Hadoop. This is explained in my blog post Big Data for the Non Geek, where in addition to installing Hadoop, I have also explained how to overcome the challenges of installing the RHadoop packages on top Hadoop on the Hortonworks platform. Hortonworks has a nice example of how to run an rHadoop program on the HDP...