blog@Yantrajaal

In the  previous post , we saw how to run a Spark - Python program in a Jupyter Notebook on a standalone EC2 instance on Amazon AWS, but the real interesting part would be to run the same program on genuine Spark Cluster consisting of one master and multiple slave machines. The process is explained pretty well in  Tom Zeng's blog post  and we follow the same strategy here. 1. Install AWS Command Line services by following these instructions . 2. Configure the AWS CLI with your AWS credentials using these instructions . in particular, the following is necessary $ aws configure AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE  AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY Default region name [None]: us-east-1 Default output format [None]: ENTER you will have to use your own  AWS Access Key ID and  AWS Secret Access Key of course! 3. Execute the following command : aws emr create-cluster --release-label emr-5.2.0 \ ...

In an earlier post I have explained how to run Python+Spark program with Jupyter on local machine and in a subsequent post , I will explain how the same can be done an AWS EMR cluster of multiple machines. In this post, I explain how this can be done on a single EC2 machine instance running Ubuntu on Amazon AWS. The strategy described in this blog post is based on strategies described in posts written by Jose Marcial Portilla and Chris Albon . We assume that you have a basic familiarity with AWS services like EC2 machines, S3 data storage and concept of keypairs and an account with Amazon AWS. You may use your Amazon eCommerce account but you may also create one on the AWS login page . This tutorial is based on Ubuntu and assumes that  you have a basic familiarity with the SSH command and other general Linux file operation commands. 1. Login to AWS Go to the AWS console  ,login with userID and password, then go to the page with EC2 services. Unless you have used AWS ...

Hadoop is history and Spark is the new kid on the block who is the darling of the Big Data community. Hadoop was unique. It was a pioneer that showed how "easy" it was to replace large, expensive server hardware with a collection, or cluster, of cheap, low end machines and crunch through gigabytes of data using a new programming style called Map-Reduce that I have explained elsewhere . But "easy" is a relative term. Installing Hadoop or writing the Java code for even simple Map-Reduce tasks was not for the faint hearted. So we had Hive and Pig to simplify matters. Then came tools like H20 and distributions like Hortonworks to make life even simpler for non-Geeks  who wanted to focus purely on the data science piece without having to bother about technology. But as I said, with the arrival of Spark, all that is now history! Spark was developed at the University of California at Berkeley and appeared on the horizon for data scientists in 2013 at an O'Reilly ...