blog@Yantrajaal

In the  previous post , we saw how to run a Spark - Python program in a Jupyter Notebook on a standalone EC2 instance on Amazon AWS, but the real interesting part would be to run the same program on genuine Spark Cluster consisting of one master and multiple slave machines. The process is explained pretty well in  Tom Zeng's blog post  and we follow the same strategy here. 1. Install AWS Command Line services by following these instructions . 2. Configure the AWS CLI with your AWS credentials using these instructions . in particular, the following is necessary $ aws configure AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE  AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY Default region name [None]: us-east-1 Default output format [None]: ENTER you will have to use your own  AWS Access Key ID and  AWS Secret Access Key of course! 3. Execute the following command : aws emr create-cluster --release-label emr-5.2.0 \ ...

In an earlier post I have explained how to run Python+Spark program with Jupyter on local machine and in a subsequent post , I will explain how the same can be done an AWS EMR cluster of multiple machines. In this post, I explain how this can be done on a single EC2 machine instance running Ubuntu on Amazon AWS. The strategy described in this blog post is based on strategies described in posts written by Jose Marcial Portilla and Chris Albon . We assume that you have a basic familiarity with AWS services like EC2 machines, S3 data storage and concept of keypairs and an account with Amazon AWS. You may use your Amazon eCommerce account but you may also create one on the AWS login page . This tutorial is based on Ubuntu and assumes that  you have a basic familiarity with the SSH command and other general Linux file operation commands. 1. Login to AWS Go to the AWS console  ,login with userID and password, then go to the page with EC2 services. Unless you have used AWS ...