Software

Use software on the campus server:

• PySpark enabled Jupyter Notebook: http://adal.etown.edu:12345

Install HDP Sandbox by yourself:

• Docker Engine: https://docs.docker.com/engine/install/. Find the correct installation guide for your operating systems.

• Hortonworks Data Platform (HDP) on Sandbox version 2.6.5: https://www.cloudera.com/downloads/hortonworks-sandbox/hdp.html You need to register a Cloudera account first, then choose the correct installation type on Docker (less resource demanding than a virtual machine). The recommended configuration is 4 CPUs and 8 GB of free RAM; however, 2 CPU and 6 GBs of RAM should be enough for our case. Reducing the amount of RAM will make some of the services unavailable.

• Follow along the tutorial to install HDP on docker engine: https://www.cloudera.com/tutorials/sandbox-deployment-and-install-guide/3.html

Install PySpark for Your Own Jupyter Server:

• Checkout my tutorial on Github: https://github.com/Peilong/PySparkOnJupyter/blob/master/README.md