Below are the basic Mac Terminal Commands to set up Anaconda, PySpark and Java on Mac for Jupyter or Colaboratory.
1. Revise in Terminal your Python version
python
Python 3.9.7 (default, Sep 16 2021, 08:50:36)
2. Download and install anaconda for Mac related to your Python version
2.1. from Web Browser
https://www.anaconda.com/products/individual
2.2. from Mac Terminal
bash ~/Downloads/Anaconda3-2019.03-MacOSX-x86_64.sh
3. Restart your Terminal and revise the installed files
conda list
# packages in environment at /opt/anaconda3:
#
# Name Version Build Channel
_ipyw_jlab_nb_ext_conf 0.1.0 py39hecd8cb5_0
alabaster 0.7.12 pyhd3eb1b0_0
anaconda 2021.11 py39_0
4. Control the opt anaconda folder and revise that it has a spark
cd opt
cd anaconda3
touch hello-spark.yml
vi hello-spark.yml
source activate hello-spark
conda create --name myenv python=3.9
conda activate myenv
5. Revise Java version
java -version
java version "1.8.0_321"
ls /Library/Java/JavaVirtualMachines
6. Download and Revise Java version
http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html
/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
/bin/bash -c $(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install.sh)
brew tap homebrew/cask-versions
brew install brew-cask
brew install brew-cask-completion
brew doctor
sudo xcode-select --install
brew install --cask adoptopenjdk8
brew install --cask homebrew/cask-versions/adoptopenjdk8
brew tap adoptopenjdk/openjdk
conda env remove -n hello-spark -y
7. Other installations in /usr/local/bin/
brew install apache-spark
apache-spark 3.2.1 is already installed and up-to-date.
brew install python3
pip3 install pyspark
brew install pyenv
brew install scala
Using Scala version 2.12.15 (OpenJDK 64-Bit Server VM, Java 11.0.12)
brew install sbt
brew info apache-spark
/usr/local/Cellar/apache-spark/3.2.1
conda install -c conda-forge findspark
conda install -c conda-forge pyspark
python -m pip install findspark
Requirement already satisfied: findspark in /opt/anaconda3/lib/python3.9/site-packages (2.0.1)
pip install pyspark
Requirement already satisfied: pyspark in /opt/anaconda3/lib/python3.9/site-packages (3.2.1)
Requirement already satisfied: py4j==0.10.9.3 in /opt/anaconda3/lib/python3.9/site-packages (from pyspark) (0.10.9.3)
jupyter notebook --profile=pyspark
8.Declarations
export JAVA_HOME=/Library/java/JavaVirtualMachines/adoptopenjdk-8.jdk/contents/Home/
export JAVA_HOME=/usr/local/Cellar/openjdk@11/11.0.12/libexec/openjdk.jdk/Contents/Home
export JRE_HOME=/Library/java/JavaVirtualMachines/openjdk-13.jdk/contents/Home/jre/
export SPARK_HOME=/usr/local/Cellar/apache-spark/3.2.1/libexec
export PATH=/usr/local/Cellar/apache-spark/3.2.1/bin:$PATH
export PYSPARK_PYTHON=/Library/Frameworks/Python.framework/Versions/3.9/bin/python3
export PYSPARK_DRIVER_PYTHON=jupyter
export PYSPARK_DRIVER_PYTHON_OPTS='notebook'
which python3
/Library/Frameworks/Python.framework/Versions/3.9/bin/python3
which python
/opt/anaconda3/bin/python
9. PySpark
9.1.Terminal Confirmation
spark-shell
9.2. Spark Shell Application User interface - Jobs, Stages, Storage, Environment, Executors
http://xxxxxxxxxxxxx-imac.home:4040
https://www.guru99.com/pyspark-tutorial.html
https://medium.com/swlh/pyspark-on-macos-installation-and-use-31f84ca61400
https://stackoverflow.com/questions/63216201/how-to-install-python3-9-with-conda
https://docs.anaconda.com/anaconda-scale/howto/spark-configuration/#scale-spark-config-sparkcontext
https://docs.datastax.com/en/jdk-install/doc/jdk-install/installOpenJdkDeb.html
https://docs.anaconda.com/anaconda-scale/howto/spark-configuration/
https://www.dataquest.io/blog/pyspark-installation-guide/
https://notadatascientist.com/install-spark-on-macos/
Comments