AttributeError: ‘SparkContext’ object has no attribute ‘list_packages’ – AWS EMR

Steffy Alen

3 years ago

I am trying to install a new package in AWS EMR and running into the error “AttributeError: ‘SparkContext’ object has no attribute ‘list_packages'”. However, this seems to be the official approach. Appreciate your help in finding what I am missing

Code

from pyspark.sql import SparkSession, Window
from pyspark import SparkConf, SparkContext
spark = SparkSession.builder.getOrCreate()
sc =SparkContext.getOrCreate()
sc.list_packages()

Config emr-5.20.0 Spark 2.4.0