Bagaimana cara membuatDataFrame dari dict? Saya menggunakan kode berikut dan menemui kesalahan.

from pyspark import SparkContext, SQLContext
sc = SparkContext.getOrCreate()
spark = SQLContext(sc)

result_dict = {'a':3,'b':44}
data = list(map(list, result_dict.items()))
f_rdd = spark.createDataFrame(data, ["A", "B"]).repartition(1)

Kesalahan:

AttributeError                      Traceback (most recent call last)
<ipython-input-10-a25453caa1c3> in <module>
      5 result_dict = {'a':3,'b':44}
      6 data = list(map(list, result_dict.items()))
----> 7 f_rdd = spark.createDataFrame(data, ["A", "B"]).repartition(1)

AttributeError: 'SQLContext' object has no attribute 'createDataFrame'
0
rosefun 3 Juli 2020, 02:40

1 menjawab

Jawaban Terbaik

Anda dapat mencoba cara ini

    from pyspark.sql import SparkSession

    spark = SparkSession.builder \
        .appName('so')\
        .getOrCreate()

    sc= spark.sparkContext

    map = {'a':3,'b':44}
    data = sc.parallelize([(k,v) for k,v in map.items()]).toDF(['A','B'])

    data.show()

    # +---+---+
    # |  A|  B|
    # +---+---+
    # |  a|  3|
    # |  b| 44|
    # +---+---+
1
kites 2 Juli 2020, 23:52