Saya membaca data berikut dari file json di pyspark:

{"positionmessage":{"callsign": "PPH1", "name": "testschip-10", "mmsi": 100,"timestamplast": "2019-08-01T00:00:08Z"}}
{"positionmessage":{"callsign": "PPH2", "name": "testschip-11", "mmsi": 200,"timestamplast": "2019-08-01T00:00:01Z"}}

Kode terlihat sebagai berikut:

from pyspark.sql import SparkSession
from pyspark.sql.types import ArrayType, StructField, StructType, StringType, IntegerType, DateType, FloatType, TimestampType

appName = "PySpark Example - JSON file to Spark Data Frame"
master = "local"
# Create Spark session
spark = SparkSession.builder \
    .appName(appName) \
    .master(master) \
    .getOrCreate()

schema = StructType([
    StructField("positionmessage",
    StructType([
    StructField('callsign', StringType(), True),
    StructField('name', StringType(), True),
    StructField('timestamplast', TimestampType(), True),    
    StructField('mmsi', IntegerType(), True)
    ]))])

file_name = "data.json"
df = spark.read.json(file_name).select("positionmessage.*")

Sekarang saya ingin menghapus sengatan "testschip-" dari "nama". saya lakukan sebagai berikut:

import pyspark.sql.functions as follows:
df = df.withColumn("name", f.split(df['name'], '\-')[1]).show() # strips the string "testschip-"

Bagaimana saya sekarang membuat "nama" bilangan bulat?

0
Jeroen 20 Mei 2020, 19:14

1 menjawab

Jawaban Terbaik

Transmisikan saja ke int.

from pyspark.sql.functions import *
from pyspark.sql.types import *
df = df.withColumn("name", f.split(df['name'], '\-')[1].cast("int")).show()
1
Shu 20 Mei 2020, 16:23