Saya melatih dan menyimpan model XGBoost seperti yang ditunjukkan di bawah ini:

XGBoost Versi 0.82

Spark Versi 2.4.2

Dapatkan Model (memanggil fungsi kereta)

def getModel(trainingData: DataFrame): PipelineModel = {
    val pipelineModel = train(trainingData)

    if (modelPathToSave != "") {
      pipelineModel.write.overwrite().save(modelPathToSave)
      println(f"Saved model to $modelPathToSave")
    }
    pipelineModel
    }

Model Kereta

def train(trainingData: DataFrame): PipelineModel = {
    val nh = new NullHandler()
      .setCols(hackyEncode(featureList))
      .setMethod("fill")

    val va = new VectorAssembler()
      .setInputCols(hackyDecode(nh.getCols).toArray)
      .setOutputCol(featuresCol)

    val xgb = new XGBoostClassifier()
      .setLabelCol("label")
      .setFeaturesCol("features")
      .setEta(0.3)
      .setMaxDepth(8)
      .setObjective("binary:logistic")
      .setEvalMetric("auc")  
      .setScalePosWeight(9)

    val pipeline = new Pipeline()
        .setStages(Array[PipelineStage](nh, va, xgb))

    pipeline.fit(trainingData)
}

Namun saya mendapatkan kesalahan ini:

Exception in thread "main" java.lang.NoSuchMethodError: shaded.json4s.jackson.JsonMethods$.parse(Lshaded/json4s/JsonInput;Z)Lshaded/json4s/JsonAST$JValue;
    at ml.dmlc.xgboost4j.scala.spark.params.DefaultXGBoostParamsWriter$$anonfun$1$$anonfun$3.apply(DefaultXGBoostParamsWriter.scala:73)
    at ml.dmlc.xgboost4j.scala.spark.params.DefaultXGBoostParamsWriter$$anonfun$1$$anonfun$3.apply(DefaultXGBoostParamsWriter.scala:71)

Meskipun memiliki json4s di file build.sbt saya.

  "org.json4s" %% "json4s-native" % "3.5.1",
  "org.json4s" %% "json4s-jackson" % "3.6.6", 

Ada yang bisa bantu tolong?

1
jxn 20 September 2019, 06:57

1 menjawab

Jawaban Terbaik

Xgboost versi 0.82 tidak kompatibel dengan Spark 2.4. Anda dapat menurunkan versi ke Spark 2.3 atau menggunakan Xgboost versi 0.90.

Referensi:

https://discuss.xgboost.ai/t/xgboost-0-8-2-and-spark-2-4-0-unable-to-save-pipeline-model-into-aws-s3/838

1
zhanwei lim 20 September 2019, 08:16