serializeToBundle object issue

[drkmd8]

If running the code as above, there's an issue with featurePipeline.serializeToBundle("jar:file:/tmp/pyspark.example.zip"). AttributeError: 'Pipeline' object has no attribute 'serializeToBundle'.

If use the following code:
featurePipeline2 = featurePipeline.fit(df2) featurePipeline2.serializeToBundle("jar:file:/tmp/pyspark.example.zip")
There is an error with self._java_obj = _jvm().ml.combust.mleap.spark.SimpleSparkSerializer(), saying "TypeError: 'JavaPackage' object is not callable"

How to solve it?

[hollinwilkins]

@drkmd8 Can you give us version information for both the Python and MLeap JVM packages as well as Spark that you are using?

[drkmd8]

I used Python 3.6.1, Mleap 0.8.1 (pip install mleap) and pyspark 2.1.1+hadoop2.7. This problem was caused by python mleap library I think, because scala seems to work fine but python requires running with external jar files where mleap classes are included.

[alexkayal]

I have the same issue at the moment as well.

[tianhongjie]

Yes, I have the same issue, my solution is add the jar file to the pyspark jars dir which at the python package path:site-packages/pyspark/jars/ .
I add some jars such as below:
mleap-base_2.11-0.10.0.jar
mleap-core_2.11-0.10.0.jar
mleap-runtime_2.11-0.10.0.jar
mleap-spark_2.11-0.10.0.jar
mleap-spark-base_2.11-0.10.0.jar
mleap-tensor_2.11-0.10.0.jar

I hope it is helpful for you.

[alexkayal]

I also solved it by adding the jars manually to /usr/lib/spark/jars.
But I guess there is a better way to do it: just sudo pip install jip
Then install MLeap. Jip is supposed to take care of your Java dependencies if i understand correctly.

[Khiem-Tran]

Hi @alexkayal & @tianhongjie, I have tried your solution, it fixed the JavaPackage issue but then I got another one.

Py4JJavaError: An error occurred while calling o261.serializeToBundle.
: java.lang.NoClassDefFoundError: com/trueaccord/scalapb/GeneratedEnum

I am not sure how it can happen, since my dataframe only has primitive types {int, double}. Do you have any ideas on it?

[samant2008]

@alexkayal @tianhongjie @Khiem-Tran ,
I am also getting the same error. Any idea how to resolve this issue?`

Py4JJavaError: **An error occurred while calling o414.serializeToBundle. : java.lang.NoClassDefFoundError: com/trueaccord/scalapb/GeneratedEnum** at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:763) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142) at java.net.URLClassLoader.defineClass(URLClassLoader.java:467) at java.net.URLClassLoader.access$100(URLClassLoader.java:73) at java.net.URLClassLoader$1.run(URLClassLoader.java:368) at java.net.URLClassLoader$1.run(URLClassLoader.java:362) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:361) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) at ml.combust.mleap.spark.SimpleSparkSerializer.serializeToBundle(SimpleSparkSerializer.scala:17) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:244) at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:357) at py4j.Gateway.invoke(Gateway.java:280) at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:132) at py4j.commands.CallCommand.execute(CallCommand.java:79) at py4j.GatewayConnection.run(GatewayConnection.java:214) at java.lang.Thread.run(Thread.java:748) Caused by: java.lang.ClassNotFoundException: com.trueaccord.scalapb.GeneratedEnum at java.net.URLClassLoader.findClass(URLClassLoader.java:381) at java.lang.ClassLoader.loadClass(ClassLoader.java:424) at java.lang.ClassLoader.loadClass(ClassLoader.java:357) ... 23 more

[elgalu]

I fixed the com/trueaccord related errors by adding lenses_2.11-0.4.12.jar

[samant2008]

@elgalu ,

Thank you so much for your prompt response.
I have included the lenses_2.11-0.4.12.jar but still I m getting the same error as above. Do you have any other suggestion to resolve this issue?

[elgalu]

Make sure is in the CLASSPATH, note Py4J has its own jars/ folder. And if you install pyspark separately it also comes with its own jars/ folder. What I do is remove all those jars directories and symlink to 1 /jars where a put together the whole set of working versions.

You can find all my working jars at:
https://github.com/elgalu/jupyter-spark-117/tree/master/spark/jars

Pending: to build an sbt or pom.xml project (instead of a bunch of jars)

[siyouhe666]

@elgalu ,

Thank you so much for your prompt response.
I have included the lenses_2.11-0.4.12.jar but still I m getting the same error as above. Do you have any other suggestion to resolve this issue?

I am same to you, have you find an answer?

[Khiem-Tran]

@elgalu @siyouhe666, I have been using spark 2.2.1 and the config --packages ml.combust.mleap:mleap-spark_2.11:0.11.0. It seems work for me

Btw, I also follow this (blog)[https://medium.com/@bogdan.cojocar/pyspark-and-xgboost-integration-tested-on-the-kaggle-titanic-dataset-4e75a568bdb] to fix xgboost dependency because somehow my mleap-xgboost does not work properly

[siyouhe666]

@elgalu @siyouhe666, I have been using spark 2.2.1 and the config --packages ml.combust.mleap:mleap-spark_2.11:0.11.0. It seems work for me

Btw, I also follow this (blog)[https://medium.com/@bogdan.cojocar/pyspark-and-xgboost-integration-tested-on-the-kaggle-titanic-dataset-4e75a568bdb] to fix xgboost dependency because somehow my mleap-xgboost does not work properly

Thanks I have solved this problem through changed my spark version to 2.4.0,
Btw, alough the official docs of mleap said they hadn't support for 2.4.0, but i found it works well

[yairdata]

i am using python 3.6 , pyspark 2.3.1.
used mleap-core_2.11-0.11.0.jar
mleap-spark-base_2.11-0.13.0.jar
mleap-runtime_2.11-0.13.0.jar
tried 2 approaches to have pyspark know mleap:
git clone and use:
sys.path.append('C:\my-mleap\mleap-master\python')
and also by:
os.environ['PYSPARK_SUBMIT_ARGS'] = '--jars ....'
or to pip install mleap (installs version 0.8.1)

when calling
model.serializeToBundle(model_file_path, sparkTransformed)
i get:
Py4JError: ml.combust.mleap.spark.SimpleSparkSerializer does not exist in the JVM

[SoloBean]

@yairdata I am same to you, have you find an answer? Thanks a lot.