Sparklint UI does not come up

[sparkuser3]

Hi ,

Tried the instruction provided in https://github.com/groupon/sparklint . When I do a spark submit I do see the logs which says the below . But when I hit the driver IP with port I cannot see the UI. Could you please help me . Are there any additional configs to be applied ?

Logs

17/08/17 14:30:16 INFO nio1.NIO1SocketServerGroup: Service bound to address /0:0:0:0:0:0:0:0:23763
17/08/17 14:30:16 INFO sparklint.Sparklint: Server started on 23763
17/08/17 14:30:16 INFO spark.SparkContext: Registered listener com.groupon.sparklint.SparklintListener

[NileshGule]

I assume you are running the Live mode (run inside spark driver node). The UI is available only when the Spark program is running. If your program finishes execution by the time you access the URL in browser you would not be able to see anything.

You can refer to an example in my repo https://github.com/NileshGule/learning-spark

I was able to see the SparkLint UI by executing the following command

spark-submit --packages com.databricks:spark-csv_2.10:1.5.0,com.groupon.sparklint:sparklint-spark163_2.10:1.0.8 \
--conf spark.logConf=true \
--conf spark.extraListeners=com.groupon.sparklint.SparklintListener \
--class com.nileshgule.movielens.RatingsCsvReader \
--master local \
--deploy-mode client \
--executor-memory 4g \
--name RatingsCsvReader \
--conf "spark.app.id=RatingsCsvReader" \
target/learning-spark-1.0.jar ml-latest/ratings.csv

Here is a screenshot of the results shown in UI

Hope this helps.

[neeraj20gupta]

I have a spark streaming job with 10 sec micro batch. My sparkling UI does not show up. Logs show that connection is accepted still no response. Do I have to put additional config for streaming job?

[Amittyagi007]

Hi,

I am also facing the same issue as sparkuser3. The SparkLint UI does not comes up even when the job is running, but I can see the below message in the logs. Am I missing any configuration here?

INFO nio1.NIO1SocketServerGroup: Service bound to address /0:0:0:0:0:0:0:0:23763
INFO sparklint.Sparklint: Server started on 23763
INFO spark.SparkContext: Registered listener com.groupon.sparklint.SparklintListener

Thanks
Amit

[Maxim341]

What platform are you running the application on?

[Amittyagi007]

Thanks Max for getting back. I am using it on a cloudera hadoop platform (cdh 15.5).

Regards
Amit

[Maxim341]

Hi! We used it on a hortonworks hadoop platform and to view the UI we had to port forward the required port. This could be a similar issue you are facing

[Amittyagi007]

Hi Max,

This is on a single server, so I am not sure port forwarding is required here. Did you make any other configuration changes or anything in setup?
I did try by enabling tcp6 and below was the o/p of netstat. Still no UI.

tcp6 0 0 :::23763 :::* LISTEN 61356/java
tcp6 583 0 xx.xx.xxx.xxx:23763 xx.xx.xxx.xxx::51283 ESTABLISHED 61356/java