LSTM Experiment. Oh and cars..

So. What happens when you take a LSTM neural network and teach it about car naming?

You get some interesting results! (These were chosen by me - in decending training epochs) Machine learning is going to be fun!

I mean, who doesn't want to drive a 2012 Bentley Classis? Or a 2007 Toyota Carooe?

1987 wawasaki cum-fk100
2004 honda nc700/shodow ace prick
1997 ford crancoer
1987 buick tesperspe
2007 ponperp w7
2004 dodge disent
2004 freightliner fland
1997 kia gonartoutin
2012 kia fourirx
2004 chevrolet conticon
1987 volkswagen book fry
2012 ferrari 0080 series v6
1993 mitsubishi calater
1997 dodge rad series montegai
1993 mitichipiizan 6i
1993 bmw r125 glancer sport
1997 chevrolet coupscorn
2018 vinysh sherd
1993 cadillac arravieno
2012 bentley classis
2007 honda efinig conventional bus chassis
2018 toyota carooe
2018 jeep rango
2004 cadillac eevolo
2000 nissan suberard

Spark2 Can't create directory errors

18/03/15 11:00:27 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1521132642246_0008_01_000007 on host: host123. Exit status: -1000. Diagnostics: Application application_1
521132642246_0008 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is reporting
main : requested yarn user is reporting
Can't create directory /cdh/0/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/1/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/10/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/11/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/12/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/13/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/14/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/15/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/16/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/17/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/18/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/19/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/2/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/20/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/21/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/22/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/23/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/3/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/4/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/5/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/6/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/7/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/8/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/9/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Did not create any app directories


18/03/15 11:00:27 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1521132642246_0008_01_000009 on host: host123. Exit status: -1000. Diagnostics: Application application_1
521132642246_0008 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is reporting
main : requested yarn user is reporting
Can't create directory /cdh/0/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/1/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/10/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/11/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/12/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/13/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/14/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/15/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/16/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/17/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied


I have seen these types of Spark2 errors since Kerberizing my cluster. Usually not service impacting, Spark2 seemed to just retry on a different node. If the job was big - the odds of a fatal failure increased, probably hits a too many failed container limit or something. (Just guessing there)

Found out that Kerberizing with these directories created pre-kerberization could cause that issue. I have since deleted en mass and have not seen anything like this since.

rm -rf [your_mount_points]/yarn/nm/usercache/reporting/appcache/* on all nodes.

Hope that helps somebody!