What am I working on now? NBP and in car video with Track Addict

I have 2 new repos on Github for attaching a 2009 CTS-V (Or pretty much any GM from that era) to track addict. Lets you log brake application, TCS/ABS/VDCS and the like.

https://github.com/tmkdev/canlogger
https://github.com/tmkdev/pynbp

I'll post video when I actually get it working 100%. I see HP Tuners updated the android app with fixes :)

GY-BNO055 (DX/Banggood/Aliexpress) conversion to serial

The GY-BNO055 sensor board from China ships jumpered for i2c on 0x29. And that's great unless you have a raspberry pi and i2c clock stretching is causing you all kinds of problems.

With a little bit of desoldering wick and flux you can desolder one of the config jumpers on the board and convert the module to UART serial.

Good luck and happy hacking.

Meatballs in guajillo sauce

Meatballs in guajillo sauce - Just make it. It's great!

Gig E problems...

Time for managed switches at home. I currently have 3 unmanaged GigE TP-S108s. One of the agg switches wasn't connecting to the main distribution switch at GigE and that switch has my NAS on it. Wifi->NAS wasn't working quite right and maxing out at 11MB/s.. Turns out one of the pairs came loose on the punch down block and once I re-punched it down, 30-50MB/s copies on WiFi. Magic.

Time for switches that can tell me what's going on. Dell 2716s on Ebay seem to be a good deal. Here goes.

LSTM Experiment. Oh and cars..

So. What happens when you take a LSTM neural network and teach it about car naming?

You get some interesting results! (These were chosen by me - in decending training epochs) Machine learning is going to be fun!

I mean, who doesn't want to drive a 2012 Bentley Classis? Or a 2007 Toyota Carooe?

1987 wawasaki cum-fk100
2004 honda nc700/shodow ace prick
1997 ford crancoer
1987 buick tesperspe
2007 ponperp w7
2004 dodge disent
2004 freightliner fland
1997 kia gonartoutin
2012 kia fourirx
2004 chevrolet conticon
1987 volkswagen book fry
2012 ferrari 0080 series v6
1993 mitsubishi calater
1997 dodge rad series montegai
1993 mitichipiizan 6i
1993 bmw r125 glancer sport
1997 chevrolet coupscorn
2018 vinysh sherd
1993 cadillac arravieno
2012 bentley classis
2007 honda efinig conventional bus chassis
2018 toyota carooe
2018 jeep rango
2004 cadillac eevolo
2000 nissan suberard

Spark2 Can't create directory errors

18/03/15 11:00:27 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1521132642246_0008_01_000007 on host: host123. Exit status: -1000. Diagnostics: Application application_1
521132642246_0008 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is reporting
main : requested yarn user is reporting
Can't create directory /cdh/0/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/1/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/10/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/11/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/12/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/13/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/14/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/15/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/16/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/17/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/18/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/19/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/2/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/20/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/21/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/22/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/23/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/3/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/4/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/5/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/6/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/7/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/8/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/9/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Did not create any app directories


18/03/15 11:00:27 WARN cluster.YarnSchedulerBackend$YarnSchedulerEndpoint: Container marked as failed: container_1521132642246_0008_01_000009 on host: host123. Exit status: -1000. Diagnostics: Application application_1
521132642246_0008 initialization failed (exitCode=255) with output: main : command provided 0
main : run as user is reporting
main : requested yarn user is reporting
Can't create directory /cdh/0/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/1/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/10/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/11/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/12/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/13/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/14/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/15/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/16/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied
Can't create directory /cdh/17/yarn/nm/usercache/reporting/appcache/application_1521132642246_0008 - Permission denied


I have seen these types of Spark2 errors since Kerberizing my cluster. Usually not service impacting, Spark2 seemed to just retry on a different node. If the job was big - the odds of a fatal failure increased, probably hits a too many failed container limit or something. (Just guessing there)

Found out that Kerberizing with these directories created pre-kerberization could cause that issue. I have since deleted en mass and have not seen anything like this since.

rm -rf [your_mount_points]/yarn/nm/usercache/reporting/appcache/* on all nodes.

Hope that helps somebody!

Weekend cooking

First attempt at squid ink pasta. Yum!

Things that make you go WTF?

[reporting utils]$ date --date='-1 month'
Sun Dec 31 07:08:07 PST 2017

[reporting utils]$ date --date='1 month'
Sat Mar  3 07:08:15 PST 2018

[reporting utils]$ date
Wed Jan 31 07:10:07 PST 2018

Cloudera Impala query fails on Java UDFs - PermGen Space

UDF WARNING: Hive UDF path=hdfs://nameservice1/libs/esri-gis.jar class=com.esri.hadoop.hive.ST_GeomFromText failed due to: ImpalaRuntimeException: UDF::evaluate() ran into a problem.

CAUSED BY: ImpalaRuntimeException: UDF failed to evaluate

CAUSED BY: InvocationTargetException: null

CAUSED BY: OutOfMemoryError: PermGen space

UDF WARNING: Hive UDF path=hdfs://nameservice1/libs/esri-gis.jar class=com.esri.hadoop.hive.ST_GeodesicLengthWGS84 failed due to: ImpalaRuntimeException: UDF::evaluate() ran into a problem.

CAUSED BY: ImpalaRuntimeException: UDF failed to evaluate

CAUSED BY: InvocationTargetException: null

CAUSED BY: NumberFormatException: For input string: "?"

Couldn't open transport for reshdp-xxx:22000 (connect() failed: Connection refused)

Couldn't open transport for reshdp-xxx:22000 (connect() failed: Connection refused)

Couldn't open transport for reshdp-xxx:22000 (connect() failed: Connection refused)

Couldn't open transport for reshdp-xxx:22000 (connect() failed: Connection refused)

Couldn't open transport for reshdp-xxx:22000 (connect() failed: Connection refused)

UDF WARNING: Hive UDF path=hdfs://nameservice1/libs/esri-gis.jar class=com.esri.hadoop.hive.ST_GeomFromText failed due to: ImpalaRuntimeException: UDF::evaluate() ran into a problem.

CAUSED BY: ImpalaRuntimeException: UDF failed to evaluate

CAUSED BY: InvocationTargetException: null

CAUSED BY: OutOfMemoryError: PermGen space

Started getting this error after kerberizing my cluster. I suspect that Java was loading kerberos libraries and running out of space for the UDFs when it came time to fix. 

Solution: Add memory to the JAVA_TOOL_OPTION for impala:

Daemon Safety Valve Config

JAVA_TOOL_OPTIONS=-XX:MaxPermSize=512m

Restart impala. Should fix it.