Wednesday, January 31, 2018

Things that make you go WTF?

[reporting utils]$ date --date='-1 month'
Sun Dec 31 07:08:07 PST 2017

[reporting utils]$ date --date='1 month'
Sat Mar  3 07:08:15 PST 2018

[reporting utils]$ date
Wed Jan 31 07:10:07 PST 2018

Tuesday, January 30, 2018

Cloudera Impala query fails on Java UDFs - PermGen Space

UDF WARNING: Hive UDF path=hdfs://nameservice1/libs/esri-gis.jar class=com.esri.hadoop.hive.ST_GeomFromText failed due to: ImpalaRuntimeException: UDF::evaluate() ran into a problem.
CAUSED BY: ImpalaRuntimeException: UDF failed to evaluate
CAUSED BY: InvocationTargetException: null
CAUSED BY: OutOfMemoryError: PermGen space

UDF WARNING: Hive UDF path=hdfs://nameservice1/libs/esri-gis.jar class=com.esri.hadoop.hive.ST_GeodesicLengthWGS84 failed due to: ImpalaRuntimeException: UDF::evaluate() ran into a problem.
CAUSED BY: ImpalaRuntimeException: UDF failed to evaluate
CAUSED BY: InvocationTargetException: null

CAUSED BY: NumberFormatException: For input string: "?"

Couldn't open transport for reshdp-xxx:22000 (connect() failed: Connection refused)
Couldn't open transport for reshdp-xxx:22000 (connect() failed: Connection refused)
Couldn't open transport for reshdp-xxx:22000 (connect() failed: Connection refused)
Couldn't open transport for reshdp-xxx:22000 (connect() failed: Connection refused)
Couldn't open transport for reshdp-xxx:22000 (connect() failed: Connection refused)
UDF WARNING: Hive UDF path=hdfs://nameservice1/libs/esri-gis.jar class=com.esri.hadoop.hive.ST_GeomFromText failed due to: ImpalaRuntimeException: UDF::evaluate() ran into a problem.
CAUSED BY: ImpalaRuntimeException: UDF failed to evaluate
CAUSED BY: InvocationTargetException: null
CAUSED BY: OutOfMemoryError: PermGen space

Started getting this error after kerberizing my cluster. I suspect that Java was loading kerberos libraries and running out of space for the UDFs when it came time to fix. 

Solution: Add memory to the JAVA_TOOL_OPTION for impala:


Daemon Safety Valve Config
JAVA_TOOL_OPTIONS=-XX:MaxPermSize=512m

Restart impala. Should fix it.