Replies: 4
After upgrading to HDP2.1, I’m now seeing the hiveserver2 thread count continually increase over time as we use it (a periodic cron job that is loading data; it’s running & then exiting.)
We ended up hitting the default(?) centos6.5 limit of 1024 processes/user for hive, at which point hive started throwing:
java.lang.RuntimeException: java.lang.OutOfMemoryError: unable to create new native thread
at org.apache.hive.service.cli.session.HiveSessionProxy.invoke(HiveSessionProxy.java:84)
at org.apache.hive.service.cli.session.HiveSessionProxy.access$000(HiveSessionProxy.java:37)
at org.apache.hive.service.cli.session.HiveSessionProxy$1.run(HiveSessionProxy.java:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
The same cron process was running the same way against the HDP2.0 cluster, and the thread count in hiveserver2 never budged. (We’ve got graphs!)
This appears to be a HDP2.1 regression. Anyone have any tips on troubleshooting (what are these threads doing?)?