Replies: 1
We have a a linux dev cluster for HDP2.2, which comes with hive0.14. We installed spark 1.2 on one of the nodes by following the instructions on the HDP2.2 Spark Technical Preview (http://hortonworks.com/hadoop-tutorial/using-apache-spark-hdp/ ) and using the tarball provided on that page.
Most of the examples worked except the ones related to Hive. The hive-site.xml for the cluster has been put into spark’s /conf folder and hive.metastore.uris has been set in that hive-site.xml.
When we tried the example under the section of “Running Hive 0.13.1 UDF” and execute the table creation statement :
hiveContext.hql(“CREATE TABLE IF NOT EXISTS TestTable (key INT, value STRING)”
we got the following exception, indicating database “default” is not found. but we do have the default database in hive. Do we miss any configurations needed to make it work (although we did follow the instructions on that webpage for HDP2.2 Spark Technical Preview? Or is it buz the instructions are only for the sandbox which might not work for the hive version that come with the actual linux HDP2.2 distribution? How can we make it work?
HIVE FAILURE OUTPUT
======================
FAILED: SemanticException [Error 10072]: Database does not exist: default
======================
END HIVE FAILURE OUTPUT
======================
org.apache.spark.sql.execution.QueryExecutionException: FAILED: SemanticException [Error 10072]: Database does not exist: default
at org.apache.spark.sql.hive.HiveContext.runHive(HiveContext.scala:309)
at org.apache.spark.sql.hive.HiveContext.runSqlHive(HiveContext.scala:276)
at org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult$lzycompute(NativeCommand.scala:35)
at org.apache.spark.sql.hive.execution.NativeCommand.sideEffectResult(NativeCommand.scala:35)
at org.apache.spark.sql.execution.Command$class.execute(commands.scala:46)
at org.apache.spark.sql.hive.execution.NativeCommand.execute(NativeCommand.scala:30)
at org.apache.spark.sql.SQLContext$QueryExecution.toRdd$lzycompute(SQLContext.scala:425)
at org.apache.spark.sql.SQLContext$QueryExecution.toRdd(SQLContext.scala:425)
at org.apache.spark.sql.SchemaRDDLike$class.$init$(SchemaRDDLike.scala:58)
at org.apache.spark.sql.SchemaRDD.<init>(SchemaRDD.scala:108)
at org.apache.spark.sql.hive.HiveContext.hiveql(HiveContext.scala:102)
at org.apache.spark.sql.hive.HiveContext.hql(HiveContext.scala:106)
at $iwC$$iwC$$iwC$$iwC.<init>(<console>:17)
at $iwC$$iwC$$iwC.<init>(<console>:22)
at $iwC$$iwC.<init>(<console>:24)
at $iwC.<init>(<console>:26)
at <init>(<console>:28)
at .<init>(<console>:32)
at .<clinit>(<console>)
at .<init>(<console>:7)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method