Replies: 0
Hi All,
I’m trying to submit a mapreduce job from a java client running on Windows7 to the Hortonworks Sandbox.
Driver code:
public static void main(String[] args) throws Exception
{
UserGroupInformation ugi = UserGroupInformation.createRemoteUser(“jackie_leslie”);
ugi.doAs(new PrivilegedExceptionAction<Object>() {
String[] jobArgs;
@Override
public Object run() throws Exception {
JobWrapper mr = new JobWrapper();
int exitCode = ToolRunner.run(mr, jobArgs);
System.exit(exitCode);
return mr;
}
private PrivilegedExceptionAction init(String[] myArgs)
{
this.jobArgs = myArgs;
return this;
}
}.init(args)
);
}
Mapper class
@Override
public int run(String[] args) throws Exception
{
Configuration config = getConf();
config.set(“fs.defaultFS”, “hdfs://sandbox.hortonworks.com:8020″);
config.set(“mapreduce.framework.name”, “yarn”);
config.set(“yarn.resourcemanager.address”, “localhost:8050″); //8025 or 8032?
config.set(“hadoop.job.ugi”, “jackie_leslie”);
@SuppressWarnings(“deprecation”)
Job job = new Job(config);
job.setJobName(“upload”);
job.setJarByClass(MyKVMapper.class);
job.setMapperClass(MyKVMapper.class);
job.setMapOutputKeyClass(ImmutableBytesWritable.class);
job.setInputFormatClass(TextInputFormat.class);
FileInputFormat.addInputPath(job, new Path(args[0]));
FileOutputFormat.setOutputPath(job, new Path(args[1]));
boolean success = job.waitForCompletion(true);
client stacktrace
Exception in thread “main” org.apache.hadoop.ipc.RemoteException(java.io.IOException): File /tmp/hadoop-yarn/staging/jackie_leslie/.staging/job_1397842310782_0004/job.split could only be replicated to 0 nodes instead of minReplication (=1). There are 1 datanode(s) running and 1 node(s) are excluded in this operation.
at org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget(BlockManager.java:1384)
at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2503)
at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:555)
at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:387)
at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:59582) etc.
Also from the logs.
2014-04-18 13:59:25,603 WARN security.UserGroupInformation (UserGroupInformation.java:getGroupNames(1355)) – No groups available for user jackie_leslie
Why is this job not being submitted to my data node?
Thanks.
