There is always a need for us to specify the queue for our MR or hive task. Here's the way:
An example for MapReduce task:
For Hive Task, inserting the following code before invoking the real HQL task:
To generalize it, we can safely conclude that most of Hadoop or Hive configurations can be set in the upper forms respectively. What the 'most' means here is that some configurations cannot be revised during runtime, or being stated as 'final'.
© 2014-2017 jason4zhu.blogspot.com All Rights Reserved
If transfering, please annotate the origin: Jason4Zhu
An example for MapReduce task:
hadoop jar $HADOOP_HOME/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi -Dmapred.job.queue.name=root.example_queue 10 10000
For Hive Task, inserting the following code before invoking the real HQL task:
set mapred.job.queue.name=root.example_queue;
To generalize it, we can safely conclude that most of Hadoop or Hive configurations can be set in the upper forms respectively. What the 'most' means here is that some configurations cannot be revised during runtime, or being stated as 'final'.
© 2014-2017 jason4zhu.blogspot.com All Rights Reserved
If transfering, please annotate the origin: Jason4Zhu
When I set the queue : set mapred.job.queue.name=long_running; and run the job. Its throwing the following error. Do you have an idea? It's not allowing me to set the queue.
ReplyDeleteFailed to run job : Application application_1465873717267_0011 submitted by user root to unknown queue: long_running
I see everyone's comments very interesting. Each person has an opinion and the opinion of their own. I will also be able to read their comments yet. Hopefully everyone will accept it.
ReplyDeleteBig data training in chennai
Check whether you have queue named long_running defined? If your hive session is running in TEZ .. use set tez.queue.name=
ReplyDeletegr8
DeleteAmazing & Great informative blog,it gives very useful practical information to developer like me. Besides that Wisen has established as Best Hibernate Training in Chennai . or learn thru Online Training mode Hibernate Online Training
ReplyDeleteNice information on QUEUE keep updating the post
ReplyDeleteBig Data Hadoop Online Training
Very nice blog,keep sharing more posts with us.
ReplyDeleteThank you for info.
big data online training