- 論壇徽章:
- 0
|
<DIV><FONT size=3 face=宋體> 最近學(xué)習(xí)Hadoop，可以使用命令行來簡單的管理Hadoop，例如使用：</FONT></DIV>
<DIV><FONT size=3 face=宋體>
<DIV style="WIDTH: 92.94%; HEIGHT: 122px" id=codeText class=codeText>
<OL style="PADDING-BOTTOM: 5px; MARGIN: 0px 1px 0px 0px; PADDING-LEFT: 0px; PADDING-RIGHT: 0px; PADDING-TOP: 5px" class=dp-css>
<LI>#查看hadoop的HDFS根節(jié)點(diǎn)情況</LI>
<LI>hadoop fs -ls /</LI>
<LI></LI>
<LI>#創(chuàng)建文件夾LGG</LI>
<LI>hadoop fs mkdir /user/root/LGG</LI></OL></DIV></FONT></DIV>
<DIV><FONT size=3 face=宋體>通過命令行接口來進(jìn)行操作。</FONT></DIV>
<DIV><FONT size=3 face=宋體> 當(dāng)然，hadoop命令行接口同樣支持自己編寫的類的參數(shù)，例如在Hadoop權(quán)威指南中的一個(gè)例子如下：</FONT></DIV>
<DIV><FONT size=3 face=宋體>
<DIV id=codeText class=codeText>
<OL style="PADDING-BOTTOM: 5px; MARGIN: 0px 1px 0px 0px; PADDING-LEFT: 0px; PADDING-RIGHT: 0px; PADDING-TOP: 5px" class=dp-css>
<LI><SPAN style="COLOR: #000000"><SPAN style="COLOR: #0000ff">import</SPAN> <SPAN style="COLOR: #ff0000">java</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">io</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">InputStream</SPAN><SPAN style="COLOR: #0000cc">;</SPAN><BR></LI>
<LI><SPAN style="COLOR: #0000ff">import</SPAN> <SPAN style="COLOR: #ff0000">java</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">net</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">URL</SPAN><SPAN style="COLOR: #0000cc">;</SPAN><BR></LI>
<LI><SPAN style="COLOR: #0000ff">import</SPAN> <SPAN style="COLOR: #ff0000">org</SPAN><SPAN style="COLOR: #0000cc">.</SPAN>apache<SPAN style="COLOR: #0000cc">.</SPAN>hadoop<SPAN style="COLOR: #0000cc">.</SPAN>fs<SPAN style="COLOR: #0000cc">.</SPAN>FsUrlStreamHandlerFactory<SPAN style="COLOR: #0000cc">;</SPAN><BR></LI>
<LI><SPAN style="COLOR: #0000ff">import</SPAN> <SPAN style="COLOR: #ff0000">org</SPAN><SPAN style="COLOR: #0000cc">.</SPAN>apache<SPAN style="COLOR: #0000cc">.</SPAN>hadoop<SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">io</SPAN><SPAN style="COLOR: #0000cc">.</SPAN>IOUtils<SPAN style="COLOR: #0000cc">;</SPAN><BR></LI>
<LI><BR></LI>
<LI><SPAN style="COLOR: #ff9900">/**<BR></LI>
<LI> * <BR></LI>
<LI> * @author LGG<BR></LI>
<LI> *<BR></LI>
<LI> */</SPAN><BR></LI>
<LI><SPAN style="COLOR: #0000ff">public</SPAN> <SPAN style="COLOR: #0000ff">class</SPAN> URLCat <SPAN style="COLOR: #0000cc">{</SPAN><BR></LI>
<LI> <SPAN style="COLOR: #0000ff">static</SPAN><SPAN style="COLOR: #0000cc">{</SPAN><BR></LI>
<LI> <SPAN style="COLOR: #ff0000">URL</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">setURLStreamHandlerFactory</SPAN><SPAN style="COLOR: #0000cc">(</SPAN><SPAN style="COLOR: #0000ff">new</SPAN> FsUrlStreamHandlerFactory<SPAN style="COLOR: #0000cc">(</SPAN><SPAN style="COLOR: #0000cc">)</SPAN><SPAN style="COLOR: #0000cc">)</SPAN><SPAN style="COLOR: #0000cc">;</SPAN><BR></LI>
<LI> <SPAN style="COLOR: #0000cc">}</SPAN><BR></LI>
<LI> <SPAN style="COLOR: #0000ff">public</SPAN> <SPAN style="COLOR: #0000ff">static</SPAN> <SPAN style="COLOR: #0000ff">void</SPAN> main<SPAN style="COLOR: #0000cc">(</SPAN><SPAN style="COLOR: #ff0000">String</SPAN><SPAN style="COLOR: #0000cc">[</SPAN><SPAN style="COLOR: #0000cc">]</SPAN> args<SPAN style="COLOR: #0000cc">)</SPAN> <SPAN style="COLOR: #0000ff">throws</SPAN> <SPAN style="COLOR: #ff0000">Exception</SPAN><SPAN style="COLOR: #0000cc">{</SPAN><BR></LI>
<LI> <SPAN style="COLOR: #ff0000">InputStream</SPAN> <SPAN style="COLOR: #ff0000">in</SPAN> <SPAN style="COLOR: #0000cc">=</SPAN> <SPAN style="COLOR: #0000ff">null</SPAN><SPAN style="COLOR: #0000cc">;</SPAN><BR></LI>
<LI> <SPAN style="COLOR: #0000ff">try</SPAN><SPAN style="COLOR: #0000cc">{</SPAN><BR></LI>
<LI> <SPAN style="COLOR: #ff0000">in</SPAN> <SPAN style="COLOR: #0000cc">=</SPAN> <SPAN style="COLOR: #0000ff">new</SPAN> <SPAN style="COLOR: #ff0000">URL</SPAN><SPAN style="COLOR: #0000cc">(</SPAN>args<SPAN style="COLOR: #0000cc">[</SPAN>0<SPAN style="COLOR: #0000cc">]</SPAN><SPAN style="COLOR: #0000cc">)</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">openStream</SPAN><SPAN style="COLOR: #0000cc">(</SPAN><SPAN style="COLOR: #0000cc">)</SPAN><SPAN style="COLOR: #0000cc">;</SPAN><BR></LI>
<LI> IOUtils<SPAN style="COLOR: #0000cc">.</SPAN>copyBytes<SPAN style="COLOR: #0000cc">(</SPAN><SPAN style="COLOR: #ff0000">in</SPAN><SPAN style="COLOR: #0000cc">,</SPAN> <SPAN style="COLOR: #ff0000">System</SPAN><SPAN style="COLOR: #0000cc">.</SPAN>out<SPAN style="COLOR: #0000cc">,</SPAN> 4096<SPAN style="COLOR: #0000cc">,</SPAN>false<SPAN style="COLOR: #0000cc">)</SPAN><SPAN style="COLOR: #0000cc">;</SPAN><BR></LI>
<LI> <SPAN style="COLOR: #0000cc">}</SPAN><SPAN style="COLOR: #0000ff">finally</SPAN><SPAN style="COLOR: #0000cc">{</SPAN><BR></LI>
<LI> IOUtils<SPAN style="COLOR: #0000cc">.</SPAN>closeStream<SPAN style="COLOR: #0000cc">(</SPAN><SPAN style="COLOR: #ff0000">in</SPAN><SPAN style="COLOR: #0000cc">)</SPAN><SPAN style="COLOR: #0000cc">;</SPAN><BR></LI>
<LI> <SPAN style="COLOR: #0000cc">}</SPAN><BR></LI>
<LI> <SPAN style="COLOR: #0000cc">}</SPAN><BR></LI>
<LI><SPAN style="COLOR: #0000cc">}</SPAN></SPAN></LI></OL></DIV></FONT></DIV>
<DIV><FONT size=3 face=宋體> 將URLCat.java編譯后生成class文件，然后可以將此類作為一個(gè)hadoop命令行接口的參數(shù)進(jìn)行使用，如下：
<DIV id=codeText class=codeText>
<OL style="PADDING-BOTTOM: 5px; MARGIN: 0px 1px 0px 0px; PADDING-LEFT: 0px; PADDING-RIGHT: 0px; PADDING-TOP: 5px" class=dp-css>
<LI>hadoop URLCat hdfs://localhost/user/tom/readme.txt</LI></OL></DIV></FONT></DIV>
<DIV><FONT size=3 face=宋體>作用是將HDFS文件里面的readme.txt輸出到System.out中，就是標(biāo)準(zhǔn)輸出。但是在執(zhí)行的時(shí)候會(huì)報(bào)以下錯(cuò)誤：</FONT></DIV>
<DIV>
<DIV id=codeText class=codeText>
<OL style="PADDING-BOTTOM: 5px; MARGIN: 0px 1px 0px 0px; PADDING-LEFT: 0px; PADDING-RIGHT: 0px; PADDING-TOP: 5px" class=dp-css>
<LI><SPAN style="COLOR: #000000"><SPAN style="COLOR: #ff0000">Exception</SPAN> <SPAN style="COLOR: #ff0000">in</SPAN> <SPAN style="COLOR: #ff0000">thread</SPAN> <SPAN style="COLOR: #ff00ff">"main"</SPAN> <SPAN style="COLOR: #ff0000">java</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">lang</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">NoClassDefFoundError</SPAN><SPAN style="COLOR: #0000cc">:</SPAN> URLCat<BR></LI>
<LI>Caused by<SPAN style="COLOR: #0000cc">:</SPAN> <SPAN style="COLOR: #ff0000">java</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">lang</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">ClassNotFoundException</SPAN><SPAN style="COLOR: #0000cc">:</SPAN> URLCat<BR></LI>
<LI> at <SPAN style="COLOR: #ff0000">java</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">net</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">URLClassLoader</SPAN><SPAN style="COLOR: #ff0000">$</SPAN>1<SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">run</SPAN><SPAN style="COLOR: #0000cc">(</SPAN><SPAN style="COLOR: #ff0000">URLClassLoader</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">java</SPAN><SPAN style="COLOR: #0000cc">:</SPAN>217<SPAN style="COLOR: #0000cc">)</SPAN><BR></LI>
<LI> at <SPAN style="COLOR: #ff0000">java</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">security</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">AccessController</SPAN><SPAN style="COLOR: #0000cc">.</SPAN>doPrivileged<SPAN style="COLOR: #0000cc">(</SPAN><SPAN style="COLOR: #0000ff">Native</SPAN> <SPAN style="COLOR: #ff0000">Method</SPAN><SPAN style="COLOR: #0000cc">)</SPAN><BR></LI>
<LI> at <SPAN style="COLOR: #ff0000">java</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">net</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">URLClassLoader</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">findClass</SPAN><SPAN style="COLOR: #0000cc">(</SPAN><SPAN style="COLOR: #ff0000">URLClassLoader</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">java</SPAN><SPAN style="COLOR: #0000cc">:</SPAN>205<SPAN style="COLOR: #0000cc">)</SPAN><BR></LI>
<LI> at <SPAN style="COLOR: #ff0000">java</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">lang</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">ClassLoader</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">loadClass</SPAN><SPAN style="COLOR: #0000cc">(</SPAN><SPAN style="COLOR: #ff0000">ClassLoader</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">java</SPAN><SPAN style="COLOR: #0000cc">:</SPAN>321<SPAN style="COLOR: #0000cc">)</SPAN><BR></LI>
<LI> at sun<SPAN style="COLOR: #0000cc">.</SPAN>misc<SPAN style="COLOR: #0000cc">.</SPAN>Launcher$AppClassLoader<SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">loadClass</SPAN><SPAN style="COLOR: #0000cc">(</SPAN>Launcher<SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">java</SPAN><SPAN style="COLOR: #0000cc">:</SPAN>294<SPAN style="COLOR: #0000cc">)</SPAN><BR></LI>
<LI> at <SPAN style="COLOR: #ff0000">java</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">lang</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">ClassLoader</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">loadClass</SPAN><SPAN style="COLOR: #0000cc">(</SPAN><SPAN style="COLOR: #ff0000">ClassLoader</SPAN><SPAN style="COLOR: #0000cc">.</SPAN><SPAN style="COLOR: #ff0000">java</SPAN><SPAN style="COLOR: #0000cc">:</SPAN>266<SPAN style="COLOR: #0000cc">)</SPAN><BR></LI>
<LI>Could <SPAN style="COLOR: #ff0000">not</SPAN> <SPAN style="COLOR: #ff0000">find</SPAN> the main <SPAN style="COLOR: #0000ff">class</SPAN><SPAN style="COLOR: #0000cc">:</SPAN> URLCat<SPAN style="COLOR: #0000cc">.</SPAN> Program will <SPAN style="COLOR: #ff0000">exit</SPAN><SPAN style="COLOR: #0000cc">.</SPAN></SPAN></LI></OL></DIV></DIV>
<DIV><FONT size=3 face=宋體>這個(gè)異常是典型的類裝入問題的異常，異常表明在類裝入的過程中，沒有找到URLCat這個(gè)類的定義。于是LGG查閱了相關(guān)資料，發(fā)現(xiàn)hadoop腳本執(zhí)行的時(shí)候，classpath的搜索路徑默認(rèn)會(huì)在當(dāng)前系統(tǒng)變量$HADOOP_CLASSPATH中去尋找，于是echo查看之，發(fā)現(xiàn)竟然顯示為空！</FONT></DIV>
<DIV><FONT size=3 face=宋體> 問題找到了，因?yàn)橄到y(tǒng)指定了classpath，所以回去這個(gè)環(huán)境變量中去尋找，但是在當(dāng)前路徑下找不到類URLCat（廢話，都是空的了）。使用export命令就可以解決了：</FONT></DIV>
<DIV><FONT size=3 face=宋體>
<DIV id=codeText class=codeText>
<OL style="PADDING-BOTTOM: 5px; MARGIN: 0px 1px 0px 0px; PADDING-LEFT: 0px; PADDING-RIGHT: 0px; PADDING-TOP: 5px" class=dp-css>
<LI>#.表示當(dāng)前目錄，即類所在目錄，只有這樣才能讓hadoop找到這個(gè)類</LI>
<LI></LI>
<LI>export HADOOP_CLASSPATH=.</LI></OL></DIV></FONT></DIV>
<DIV><FONT size=3 face=宋體>然后再執(zhí)行：</FONT></DIV>
<DIV><FONT size=3 face=宋體>
<DIV id=codeText class=codeText>
<OL style="PADDING-BOTTOM: 5px; MARGIN: 0px 1px 0px 0px; PADDING-LEFT: 0px; PADDING-RIGHT: 0px; PADDING-TOP: 5px" class=dp-css>
<LI>hadoop URLCat hdfs://localhost/user/tom/readme.txt</LI></OL></DIV></FONT></DIV>
<DIV><FONT size=3 face=宋體>就搞定了~~</FONT></DIV>
<DIV><FONT size=3 face=宋體></FONT> </DIV>
<DIV><FONT size=3 face=宋體></FONT> </DIV>
<DIV><FONT size=3 face=宋體></FONT> </DIV>
<DIV><FONT size=3 face=宋體></FONT> </DIV> |
|
免費(fèi)注冊
發(fā)表于 2011-12-22 08:52