詳解HBase Compaction

dingdejun

HBase 中有一種數(shù)據(jù)行為叫Compaction，從字面的意思就是數(shù)據(jù)文件合并，本文對Compaction的目的，控制方法，具體實施過程等幾個方面對HBase 的 Compaction 行為進(jìn)行了介紹。
1. Compaction是什么

合并多個HFile生成一個HFile

Compaction有兩種：

    Minor Compaction（部分文件合并）
    Major Compaction（完整文件合并）

2.為什么要Compaction？

    減少HFile文件的個數(shù)
    提高性能
    清除過期和刪除數(shù)據(jù)

3.配置

修改Hbase配置文件可以控制compaction行為
鍵值         默認(rèn)值         意義
hbase.regionserver.thread.splitcompactcheckfrequency         20s         compaction檢查周期
hbase.hstore.compactionThreshold         3         最小minor compaction的文件個數(shù)
hbase.hstore.blockingStoreFiles         7         Block flush操作的Store個數(shù)
hbase.hstore.blockingWaitTime         90s         Block flush操作的等待時間
hbase.hstore.compaction.max         10         最大minor compaction的文件個數(shù)
hbase.hregion.majorcompaction         1 day         Major compaction的周期
4.流程

Compaction是一個Async的過程,可以由客戶端發(fā)起，也可能是服務(wù)器端自己檢查發(fā)起compaction.
1）客戶端發(fā)起

Client端：

    HBaseAdmin::compaction or majorCompaction

    ==>HMaster modifyTable

    ==>RegionManager::startAction

    ==> put into map regionsToCompact and regionsToMajorCompact

    ==>Send to HRegionServer

Server端：

    HRegionServer::run forward the request to CompactionSplitThread

    ==>CompactionSplitThread handle the request from queue

    ==>HRegion::compactStores

    ==>Do compaction preparations, create the compaction folder

    ==>HStore::compaction

    ==>Create a HFile.Writer for writing

    ==>Create a StoreScanner for major compaction

    ==>Create a MinorCompactionStoreScanner for minor compaction

    ==>Scan the scanner and write to the hfile

    ==>Complete the compaction,delete old files and move the file to store folder

2) Server檢查發(fā)起

    Major compaction:

    Major compaction由region server定期檢查

    ==>HRegionServer::MajorCompactionChecker

    ==>Send the request to CompactionSplitThread

    Minor compaction:

    Minor compaction由Memstore flush到HDFS前檢查

    ==>MemStoreFlusher::flushRegion

    ==>Send the request to CompactionSplitThread

原文鏈接：http://www.spnguru.com/?p=271