hadoop如何實(shí)現(xiàn)關(guān)聯(lián)計(jì)算

jiongtoast

    選擇Hadoop，低成本和高擴(kuò)展性是主要原因，但但它的開發(fā)效率實(shí)在無法讓人滿意。

    以關(guān)聯(lián)計(jì)算為例。
    假設(shè)：HDFS上有2個(gè)文件，分別是客戶信息和訂單信息，customerID是它們之間的關(guān)聯(lián)字段。如何進(jìn)行關(guān)聯(lián)計(jì)算，以便將客戶名稱添加到訂單列表中？
    一般方法是：輸入2個(gè)源文件。根據(jù)文件名在Map中處理每條數(shù)據(jù)，如果是Order，則在foreign key上加標(biāo)記”O(jiān)”，形成combined key；如果是Customer則做標(biāo)記”C”。Map之后的數(shù)據(jù)按照key分區(qū)，再按照combined key分組排序。最后在reduce中合并結(jié)果再輸出。
實(shí)現(xiàn)代碼：
public static class JMapper extends Mapper<LongWritable, Text, TextPair, Text> {
    //mark every row with "O" or "C" according to file name
    @Override
    protected void map(LongWritable key, Text value, Context context) throws IOException, InterruptedException {
    String pathName = ((FileSplit) context.getInputSplit()).getPath().toString();
    if (pathName.contains("order.txt")) {//identify order by file name
            String values[] = value.toString().split("\t");
            TextPair tp = new TextPair(new Text(values[1]), new Text("O"));//mark with "O"
            context.write(tp, new Text(values[0] + "\t" + values[2]));
        }
   if (pathName.contains("customer.txt")) {//identify customer by file name
           String values[] = value.toString().split("\t");
           TextPair tp = new TextPair(new Text(values[0]), new Text("C"));//mark with "C"
           context.write(tp, new Text(values[1]));
        }
    }
}
public static class JPartitioner extends Partitioner<TextPair, Text> {
    //partition by key, i.e. customerID
    @Override
    public int getPartition(TextPair key, Text value, int numParititon) {
        return Math.abs(key.getFirst().hashCode() * 127) % numParititon;
    }
}
public static class JComparator extends WritableComparator {
    //group by muti-key
    public JComparator() {
        super(TextPair.class, true);
    }
    @SuppressWarnings("unchecked")
    public int compare(WritableComparable a, WritableComparable b) {
        TextPair t1 = (TextPair) a;
        TextPair t2 = (TextPair) b;
        return t1.getFirst().compareTo(t2.getFirst());
    }
}
public static class JReduce extends Reducer<TextPair, Text, Text, Text> {
    //merge and output
    protected void reduce(TextPair key, Iterable<Text> values, Context context) throws IOException,InterruptedException {
    Text pid = key.getFirst();
    String desc = values.iterator().next().toString();
    while (values.iterator().hasNext()) {
        context.write(pid, new Text(values.iterator().next().toString() + "\t" + desc));
   }
    }
}
public class TextPair implements WritableComparable<TextPair> {
    //make muti-key
    private Text first;
    private Text second;
    public TextPair() {
        set(new Text(), new Text());
    }
    public TextPair(String first, String second) {
        set(new Text(first), new Text(second));
    }
    public TextPair(Text first, Text second) {
        set(first, second);
    }
    public void set(Text first, Text second) {
  this.first = first;
  this.second = second;
    }
    public Text getFirst() {
  return first;
    }
    public Text getSecond() {
  return second;
    }
    public void write(DataOutput out) throws IOException {
  first.write(out);
  second.write(out);
    }
    public void readFields(DataInput in) throws IOException {
  first.readFields(in);
  second.readFields(in);
    }
    public int compareTo(TextPair tp) {
  int cmp = first.compareTo(tp.first);
  if (cmp != 0) {
       return cmp;
  }
    return second.compareTo(tp.second);
    }
}
public static void main(String agrs[]) throws IOException, InterruptedException, ClassNotFoundException {
    //job entrance
    Configuration conf = new Configuration();
    GenericOptionsParser parser = new GenericOptionsParser(conf, agrs);
    String[] otherArgs = parser.getRemainingArgs();
    if (agrs.length < 3) {
   System.err.println("Usage: J <in_path_one> <in_path_two> <output>");
   System.exit(2);
    }
    Job job = new Job(conf, "J");
    job.setJarByClass(J.class);//Join class
    job.setMapperClass(JMapper.class);//Map class
    job.setMapOutputKeyClass(TextPair.class);//Map output key class
    job.setMapOutputValueClass(Text.class);//Map output value class
    job.setPartitionerClass(JPartitioner.class);//partition class
    job.setGroupingComparatorClass(JComparator.class);//condition group class after partition
    job.setReducerClass(Example_Join_01_Reduce.class);//reduce class
    job.setOutputKeyClass(Text.class);//reduce output key class
    job.setOutputValueClass(Text.class);//reduce ouput value class
    FileInputFormat.addInputPath(job, new Path(otherArgs[0]));//one of source files
    FileInputFormat.addInputPath(job, new Path(otherArgs[1]));//another file
    FileOutputFormat.setOutputPath(job, new Path(otherArgs[2]));//output path
    System.exit(job.waitForCompletion(true) ? 0 : 1);//run untill job ends
}

    不能直接使用原始數(shù)據(jù)，而是要搞一堆代碼處理標(biāo)記，并繞過MapReduce原本的架構(gòu)，最后從底層設(shè)計(jì)并計(jì)算數(shù)據(jù)之間的關(guān)聯(lián)關(guān)系。這還是最簡(jiǎn)單的關(guān)聯(lián)計(jì)算，如果用MapReduce進(jìn)行多表關(guān)聯(lián)或邏輯更復(fù)雜的關(guān)聯(lián)計(jì)算，復(fù)雜度會(huì)呈幾何級(jí)數(shù)遞增。

歡迎訪問我的blog，大數(shù)據(jù)相關(guān)特別是hive：http://blog.chinaunix.net/uid/29242841.html

aluop

                                100% pass ,220-701 - CompTIA A+ Essentials (2009 Edition) Certification Exam
100% pass ,220-702 - CompTIA A+ Practical Application (2009 Edition) Certification Exam
100% pass ,220-801 - CompTIA A+ Certification Exam
100% pass ,220-802 - CompTIA A+ Certification Exam
100% pass ,225-030 - CompTIA CDIA+ Certification Exam
100% pass ,ADR-001 - CompTIA Mobile App Security+ Certification Exam (Android Edition)
100% pass ,CAS-001 - CompTIA Advanced Security Practitioner (CASP) Exam
100% pass ,CD0-001 - CompTIA CDIA+ Certification Exam
100% pass ,CLO-001 - CompTIA Cloud Essentials Exam

100% guaranteed pass ,Once failed 100% refund
The Fastest and Guaranteed Way to Certify Now
[url]http://www.eliteguides.cn[/url]
certification,certify,certified,braindumps,braindump,dumps,exam,exams,study guides,study guide,test,tests,testing,training,free,book,pdf,question,questions

100% pass ,CV0-001 - CompTIA Cloud+ Certification Exam
100% pass ,FC0-101 - CompTIA Strata PC Hardware Engineer Exam
100% pass ,FC0-GR1 - CompTIA Strata Green IT Exam
100% pass ,FC0-TS1 - CompTIA IT for Sales Specialist Exam
100% pass ,FC0-U11 - CompTIA Strata Fundamentals of PC Functionality Exam
100% pass ,FC0-U21 - CompTIA Strata Fundamentals of PC Technology Exam
100% pass ,FC0-U41 - CompTIA Strata IT Fundamentals Exam
100% pass ,HIT-001 - CompTIA Healthcare IT Technician Exam
100% pass ,iCLO-001 - Invitation Only - CompTIA Cloud Essentials Exam
100% pass ,iFC0-U41 - Invitation Only - CompTIA Strata IT Fundamentals Exam
100% pass ,IOS-001 - CompTIA Mobile App Security+ Certification Exam (iOS Edition)
100% pass ,ISS-001 - Intel? Server Specialist Certification Exam
100% pass ,JK0-017 - CompTIA Academic/E2C Project+ Certification Exam
100% pass ,JK0-018 - CompTIA Academic/E2C Security+ Certification Exam

100% guaranteed pass ,Once failed 100% refund
The Fastest and Guaranteed Way to Certify Now
[url]http://www.eliteguides.cn[/url]
certification,certify,certified,braindumps,braindump,dumps,exam,exams,study guides,study guide,test,tests,testing,training,free,book,pdf,question,questions

100% pass ,JK0-019 - CompTIA Academic/E2C Network+ Certification Exam
100% pass ,JK0-020 - CompTIA Academic/E2C Cloud+ Certification Exam
100% pass ,JK0-701 - CompTIA Academic/E2C A+ Essentials (2009 Edition) Exam
100% pass ,JK0-702 - CompTIA Academic/E2C A+ Practical Application (2009 Edition) Exam
100% pass ,JK0-801 - CompTIA Academic/E2C A+ Certification Exam
100% pass ,JK0-802 - CompTIA Academic/E2C A+ Certification Exam
100% pass ,JK0-U11 - CompTIA Academic/E2C Strata Fundamentals of PC Functionality Exam
100% pass ,JK0-U21 - CompTIA Academic/E2C Strata Fundamentals of PC Technology Exam
100% pass ,JK0-U31 - CompTIA Academic/E2C Strata IT Fundamentals Exam
100% pass ,LX0-101 - CompTIA Linux+ [Powered by LPI] Exam 1
100% pass ,LX0-102 - CompTIA Linux+ [Powered by LPI] Exam 2
100% pass ,MB0-001 - CompTIA Mobility+ Certification Exam
100% pass ,N10-005 - CompTIA Network+ Certification Exam
100% pass ,PD0-001 - CompTIA PDI+ Certification Exam

100% guaranteed pass ,Once failed 100% refund
The Fastest and Guaranteed Way to Certify Now
[url]http://www.eliteguides.cn[/url]
certification,certify,certified,braindumps,braindump,dumps,exam,exams,study guides,study guide,test,tests,testing,training,free,book,pdf,question,questions

100% pass ,PK0-003 - CompTIA Project+ Certification Exam
100% pass ,SG0-001 - CompTIA Storage+ Powered By SNIA Certification Exam
100% pass ,SK0-003 - CompTIA Server+ Certification Exam
100% pass ,SY0-301 - CompTIA Security+ Certification Exam
100% pass ,TE0-301 - OS X Directory Services Specialist Certification Exam
100% pass ,TE0-302 - OS X Deployment Specialist Certification Exam
100% pass ,TE0-303 - OS X Mobile Device and Profile Specialist Certification Exam
100% pass ,TK0-201 - CompTIA CTT+ Essentials Certification Exam
100% pass ,UK0-001 - UKI Social Media Security Professional (SMSP) Certification exam
100% pass ,zUK1-001 - Invitation Only - UKI Social Media Professional (SMP) Certification Beta Exam

100% guaranteed pass ,Once failed 100% refund
The Fastest and Guaranteed Way to Certify Now
[url]http://www.eliteguides.cn[/url]
certification,certify,certified,braindumps,braindump,dumps,exam,exams,study guides,study guide,test,tests,testing,training,free,book,pdf,question,questions