Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-10591

Support limited integer type promotion in ORC

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 1.3.0
    • 1.3.0, 2.0.0
    • None
    • None

    Description

      ORC currently does not support schema-on-read. If we alter an ORC table with 'int' type to 'bigint' and if we query the altered table ClassCastException will be thrown as the schema on read from table descriptor will expect LongWritable whereas ORC will return IntWritable based on file schema stored within ORC file. OrcSerde currently doesn't do any type conversions or type promotions for performance reasons in inner loop. Since smallints, ints and bigints are stored in the same way in ORC, it will be possible be allow such type promotions without hurting performance. Following type promotions can be supported without any casting
      smallint -> int
      smallint -> bigint
      int -> bigint

      Tinyint promotion is not possible without casting as tinyints are stored using RLE byte writer whereas smallints, ints and bigints are stored using RLE integer writer.

      Attachments

        1. HIVE-10591.1.patch
          29 kB
          Prasanth Jayachandran
        2. HIVE-10591.2.patch
          32 kB
          Szehon Ho
        3. HIVE-10591.2.patch
          32 kB
          Prasanth Jayachandran
        4. HIVE-10591.3.patch
          35 kB
          Prasanth Jayachandran
        5. HIVE-10591.3.patch
          35 kB
          Prasanth Jayachandran
        6. HIVE-10591.3.patch
          35 kB
          Prasanth Jayachandran

        Activity

          People

            prasanth_j Prasanth Jayachandran
            prasanth_j Prasanth Jayachandran
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: