暗能星系
    • 登录
    • 搜索

    Hbase批量导入数据

    大数据
    1
    12
    54
    正在加载更多帖子
    • 从旧到新
    • 从新到旧
    • 最多赞同
    回复
    • 在新帖中回复
    登录后回复
    此主题已被删除。只有拥有主题管理权限的用户可以查看。
    • A
      anneng 最后由 编辑

      因为blastdbcmd导出的数据有很多字符 为了满足 importtsv 的格式要求 我们需要以tab分割 分割tab有两个方法：
      1.把下面的命令保存到sh文件中 方便输入tab
      blastdbcmd -db nt -entry all -out newnt1.fa -outfmt '%a %t %T %s'

      2.在命令行首先输入ctrl+v 然后输入tab 就可以直接把tab输入到命令行

      1 条回复 最后回复 回复 引用 0
      • A
        anneng 最后由 编辑

        hbase的cell有最大长度限制 默认10MB 需要禁用掉
        <property>
        <name>hbase.client.keyvalue.maxsize</name>
        <value>0</value>
        </property>

        配置修改后要同步到各个节点 然后重启 hbase
        stop-hbase.sh
        start-hbase.sh

        1 条回复 最后回复 回复 引用 0
        • A
          anneng 最后由 anneng 编辑

          https://stackoverflow.com/questions/14326308/how-to-include-hbase-site-xml-in-the-classpath

          如何把hbase-site.xml加入到classpath 使其生效

          export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/opt/hadoop/hbase-2.3.4/lib/:/opt/hadoop/hadoop-3.2.2/lib/native/:/opt/hadoop/hbase-2.3.4/conf

          /opt/hadoop/hbase-2.3.4/conf 必须加上这个让客户端读取hbase-site.xml

          1 条回复 最后回复 回复 引用 0
          • A
            anneng 最后由 编辑

            2021-04-07 17:37:09,260 INFO [htable-pool3866-t1] client.AsyncRequestFutureImpl (AsyncRequestFutureImpl.java:resubmit(763)) - id=2899, table=nt, attempt=12/16, failureCount=1ops, last exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 M, regionName=1efaf86c7220c641629a386589e1a8ef, server=anneng01,16020,1617700403892
            at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4535)
            at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4101)
            at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4041)
            at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1081)
            at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:1013)
            at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:978)
            at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2828)
            at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44870)
            at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393)
            at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
            at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
            at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
            on anneng01,16020,1617700403892, tracking started null, retrying after=20194ms, operationsToReplay=1

            hbase.hregion.memstore.flush.size=1024M
hbase.hregion.memstore.block.multiplier=4
            1 条回复 最后回复 回复 引用 0
            • A
              anneng 最后由 编辑

              Hbase支持两种调用方法　一种直接调用类　一种使用hadoop的driver机制
              Explicit Classname
              $ bin/hbase org.apache.hadoop.hbase.tool.LoadIncrementalHFiles hdfs://storefileoutput <tablename>
              Driver
              HADOOP_CLASSPATH=${HBASE_HOME}/bin/hbase classpath ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-mapreduce-VERSION.jar completebulkload hdfs://storefileoutput <tablename>

              https://www.oreilly.com/library/view/learning-hbase/9781783985944/ch06s05.html
              hadoop的driver机制　还可以调用下面这些hbase的功能:
              completebulkload: This is for a bulk data load
              copytable: This is to export a table data from the local to peer cluster
              export: This is to export data from an HBase table to HDFS as a sequence file
              import: This is to import data written by export
              importtsv: This is to import data in TSV format to HBase
              rowcounter: This is to count rows in an HBase table using MapReduce
              verifyrep: This is to compare the data from tables of different clusters

              1 条回复 最后回复 回复 引用 0
              • A
                anneng 最后由 编辑

                有一次运行 下面的参数写错了 应该是importtsv.bulk.output
                -Dimporttst.bulk.output=hdfs://192.168.1.2:7000/nt/hfiles/

                importtsv 在这个参数错误的情况下 顺利执行了 但是不知道把结果写入到哪里去了

                1 条回复 最后回复 回复 引用 0
                • A
                  anneng 最后由 anneng 编辑

                  java.lang.reflect.InvocationTargetException
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:498)
                  at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:64)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:498)
                  at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
                  at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
                  Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.client.HdfsDataInputStream.getReadStatistics()Lorg/apache/hadoop/hdfs/DFSInputStream$ReadStatistics;
                  at org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.updateInputStreamStatistics(FSDataInputStreamWrapper.java:253)
                  at org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.close(FSDataInputStreamWrapper.java:300)
                  at org.apache.hadoop.hbase.io.hfile.HFile.isHFileFormat(HFile.java:590)
                  at org.apache.hadoop.hbase.io.hfile.HFile.isHFileFormat(HFile.java:571)
                  at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.visitBulkHFiles(LoadIncrementalHFiles.java:1072)
                  at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.discoverLoadQueue(LoadIncrementalHFiles.java:988)
                  at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.prepareHFileQueue(LoadIncrementalHFiles.java:249)
                  at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:356)
                  at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:1216)
                  at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:1229)
                  at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:1264)
                  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
                  at org.apache.hadoop.hbase.tool.BulkLoadHFilesTool.main(BulkLoadHFilesTool.java:66)
                  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
                  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
                  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
                  at java.lang.reflect.Method.invoke(Method.java:498)
                  at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
                  at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
                  at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:152)
                  ... 11 more
                  Command exited with non-zero status 1

                  不知道什么原因 执行下面的方法就好了
                  java -cp /opt/hadoop/hbase-2.3.4/lib/hbase-mapreduce-2.3.4.jar:/opt/hadoop/hbase-2.3.4/lib/hbase-server-2.3.4.jar:/opt/hadoop/hbase-2.3.4/lib/:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/ org.apache.hadoop.hbase.tool.BulkLoadHFilesTool hdfs://192.168.1.2:7000/nt/hfiles/ nt

                  1 条回复 最后回复 回复 引用 0
                  • A
                    anneng 最后由 编辑

                    可以提前把文件下载好 然后执行krakenuniq-download 就会自动检测已经下载好的文件
                    293d9cbe-faae-48f8-a0e9-77481cf57599-image.png

                    1 条回复 最后回复 回复 引用 0
                    • A
                      anneng 最后由 anneng 编辑 

                      java -cp /opt/hadoop/hbase-2.3.4/lib/hbase-mapreduce-2.3.4.jar:/opt/hadoop/hbase-2.3.4/lib/hbase-server-2.3.4.jar:/opt/hadoop/hbase-2.3.4/lib/*:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/*:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/* org.apache.hadoop.hbase.tool.BulkLoadHFilesTool hdfs://192.168.1.2:7000/nt/hfiles/ nt

                      加载上一步生成的文件 这一步时间很快 可以忽略

                      1 条回复 最后回复 回复 引用 0
                      • A
                        anneng 最后由 anneng 编辑

                        2021-04-09 18:38:33,008 ERROR [main] tool.LoadIncrementalHFiles (LoadIncrementalHFiles.java:checkHFilesCountPerRegionPerFamily(610)) - Trying to load more than 32 hfiles to family seq of region with start key
                        2021-04-09 18:38:33,019 INFO [main] client.ConnectionImplementation (ConnectionImplementation.java:closeMasterService(1898)) - Closing master protocol: MasterService
                        Exception in thread "main" java.io.IOException: Trying to load more than 32 hfiles to one family of one region
                        at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.performBulkLoad(LoadIncrementalHFiles.java:455)
                        at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:367)
                        at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:1216)
                        at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:1229)
                        at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:1264)
                        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
                        at org.apache.hadoop.hbase.tool.BulkLoadHFilesTool.main(BulkLoadHFilesTool.java:66)

                        java -cp /opt/hadoop/hbase-2.3.4/lib/hbase-mapreduce-2.3.4.jar:/opt/hadoop/hbase-2.3.4/lib/hbase-server-2.3.4.jar:/opt/hadoop/hbase-2.3.4/lib/:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/* org.apache.hadoop.hbase.tool.BulkLoadHFilesTool -Dhbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily=1024 hdfs://192.168.1.2:7000/nt/hfiles2/ nt

                        1 条回复 最后回复 回复 引用 0
                        • First post
                          Last post
                        Powered by 暗能星系