<![CDATA[Hbase批量导入数据]]>对于tsv形式的数据　可以用下面的方式导入

将数据加入hdfs
hdfs dfs -put /ceph_disk1/gene_data/MetaDatabase/NCBI_blast_db_FASTA/nt/nt_exported_fasta/newnt1.fa /nt/

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/opt/hadoop/hbase-2.3.4/lib/*:/opt/hadoop/hadoop-3.2.2/lib/native/*:/opt/hadoop/hbase-2.3.4/conf
time hadoop jar /opt/hadoop/hbase-2.3.4/lib/hbase-mapreduce-2.3.4.jar importtsv -Dimporttsv.columns=HBASE_ROW_KEY,seq:seqt,seq:taxid,seq:seqs -Dimporttsv.bulk.output=hdfs://192.168.1.2:7000/nt/hfiles/ nt hdfs://192.168.1.2:7000/nt/newnt1.fa &

默认用tab分割 不用下面的&的分割
nohup time hadoop jar /opt/hadoop/hbase-2.3.4/lib/hbase-mapreduce-2.3.4.jar importtsv '-Dimporttsv.separator=&' -Dimporttsv.columns=HBASE_ROW_KEY,seq:seqt,seq:taxid,seq:seqs -Dimporttsv.bulk.output=hdfs://192.168.1.2:7000/nt/hfiles/ nt hdfs://192.168.1.2:7000/nt/ &

生成HFiles的步骤 370GB NT数据 大概用了不到4个小时
Map-Reduce Framework
Map input records=74547430
Map output records=74547430
Map output bytes=408633905926
Map output materialized bytes=408933633110
Input split bytes=302394
Combine input records=74681364
Combine output records=74681364
Reduce input groups=74547430
Reduce shuffle bytes=408933633110
Reduce input records=74547430
Reduce output records=223642290
Spilled Records=223598923
Shuffled Maps =320358
Failed Shuffles=0
Merged Map outputs=320358
GC time elapsed (ms)=137615
Total committed heap usage (bytes)=27134307336192
ImportTsv
Bad Lines=0
Shuffle Errors
BAD_ID=0
CONNECTION=0
IO_ERROR=0
WRONG_LENGTH=0
WRONG_MAP=0
WRONG_REDUCE=0
File Input Format Counters
Bytes Read=421141495842
File Output Format Counters
Bytes Written=410423108883
9057.42user 3919.59system 3:40:55elapsed 97%CPU (0avgtext+0avgdata 31694828maxresident)k

]]>http://an.forum.genostack.com/topic/272/hbase批量导入数据RSS for NodeSat, 13 Jun 2026 12:32:44 GMTThu, 01 Apr 2021 12:19:33 GMT60<![CDATA[Reply to Hbase批量导入数据 on Fri, 09 Apr 2021 10:48:29 GMT]]>2021-04-09 18:38:33,008 ERROR [main] tool.LoadIncrementalHFiles (LoadIncrementalHFiles.java:checkHFilesCountPerRegionPerFamily(610)) - Trying to load more than 32 hfiles to family seq of region with start key
2021-04-09 18:38:33,019 INFO [main] client.ConnectionImplementation (ConnectionImplementation.java:closeMasterService(1898)) - Closing master protocol: MasterService
Exception in thread "main" java.io.IOException: Trying to load more than 32 hfiles to one family of one region
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.performBulkLoad(LoadIncrementalHFiles.java:455)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:367)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:1216)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:1229)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:1264)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.hbase.tool.BulkLoadHFilesTool.main(BulkLoadHFilesTool.java:66)

java -cp /opt/hadoop/hbase-2.3.4/lib/hbase-mapreduce-2.3.4.jar:/opt/hadoop/hbase-2.3.4/lib/hbase-server-2.3.4.jar:/opt/hadoop/hbase-2.3.4/lib/:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/* org.apache.hadoop.hbase.tool.BulkLoadHFilesTool -Dhbase.mapreduce.bulkload.max.hfiles.perRegion.perFamily=1024 hdfs://192.168.1.2:7000/nt/hfiles2/ nt

]]>http://an.forum.genostack.com/post/577http://an.forum.genostack.com/post/577Fri, 09 Apr 2021 10:48:29 GMT<![CDATA[Reply to Hbase批量导入数据 on Fri, 09 Apr 2021 03:48:08 GMT]]>java -cp /opt/hadoop/hbase-2.3.4/lib/hbase-mapreduce-2.3.4.jar:/opt/hadoop/hbase-2.3.4/lib/hbase-server-2.3.4.jar:/opt/hadoop/hbase-2.3.4/lib/*:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/*:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/* org.apache.hadoop.hbase.tool.BulkLoadHFilesTool hdfs://192.168.1.2:7000/nt/hfiles/ nt

加载上一步生成的文件 这一步时间很快 可以忽略

]]>http://an.forum.genostack.com/post/569http://an.forum.genostack.com/post/569Fri, 09 Apr 2021 03:48:08 GMT<![CDATA[Reply to Hbase批量导入数据 on Thu, 08 Apr 2021 08:31:17 GMT]]>可以提前把文件下载好 然后执行krakenuniq-download 就会自动检测已经下载好的文件
293d9cbe-faae-48f8-a0e9-77481cf57599-image.png

]]>http://an.forum.genostack.com/post/568http://an.forum.genostack.com/post/568Thu, 08 Apr 2021 08:31:17 GMT<![CDATA[Reply to Hbase批量导入数据 on Thu, 08 Apr 2021 11:16:45 GMT]]>java.lang.reflect.InvocationTargetException
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.hbase.mapreduce.Driver.main(Driver.java:64)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.RunJar.run(RunJar.java:323)
at org.apache.hadoop.util.RunJar.main(RunJar.java:236)
Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.hdfs.client.HdfsDataInputStream.getReadStatistics()Lorg/apache/hadoop/hdfs/DFSInputStream$ReadStatistics;
at org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.updateInputStreamStatistics(FSDataInputStreamWrapper.java:253)
at org.apache.hadoop.hbase.io.FSDataInputStreamWrapper.close(FSDataInputStreamWrapper.java:300)
at org.apache.hadoop.hbase.io.hfile.HFile.isHFileFormat(HFile.java:590)
at org.apache.hadoop.hbase.io.hfile.HFile.isHFileFormat(HFile.java:571)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.visitBulkHFiles(LoadIncrementalHFiles.java:1072)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.discoverLoadQueue(LoadIncrementalHFiles.java:988)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.prepareHFileQueue(LoadIncrementalHFiles.java:249)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.doBulkLoad(LoadIncrementalHFiles.java:356)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:1216)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:1229)
at org.apache.hadoop.hbase.tool.LoadIncrementalHFiles.run(LoadIncrementalHFiles.java:1264)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.hbase.tool.BulkLoadHFilesTool.main(BulkLoadHFilesTool.java:66)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:71)
at org.apache.hadoop.util.ProgramDriver.run(ProgramDriver.java:144)
at org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:152)
... 11 more
Command exited with non-zero status 1

不知道什么原因 执行下面的方法就好了
java -cp /opt/hadoop/hbase-2.3.4/lib/hbase-mapreduce-2.3.4.jar:/opt/hadoop/hbase-2.3.4/lib/hbase-server-2.3.4.jar:/opt/hadoop/hbase-2.3.4/lib/:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/:/opt/hadoop/hadoop-3.2.2/share/hadoop/common/lib/ org.apache.hadoop.hbase.tool.BulkLoadHFilesTool hdfs://192.168.1.2:7000/nt/hfiles/ nt

]]>http://an.forum.genostack.com/post/567http://an.forum.genostack.com/post/567Thu, 08 Apr 2021 11:16:45 GMT<![CDATA[Reply to Hbase批量导入数据 on Thu, 08 Apr 2021 03:19:03 GMT]]>有一次运行 下面的参数写错了 应该是importtsv.bulk.output
-Dimporttst.bulk.output=hdfs://192.168.1.2:7000/nt/hfiles/

importtsv 在这个参数错误的情况下 顺利执行了 但是不知道把结果写入到哪里去了

]]>http://an.forum.genostack.com/post/565http://an.forum.genostack.com/post/565Thu, 08 Apr 2021 03:19:03 GMT<![CDATA[Reply to Hbase批量导入数据 on Thu, 08 Apr 2021 03:01:14 GMT]]>Hbase支持两种调用方法　一种直接调用类　一种使用hadoop的driver机制
Explicit Classname
$ bin/hbase org.apache.hadoop.hbase.tool.LoadIncrementalHFiles hdfs://storefileoutput <tablename>
Driver
HADOOP_CLASSPATH=${HBASE_HOME}/bin/hbase classpath ${HADOOP_HOME}/bin/hadoop jar ${HBASE_HOME}/hbase-mapreduce-VERSION.jar completebulkload hdfs://storefileoutput <tablename>

https://www.oreilly.com/library/view/learning-hbase/9781783985944/ch06s05.html
hadoop的driver机制　还可以调用下面这些hbase的功能:
completebulkload: This is for a bulk data load
copytable: This is to export a table data from the local to peer cluster
export: This is to export data from an HBase table to HDFS as a sequence file
import: This is to import data written by export
importtsv: This is to import data in TSV format to HBase
rowcounter: This is to count rows in an HBase table using MapReduce
verifyrep: This is to compare the data from tables of different clusters

]]>http://an.forum.genostack.com/post/564http://an.forum.genostack.com/post/564Thu, 08 Apr 2021 03:01:14 GMT<![CDATA[Reply to Hbase批量导入数据 on Wed, 07 Apr 2021 09:40:35 GMT]]>2021-04-07 17:37:09,260 INFO [htable-pool3866-t1] client.AsyncRequestFutureImpl (AsyncRequestFutureImpl.java:resubmit(763)) - id=2899, table=nt, attempt=12/16, failureCount=1ops, last exception=org.apache.hadoop.hbase.RegionTooBusyException: org.apache.hadoop.hbase.RegionTooBusyException: Over memstore limit=512.0 M, regionName=1efaf86c7220c641629a386589e1a8ef, server=anneng01,16020,1617700403892
at org.apache.hadoop.hbase.regionserver.HRegion.checkResources(HRegion.java:4535)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4101)
at org.apache.hadoop.hbase.regionserver.HRegion.batchMutate(HRegion.java:4041)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doBatchOp(RSRpcServices.java:1081)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicBatchOp(RSRpcServices.java:1013)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.doNonAtomicRegionMutation(RSRpcServices.java:978)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.multi(RSRpcServices.java:2828)
at org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:44870)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
on anneng01,16020,1617700403892, tracking started null, retrying after=20194ms, operationsToReplay=1

hbase.hregion.memstore.flush.size=1024M
hbase.hregion.memstore.block.multiplier=4
]]>http://an.forum.genostack.com/post/549http://an.forum.genostack.com/post/549Wed, 07 Apr 2021 09:40:35 GMT<![CDATA[Reply to Hbase批量导入数据 on Wed, 07 Apr 2021 12:57:19 GMT]]>https://stackoverflow.com/questions/14326308/how-to-include-hbase-site-xml-in-the-classpath

如何把hbase-site.xml加入到classpath 使其生效

export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/opt/hadoop/hbase-2.3.4/lib/:/opt/hadoop/hadoop-3.2.2/lib/native/:/opt/hadoop/hbase-2.3.4/conf

/opt/hadoop/hbase-2.3.4/conf 必须加上这个让客户端读取hbase-site.xml

]]>http://an.forum.genostack.com/post/547http://an.forum.genostack.com/post/547Wed, 07 Apr 2021 12:57:19 GMT<![CDATA[Reply to Hbase批量导入数据 on Tue, 06 Apr 2021 03:26:01 GMT]]>hbase的cell有最大长度限制 默认10MB 需要禁用掉
<property>
<name>hbase.client.keyvalue.maxsize</name>
<value>0</value>
</property>

配置修改后要同步到各个节点 然后重启 hbase
stop-hbase.sh
start-hbase.sh

]]>http://an.forum.genostack.com/post/541http://an.forum.genostack.com/post/541Tue, 06 Apr 2021 03:26:01 GMT<![CDATA[Reply to Hbase批量导入数据 on Fri, 02 Apr 2021 02:24:13 GMT]]>因为blastdbcmd导出的数据有很多字符 为了满足 importtsv 的格式要求 我们需要以tab分割 分割tab有两个方法：
1.把下面的命令保存到sh文件中 方便输入tab
blastdbcmd -db nt -entry all -out newnt1.fa -outfmt '%a %t %T %s'

2.在命令行首先输入ctrl+v 然后输入tab 就可以直接把tab输入到命令行

]]>http://an.forum.genostack.com/post/539http://an.forum.genostack.com/post/539Fri, 02 Apr 2021 02:24:13 GMT<![CDATA[Reply to Hbase批量导入数据 on Thu, 01 Apr 2021 12:19:42 GMT]]>java.lang.Exception: java.lang.IllegalArgumentException: TsvParser only supports single-byte separators
at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.lang.IllegalArgumentException: TsvParser only supports single-byte separators
at org.apache.hbase.thirdparty.com.google.common.base.Preconditions.checkArgument(Preconditions.java:142)
at org.apache.hadoop.hbase.mapreduce.ImportTsv$TsvParser.<init>(ImportTsv.java:161)
at org.apache.hadoop.hbase.mapreduce.TsvImporterMapper.setup(TsvImporterMapper.java:108)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)

]]>http://an.forum.genostack.com/post/538http://an.forum.genostack.com/post/538Thu, 01 Apr 2021 12:19:42 GMT