构建本地nt/nr数据库

[anneng]

https://blast.ncbi.nlm.nih.gov/Blast.cgi?PAGE_TYPE=BlastDocs&DOC_TYPE=Download
1.为什么要构建本地nt/nr数据库？
Do you have difficulties running high volume BLAST searches?
Do you have proprietary sequence data to search and cannot use the NCBI BLAST web site?
Do you have access to your own server?
Do you have your own research pipeline?
Have security or IP concerns about sending searches outside of your organization?

2.下载blast+软件
ftp://ftp.ncbi.nlm.nih.gov/blast/executables/blast+/LATEST/
软件手册:
https://www.ncbi.nlm.nih.gov/books/NBK279690/
软件编译安装:
cd c++
./configure
cd ReleaseMT/build
make all_r

包含的软件：



https://open.oregonstate.education/computationalbiology/chapter/command-line-blast/
配置：
程序所在路径
export PATH=$PATH:$HOME/ncbi-blast-2.10.1+/bin
数据库所在路径
export BLASTDB=$HOME/blastdb

3.下载数据库
ftp://ftp.ncbi.nlm.nih.gov/blast/db/
cd $HOME/blastdb
perl ../bin/update_blastdb.pl --passive --decompress 16S_ribosomal_RNA
更新数据库
perl ../bin/update_blastdb.pl --passive --decompress 16S_ribosomal_RNA

一个例子:
blastdbcmd -db 16S_ribosomal_RNA -entry nr_025000 -out 16S_query.fa
blastn -db 16S_ribosomal_RNA -query 16S_query.fa -task blastn -dust no -outfmt "7 delim=, qacc sacc evalue bitscore qcovus pident" -max_target_seqs 5

BLAST Database error: No alias or index file found for nucleotide database [./16S_ribosomal_RNA] in search path [/home/bioinfo/gene_data/blastdb::]
加一个全局变量即可
export BLASTDB=/home/bioinfo/gene_data/blastdb/16S_ribosomal_RNA/

4.添加自己的序列
序列使用 fast格式　建议取一个唯一的名字
$ cat test.fsa

seq1
MSFSTKPLDMATWPDFAALVERHNGVWGGCWCMAFHAKGSGAVGNREAKEARVREGSTHAALVFDGSACVGWCQFGPTGE
LPRIKHLRAYEDGQAVLPDWRITCFFSDKAFRGKGVAAAALAGALAEIGRLGGGTVESYPEDAQGRTVAGAFLHNGTLAM
这个序列是哪种物种？用下面的文件描述:
包括identifiers to taxids的对应关系
$ cat test_map.txt
seq1 68287
$ makeblastdb -in test.fsa -parse_seqids -blastdb_version 5 -taxid_map test_map.txt -title "Cookbook demo" -dbtype prot

https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/accession2taxid/
https://ncbiinsights.ncbi.nlm.nih.gov/2018/02/22/new-taxonomy-files-available-with-lineage-type-and-host-information/
https://www.ncbi.nlm.nih.gov/Taxonomy/TaxIdentifier/tax_identifier.cgi
也可以把fa文件当作数据库来查询：
blastn -task megablast -db Dmel_transcripts_Ensembl/Dmel_genes_all.fa -query blast_query.txt -dust no -max_target_seqs 1 -outfmt "6 qseqid sseqid evalue pident stitle" -out outputfile.txt
5.nt/nr的数据库格式

6.按物种id查询
blastn –db nt –query QUERY –taxids 9606 –outfmt 7 –out OUTPUT.tab
这样可以降低查询数据量　也反应了使用sql数据库的好处

7.导出数据
blastdbcmd -db 16S_ribosomal_RNA -entry all -out 16S_query.fa
-outfmt "%f"
%f - Output all sequence data and metadata in FASTA format (this is the default behavior of the command)
%s - Sequence data
%a - Database specific accession ID
%g - NCBI sequence ID (i.e. gi)
%i - Sequence ID
%l - Sequence length
更多格式
/home/bioinfo/software/blast_debug/ncbi-blast-2.11.0+-src/c++/ReleaseMT/bin/blastdbcmd -db ./16S_ribosomal_RNA -entry all -out 16S_query.fa -outfmt "%T %a %g %i %l %t"

-outfmt <String>
Output format, where the available format specifiers are:
%f means sequence in FASTA format
%s means sequence data (without defline)
%a means accession
%g means gi
%o means ordinal id (OID)
%i means sequence id
%t means sequence title
%l means sequence length
%h means sequence hash value
%T means taxid
%X means leaf-node taxids
%e means membership integer
%L means common taxonomic name
%C means common taxonomic names for leaf-node taxids
%S means scientific name
%N means scientific names for leaf-node taxids
%B means BLAST name
%n means a list of links integers separated by ';'
%K means taxonomic super kingdom
%P means PIG
%d means defline in text ASN.1 format
%b means Bioseq in text ASN.1 format
%m means sequence masking data.
Masking data will be displayed as a series of 'N-M' values
separated by ';' or the word 'none' if none are available.
If '%f' or '%d' are specified, all other format specifiers are ignored.
For every format except '%f' and '%d', each line of output will correspond
to a sequ

[anneng]

Database resources of the National Center for Biotechnology Information
https://academic.oup.com/nar/article/42/D1/D7/1054454

[anneng]

Nr database encompasses sequences from both non-curated and curated databases:

Non-curated databases (low quality):

GenBank/GenPept - unreviewed sequences submitted from individual laboratories and large-scale sequencing projects. Since these sequence records are owned by the original submitters and can not be altered, GenBank might contain many low quality sequences.
trEMBL - unreviewed section of UniProt. This section contains a computer-annotated supplement of SwissProt that contains all the translations of EMBL nucleotide sequence entries not yet integrated in SwissProt
Curated databases (high quality):

RefSeq - GenBank sequences that are manually curated by the NCBI staff. RefSeq records are owned by NCBI and can be updated as needed to maintain current annotation or to incorporate additional information.
SwissProt - manually annotated and reviewed protein sequences
PIR - non-redundant annotated protein sequence database
PDB - experimentally-determined structures of proteins, nucleic acids, and complex assemblies

https://www.biostars.org/p/164641/

[anneng]

https://ftp.ncbi.nlm.nih.gov/blast/db/blastdbv5.pdf

[anneng]

                     The BLAST Databases
                Last updated on February 3, 2020

IMPORTANT: As of February 4, 2020, the BLAST databases on the FTP site are version 5 (v5).
At the same time, the databases offered has been changed. This document reflects those changes.
Information on newly enabled features with the v5 databases at
https://ftp.ncbi.nlm.nih.gov/blast/db/blastdbv5.pdf

This document describes the BLAST databases available on the NCBI FTP site under
the /blast/db directory. The direct URL is ftp://ftp.ncbi.nlm.nih.gov/blast/db

1. Quick Start

   • Get all numbered files for a database with the same base name:
     Each of these files represents a subset (volume) of that database,
     and all of them are needed to reconstitute the database.
   • After extraction, there is no need to concatenate the resulting files:
     Call the database with the base name, for nr database files, use "-db nr".
   • For easy download, use the update_blastdb.pl script from the blast+ package.
   • Incremental update is not available.

2. General Introduction

BLAST search pages under the Basic BLAST section of the NCBI BLAST home page
(http://blast.ncbi.nlm.nih.gov/) use a standard set of BLAST databases for
nucleotide, protein, and translated BLAST searches. These databases are made
available as compressed archives of pre-formatted form) and can be donwloaed from
the /db directory of the BLAST ftp site (ftp://ftp.ncbi.nlm.nih.gov/blast/db/).
The FASTA files reside under the /FASTA subdirectory.

The pre-formatted databases offer the following advantages:
* Pre-formatting removes the need to run makeblastdb;
* Species-level taxonomy ids are included for each database entry;
* Databases are broken into smaller-sized volumes and are therefore easier
to download;
* Sequences in FASTA format can be generated from the pre-formatted databases
by using the blastdbcmd utility;
* A convenient script (update_blastdb.pl) is available in the blast+ package
to download the pre-formatted databases.

Pre-formatted databases must be downloaded using the update_blastdb.pl script or
via FTP in binary mode. Documentation for this script can be obtained by running
the script without any arguments; Perl installation is required.

The compressed files downloaded must be inflated with gzip or other decompress
tools. The BLAST database files can then be extracted out of the resulting tar
file using the tar utility on Unix/Linux, or WinZip and StuffIt Expander on
Windows and Macintosh platforms, respectively.

Large databases are formatted in multiple one-gigabyte volumes, which are named
using the basename.##.tar.gz convention. All volumes with the same base name are
required. An alias file is provided to tie individual volumes together so that
the database can be called using the base name (without the .nal or .pal
extension). For example, to call the est database, simply use "-db est" option
in the command line (without the quotes).

For other genomic BLAST databases, please check the genomes ftp directory at:
ftp://ftp.ncbi.nlm.nih.gov/genomes/

3. Contents of the /blast/db/ directory

The pre-formatted BLAST databases are archived in this directory. The names of
these databases and their contents are listed below.

+-----------------------------+------------------------------------------------+
File Name | Content Description
+-----------------------------+------------------------------------------------+
README | README for this subdirectory (this file)
nr.tar.gz | Non-redundant protein sequences from GenPept,
Swissprot, PIR, PDF, PDB, and NCBI RefSeq
nt.tar.gz | Partially non-redundant nucleotide sequences from
all traditional divisions of GenBank, EMBL, and DDBJ
excluding GSS,STS, PAT, EST, HTG, and WGS.
landmark.tar.gz | Proteome of 27 model organisms, see
https://blast.ncbi.nlm.nih.gov/smartblast/smartBlast.cgi?CMD=Web&PAGE_TYPE=BlastDocs#searchSets
16S_ribosomal_RNA | 16S ribosomal RNA (Bacteria and Archaea type strains)
18S_fungal_sequences.tar.gz | 18S ribosomal RNA sequences (SSU) from Fungi type and reference material (BioProject PRJNA39195)
28S_fungal_sequences.tar.gz | 28S ribosomal RNA sequences (LSU) from Fungi type and reference material (BioProject PRJNA51803)
ITS_RefSeq_Fungi.tar.gz | Internal transcribed spacer region (ITS) from Fungi type and reference material (BioProject PRJNA177353)
ITS_eukaryote_sequences.tar.gz| Internal transcribed spacer region (ITS) for eukaryotic sequences
LSU_eukaryote_rRNA.tar.gz | Large subunit ribosomal RNA sequences for eukaryotic sequences
LSU_prokaryote_rRNA.tar.gz | Large subunit ribosomal RNA sequences for prokaryotic sequences
SSU_eukaryote_rRNA.tar.gz | Small subunit ribosomal RNA sequences for eukaryotic sequences
ref_euk_rep_genomestar.gz | Refseq Representative Eukaryotic genomes (1000+ organisms)
ref_prok_rep_genomestar.gz | Refseq Representative Prokaryotic genomes (5700+ organisms)
ref_viruses_rep_genomestar.gz | Refseq Representative Virus genomes (9000+ organisms)
ref_viroids_rep_genomestar.gz | Refseq Representative Viroid genomes (46 organisms)
refseq_protein.*tar.gz | NCBI protein reference sequences
refseq_rna.*tar.gz | NCBI Transcript reference sequences
swissprot.tar.gz | Swiss-Prot sequence database (last major update)
pataa.*tar.gz | Patent protein sequences
patnt.*tar.gz | Patent nucleotide sequences. Both patent databases
are directly from the USPTO, or from the EPO/JPO
via EMBL/DDBJ
pdbaa.*tar.gz | Sequences for the protein structure from the
Protein Data Bank
pdbnt.*tar.gz | Sequences for the nucleotide structure from the
Protein Data Bank. They are NOT the protein coding
sequences for the corresponding pdbaa entries.
taxdb.tar.gz | Additional taxonomy information for the databases
listed here providing common and scientific names
FASTA/ | Subdirectory for FASTA formatted sequences
v4/ | BLAST databases in version 4 (v4). These files are no
longer being updated.
cloud/ | Subdirectory of databases for BLAST AMI; see
http://1.usa.gov/TJAnEt
+-----------------------------+------------------------------------------------+

4. Contents of the /blast/db/FASTA directory

This directory contains FASTA formatted sequence files. The file names
and database contents are listed below. These files must be unpacked before
use. They are provided as a convenience for users needing these sets in
FASTA format. For use with BLAST, it is preferable to use the BLAST database
on the FTP site.

+-----------------------+-----------------------------------------------------+
|File Name | Content Description |
+-----------------------+-----------------------------------------------------+
nr.gz* | non-redundant protein sequence database with entries
from GenPept, Swissprot, PIR, PDF, PDB, and RefSeq
nt.gz* | nucleotide sequence database, with entries from all
traditional divisions of GenBank, EMBL, and DDBJ;
excluding bulk divisions (gss, sts, pat, est, htg)
and wgs entries. Partially non-redundant.
pdbaa.gz* | protein sequences from pdb protein structures
swissprot.gz* | swiss-prot database (last major release)
+-----------------------+---------------------------------------------------+
NOTE:
(1) For screening for vector contamination, use the UniVec database:
ftp://ftp.ncbi.nlm.nih.gov/pub/UniVec/

• marked files have pre-formatted counterparts.

5. Database updates

The BLAST databases are updated regularly. There is no established incremental
update scheme. We recommend downloading the complete databases regularly to
keep their content current.

6. Non-redundant defline syntax

The non-redundant databases are nr, nt and pataa. Identical sequences are
merged into one entry in these databases. To be merged two sequences must
have identical lengths and every residue at every position must be the
same. The FASTA deflines for the different entries that belong to one
record are separated by control-A characters invisible to most
programs. In the example below both entries Q57293.1 and AAB05030.1
have the same sequence, in every respect:

Q57293.1 RecName: Full=Fe(3+) ions import ATP-binding protein FbpC ^AAAB05030.1 afuC
[Actinobacillus pleuropneumoniae] ^AAAB17216.1 afuC [Actinobacillus pleuropneumoniae]
MNNDFLVLKNITKSFGKATVIDNLDLVIKRGTMVTLLGPSGCGKTTVLRLVAGLENPTSGQIFIDGEDVTKSSIQNRDIC
IVFQSYALFPHMSIGDNVGYGLRMQGVSNEERKQRVKEALELVDLAGFADRFVDQISGGQQQRVALARALVLKPKVLILD
EPLSNLDANLRRSMREKIRELQQRLGITSLYVTHDQTEAFAVSDEVIVMNKGTIMQKARQKIFIYDRILYSLRNFMGEST
ICDGNLNQGTVSIGDYRFPLHNAADFSVADGACLVGVRPEAIRLTATGETSQRCQIKSAVYMGNHWEIVANWNGKDVLIN
ANPDQFDPDATKAFIHFTEQGIFLLNKE

Individual sequences are now identifed simply by their accession.version.

For databases whose entries are not from official NCBI sequence databases,
such as Trace database, the gnl| convention is used. For custom databases,
this convention should be followed and the id for each sequence must be
unique, if one would like to take the advantage of indexed database,
which enables specific sequence retrieval using blastdbcmd program included
in the blast executable package. One should refer to documents
distributed in the standalone BLAST package for more details.

7. Formatting a FASTA file into a BLASTable database

FASTA files need to be formatted with makeblastdb before they can be used in local
blast search. For those from NCBI, the following makeblastdb commands are
recommended:

For nucleotide fasta file: makeblastdb -in input_db -dbtype nucl -parse_seqids
For protein fasta file: makeblastdb -in input_db -dbtype prot -parse_seqids

In general, if the database is available as BLAST database, it is better to use the
preformatted database.

8. Technical Support

Questions and comments on this document and NCBI BLAST related questions
should be sent to the blast-help group at:
blast-help@ncbi.nlm.nih.gov

For information about other NCBI resources/services, please send email to
NCBI User Service at:
info@ncbi.nlm.nih.gov

[anneng]

http://arep.med.harvard.edu/seqanal/db.html
What is Redundancy?
A key concept in comparing databases is the issue of redundancy. Many databases try to be "non-redundant". Unfortunately, biological data is too complex to fit a simple definition of redundancy. Are two alleles of the same locus redundant? Two isozymes in the same organism? The same locus in two closely related organisms? Hence, each "non-redundant" database has its own definition of redundancy. Some use automated measures, while others use manual culling; the former are amenable to large projects, the latter give higher quality. Other databases don't attempt to be non-redundant, but rather sacrifice this goal in favor of ensuring completeness.
Databases
Nucleotide (DNA & RNA)
nr (NCBI)
The nr nucleotide database maintained by NCBI as a target for their BLAST search services is a composite of GenBank, GenBank updates, and EMBL updates.
Non-redundant: Entries with absolutely identical sequences have been merged.
GenBank / EMBL / DDBJ
In theory, GenBank, the EMBL Datalibrary, and the DNA Databank of Japan (DDBJ) are just names for the same database. In reality, small timelags in propagating data between the database centers causes minor differences in these databases. However, if one of these libraries is merged with the updates to all of these databases, a complete set of sequences is formed.
Redundant: Little to no attempts to reduce redundancy
dbEST (Boguski, Lowe, & Tolstoshev. Nature Genetics 4:332 1993) is a library of Expressed Sequence Tags (Science 252:1651), single-pass cDNA sequences generated from automated sequencers.
CAUTION: ESTs are blindly sequenced from cDNA libraries with little or no human intervention; they are therefore likely to contain sequencing errors and are frequently contaminated with heterologous sequences and transcribed repetitive elements.

Redundant: no attempts made to reduce redundancy
Protein
nr (NCBI)
The nr protein database maintained by NCBI as a target for their BLAST search services is a composite of SwissProt, SwissProt updates, PIR, PDB. Entries with absolutely identical sequences have been merged.
SwissProt
SwissProt is maintained by Amos Bairoch at the University of Geneva. SwissProt is a highly-curated, highly-crossreferenced, non-redundant database. Unfortunately, the cost of this labor-intensive quality enhancement process is that not every sequence is in SwissProt. If you wish to look up information about a sequence, SwissProt is the first place to look.
Non-redundant: manual curation used to provide only one entry per protein product; variants are annotated in entry.
Highly-cross-referenced to other databases.
PIR
The Protein Identification Resource was originated by the late Margaret Dayhoff. It attempts to enjoy the advantages of a complete and a non-redundant database.
Non-redundant: PIR1 section contains only one entry per protein product.
Redundant: Complete database (PIR1+PIR2+PIR3) has many redundancies
PDB
The Protein Data Bank, maintained by Brookhaven National Laboratory (Long Island, New York, USA), contains all publically available solved protein structures. Searches against the pdb can be used to ask whether any known 3D structures are similar to your query protein.
Non-redundant: Only the "best" determination of a given structure is left in the database; however, multiple structures for one molecule may exist due to other components (i.e. one entry uncomplexed, one complexed).
OWL
Prot. Eng. 3:153
Non-redundant: Automatically generated from component databases (see reference for further info).
Protein Motifs
Prosite
Prosite is a database of protein motifs maintained by Amos Bairoch at the University of Geneva (NAR 19:2241, 1991). Each motif (defined by either a regular expression or a profile) is accompanied by a description of the motif and what is known about it's biology, as well as a listing of the true positive, false negative, and false positive SwissProt entries for the pattern.
BLOCKS
BLOCKS is a database developed by Steve Henikoff and colleagues. A block is a gap-free multiple alignment of sequences based on Prosite (Henikoff & Henikoff, NAR 19:6565 1991).

[anneng]

https://www.ncbi.nlm.nih.gov/books/NBK279670/

[anneng]

blastdbv5.pdf

[anneng]

https://docs.oracle.com/cd/B19306_01/datamine.102/b14340/blast.htm
Oracle对blast的支持

[anneng]

Bioinformatics_ introduction to using BLAST with Ubuntu.pdfBioinformatics_ managing BLAST data sources.pdf

[anneng]

https://dbsloan.github.io/TS2019/exercises/local_blast.html
Running Local BLAST and Parsing Output

makeblastdb -in Ecoli.proteins.fas -dbtype prot

makeblastdb -in Ecoli.genome.fas -dbtype nucl

blastn -task blastn  -query Salmonella.genome.fas -db Ecoli.genome.fas -evalue 1e-20 -num_threads 4 -out blastn.txt

pdf ("my_dotplot.pdf")
plot (blastnData$Query_Start, blastnData$Hit_Start, cex = .25)
dev.off()
quit()

[anneng]

Extracting data from BLAST databases with blastdbcmd
https://www.ncbi.nlm.nih.gov/books/NBK279689/

[anneng]

Preformatted BLAST vs Fasta
https://www.ncbi.nlm.nih.gov/books/NBK62345/
Getting the preformatted database files
Preformatted BLAST database files offer several advantages over the FASTA files:

The preformatted databases are broken into smaller volumes and therefore can be downloaded more readily with fewer errors
A convenient Perl script (update_blastdb.pl found in the bin directory of a locally installed blast+ package) is available to simplify the download of these preformatted databases
Preformatted database files remove the makeblastdb formatting steps, and saves valuable processing time and diskspace
Taxonomic information is encoded within the preformatted databases and can be used to limit the scope of a blast search, and sequence retrieval, and scientific name addition through the included taxdb files
Sequences in FASTA format can be generated easily from the preformatted databases using the blastdbcmd utility when needed

[anneng]

Annotating BLAST Reports with Taxonomy Information
https://www.mathworks.com/matlabcentral/mlc-downloads/downloads/submissions/15970/versions/2/previews/taxoblastdemo/html/taxoblastdemo.html?access_key=

[anneng]

https://github.com/lskatz/taxdb
一个工具　可以把taxdump导入sqlite

[anneng]

The BLAST taxonomy database is required in order to print the scientific name, common name, blast name, or super kingdom as part of the BLAST report or in a report with blastdbcmd. The BLAST database contains only the taxid (an integer) for each entry, and the taxonomy database allow BLAST to retrieve the scientific name etc. from a taxid. The BLAST taxonomy database consists of a pair of files (taxdb.bti and taxdb.btd) that are available as a compressed archive from the NCBI BLAST FTP site (ftp://ftp.ncbi.nlm.nih.gov/blast/db/taxdb.tar.gz). The update_blastdb.pl script can be used to download and update this archive; it is recommended that the uncompressed contents of the archive be installed in the same directory where the BLAST databases reside. Assuming proper file permissions and that the BLASTDB environment variable contains the path to the installation directory of the BLAST databases, the following commands accomplish that:

Download the taxdb archive

perl update_blastdb.pl taxdb

Install it in the BLASTDB directory

gunzip -cd taxdb.tar.gz | (cd $BLASTDB; tar xvf - )

[zhangfanglin]

nr、nt导出规范

**%T %a %i  %t %s**    
1、物种ID 
2、accession 序列ID           
3、sequence title
4、描述
5、序列

上面的导出%i 应该是序列id 我们可以不用这个字段
%a %t %T %s

[anneng]

-outfmt <String>
Output format, where the available format specifiers are:
%f means sequence in FASTA format
%s means sequence data (without defline)
%a means accession
%g means gi
%o means ordinal id (OID)
%i means sequence id
%t means sequence title
%l means sequence length
%h means sequence hash value
%T means taxid
%X means leaf-node taxids
%e means membership integer
%L means common taxonomic name
%C means common taxonomic names for leaf-node taxids
%S means scientific name
%N means scientific names for leaf-node taxids
%B means BLAST name
%K means taxonomic super kingdom
%P means PIG
%m means sequence masking data.
Masking data will be displayed as a series of 'N-M' values
separated by ';' or the word 'none' if none are available.
If '%f' is specified, all other format specifiers are ignored.
For every format except '%f', each line of output will correspond
to a sequence.
Default = `%f'

[anneng]

如果想把一个fasta文件中的序列都当作一个物种对待 那么可以使用taxid参数
合并两个数据库：
makeblastdb -in mysequences.fna -dbtype nucl -title "some sequences I found" -out mysequences -parse_seqids
blastdb_aliastool -dblist nt mysequences -dbtype nucl -title "nt database + my own sequences" -out ntandmore
如果有多个fasta 文件 每个文件是一个物种 可以先分别建库 然后用blastdb_aliastool合并