intermine和bluegene技术验证

anneng

1.i架构

2.安装
http://intermine.org/im-docs/docs/get-started/tutorial/index

anneng

biotestmine验证
biotestmine是intermine的一个测试项目
https://github.com/intermine/biotestmine

验证过程：
clone项目
git clone https://ghproxy.com/https://github.com/intermine/biotestmine.git
1.安装perl依赖

sudo cpan XML::Parser::PerlSAX Text::Glob Cwd Getopt::Std

cpan网络不稳定 perl的包都可以直接用ubuntu的apt安装

2.安装postgresql
最好不要用root 使用ubuntu当前用户在pg中创建一个role

CREATE USER role_name(和ubuntu的一个本地用户同名);
alter role role_name with CREATEDB;

参考
https://www.digitalocean.com/community/tutorials/how-to-use-roles-and-manage-grant-permissions-in-postgresql-on-a-vps-2
3.安装Solr

//下载时间比较长 找我获取
wget https://archive.apache.org/dist/lucene/solr/8.9.0/solr-8.9.0.tgz
bash solr-8.9.0/bin/install_solr_service.sh solr-8.9.0.tgz
sudo ufw allow 8983
./bin/solr create_core -c demo
./bin/post -c demo2 example/exampledocs/*xml

http://192.168.0.157:8983/solr/#/demo2/core-overview
biotestmine怎么使用Solr的还需要细看下
https://intermine.readthedocs.io/en/latest/system-requirements/software/solr/
4.创建配置目录
mkdir ~/.intermine
cp biotestmine/data/biotestmine.properties ~/.intermine
修改PSQL_USER and PSQL_PWD为PG的新建role_name(实际上就是ubuntu的一个本地用户)

5.执行安装
DB_ENCODING=UTF-8 ./setup.sh
由于maven比较慢使用腾讯镜像
vi build.gradle

buildscript {
    repositories {
**maven { url 'https://mirrors.cloud.tencent.com/nexus/repository/maven-public/' }**
        mavenCentral()
    }

    dependencies {
        classpath group: 'org.intermine', name: 'plugin', version: System.getProperty("imVersion")
        classpath 'com.bmuschko:gradle-cargo-plugin:2.8.0'
    }
}

ext {
    mineRelease = hasProperty('release') ? "." + release : ''
    mineName = "biotestmine"
    minePropertyFileName = "${mineName}.properties$mineRelease"
    minePropertyFile = "${System.env.HOME}/.intermine/$minePropertyFileName"
    if( !file(minePropertyFile).exists() ) {
        minePropertyFile = "$project.rootDir/dbmodel/resources/biotestmine.properties"
    }
    println "Using properties file: '$minePropertyFile'"
}

subprojects {
    apply plugin: 'java'
    apply plugin: 'maven'

    sourceCompatibility = 1.8
    targetCompatibility = 1.8

    repositories {
**maven { url 'https://mirrors.cloud.tencent.com/nexus/repository/maven-public/' }**
        mavenLocal()
        mavenCentral()
        maven {
            url "https://clojars.org/repo"
        }
    }

其他参考
http://intermine.org/im-docs/docs/get-started/tutorial/index
https://github.com/intermine/biotestmine

anneng

我们使用intermine来构建这三个薯类物种
https://data.jgi.doe.gov/refine-download/phytozome
manihot esculenta v8.1 木薯 taxid 3983
https://www.ncbi.nlm.nih.gov/search/all/?term=manihot+esculenta
Solanum tuberosum v6.1 马铃薯
https://www.ncbi.nlm.nih.gov/search/all/?term=Solanum+tuberosum taxid 4113

甘薯: taxid:4120
https://www.ncbi.nlm.nih.gov/search/all/?term=Ipomoea batatas
（http://public-genomes-ngs.molgen.mpg.de/sweetpotato/DOWNLOADS/）
lpomoea batatas
ipoBat4.fa 基因组
ipoBat4.transcript.bed12 基因
ipoBat4.transcript.fa 转录本
ipoBat4_final.mod.vcf.gz 变异

zhanglu

Intermine+bluegene+jbrow部署配置

intermine
（1）编译

git clone https://github.com/intermine/biotestmine
cd biotestmine
./gradlew war

注意：执行编译打包时会因为网络问题，下载依赖失败，配上aliyun或者腾旭仓库后会下载依赖会提速，但intermine的jar包无法下载；

（2）导入数据：
启动数据库

docker run -d -p 8983:8983 -v /data_raid1/bioinfo/app/solr-7.2.1/data:/var/solr/data -t --privileged=true solr:8.11.1

docker run  -v /data_raid1/bioinfo/app/postgresql_data:/var/lib/postgresql/data -e POSTGRES_PASSWORD=postgres -p 5432:5432 -d postgres:10

修改数据库配置文件

mkdir ~/.intermine
cp ./biotestmine/data/biotestmine.properties ~/.intermine/

使用样例数据：biotestmine/data/malaria-data.tar.gz

tar -zxvf malaria-data.tar.gz
cp biotestmine/data/project.xml biotestmine

配置project数据路径
导入数据：

./gradlew buildDB
./gradlew integrate -Psource=uniprot-malaria --stacktrace
./gradlew integrate -Psource=malaria-gff --stacktrace
./gradlew integrate -Psource=malaria-chromosome-fasta --stacktrace
./gradlew integrate -Psource=go --stacktrace
./gradlew integrate -Psource=go-annotation --stacktrace
./gradlew integrate -Psource=update-publications --stacktrace
./gradlew integrate -Psource=entrez-organism --stacktrace

./gradlew postprocess -Pprocess=create-references
./gradlew postprocess -Pprocess=transfer-sequences
./gradlew postprocess -Pprocess=summarise-objectstore

solr create -c biotestmine-search
solr create -c biotestmine-autocomplete

./gradlew postprocess -Pprocess=create-search-index
./gradlew postprocess -Pprocess=create-autocomplete-index

（3）启动biotestmine

docker run   -p 8080:8080 -d -v /data_raid1/bioinfo/app/apache-tomcat-10.1.7/webapps/:/usr/local/tomcat/webapps/ tomcat:9.0

将war包映射到tomcat下，使用docker方式启动；
配置文件介绍
（1）配置postgre数据库：webapps/biotestmine/WEB-INF/classes/intermine.properties
（2）配置solr数据库,在webapps/biotestmine/WEB-INF/classes/ 新建keyword_search.properties

index.solrurl = http://192.114.101.5:8983/solr/biotestmine-search
index.batch.size = 1000
autocomplete.solrurl = http://192.114.101.5:8983/solr/biotestmine-autocomplete

（3）biotestmine/WEB-INF/web.properties

webapp.baseurl=http://103.114.101.5:8080
webapp.deploy.url=http://103.114.101.5:8080

注意：
webapps/biotestmine/WEB-INF/classes/intermine.properties下有webapp.baseurl和webapp.deploy.url的配置，但是web.properties优先级更高；

zhanglu

gradle.properties添加
org.gradle.jvmargs=-Xmx10096m -XX:MaxPermSize=10096m -XX:+HeapDumpOnOutOfMemoryError -Dfile.encoding=UTF-8

zhanglu

docker run -p 8080:8080 -d -v /data_raid1/bioinfo/app/apache-tomcat-10.1.7/webapps/:/usr/local/tomcat/webapps/ tomcat:9.0

docker run -p 5000:5000 --env-file /data_raid1/bioinfo/app/bluegene/bluegenes.env -v /data_raid1/bioinfo/app/bluegene/tools:/tools -d --restart unless-stopped intermine/bluegenes:latest

docker run -d -v /data_raid1/bioinfo/app/jbrowse/jbrowse_conf.json:/conda/opt/jbrowse/jbrowse_conf.json -v /data_raid1/bioinfo/app/jbrowse/index.html:/conda/opt/jbrowse/index.html -p 8079:80 bokebiotech/jbrowser:v1.16.11

zhanglu

docker run -p 5000:5000 --env-file /data_raid1/bioinfo/app/bluegene/bluegenes.env -v /data_raid1/bioinfo/app/bluegene/tools:/tools -d --restart unless-stopped intermine/bluegenes:latest

docker run -d -v /data_raid1/bioinfo/app/jbrowse/jbrowse_conf.json:/conda/opt/jbrowse/jbrowse_conf.json -v /data_raid1/bioinfo/app/jbrowse/index.html:/conda/opt/jbrowse/index.html -p 8079:80 bokebiotech/jbrowser:v1.16.11