Installation guide: configuration

From ZENBU documentation wiki
Jump to: navigation, search

This section of the Installation guide we will discuss the steps of configuring the server, creating the support mysql databases, loading data via the commandline, and adding new genomes.

Configuring Apache to execute ZENBU cgi

Depending on how your apache was previously configured, you will most likely need to modify the apache config to allow ZENBU cgi webservices to execute. The general procedure for configuring cgi in Apache 2.2 is describe here http://httpd.apache.org/docs/2.2/en/howto/cgi.html and http://httpd.apache.org/docs/trunk/howto/cgi.html


On debian7 linux systems for example, apache2 is configured in /etc/apache2/. There is a directory structure as follows 

-rw-r--r-- 1 root root  9640 Mar  4  2013 apache2.conf
drwxr-xr-x 2 root root  4096 Feb  7 20:05 conf.d
-rw-r--r-- 1 root root  1465 Mar  4  2013 envvars
-rw-r--r-- 1 root root 31063 Oct 21  2012 magic
drwxr-xr-x 2 root root  4096 Feb  8 02:11 mods-available
drwxr-xr-x 2 root root  4096 Feb  7 21:33 mods-enabled
-rw-r--r-- 1 root root   750 Mar  3  2013 ports.conf
drwxr-xr-x 2 root root  4096 Feb  8 02:13 sites-available
drwxr-xr-x 2 root root  4096 Feb  8 02:13 sites-enabled

Check mods-enabled and make sure there are the following for cgi and fcgi 

 rwxrwxrwx 1 root root 27 Feb  7 20:05 cgid.conf -> ../mods-available/cgid.conf
 rwxrwxrwx 1 root root 27 Feb  7 20:05 cgid.load -> ../mods-available/cgid.load
 rwxrwxrwx 1 root root 28 Feb  7 21:33 fcgid.conf -> ../mods-available/fcgid.conf
 rwxrwxrwx 1 root root 28 Feb  7 21:33 fcgid.load -> ../mods-available/fcgid.load

Then in the sites-enabled subdirectory there should be a default config 

 000-default

add the following section 

       <Directory "/var/www/zenbu/">
               Options +ExecCGI
               AddHandler cgi-script .cgi .fcgi .pl
               AddHandler fcgid-script .fcgi
       </Directory>

Restart your apache server 

sudo /etc/init.d/apache restart

Finally check to make sure your apache/cgi is setup and running correctly. In a web browser point at your newly installed zenbu and go to the cgi directory. For example 

 http://localhost/zenbu/cgi
 http://localhost/zenbu/cgi/eedb_search.cgi  
 http://localhost/zenbu/cgi/eedb_search.fcgi

If apache is setup and the compile was successful, it should execute and return an HTML page with some status information. The eedb_search.fcgi should return with increasing "invocation number" and the same PID on successive reloads.

If there are problems, check the apache error log 

 sudo tail -f /var/log/apache2/error.log

Mysql support database creation

In version 2.8 and 2.9 ZENBU used the mysql databases for many aspects. In version 2.10 and on, we are using mysql only for the zenbu_users database for user/collaboration and configuration management. This documentation is for the older 2.9 version. For 2.10 please ignore the references to zenbu_public_share and zenbu_curated databases

ZENBU uses a mysql server for the user system, visualization view/track/script config saving, data sharing and for genome sequence. ZENBU needs two specific users to be added onto the mysql server. The user "read" must be created with password "read". The user "zenbu_admin" can have any password but we recommend using the password "zenbu_admin" 

CREATE USER 'read'@'%' IDENTIFIED BY 'read';
CREATE USER 'zenbu_admin'@'%' IDENTIFIED BY 'zenbu_admin';

The "read" user will need to have the follow grant permissions for each relevant database 

GRANT SELECT, CREATE TEMPORARY TABLES, LOCK TABLES on specific_db.* to 'read'@"%";


STEP 2: next you need to create the three databases mentioned above

  • create "user" database maybe like "zenbu_users"
  • create "public sharing" database maybe like "zenbu_public_share"
  • create "curation" database maybe like "zenbu_curated"

you create a zenbu database by the mysql commands. You may choose different database names if you would like. 

 CREATE DATABASE zenbu_users;
 CREATE DATABASE zenbu_public_share;
 CREATE DATABASE zenbu_curated;

 GRANT SELECT, CREATE TEMPORARY TABLES, LOCK TABLES on zenbu_curated.* to 'read'@"%";
 GRANT SELECT, CREATE TEMPORARY TABLES, LOCK TABLES on zenbu_public_share.* to 'read'@"%";
 #DO NOT do give read:read grants to the user database
 GRANT SELECT, CREATE TEMPORARY TABLES, LOCK TABLES, INSERT, UPDATE, CREATE, ALTER, DELETE, INDEX on zenbu_users.* to 'zenbu_admin'@"%";
 GRANT SELECT, CREATE TEMPORARY TABLES, LOCK TABLES, INSERT, UPDATE, CREATE, ALTER, DELETE, INDEX on zenbu_curated.* to 'zenbu_admin'@"%";
 GRANT SELECT, CREATE TEMPORARY TABLES, LOCK TABLES, INSERT, UPDATE, CREATE, ALTER, DELETE, INDEX on zenbu_public_share.* to 'zenbu_admin'@"%";

for the "user database" 

 cmdline> mysql -hXXXX -uYYYY -pZZZZ -PWWWW   zenbu_users  < /zenbu/src/ZENBU_x.x.x/sql/schema.sql
 cmdline> mysql -hXXXX -uYYYY -pZZZZ -PWWWW   zenbu_users  < /zenbu/src/ZENBU_x.x.x/sql/system_tables.sql
 cmdline> /zenbu/bin/zenbu_register_peer -url "mysql://zenbu_admin:zenbu_admin@mysql_hostname:3306/zenbu_users" -newpeer

for the "collaboration" databases 

 cmdline> mysql -hXXXX -uYYYY -pZZZZ -PWWWW   zenbu_public_share  < /zenbu/src/ZENBU_x.x.x/sql/schema.sql
 cmdline> mysql -hXXXX -uYYYY -pZZZZ -PWWWW   zenbu_curated  < /zenbu/src/ZENBU_x.x.x/sql/schema.sql
 cmdline> /zenbu/bin/zenbu_register_peer -url "mysql://zenbu_admin:zenbu_admin@mysql_hostname:3306/zenbu_public_share" -newpeer
 cmdline> /zenbu/bin/zenbu_register_peer -url "mysql://zenbu_admin:zenbu_admin@mysql_hostname:3306/zenbu_curated" -newpeer

Note that ZENBU uses a URL-like notation to pointing to sql databases. The format is 

 mysql://<mysql_username>:<mysql_password>@<mysql_hostname>:<mysql_port>/<database_name>
 sqlite:///<full_path_to_sqlite_database>

in addition you will need to enable zenbu_admin with some "super user" privileges 

GRANT CREATE, ALTER, DELETE, CREATE TEMPORARY TABLES, INDEX, SHOW DATABASES, LOCK TABLES on *.* to 'zenbu_admin'@"%";
GRANT CREATE, ALTER, DELETE, CREATE TEMPORARY TABLES, INDEX,  INSERT, SHOW DATABASES, UPDATE, SELECT, LOCK TABLES on zenbu*.* to 'zenbu_admin'@"%";

if you can use a ZENBU specific mysql server, it will be easier to just do the following 

GRANT CREATE, ALTER, DELETE, CREATE TEMPORARY TABLES, INDEX,  INSERT, SHOW DATABASES, UPDATE, SELECT, LOCK TABLES on *.* to 'zenbu_admin'@"%";

Server configuration

First step, copy the example server configuration file to the /zenbu/server_config directory 

cp /zenbu/src/ZENBU_2.9.0/www/zenbu/cgi/EXAMPLE_eedb_server_config.xml /zenbu/server_config/zenbu_2.9.0_config.xml
cd /zenbu/server_config
ln -s zenbu_2.9.0_config.xml active_config.xml

Next you will need to edit the file related to your specific server setup. The server configuration file is in XML and hopefully is easy to read. Here is an example from the ZENBU virtual machine 

 <zenbu_server_config>
   <eedb_root>/zenbu/src/ZENBU_2.9.0/</eedb_root>
   <eedb_user_rootdir>/zenbu/users</eedb_user_rootdir>
   <cache_dir>/zenbu/cache</cache_dir>

   <user_db>mysql://zenbu_admin:zenbu_admin@localhost:3306/zenbu_vbox_users</user_db>
   <user_admin_password>zenbu_admin</user_admin_password>

   <smtp_server>smtps://postman.riken.jp</smtp_server>
   <smtp_user></smtp_user>
   <smtp_password></smtp_password>
   <smtp_from>zenbu@riken.jp</smtp_from>
   <note> you should change to your local SMTP server. Zenbu use SMTP to send emails to users for account creation and notifications</note>

   <federation_seeds>
       <seed>zenbu://fantom.gsc.riken.jp/zenbu</seed>
   </federation_seeds>

   <public_collaboration>
       <database>mysql://read:read@localhost:3306/zenbu_vbox_public_share</database>
       <name>public sharing - ZENBU VBOX server</name>
       <description>collaboration for sharing public data and public views/tracks/scripts on ZENBU virtual machine</description>
   </public_collaboration>

   <curated_collaboration>
       <database>mysql://read:read@localhost:3306/zenbu_vbox_curated</database>
       <name>curated - ZENBU VBOX server</name>
       <description>curated data collaboration for ZENBU virtual machine</description>
       <curators>
         <user>jessica.severin@gmail.com</user>
       </curators>
   </curated_collaboration>

   <default_genome>hg19</default_genome>

   <session_name>ZENBU_VBOXSERVER_SESSID</session_name>
   <server_name>localhost</server_name>
   <web_root>http://localhost/zenbu</web_root>
   <web_uuid>4e6958e6-f404-4219-ac90-0d7c95f63742</web_uuid>

 </zenbu_server_config>

The three databases you created above are named in this configuration file.


In addition there is another perl specific configuration file used by the OpenID login processes. This configuration file exists inside the zenbu/cgi directory and is called eedb_server.conf Here is the example from the ZENBU virtual machine. In a future release we are planning to consolidate all the configuration into the XML formated file. 

[
  { TYPE  => 'EEDB_URL',
    'user_db' => 'mysql://zenbu_admin:zenbu_admin@localhost:3306/zenbu_vbox_users',
    'session_name' => "ZENBU_VBOXSERVER_SESSID",
  },
  { TYPE  => 'EEDB_ENV',
    'EEDB_ROOT' => "/zenbu/src/ZENBU_2.9.0/",
    'EEDB_USER_ROOTDIR' => "/zenbu/users",
  },
  { TYPE  => 'ZENBU_WEB',
    'WEB_ROOT'    => "http://localhost/zenbu",
    'SERVER_NAME' => "localhost"
  },
  { TYPE => 'END' }
]

Running background processing agents

Although ZENBU is based on real-time stream processing, it still requires background processes for data-upload and whole-genome track building. There are two different agents, but they are launched from a single script.

Simply open another terminal window and run this command as root or sudo. This should run in the background at all times. 

 sudo /zenbu/bin/zenbu_agent_launcher.sh &

Do not run this script in a cron job. It internally performs a periodic launch of the required agents every 10 seconds. The code of the agent launcher is a very simple infinite loop, and can be modified depending on your server hardware. You can decrease the sleep frequency to reduce latency for job pickup. Or increase the -loadlimit if you want more of your CPU cycles devoted to background track-building compared to web-server responsiveness. 

#!/bin/bash
while [ 1==1 ]; do  
echo -n "launch zenbu agents :: "  
date
/zenbu/bin/zenbu_track_builder -build -buildtime 583 -loadlimit 0.5&
/zenbu/bin/zenbu_job_runner -run&
sleep 10;
done
Retrieved from "https://zenbu-wiki.gsc.riken.jp/zenbu/wiki/index.php?title=Installation_guide:_configuration&oldid=4006"