Ubuntu enforce memory freedom

Sometimes it occurs that a device runs full with memory. Simply deleting the files on the device doesn’t help. Ubuntu stills shows that the device is 100% full, even if there are no files on it anymore. This command usually helps to give free the memory.

sudo tune2fs -m 0 /dev/sda5

If it doesn’t help you need to execute this command as well. It quits all processes which are using the device.

sudo fuser -km /mnt/share

Docker Introduction

Docker is one of the most promising technologies these days. It is a container technology based on Linux. A very lightweight form of virtualization. Docker containers can be as small as 50 MB. Much smaller than traditional VMs.

The Docker project was started in March 2013 by dotCloud. In the mean while the makers of dotCloud sold dotCloud to CloudControl and raised $55 Million to focus only on the Docker development.

Check out my slides to “Docker Introduction”. I did this talk at the Webmontag in Mannheim. Feedback was very good.

I’m using Docker since beginning of 2014 in production and I love it. It’s a great technology!

daemon script

This shell script runs forever and checks if the rails worker is running and if not it starts it again:

while :
do
  if ps ax | grep -v grep | grep 'rails worker' > /dev/null
  then
      echo "service running, everything is fine"
      sleep 5
  else
      echo "service is not running. Lets start again"
      cd $APP_ROOT
      $BUNDLE exec unicorn_rails -D -c $CONF
      echo "restarted on $(cat /rails/pids/unicorn.pid)"
      sleep 15
  fi
done

Linux Tage 2013 in Berlin

Linux Tage 2013 in Berlin is the biggest Linux Conference in Germany. It goes from Wednesday to Saturday. There are all day long sessions & tech talks. And of course a big exhibition hall, where you can meet some of the vendors. It is a place there .com meets .org.

I have been there today and will be there tomorrow and on Saturday. You can find me at the Yunicon stand in the exhibition hall. Yunicon is an awesome private cloud provider from Berlin. They have the coolest stand in the exhibition hall because they offer free cocktails! 🙂

LinuxTage_VersionEye

I am doing every day at 15:30 a talk about Continuous Updating with VersionEye.

LinuxTage_VersionEye_Talk

If you want to learn something about Continuous Updating or VersionEye, join the talk. Grab a free cocktail and ask me your questions. I am happy to answer them.

LinuxTage_VersionEye_TechTalk

See you tomorrow.

Capistrano + Rails + Unicorn + NGinx + RVM

Capistrano is a ruby based deployment tool which executes commands in parallel on multiple remote machines, via the SSH protocol. With Capistrano you can deploy your application on N server with one single command from your dev machine. You don’t need to login via SSH to your server.

Screen Shot 2013-01-15 at 8.15.33 PM

If you don’t deploy your app on Heroku or CloudControl, if you have dedicated servers or you are using AWS, than Capistrano is the way to go.

Before you start with Capistrano, you have to implement SSH with authentification keys instead of password. In that way you can just login to your server with a simple “ssh user@server” without password. That is possible if your public ssh certificates are on the server. In that way the server “knows” you.

First you need to add the Gem to your Gemfile.

gem 'capistrano'
gem 'rvm-capistrano'

I am adding here “rvm-capistraon”, too. Because I am working with RVM on my dev machine and on the server. Now you have to run bundler.

bundle install

As next step you have to capify your rails project. Just run:

capify .

That will create some files in your project.

[add] writing './Capfile'
[add] writing './config/deploy.rb'
[done] capified!

Your Capfile is loading the deploy.rb script. The Capfile should look like this:

load 'deploy'
# Uncomment if you are using Rails' asset pipeline
# load 'deploy/assets'
load 'config/deploy'

The magic happens in the deploy.rb file. The full documentation to this script you will find here: https://github.com/capistrano/capistrano. It basically executes a couple of commands on your server. It fetches the current code from your GIT server, runs bundler, rake db:migrate, precompiles your assets and runs the ruby app server.

Here is my deploy.rb with some additional comments.


# Automatically precompile assets
load "deploy/assets"

# Execute "bundle install" after deploy, but only when really needed
require "bundler/capistrano"

# RVM integration
require "rvm/capistrano"

# Name of the application in scm (GIT)
set :application, "your_app"
set :repository, "git@github.com:your_company/your_project/project.git"

# Source Control Management
set :scm, :git

set :deploy_to, "/var/www/#{application}"

# server there the web server is running (nginx)
role :web, "192.168.0.12"

# server there the app server is running (unicorn)
role :app, "192.168.0.12"

# server there the db is running
# This is where Rails migrations will run
role :db, "192.168.0.12", :primary => true

set :rails_env, :production

# user on the server
set :user, "project_user"
set :use_sudo, false

# Target ruby version
set :rvm_ruby_string, '1.9.3'

# System-wide RVM installation
set :rvm_type, :system

# Apply default RVM version for the current account
after "deploy:setup", "deploy:set_rvm_version"

namespace :deploy do

  task :set_rvm_version, :roles => :app, :except => { :no_release => true } do
    run "source /etc/profile.d/rvm.sh && rvm use #{rvm_ruby_string} --default"
  end

  task :start, :roles => :app, :except => { :no_release => true } do
    run "/etc/init.d/unicorn start"
  end

  task :stop, :roles => :app, :except => { :no_release => true } do
    run "/etc/init.d/unicorn stop"
  end

  task :restart, :roles => :app, :except => { :no_release => true } do
    run "/etc/init.d/unicorn restart"
  end

  # Precompile assets
  namespace :assets do
    task :precompile, :roles => :web, :except => { :no_release => true } do
      run %Q{cd #{latest_release} && #{rake} RAILS_ENV=#{rails_env} #{asset_env} assets:precompile}
    end
  end

end

The script for starting and stoping unicorn you can find here: https://robert-reiz.com/2012/02/29/running-unicorn-as-a-service-on-debian-linux/.

On the Linux server you should install a couple things before you deploy:

apt-get install make
apt-get install g++
gem update --system
gem install bundler

Now you can deploy with:

cap deploy

And this command shows you all possible Capistrano tasks:

cap -T

Let me know if you have questions.

Rewriting URLs with Nginx

Rewriting URLs with Apache HTTPD can be pretty ugly. With Nginx it is a breeze. Take a look to this example:

server {
 listen 80;
 server_name my_old_domain.com;
 rewrite ^/(.*) https://www.my_new_domain.com/$1 permanent;
}

This will not just redirect your old URL to your new URL. It passes all the parameter to your new URL, too. If you are migrating a Web Application to new URL, this what you want. All old bookmarked URL are still working with this.

Load balancing with Nginx And Unicorn

Nginx is a pretty awesome lightweight Web Server for Linux. You can use it to deliver static web content. But you can use at as a load balancer, too. If you have a Ruby on Rails Application running on multiple unicorn servers, than Nginx can do the load balancing for the unicorn servers.

Just add the following lines to your “/etc/nginx/conf.d/default.conf”:

upstream unicorns { 
  ip_hash; 
  server XXX.XXX.XXX.XXX:8080; 
  server <IP_ADDRESS>:<PORT>; 
}

XXX stands for an IP Address. You can add as many server lines as you want. The “ip_hash” is not mandatory. This just if you want to force nginx to send the requests from a particular user every time to the same app server. If your Web Application is REST ful and Stateless you don’t need it. Otherwise you should add the line.

Now you just have to add one block for your domain:

server {
  listen 80;
  server_name my-awesome-domain.com;
  location / { 
    proxy_pass http://unicorns; 
  } 
}

And that’s it.

One more thing. If you want limit the file upload size or you have problems with big HTML5 AJAX requests, you should add a little bit more. Here is a more complete example:

server {
  listen 80;
  server_name my-awesome-domain.com;
  client_max_body_size 1M; 
  proxy_buffer_size 128k; 
  proxy_buffers 4 256k; 
  proxy_busy_buffers_size 256k; 
  location / { 
    proxy_pass http://unicorns; 
  } 
}

Bad Gateway with NGinx & Unicorn

I am using NGinx as proxy to load balance the traffic. Behind Nginx their are some unicorn servers. On Friday I did some changes on the server and I realized that I am always running into a Bad Gateway error. The way from Nginx to unicorn worked. But the way back not really.

I googled for it and I found out that the problem is the buffer size on NGinx. After setting the buffer sizes up like this here, it worked fine for me.

proxy_buffer_size   128k;
proxy_buffers   4 256k;
proxy_busy_buffers_size   256k;

You have to add this to your default.conf file in “/etc/nginx/conf”.

Running Unicorn as a Service on Debian Linux

Unicorn is a very fast app server for Ruby on Rails. You just have to go to the APP_ROOT and type in “unicorn_rails”. And your app is running.

The problem with that is, if your server reboots your app is down. And you have to start it again by hand. That is not so cool!

Wouldn’t it be cool if you could start your Ruby on Rails like that:

/etc/init.d/unicorn start

and stop it like that:

/etc/init.d/unicorn stop

like all other services on Linux? And after a reboot unicorn starts automatically.

That is possible. Just use this script here for starting and stopping unicorn.

#!/bin/bash

### BEGIN INIT INFO
# Provides: unicorn
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start daemon at boot time
# Description: Enable service provided by daemon.
### END INIT INFO

PATH="/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin"
BUNDLE=/usr/local/bin/bundle
UNICORN=/usr/local/bin/unicorn_rails
KILL=/bin/kill
APP_ROOT=/var/www/my_app/current
PID=$APP_ROOT/pids/unicorn.pid
OLD_PID=$APP_ROOT/pids/unicorn.pid.oldbin
CONF=$APP_ROOT/unicorn.conf.rb
GEM_HOME=/var/www/my_app/shared/bundle/ruby/2.1.0/gems/

sig () {
  test -s "$PID" && kill -$1 `cat $PID`
}

case "$1" in
  start)
    echo "Starting unicorn rails app ..."
    cd $APP_ROOT
    if [ $USER == "ubuntu" ]
    then
      $BUNDLE exec unicorn_rails -D -c $CONF
    else
      su ubuntu -c "$BUNDLE exec unicorn_rails -D -c $CONF"
    fi
    echo "Unicorn rails app started!"
    ;;
  stop)
    echo "Stoping unicorn rails app ..."
    sig QUIT && exit 0
    echo "Not running"
    ;;
  restart)
    if [ -f $PID ];
    then
      echo "Unicorn PID $PID exists"
      /bin/kill -s USR2 `/bin/cat $PID`
      sleep 30
      echo "Old PID $OLD_PID"
      /bin/kill -9 `/bin/cat $OLD_PID`
    else
      echo "Unicorn rails app is not running. Lets start it up!"
      $0 start
    fi
    ;;
  status)
    ;;
  *)
    echo "Usage: $0 {start|stop|restart|status}"
    ;;
esac

Just customize the paths to your server and place the script in “/etc/init.d”. And ensure that the script is executable.

chmod ugo+x <YOUR_SCRIPT>

Than navigate to the direcotry “/etc/init.d” and execute this here:

update-rc.d <YOUR_SCRIPT> defaults

That will add your script to the default runlevels. So that your script will be executed by every reboot.

Installing Ruby 1.9.X on Debian Linux

It is pretty easy to install Ruby on Debian. Just execute this as root:

apt-get install ruby1.9.1

That’s it! Depending on that GEMs you want to install you should install this here, too.

apt-get install make bcrypt libxml-ruby1.9.1 libxml2-dev libxslt-dev libpq-dev g++

Your GEMs are at “/var/lib/gem/1.9.1/”. You should add this here your path:

/var/lib/gem/1.9.1/bin

That’s it.

Access PostgreSQL from outside

After you installed PostgreSQL on a debian server, by default it is not accessible from outside. From other servers! It is just accessible from localhost. To change that you need to make it listen to a special ip address and port. On Debian 6 you have to switch to this directory:

/etc/postgresql/8.4/main

Here are the config files for PostgreSQL. You have to edit the “postgresql.conf” file. There is one line called “listen_addresses”. This line you have to enhance:

listen_addresses = 'localhost, <YOUR_SERVER_IP>' # what IP address(es) to listen on;
 # comma-separated list of addresses;
 # defaults to 'localhost', '*' = all
 # (change requires restart)

And with this lines you are controlling the port and the connections.

port = 5432 # (change requires restart)
max_connections = 100 # (change requires restart)

Now you have to restart the postgres service:

/etc/init.d/postgresql restart

That alone is not enough. Now PostgresSQL is listen but you can anyway not access it 🙂

For security reasons you have to edit the “pg_hba.conf” file, too. Here you say which IP address is trustful enough to access the service. Here you have to add a line like this:

host    all         all         192.168.0.19/24        md5

That means that all ip from your network can access all databases and all users.

Setting password for PostgreSQL

After a standard installation of PostgreSQL on Debian linux, there is no password set for the default user postgres. You can set the password by login in as postgres user. If you are root, just type in that:

su postgres

Than you can start the postgres client:

psql

And now execute this command here to set up a password for user postgres.

ALTER USER Postgres WITH PASSWORD '<password>';

That’s it.

screen

How can somebody be so stupid to choose “screen” as product name?

I mean the word “screen” has already a meaning! A very strong meaning! That means it is very difficult to google for the product “screen”. So you have to add more keywords into your search. It is a nightmare!

custom_require>:29:in `require’: no such file to load — mkmf (LoadError)

Today I installed Ruby 1.9.X on a Debian Linux Server. Worked pretty smoothy with “apt-get install ruby1.9.X”. After “gem1.9.1 install rails” I got this Exception:

custom_require>:29:in `require': no such file to load -- mkmf (LoadError)

If you get this error you have to install two more things:

apt-get install ruby1.9.1-dev
apt-get install make

After this update I could install rails.

Install bash-completion

If you are working with Debian Linux you have to the package manager “apt-get” to install packages. What is pretty cool. 1 mil. times better the the shitty RPM System from Red Had. If you want to have autocompletion for apt-get install the package “bash-completion”.

apt-get install bash-completion

and than add this lines to your “/root/.bashrc” file:

if [ -f /etc/bash_completion ]; then
. /etc/bash_completion
fi

Installing MongoDB on YDL

MongoDB is a pretty good Document Database. It is NOT a Object Database. Anyway. If you want learn more about MongoDB check out this site: http://www.mongodb.org

As far as I know MongoDB is not available over the yum package manager from Yellow Dog Linux (YDL). But it is pretty easy to install. Just download the linux binaries and unpack it.

cd /opt/
sudo wget http://downloads.mongodb.org/linux/mongodb-linux-x86_64-1.8.1.tgz
sudo tar -xzf mongodb-linux-x86_64-1.8.1.tgz

For convenient reasons you can create a short symlink

sudo ln -s mongodb-linux-x86_64-1.8.1 mongodb

Now you can start all mongo binaries under “/opt/mongodb/bin”. For convenient reasons you can create more symlinks.

cd /usr/bin
sudo ln -s /opt/mongodb/bin/mongod
sudo ln -s /opt/mongodb/bin/mongo

MongoDB will save all the database stuff in the directory “/data/db”. If this directory does not exist you will get an error. Just create the directory.

sudo mkdir /data
sudo mkdir /data/db

Now you can start the MongoDB server

sudo mongod

Your MongoDB Server is up and running. In a diffrent shell you can use the mongo client to connect to the server.

mongo

Now you are in the Mongo Shell. By default the mongo client connects to the database “test”. Try this:

db.foo.save({"b":"2"})
db.foo.find()

Now you should see something like this:

{ "_id" : ObjectId("4dd975e60994b04312264239"), "b" : "2" }

Congratulation! You can exit the shell with this command:

exit

Installing Nginx 1.0.2 on Yellow Dog Linux

I just got an Yellow Dog Linux (YDL) up and running on Amazon EC2. YDL is based on Red Hat. The installation tool is called “yum”. It is similar to debians “apt-get” tool, but it is based on RPMs.

Anyway. You can install nginx via yum with this command:

sudo yum install nginx

But this is not the newest version. If you want the newest version you should install it from the source code. But at first you need to install some additional tools.

sudo yum install gcc
sudo yum install pcre
sudo yum install pcre-devel
sudo yum install zlib
sudo yum install zlib-devel
sudo yum install make

Now you can download the newest version with weget from the nginx page: http://wiki.nginx.org/Install

After the download you should unpack the *tar.gz file und navigate into the directory. Now you can install it with this 3 commands:

sudo ./configure
sudo make
sudo make install

If everything works fine it is installed here:

/usr/local/nginx/

Now you can add “/usr/local/nginx/sbin” to your PATH or you just create a symlink in “/usr/bin”

cd /usr/bin
ln -s /usr/local/nginx/sbin/nginx

For security reason you should create a nginx group.

sudo useradd -s /sbin/nologin -r nginx
sudo usermod -a -G nginx <USERNMAE>

And you should have a place where you deploy your webapps.

sudo mkdir /var/www
sudo chgrp -R nginx /var/www
sudo chmod -R 755 /var/www

Now you can start nginx with this command:

sudo nginx

And stop it with this command:

sudo nginx -s stop

.gitignore

If you want that git is ignoring some files for you you just have to create the “.gitignore” file in your project root. Here you can add all the files you don’t want have in your git repository. For example some meta files from IntelliJ IDEA.

*.iml
*.ipr
*.iws
.idea/

psql client for PostgreSQL

psql is a command line client for the PostgreSQL Database Engine. If you install postgreSQL on Debian psql is installed by default. After you switched to the postgteSQL user:

su postgres

you can access the client with this simpel command:

psql

and you can exit with this command:

\q

You can set a password with this command inside of the psql client:

ALTER ROLE postgres WITH ENCRYPTED PASSWORD 'postgres';

you can call the help with this command:

\?

It’s a mess! I think more “NOT SEXY” is not possible! MySQL is maybe not as good as PostgreSQL, but from the usability it is 10 times better. In MySQL the command for exit is “exit” not “\q” and for showing the databases the command is “show databases” and for showing tables “show tables”. Much more intuitive than PostgreSQL!