GitHub API : User Agent Now Mandatory

GitHub now enforces the User Agent. If you do API calls without user agent you will get a 403 error message back. Check out this post: http://developer.github.com/changes/2013-04-24-user-agent-required/.

If you are using the HTTParty GEM in Ruby to do the API calls you have to set the headers. Here an example:

body = HTTParty.get(url, :headers => {"User-Agent" => "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_8_2) AppleWebKit/537.17 (KHTML, like Gecko) Chrome/24.0.1309.0 Safari/537.17" } ).response.body

VersionEye API : Dependency stable

At the last DevCamp in KA I had some good discussion about the VersionEye API. That’s why I enhanced it a little bit. I just introduced a new attribute to the project dependencies. The “stable” attribute. If this is true than that means that the newest version is stable, otherwise the newest version is unstable. In that case it could be an alpha or beta version. If this is the case we display the dependency with an yellow background instead of a red background.

Screen Shot 2013-03-25 at 11.19.46 AM Here is the color code we are using at VersionEye: 

if unknown == true
  return "gray"
elsif outdated == true && stable == true
  return "red"
elsif outdated == true && stable == false
  return "yellow"
else
  return "green"
end

Ruby Link Checker

This code checks if the given link exists and returns an HTTP Code 200.


def self.link_ok? link
  url = URI.parse( link )
  req = Net::HTTP.new(url.host, url.port)
  req.use_ssl = true
  req.verify_mode = OpenSSL::SSL::VERIFY_NONE
  res = req.request_head(url.path)
  res.code.to_i == 200
rescue => e
  p "ERROR #{e.message}"
  return false
end

Parsing a CSV in Ruby

Parsing is CSV file in Ruby is very easy. Check out this code:

require "csv"
csv_file_path = "ABSOLUTE_PATH_TO_YOUR_CSV_FILE"
file = File.open(csv_file_path, "r:ISO-8859-1")
csv = CSV.parse file

That’s it. This is how you get the first row:

csv.first

This is how you get the first column of the first row:

csv.first.first

And this is how you iterate over all rows:

csv.each do |row|
  p row
end

Easy! Right?

How to get a list of all licenses for your Software Project

Through the new VersionEye API you can get a list of all licenses for your software project. VersionEye now supports 7 different package managers. In this example I will use a Ruby Gemfile to demonstrate how easy it is to get the license list.

First of all you need to signup at VersionEye to get your API KEY. It’s free and takes less than 30 seconds. With your API KEY you can take advantage of the VersionEye API.

The “/api/v1/projects.json” resource supports file uploads via the API.

Screen Shot 2013-02-16 at 3.39.11 PM

Just send your Gemfile via POST to this resource and receive a JSON object as response with all the dependencies in the file.

Screen Shot 2013-02-16 at 3.39.35 PM

In the JSON response you will receive an “project_key”. This project key is unique per user. You can use it to fetch all licenses for the project. Just send a GET request to “/api/v1/projects/PROJECT_KEY/licenses.json?api_key=API_KEY” and you will receive the license list as JSON.

Screen Shot 2013-02-16 at 3.40.14 PM

That’s it. You need only 2 Requests to get the licenses list. Let me know how you like it.

Ruby on Rails + MySQL on CloudControl

CloudControl is a pretty cool StartUp from Berlin. It is a PaaS Solution like Heroku.  CloudControl is hosting on AWS in Ireland. They are more flexible than Heroku. Beside the number of web processes you can choose how much RAM each instance should have. This is a cool features what you don’t get on Heroku. And you can have different environments on CloudControl for 1 single app. Check out their page: https://www.cloudcontrol.com/

Cloud-Control-logo-centered

I want to show here how you get started with your Ruby on Rails on CloudControl. First of all you have to sign up at CloudControl:  https://console.cloudcontrolled.com/

I assume you are familiar with GIT, Ruby and Linux/Mac OS X. First of all you have to install the CloudControl command line tool.

$ sudo easy_install pip
$ sudo pip install cctrl

The command line tool will be installed via pip, the python package manager. the CloudControl command line tool works similar to the Heroku command line tool. Let’s add your public SSH Key:

cctrluser key.add

In that way you don’t have to enter every time your credentials.

As next switch to the app root directory of your Rails App you want to deploy on CloudControl. Create an app on CloudControl:

cctrlapp APP_NAME create ruby

And now push your code to CloudControl:

cctrlapp APP_NAME push

The command above pushes your code but it doesn’t deploy it. Use this command to make a deployment:

cctrlapp APP_NAME deploy

Now check your app on :

http[s]://APP_NAME.cloudcontrolled.com

It is very likely that it is not running 🙂 No problem. You have to make some small changes to your Rails App. First of all, if you create an app on CloudControl it is without any database. That is a big difference to Heroku. Of course you can add a MySQL or Postgres DB as AddOn. Let’s add a MySQL:

cctrlapp APP_NAME/default addon.add mysqls.free

Don’t forget to add the “mysql2” GEM to your Gemfile. This is the driver for MySQL.

Now you have to customise your database.yml like this:

production:
  adapter: mysql2
  encoding: utf8
  database: <%= "'#{ ENV['MYSQLS_DATABASE'] }'" %>
  host: <%= "'#{ ENV['MYSQLS_HOSTNAME'] }'" %>
  port: <%= ENV["MYSQLS_PORT"] %>
  username: <%= "'#{ ENV['MYSQLS_USERNAME'] }'" %>
  password: <%= "'#{ ENV['MYSQLS_PASSWORD'] }'" %>
  pool: 100
  timeout: 5000

All right. One more step. You have to have a “Procfile” in the root of your Rails App with this content:

web: bundle exec rails s -p $PORT

That’s it. Now try again:

cctrlapp APP_NAME push
cctrlapp APP_NAME deploy

And check the URL :

http[s]://APP_NAME.cloudcontrolled.com

For me it worked quiet well. Let me know how it works for you.

Getting a list of all Licenses in a Project

In a regular project you are using a handfull Software Libraries. Every Library in your project can use a different license. To get a list of all licenses which are used in your project can be difficult. You can double check the license for every single library in your project manually. But that is time intensive and a pain in the ass!

VersionEye is offering a quick solution for that. You can get the license list in less than 1 minute. After your login you are in “My Projects”. Click the link “Add New Project” in the left nav bar, to create a new project.

Screen Shot 2013-02-04 at 5.40.07 PM

Simply upload a project file. A pom.xml, Gemfile, composer.json, package.json, dependency.gradle or another supported file with your dependencies. After the upload you will see a tab with all the dependencies.

Screen Shot 2013-02-04 at 5.40.53 PM

Simply click on the “License” tab to see all licenses in this project. And here it is.

Screen Shot 2013-02-04 at 5.41.17 PM

In this example most Libraries are using the MIT License. Some of them are under “Ruby” license and for 1 Library VersionEye was not able to detect a license.

It’s that simple. Let me know if you have questions to this.

Capistrano + Rails + Unicorn + NGinx + RVM

Capistrano is a ruby based deployment tool which executes commands in parallel on multiple remote machines, via the SSH protocol. With Capistrano you can deploy your application on N server with one single command from your dev machine. You don’t need to login via SSH to your server.

Screen Shot 2013-01-15 at 8.15.33 PM

If you don’t deploy your app on Heroku or CloudControl, if you have dedicated servers or you are using AWS, than Capistrano is the way to go.

Before you start with Capistrano, you have to implement SSH with authentification keys instead of password. In that way you can just login to your server with a simple “ssh user@server” without password. That is possible if your public ssh certificates are on the server. In that way the server “knows” you.

First you need to add the Gem to your Gemfile.

gem 'capistrano'
gem 'rvm-capistrano'

I am adding here “rvm-capistraon”, too. Because I am working with RVM on my dev machine and on the server. Now you have to run bundler.

bundle install

As next step you have to capify your rails project. Just run:

capify .

That will create some files in your project.

[add] writing './Capfile'
[add] writing './config/deploy.rb'
[done] capified!

Your Capfile is loading the deploy.rb script. The Capfile should look like this:

load 'deploy'
# Uncomment if you are using Rails' asset pipeline
# load 'deploy/assets'
load 'config/deploy'

The magic happens in the deploy.rb file. The full documentation to this script you will find here: https://github.com/capistrano/capistrano. It basically executes a couple of commands on your server. It fetches the current code from your GIT server, runs bundler, rake db:migrate, precompiles your assets and runs the ruby app server.

Here is my deploy.rb with some additional comments.


# Automatically precompile assets
load "deploy/assets"

# Execute "bundle install" after deploy, but only when really needed
require "bundler/capistrano"

# RVM integration
require "rvm/capistrano"

# Name of the application in scm (GIT)
set :application, "your_app"
set :repository, "git@github.com:your_company/your_project/project.git"

# Source Control Management
set :scm, :git

set :deploy_to, "/var/www/#{application}"

# server there the web server is running (nginx)
role :web, "192.168.0.12"

# server there the app server is running (unicorn)
role :app, "192.168.0.12"

# server there the db is running
# This is where Rails migrations will run
role :db, "192.168.0.12", :primary => true

set :rails_env, :production

# user on the server
set :user, "project_user"
set :use_sudo, false

# Target ruby version
set :rvm_ruby_string, '1.9.3'

# System-wide RVM installation
set :rvm_type, :system

# Apply default RVM version for the current account
after "deploy:setup", "deploy:set_rvm_version"

namespace :deploy do

  task :set_rvm_version, :roles => :app, :except => { :no_release => true } do
    run "source /etc/profile.d/rvm.sh && rvm use #{rvm_ruby_string} --default"
  end

  task :start, :roles => :app, :except => { :no_release => true } do
    run "/etc/init.d/unicorn start"
  end

  task :stop, :roles => :app, :except => { :no_release => true } do
    run "/etc/init.d/unicorn stop"
  end

  task :restart, :roles => :app, :except => { :no_release => true } do
    run "/etc/init.d/unicorn restart"
  end

  # Precompile assets
  namespace :assets do
    task :precompile, :roles => :web, :except => { :no_release => true } do
      run %Q{cd #{latest_release} && #{rake} RAILS_ENV=#{rails_env} #{asset_env} assets:precompile}
    end
  end

end

The script for starting and stoping unicorn you can find here: https://robert-reiz.com/2012/02/29/running-unicorn-as-a-service-on-debian-linux/.

On the Linux server you should install a couple things before you deploy:

apt-get install make
apt-get install g++
gem update --system
gem install bundler

Now you can deploy with:

cap deploy

And this command shows you all possible Capistrano tasks:

cap -T

Let me know if you have questions.

Ruby on Rails + ElasticSearch

This is a tutorial how to use ElasticSearch with Ruby on Rails. ElasticSearch is a distributed RESTful Search Engine build on top of Apache Lucene.

Sure! You can use your SQL database for search. But that is usually slow and you will not get very good search results. With ElasticSearch you can deliver a fuzzy search. Let’s say you have a record “Hibernate” in the database. If somebody is doing a search for “hibernate” you will get a match with a simple SQL query. But what if your customers input looks like this:

  • hibernate 3.2
  • hibernate.jar
  • hibernate.jar 3.5

In this cases you will have 0 results with a simple SQL query. With ElasticSearch you would still have results. Depending on your configuration. So let’s start!

ES

Just download the current version from here: http://www.elasticsearch.org/download/. Unpack it and run the ES server with this command:

./bin/elasticsearch -f

The ES server is now running on localhost:9200. If you type in “http://localhost:9200/_search” into your browser you should get some basic results.

I assume that you know already Ruby on Rails and ActiveRecord. Of course there is a GEM to interact with ElasticSearch. Checkout the Tire GEM. That is a pretty good wrapper for ElastisSearch. Just add it to your Gemfile.

gem 'tire', '0.5.4'

and run:

bundle update

In your application.rb you need to require the new package:

require 'tire'

And in environment.rb you need to init the package:


begin
  Tire.configure do
    logger STDERR
    url Settings.elasticsearch_url
  end
rescue => e
  p "Wrong configuration: #{e}"
end

On the GitHub Page there is a description how to integrate Tire into your model. But honestly I don’t like that very much. That just blows up the model class. I prefer a clear separation between my models and the interaction with ElasticSearch.

The model I wanted to map and to make searchable with ElastiSearch is “product.rb”. It is located in “app/models/”. I created another directory called “app/elastics/”. And here I placed a new file “product_elastic.rb”, which is mapping my Product class to ElasticSearch and is responsible for the interaction with the ES server.

The first thing you have to do is to create a mapping. You have to map your properties from your model to ElasticSearch. This is how I did my first mapping:


def self.create_index_with_mappings
  Tire.index Settings.elasticsearch_product_index do
    create :mappings => {
      :product => {
        :properties => {
          :_id => { :type => 'string', :analyzer => 'keyword', :include_in_all => false },
          :name => {:type => 'string', :analyzer => 'snowball', :boost => 100},
          :description => { :type => 'string', :analyzer => 'snowball' },
          :description_manual => { :type => 'string', :analyzer => 'snowball' },
          :language => { :type => 'string', :analyzer => 'keyword'}
        }
      }
    }
  end
 end

The analyzers are documented on the ElasticSearch homepage: http://www.elasticsearch.org/guide/. Here is the magic happening 😉

Than I wrote 2 more methods (clean_all and reset) to delete the “product” index at ES and to create the mappings.


def self.clean_all
  Tire.index( Settings.elasticsearch_product_index ).delete
end

def self.reset
  self.clean_all
  self.create_index_with_mappings
end

Don’t call the reset method in production 😉 With this 3 methods you can delete old mapping and create a new one. You can now run the rails console and try out the methods. That should all work fine.

If creating the mapping was successful, the next step is to index the data in the database. To index a model, the model itself must offer a method which returns their values as JSON. I added a “to_indexex_json” method to the Product class:


def to_indexed_json
 {
   :_id => self.id.to_s,
   :_type => "product",
   :name => self.name,
   :description => self.description ? self.description : "" ,
   :description_manual => self.description_manual ? self.description_manual : "" ,
   :language => self.language,
   :group_id => self.group_id ? self.group_id : "",
   :prod_key => self.prod_key,
 }
 end

The first 2 attributes are required by Tire, to link the response from ES with your model. And here is the method in “product_elastic.rb” to index one record.


def self.index( product )
  Tire.index Settings.elasticsearch_product_index do
    store product.to_indexed_json
    product.update_attribute(:reindex, false)
  end
rescue => e
  p "ERROR in index(product) Message: #{e.message}"
  p "ERROR in index(product) backtrace: #{e.backtrace}"
end

And this here is the method to index all products from the DB:


def self.index_all
  Product.all.each do |product|
    ProductElastic.index product
  end
  self.refresh
end

def self.refresh
  Tire.index( Settings.elasticsearch_product_index ).refresh
end

Easy! Right? You can try the methods in the rails console. All right. You can now create mappings and index data. The only thing which is missing now is the search method. There is a lot to say about the search. I really recommend that you take your time and read the documentation about the search at the Tire homepage and the ES homepage. But here is one example.


def self.search(q, page_count = 1)
  if (q.nil? || q.empty?)
    raise ArgumentError, "query is empty! This is not allowed"
  end

  page_count = 1 if page_count.nil? || page_count.to_i < 1
  results_per_page = 30
  from = results_per_page * (page_count.to_i - 1)

  q = "*" if !q || q.empty?

  s = Tire.search( Settings.elasticsearch_product_index,
   load: true,
   from: from,
   per_page: results_per_page,
   size: results_per_page) do |search|

    search.sort { by [{:_score => 'desc'}] }

    search.query do |query|
      query.string "name:" + q
    end

  end

  s.results
end

The method above is with paging. You can easily use it with the “will_paginate” GEM.

There is much more to say about the mapping and the search. But this is the part there you have to invest time to figure it out, to deliver the search results you want. You will find a pretty good documentation on the ElasticSearch homepage: http://www.elasticsearch.org/guide/. And the core committer from the Tire project always responded in less than 24 hours to my tickets. The project support is very good.

Together with Timo I integrated ElasticSearch into VersionEye. Since them the search is much faster and the results are better. Even if there is no perfect match you will get results.

Installing native pg gem on Mountain Lion

I tried to install the native pg gem on Mac OS X Mountain Lion (10.8.2):

gem install pg

and got this exception:

Building native extensions. This could take a while...
ERROR: Error installing pg:
 ERROR: Failed to build gem native extension.
/Users/robertreiz/.rvm/rubies/ruby-1.9.3-p327/bin/ruby extconf.rb
checking for pg_config... yes
Using config values from /usr/bin/pg_config
checking for libpq-fe.h... yes
checking for libpq/libpq-fs.h... yes
checking for pg_config_manual.h... yes
checking for PQconnectdb() in -lpq... yes
checking for PQconnectionUsedPassword()... yes
checking for PQisthreadsafe()... yes
checking for PQprepare()... yes
checking for PQexecParams()... yes
checking for PQescapeString()... yes
checking for PQescapeStringConn()... yes
checking for PQescapeLiteral()... yes
checking for PQescapeIdentifier()... yes
checking for PQgetCancel()... yes
checking for lo_create()... yes
checking for pg_encoding_to_char()... yes
checking for pg_char_to_encoding()... yes
checking for PQsetClientEncoding()... yes
checking for PQlibVersion()... yes
checking for PQping()... yes
checking for rb_encdb_alias()... yes
checking for rb_enc_alias()... yes
checking for PGRES_COPY_BOTH in libpq-fe.h... no
checking for PGRES_SINGLE_TUPLE in libpq-fe.h... no
checking for struct pgNotify.extra in libpq-fe.h... yes
checking for unistd.h... yes
checking for ruby/st.h... yes
creating extconf.h
creating Makefile
make
compiling pg.c
pg.c: In function ‘Init_pg_ext’:
pg.c:384: error: ‘PQPING_OK’ undeclared (first use in this function)
pg.c:384: error: (Each undeclared identifier is reported only once
pg.c:384: error: for each function it appears in.)
pg.c:386: error: ‘PQPING_REJECT’ undeclared (first use in this function)
pg.c:388: error: ‘PQPING_NO_RESPONSE’ undeclared (first use in this function)
pg.c:390: error: ‘PQPING_NO_ATTEMPT’ undeclared (first use in this function)
make: *** [pg.o] Error 1

I could fix it by installing postgres via Homebrew.

brew update
brew install postgresql

and after that this here worked perfect:

gem install pg

Brew is awesome!

Reset Postgres DB on Heroku

In a Rails application you can you this tasks on localhost to reset the database.

rake db:drop
rake db:create
rake db:migrate

And that will delete the db schema, create a new one and run all migrations to create the tables. This is awesome!
But if try to run this on heroku:

heroku run rake db:drop

Than you will get an exception, because you don’t have permission to delete dbs on Heroku. If you want to reset your database on Heroku you have to use this command:

heroku pg:reset DATABASE_URL

this is equal to “rake db:drop” and “rake db:create”. After that you still have to run:

heroku run rake db:migrate

That worked pretty good for me.
Just wanted to share my experience.

CSS broke Heroku Deployment

Today I did a “git push heroku master” do deploy an application. And than suddenly I got a big exception and the build failed.

CssBrakesBuild

The funny thing is that CSS is the root of the problem. This CSS line here broke the deployment:

height: inherit important!;

Indeed it should be:

height: inherit !important;

Heroku is very sensible 🙂

Rails ActiveRecord by-pass validation

ActiveRecord is my favorite way to access a SQL database. It offers very useful validation inside of a model. But sometimes you just want to save an attribute quickly, without validating the whole object. For that you can pass a parameter to the save method.

object.save(:validate => false)

That saved me a lot of time. Awesome!

3 Rake commands

Not for me. The 3 most important Rake commands in a Ruby on Rails application are:

rake db:drop
rake db:create
rake db:migrate

First command drops the database. Second command creates the database. Third command runs all migrations and creates basically the schema. This is incredible useful.

through mapping causes rspec error

by executing this command:

rails generate rspec:install

I got this error message.

~.rvm/gems/ruby-1.9.3-p194@ms/gems/activerecord-3.2.8/lib/active_record/dynamic_matchers.rb:50:in `method_missing': undefined local variable or method `consultant_id' for #<Class:0x007fdb93a08bd0> (NameError)

the problem was that in one of the models I used

:through => consultant_id

As soon I resolved this option it worked. Strange!

spec_helper LoadError

I just tried to run “rspec” and got this error.

ruby/site_ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- spec_helper (LoadError)

The problem was that the file “spec_helper” was not existing. a simple:

rails generate rspec:install

solved the problem.

Rails Migration column_change

I just created a rails migration for one of my tables. Wanted to change the type of a column from string to integer. My migration looked like that:


class ChangeColumnTypeInConsultants < ActiveRecord::Migration
  def change
    change_column :consultants, :title_id, :integer
  end
end

Unfortunately it didn’t worked out. I got this Exception:

PG::Error: ERROR: column "title_id" cannot be cast to type integer
: ALTER TABLE "consultants" ALTER COLUMN "title_id" TYPE integer

And on pgAdmin I got this error:

ERROR: column "title_id" cannot be cast to type integer
SQL state: 42804

I couldn’t figure it out how to solve this problem. I just wrote 2 more migrations, for deleting the columns and adding them new with the correct data type.


class RemoveColumnsFromConsultants < ActiveRecord::Migration
  def change
    remove_column :consultants, :title_id
  end
end

and than for adding it new.


class AddColumnsToConsultants < ActiveRecord::Migration
  def change
    add_column :consultants, :title_id, :integer
  end
end

That worked for me.

Seamless Deployment with Heroku

By default there is no seamless deployment on Heroku. That means, if you do a git push, your app gets deployed and there is a little downtime. That’s just because how Heroku works. At first they kill your running dynos, then they build the new app, deploy it on new dynos and finally boot up the new dynos.

But there is a new “preboot” feature in the Heroku labs. With that the procedure looks like that:

  1. Code push
  2. Run buildpack; store slug
  3. Boot new dynos
  4. Switch routing
  5. Kill old dynos

With that “preboot” Feature you don’t have any down times. You can activate the “preboot” feature for your app with this command:

heroku labs:enable -a myapp preboot

Now you can directly do “git push heroku master” and 3 minutes later your new version is online without any downtimes. I just tried it for VersionEye and it works perfectly for me.

That also means that there 2 different versions of the app online for round about 2 minutes. There is a short overlap for a short amount of time. That can be dangerous if you do big db migrations. For this kind of deployments you should disable the feature with

heroku labs:disable -a myapp preboot