I know things and I build stuff
This shell script runs forever and checks if the rails worker is running and if not it starts it again:
while :
do
if ps ax | grep -v grep | grep 'rails worker' > /dev/null
then
echo "service running, everything is fine"
sleep 5
else
echo "service is not running. Lets start again"
cd $APP_ROOT
$BUNDLE exec unicorn_rails -D -c $CONF
echo "restarted on $(cat /rails/pids/unicorn.pid)"
sleep 15
fi
done
Last week it happened again. Geek2Geek!
This time we came together at Flyeralarm in Berlin to talk about centralized logging. That is an interesting topic for all companies which have to scale. As soon you have more than 1 server you need to think about how you collect and analyze your log files in a distributed system. There are a couple good solutions out there for this problem.
Jilles van Gurp did the first talk about the ELKstack. ELK stands for E = Elasticsearch, L = Logstash and K = Kibana. All three products belong to the Elasticsearch company and they work all together smoothly in harmony. Jilles showed us how they use the ELK stack at Linko to build the LinkoApp.
Jilles gave us a short intro to the technology on a couple slides before he switched to the live demo. It was very interesting to listen to his real-world experiences with the ELK stack.
The learning from the past couple months are, it is easy to setup but you should be careful with the Elasticsearch cluster. Don’t shut it down all at once đ
After the first presentation the Pizza arrived and we took a little break with Pizza & Beer.
Lennart is THE guy behind Graylog2. He started the project a couples years ago at Jimdo. The very first version was implemented in Ruby. Graylog2 is a completely rewrite in Java. Lennart is also CoFounder of Torch, the company behind Graylog2.
Lennart gave a short intro about the history, intention and philosophy behind Graylog2.
I was impressed how much he knows about the other logging solutions, such as Logstash/Kibana and Splunk. He was not afraid to talk about feature comparisons and pros & cons of the different solutions.
Graylog2 is build for Enterprise usage. It is optimized for speed and high volume data. The interesting thing is that you can use it together with Elasticsearch and Logstash.
Many thanks for to Jilles and Lennart for the great talks. Both solutions are very interesting. If you still read logs on the server with “less” you should definitely check out these 2 great solutions!
By the way. I also tried to organize a Splunk talk, but unfortunately I couldn’t find any Splunkies willing to give a talk about Splunk at Geek2Geek.
Many thanks to Flyeralarm for sponsoring Location, Pizza and Beer! You guys are awesome!
By the way Flyeralarm just opened a new branch in Berlin. They have a really nice office. This is their meeting room for example.
And they are currently looking for experienced PHP developers. If you are interested you should contact Thomas.
Note for me! How to sign a file.
gpg –armor –detach-sign -r <email_address>Â <file>
The next Geek2Geek event is this Friday in Berlin Kreuzberg. The event is hosted by Flyeralarm. They even sponsor Beer & Pizza. Many Thanks for that!
This time the topic is centralized logging. We have 2 awesome speakers for the event!
Lennart, is THE guy behind Graylog2, the cutting edge logging technology for the new Enterprise. He will give us a short intro into the framework and show the newest features.
Jilles is the lead architect at Linko. He will show us how he is using the ELK stack to build Linko. ELK stands for = ElasticSearch, Logstash and Kibana.
Don’t miss it! Come to Geek2Geek, learn something new, eat a Pizza and drink a Beer!
Recently I made a db dump on a MongoDB 2.4 server like this:
mongodump --db veye_dev
And I tried to restore it on a MongoDB 2.6 server like this:
mongorestore dump/veye_dev
Unfortunately at some point the restore process failed with this error message:
mongo error: "Btree::insert: key too large to index, failing
There are some restrictions for the size of the index in MongoDB 2.6. Luckily I found the issue on MongoDBs ticket system. The import can succeed if the mongod process is running with the parameter “failIndexKeyTooLong=false”. Just start it like that:
sudo mongod --setParameter failIndexKeyTooLong=false
And now execute the mongorestore again. That worked for me.
Capistrano is a ruby based deployment tool which executes commands in parallel on multiple remote machines, via the SSH protocol. With Capistrano you can deploy your Rails application on N servers with one single command from your dev machine. You even don’t need to login via SSH to your server. This command can rollout your application on N servers:
cap production deploy
And if something goes wrong you can easily rollback to the last stable deployment. Just like this.
cap production deploy:rollback
Capistrano is pretty cool. I used already the previous version 2.X. The new version 3.X I’m using already since a couple months in production and it is super stable.
If you are deploying your Rails application to dedicated servers or instances on AWS, than Capistrano is the way to go!
Before you start with Capistrano, you have to implement SSH with authentification keys instead of password. In that way you can just login to your server with a simple “ssh user@server” without password. That is possible if your public ssh certificates are on the server. In that way the server “knows” you.
First of all you need to add the Gem to your Gemfile.
gem 'capistrano'
And if you are using Rails and Bundler you want to add this 2 lines as well.
gem 'capistrano-rails' , '~> 1.1.1' gem 'capistrano-bundler', '~> 1.1.2'
Now you have to run bundler, to install the packages.
bundle install
As next step you have to capify your rails project. Just run:
capify .
That will create some files in your project.
[add] writing './Capfile' [add] writing './config/deploy.rb' [add] writing './config/deploy/production.rb' [add] writing './config/deploy/staging.rb' [add] writing './config/deploy/test.rb' [done] capified!
In the Capfile you can require some capistrano packages. For a Rails App it will look like this.
require 'capistrano/setup'
require 'capistrano/deploy'
require 'capistrano/bundler'
require 'capistrano/rails'
require 'capistrano/rails/assets'
require 'capistrano/rails/migrations'
# Loads custom tasks from `lib/capistrano/tasks' if you have any defined.
Dir.glob('lib/capistrano/tasks/*.cap').each { |r| import r }
In Capistrano 3 most of the magic happens in the deploy.rb file, which is the central configuration file for Capistrano. In general it fetches the current code from your Git server, runs bundler, rake db:migrate, precompiles your assets and starts/restarts the ruby app server.
Here is my deploy.rb with some additional comments.
# Force rake through bundle exec
SSHKit.config.command_map[:rake] = "bundle exec rake"
# Force rails through bundle exec
SSHKit.config.command_map[:rails] = "bundle exec rails"
set :migration_role, 'app' # Defaults to 'db'
set :assets_roles, [:app] # Defaults to [:web]
# The name of your application
set :application, 'myapp'
# Configuration for the source control management system
set :scm , :git
set :repo_url, 'git@github.com:myorga/myapp.git'
set :branch , "master"
# This forwards the user agents and uses the local
# user for the git authentification.
set :ssh_options, {:forward_agent => true}
# User on remote server
set :user , "ubuntu"
# Application root directory on remote server
set :deploy_to , '/var/www/myapp'
# Shared directories over different deployments
set :linked_dirs, %w(pids log)
# Configuring capistrano log output
set :format , :pretty
set :log_level, :info # :debug :error :info
# Keeps the last 5 deployments on the server for rollback scenarios
set :keep_releases, 5
namespace :deploy do
desc 'Start application'
task :start do
on roles(:app), in: :sequence, wait: 5 do
execute "/etc/init.d/unicorn.sh start"
end
end
desc 'Stop application'
task :stop do
on roles(:app), in: :sequence, wait: 5 do
execute "/etc/init.d/unicorn.sh stop"
end
end
desc 'Restart application'
task :restart do
on roles(:app), in: :sequence, wait: 5 do
execute "/etc/init.d/unicorn.sh restart"
end
end
after :finishing, 'deploy:restart'
after :finishing, 'deploy:cleanup'
end
The script for starting and stoping unicorn you can find here:Â https://robert-reiz.com/2012/02/29/running-unicorn-as-a-service-on-debian-linux/.
In Capistrano you have different environments. For example “test”, “staging” and “production”. You can define as much as you want. Each environment has his own configuration file under “config/deploy/”. For example “config/deploy/production.rb”. Which might look like this:
set :stage, :production
# Setting RAILS_ENV environment variable on server
set :rails_env, :production
set :normalize_asset_timestamps, %{public/images public/javascripts public/stylesheets}
role :app, %w{ubuntu@myapp_server}
set :ssh_options, {
forward_agent: true # , auth_methods: %w(password)
}
The most important line is the one with the role. In capistrano you can define different roles and assign them to different servers. So that some deployment commands will be only executed on specific servers. You can read more to that in the official docu. For this article I keep it simple and go ahead with only 1 role and 1 server.
On the remote server(s) you have to create the application root directory. If your application has the name “myapp” it would look like this:
/var/www/myapp /var/www/myapp/releases /var/www/myapp/shared
Make sure that the user you defined in the deploy.rb file has full read and write access to this directories. Capistrano will create for each deployment a separate directory in the “release” directory, named with the timestamp of the deployment. The last deployment will be linked to “/var/www/myapp/current”. The “current” directory is a symbolic link to the latest deployment in “/var/www/myapp/releases”.
Now you can deploy with:
cap production deploy
If you have done everything right the deployment will run through and deploy your application.
This command shows you all possible Capistrano tasks:
cap -T
If you don’t deploy on Heroku or CloudControl, than Capistrano is a big help. It makes life much easier đ
Let me know if you have questions. Either in the comments or on Twitter.
I’m using Capybara and Selenium together with RSpec to test the Web Interface for VersionEye. That works very well. For an integration test I needed a callback on localhost:3000/auth/*. By Default Capybara is starting the tests on an odd host and port name to avoid conflicts with locahost:3000, which is the default host and port for Rails Apps in development. It took me something like 30 min. to find out how to force capybara to run all Tests on localhost:3000. That’s why I think it’s worth blogging đ
Either in your `spec_helper.rb` or in `spec/support/capybara.rb` you will have this imports:
require 'capybara/rails' require 'capybara/rspec' require 'capybara/firebug'
Below that you can configure Capybara like this.
Capybara.app_host = "http://localhost:3000" Capybara.server_host = "localhost" Capybara.server_port = "3000"
That worked for me.
I just started a new project with Rails 4.1 and explored the has_secure_password feature. Really awesome stuff!
I hope you are not storing passwords in clear text to your database! You should always store some kind of hashed values instead of clear text passwords. In case somebody steals your database he/she still doesn’t has the passwords.
There are a couple good tutorials how to hash and store passwords in a secure way. I implemented it a couples times by myself with Ruby.
A more sophisticated solution is devise, a very robust gem for security and authentication.
However. Since Rails 3.1Â you can take advantage of has_secure_password. This mechanism in Rails takes care of password validation and encryption. It requires a field ‘password_digest’ in your model, where it will store the encrypted password. Let’s generate a simple model.
rails g model user username:string password_digest:string
Let’s add this line to the user model.
has_secure_password
This will add an attribute password and password_confirmation to your model. This 2 fields are now part of your model but not part of the database schema! Because we don’t want to store cleartext passwords.
Let’s add some tests.
require 'spec_helper'
describe User do
it "fails because no password" do
User.new({:username => "hans"}).save.should be_false
end
it "fails because passwrod to short" do
User.new({:username => "hans", :password => 'han'}).save.should be_false
end
it "succeeds because password is long enough" do
User.new({:username => "hans", :password => 'hansohanso'}).save.should be_true
end
end
3 very simple tests. Persisting a new user without password should fail. Persisting a new user with too short password should fail as well. And creating a new user with a long password should succeed. If you run this tests with RSpec the 2nd test will fail. By default Rails doesn’t has a validation for the length of the password. So let’s add it to our user model.
class User < ActiveRecord::Base
has_secure_password
validates :password, :length => { :minimum => 5 }
end
If you run the tests again, they will be green. If you take a look into your database you will see that the user table/collection has a column password_digest with a very cryptical value. But there are no columns for password! That’s exactly what we wanted.
Now lets do the authentication. Assume a new user signed up at your portal and now he wants to login. This is how you authenticate him.
user = User.find_by_username("USERNAME").authenticate("CLEAR_TEXT_PASSWORD")
If the username and the clear text password from the HTTP Request is correct it will return a valid user from the database. Otherwise it will return nil!
has_secure_password validates the password on creation time, the very first time you persist it. It doesn’t check the password field after that, for updates for example. And that’s OK. Because that means you can load a user later from db, change it and persist it without knowing the password.
Another feature of this mechanism is password confirmation. has_secure_password also adds an attribute password_confirmation to your model. This attribute gets only validated if it’s not nil. If it’s not nil it must be equal to the password attribute. Let’s add 2 more tests for that one.
it "fails because password confirmation doesnt match" do
User.new({:username => "hans",
:password => 'hansohanso',
:password_confirmation => 'aa'}).save.should be_false
end
it "succeeds because password & confirmation match" do
User.new({:username => "hans",
:password => 'hansohanso',
:password_confirmation => 'hansohanso'}).save.should be_true
end
To make this tests pass you have to add one more line to the model.
class User < ActiveRecord::Base
has_secure_password
validates :password, :length => { :minimum => 5 }
validates_confirmation_of :password
end
The line “validates_confirmation_of :password” will check the password confirmation.
Rails doesn’t force you to have a password confirmation for your model, but if you want it you can turn it on.
I really like this feature because it saves me a lot of code and development time. And for most applications this is really enough.
Let me know what you think about this, either in the comments or on Twitter.
I went to the Technology Radar event from ThoughtWorks in Hamburg, a couple days ago. they presented the latest trends from the last year and explained their decision making decision. It was a very interesting event.
I wrote a detailed blog post about this event on the VersionEye blog. Check it out here.
Join the next Geek2Geek in Berlin this Friday night. We will talk about NoSQL databases and have talks to Neo4J, CouchDB and ArangoDB.
The MeetUp will happen at CO.UP in Kreuzberg. Which is a pretty cool coworking space.
Locafox, a young StartUp from Berlin, will sponsor pizza & beer. They are currently hiring in all areas. Check out their job listings here:Â http://locafox.de/jobs. And many thanks for the pizza. We really appreciate that.
I have to work with a lot (9) of different package managers at my daily work at VersionEye. Part of our mission is it to make manual updating of dependencies extinct, because itâs a manual and time consuming task which nobody enjoys. Thatâs why we are building a notification system for open source software libraries to make Continuous Updating easy and fun. And since we support several programming languages â 8 at this point! â I get to write crawlers and parsers for all of them. To give you a better overview over the strengths and weaknesses of these package managers, I picked the most popular one for each language and will compare them. The contenders are:
Here are the results of my comparison.
You can read the complete article on the VersionEye Blog and follow the discussion on hacker news and Reddit.
Do you know semantic versioning? You should! It describes how to name version numbers. Check it out here:Â semver.org.
This is the pattern it describes:
MAJOR.MINOR.PATCH
MAJOR version when you make incompatible API changesÂ
MINOR version when you add functionality in a backwards-compatible manner
PATCH version when you make backwards-compatible bug fixes
The cool thing here is that on the version number itself you can already see how big the changes are in the new release. A typical semantic version number is this:
3.2.1Â
Let’s say I am using version “3.2.0” in my project. Now I can immediately see that the new version of the package “only” contains a patch. That means for me that I can update without worrying. On the other side if this version comes out:
4.0.0Â
And I am using version “3.2.1” of the package in my project, I can now immediately see that this update will very likely break my build! In this case I have to look into the change logs and follow the migration paths.
Semantic versioning even addresses alpha and beta versions. If you are working on version “4.0.0” but it’s not quiet ready but you wanna release anyway something, you can name it like this:
4.0.0-a
That means that is version “4.0.0” alpha. And this here would be the beta version:
4.0.0-bÂ
Another convention is “RC”, that means “Release Candidate”. You can use it like this:
4.0.0-RC1Â 4.0.0-RC2
The complete order over all of them is like this, the highest and newest version is on the top.
4.0.0 4.0.0-RC2Â 4.0.0-RC1 4.0.0-b 4.0.0-a
That basically means 4.0.0 > 4.0.0-RC2 >Â 4.0.0-RC1 >Â 4.0.0-b >Â 4.0.0-a.
I’m the author of naturalsorter. That is an open source library to sort semantic version numbers in the correct way.
This week we finished the CocoaPods integration for VersionEye. VersionEye tracks now 2857 Objective-C packages on a daily basis. This here is the VersionEye page JSONKit.
You can follow the Objective-C packages you are using in your projects for your iPhone dev work and as soon the next version is released you will get notified via email.
If you are using CocoaPods as package manager you can directly upload your Podfile to VersionEye and you will see the out-dated dependencies. This here is a simple Podfile.
platform :ios pod 'SSToolkit' pod 'AFNetworking', '>= 0.5.1' pod 'CocoaLumberjack' pod 'Objection', :head # 'bleeding edge' generate_bridge_support!
It defines 4 packages for the project. If your project is on GitHub, VersionEye can directly monitor your Podfile in your GitHub repository and notify you as soon there are updates available for your project.
The CocoaPods integration is a very new feature. We rely on your feedback. Please let me know if you find something odd or your have further feature requests.
If you want to find out which version of a software library for your programming language of choice is the latest, you had to go the librariesâ website or the repository. Finding the website, the latest version and getting reminded on important bugfixes for the libraries you use can cost a lot of time.
VersionEye is a website that tracks software libraries in 8 different programming languages and 7 package managers. Eg. if you are a Ruby developer you might be interested in the latest Rails, Sinatra or Capybara version. As a Java developer the latest versions of Spring, Guice or Hibernate might be of interest for you.
If you regularly search for software libraries you may want to have a search engine directly in your browser. VersionEye now supports OpenSearch so you can add VersionEye as a search engine to your browser and quickly find out whatâŚ
View original post 201 more words
A couple weeks ago I went to the GOTO Berlin Conference. My StartUp VersionEye sponsored the event. GOTO is a conference with a wide scope. You have sessions related to all kind of computer languages and technologies. And there have been many talks to agile. I am enjoying the wide mix of technologies that’s why I decided to go to this conference.
There have been many awesome sessions. But specially I enjoyed 2 of the KeyNotes. Nathan Marz, Inventor of the Storm framework, did a KeyNote about “The Epistemology of Software Engineering“. His talk was starting lame because he talked about philosophy and I was wondering if I am in the right room and at the right conference. But then it made a lot of sense, because he questioned all our propositions. He encouraged us to build failure tolerant systems. Check out his slides! It’s worth it!
The last Keynote was from Mike Lee. He talked about “The App Universe After the Big Bang“. Very inspiring talk! At the last Keynote they removed the chairs. Everybody was standing in the big room. He asked a lot of questions in his talk but he gave answers, too!
“What kind of ideas are worth pursuing? Are app-based startups a thing anymore? What kind of business plans make sense? Is there room for indie apps? How about indie games? Can we still make a living doing this? Can we still live the dream and be our own boss? Can we still be artists?” – Mike Lee
He talked about the Silver Age of App Development.
The Gold Age of App Development we passed already. But there is nothing wrong with Silver!
He talked a lot about the StartUp life. How it is to start something and to pursue it with all your passion and all your time. One of the consequences is that you will lose friends.
If you are really passionated about something you will work day and night on it. You will skip parties, birthdays and weddings. Your attention is focused around problems related to your startup / product.
Another truth is that you will need help.
You can not build something really awesome on your own. You need good people around you. People who are completing you. People who are in certain areas much better than you are!
There is 1 single thing Mike wanted to share with us more than anything else.
Don’t make games!
There are every day a couple hundred new games in the Apple App Store. If you submit your game, the chances are very high that most people even don’t notice it. Building games for the Apple App Store is a bad business model!
He talked a little bot longer than expected but it was so interesting that most people stayed until the end. Even without chairs!
Check out his slides. And his blog:Â http://mur.mu.rs/.
During the summer 2013 I started the Geek2Geek MeetUp in Berlin. I started it because I wanted to have a MeetUp which doesn’t have a strong focus on 1 single programming language. Instead of that I did it the other way around. at Geek2Geek we pick one single topic for every MeetUp and we invite speakers from different communities to talk to the topic.
Our very first Geek2Geek MeetUp we did in our office in Berlin. The topic was “IT Infrastructure for DevOps“. Our small meeting room was full with people and Timwi did a spontaneously talk đ
And we had 2 more awesome speakers. Christoph Beckmann from KaufDA showed us how they manage their IT Infrastructure with Puppet in a Java ecosystem.
After a couple beers Tobias from Blinkist showed us how their manage they infrastructure in the cloud without any DevOps.
We had Java devs, Ruby devs and even PHP devs in 1 single room! And it was awesome. Because everybody was talking to each other. Ruby guys talked with PHP guys and Java guys with Ruby guys and so on. Everybody got new inspirations and ideas. And of course we had Pizza and beer! đ
And because the first one was a huge success and a lot of fun we did it 1 month later again! This time we did it at KaufDA.
Many Thanks to KaufDA for hosting the 2nd Geek2Geek event in Berlin! At the 2nd G2G MeetUp the topic was “Non Mainstream Languages“. Timo started with an introduction to Clojure.
Timwi did an awesome presentation of Funciton, the esoteric language with only 4 instructions.
After a break Paolo gave us an introduction to Erlang. A powerful language which powers many databases and the backend of big gaming companies.
The last talk was about Go from Peter at SoundCloud. He gave us a really good intro with some live coding.
The kitchen + meeting room at KaufDA was full packed with people.
And people with different backgrounds started again to talk with each other. They started to exchange ideas and we continued our discussions in a Bar in Friedrichshain, Berlin.
The last Geek2Geek MeetUp was roundabout 1 month ago, hosted by Fashion4Home. And we had an very unusual venue đ It was the furniture show room of Fashion4Home in Berlin. A really awesome location for a tech MeetUp! The topic was “Hardware Hacks”.
Ben did the first talk about RaspberryPi Jams.
He showed us use case with the RaspberyPi, like the smart Pi Microwave.
And he gave an intro about the RaspberryPi community in Great Britain. He showed us what kids can do with a little bit Hardware.
The 2nd talk was from Sam about “Read/write society and reverse engineering of objects”.
It was a very inspiring talk about Hardware and open source in general. About what we give up if we don’t use open source.
He talked about loosing control by not using open source. It was a talk which makes me think!
After the 2 long talks we ended again up in open discussions by Pizza and Beers until the pizza was empty.
Unfortunately this month (October) we will skip the Geek2Geek MeetUp. I am currently to busy with my StartUp and there are anyways more MeetUps, UGs and conferences in October than in other months. We do again a Geek2Geek MeetUp in November!
CU in November.
I am now since 1 week in San Francisco and I am surprised that I don’t hear here anything about PRISM. The program from the NSA to read our private emails & telecommunications. In Germany it is since months a big thing. The TV stations are talking about PRISM every day and at the biggest online tech news page heise.de there are every day new articles and details about PRISM and how it violets our privacy.
Privacy is a big thing in Germany and since PRISM many business changed their strategy. Some business decided to move out of the cloud and others who planed to move to the cloud don’t do it. Instead of public cloud solutions there are now more private cloud providers and IaaS providers who are doing good business in Germany, since PRISM.
I am just totally surprised that here in SF nobody is talking about it.
We just launched the “muted” feature a couple days ago. A highly requested feature from the VersionEye community. This is how it works. Let’s say you have a couple out-dated dependencies in your project. For example jquery-rails:
For some reason you don’t want to use the current version 3.0.4. But VersionEye keeps telling you that you are out-dated and you get every week the email notification.
Now you can mute that specific version. Just click on the icon in the left column and VersionEye will ignore this version in that project. If a newer version comes out, let’s say 3.0.5 than the “mute” feature gets automatically reseted. In that you you still get the notification about new versions but you can turn off the weekly reminder for a project.
Let me know what you think.
VersionEye is doing a raffle. You can win a RaspberryPi, Cubieboard or an iPad mini. This is how you participate:
The winner will be announced by end of October.
Good luck đ
Robert Reiz 