Inject SSH pub key to Vagrant image

Usually if you create a Vagrant VM an insecure private key gets injected into the VM, which is located at ~/.vagrant.d/insecure_private_key. In Ansible you can reference that key to ensure a passwordless login to the VM. Since Vagrant 1.8.5 this doesn’t work anymore, because of security reasons. That’s why I use now this shell provisioner with a bit Ruby code to inject my public SSH key to the VM:

config.vm.provision "shell" do |s|
  ssh_pub_key = File.readlines("#{Dir.home}/.ssh/id_rsa.pub").first.strip
  s.inline = <<-SHELL
    echo #{ssh_pub_key} >> /home/ubuntu/.ssh/authorized_keys
    echo #{ssh_pub_key} >> /root/.ssh/authorized_keys
    apt-get -y install python-simplejson
  SHELL
end

The 2nd line is pure Ruby code. It reads the public SSH key from the default .ssh directory from the home directory and stores the content in the ssh_pub_key variable. The first 2 lines of the shell provisioner are injecting the SSH key to the authorized keys for the user ubuntu and root.

With that the VM is build together with my own public SSH key and I can login to the VM via SSH without entering a password. That makes it also super easy to handle the VM later with Ansible.

Setting up a Server for Ansible

Ansible is a great tool for automation and configuration management. Maybe you heard already about Chef and Puppet. Ansible is in the same field, but with a much simpler approach. The big advantage of Ansible is that you don’t need a central master server for it and you don’t need to install an “Ansible” client on the servers at all. Ansible only needs to be installed on your local machine and it works completely via SSH. AnsibleLogo_transparent_web Although you don’t need to install a client on the server, Ansible makes a couple assumptions about the server. Ansible assumes that the server is accessible via SSH. That’s the only real requirement. In the Playbooks you can configure username and password to access the server.

However life becomes much easier if we can assume that you:

  • have a user on the system with username “ubuntu”.
  • have the ssh daemon up and running on the server.
  • be able to login as user “ubuntu” to the server via ssh without password.
  • be able to run sudo su without password popup.

All this assumptions are default on AWS EC2 instances. If you setup a new server somewhere else you should take care of this. This blog post describes how to ensure this assumptions on Ubuntu 14.04.

User ubuntu

If you create a new virtual machine make sure that the default user is ubuntu. If you have already a linux instance up and running you can add a new user like this:

sudo adduser ubuntu

SSH Daemon

This command will return the status of the SSH daemon:

sudo service ssh status

If the feedback is that the service ssh is unknown you can install the SSH Server with this command:

sudo apt-get install openssh-server

Now try again the first command. It should response with status running.

Login without password

Assuming you have already a public/private key pair and the public key on your development machine is under .ssh/id_rsa.pub you can achieve login to the server without password with this 2 commands:

ssh ubuntu@IP_ADDRESS mkdir -p .ssh
cat .ssh/id_rsa.pub | ssh ubuntu@IP_ADDRESS 'cat >> .ssh/authorized_keys'

Replace “IP_ADDRESS” with the IP address of your server! The commands above will still require that you type in your password, but after that you should be able to login to the server without password. Try:

ssh ubuntu@IP_ADDRESS

The full instructions can be found here.

sudo su without password

Simply run this command on the server:

sudo visudo

And add this line to the bottom of the file:

ubuntu ALL=(ALL) NOPASSWD: ALL

Now logout and login again as user ubuntu to the server. Try:

sudo su

Now you should be root on the server. For more questions check out this page.

Done

If you executed the steps above your server is ready for Ansible. Now you can point Ansible Playbooks to your server and use it for provisioning. I’m using Ansible since 2 years together with Docker for the VersionEye Project and so far I really like it!

vagrant push => “error starting upload: upload: resource not found”

I’m just playing around with Vagrant and as soon I tried:

vagrant push

I got this error message:

error starting upload: upload: resource not found

Luckily there is already a ticket to that on GitHub. And the workaround is this export:

export ATLAS_TOKEN=`cat ~/.vagrant.d/data/vagrant_login_token`

After executing the export above the command worked for me.

Ubuntu enforce memory freedom

Sometimes it occurs that a device runs full with memory. Simply deleting the files on the device doesn’t help. Ubuntu stills shows that the device is 100% full, even if there are no files on it anymore. This command usually helps to give free the memory.

sudo tune2fs -m 0 /dev/sda5

If it doesn’t help you need to execute this command as well. It quits all processes which are using the device.

sudo fuser -km /mnt/share

Docker Introduction

Docker is one of the most promising technologies these days. It is a container technology based on Linux. A very lightweight form of virtualization. Docker containers can be as small as 50 MB. Much smaller than traditional VMs.

The Docker project was started in March 2013 by dotCloud. In the mean while the makers of dotCloud sold dotCloud to CloudControl and raised $55 Million to focus only on the Docker development.

Check out my slides to “Docker Introduction”. I did this talk at the Webmontag in Mannheim. Feedback was very good.

I’m using Docker since beginning of 2014 in production and I love it. It’s a great technology!

Java HTTP Request with Basic Auth

This is how you do a simple HTTP request with Java. These code performs the actual HTTP request and saves the response in a String variable.

URL url = new URL(address);
URLConnection conn = url.openConnection();
conn.setConnectTimeout(30000); // 30 seconds time out
String line = "";
StringBuffer sb = new StringBuffer();
BufferedReader input = new BufferedReader(new InputStreamReader(conn.getInputStream()) );
while((line = input.readLine())!=null)
  sb.append(line);
input.close();
String response = sb.toString();

If the HTTP server requires Baisc Auth this code will fail. To make it work for Basic Auth these 3 additional lines are required.

String user_pass = username + ":" + password;
String encoded = Base64.encodeBase64String( user_pass.getBytes() );
conn.setRequestProperty("Authorization", "Basic " + encoded);

The whole method looks like that:

public String getHttpResponse(String address, String username, String password) throws Exception {
  URL url = new URL(address);
  URLConnection conn = url.openConnection();
  conn.setConnectTimeout(30000); // 30 seconds time out

  if (username != null && password != null){
    String user_pass = username + ":" + password;
    String encoded = Base64.encodeBase64String( user_pass.getBytes() );
    conn.setRequestProperty("Authorization", "Basic " + encoded);
  }

  String line = "";
  StringBuffer sb = new StringBuffer();
  BufferedReader input = new BufferedReader(new InputStreamReader(conn.getInputStream()) );
  while((line = input.readLine()) != null)
    sb.append(line);
  input.close();
  return sb.toString();
}

MongoDB Map & Reduce with Date filter

We are using MongoDB as primary DB at VersionEye, together with MongoID. Software package is a document in the “products” collection. These products collections has a subcollection with “versions”. Assume we want to know how many versions/artifacts existed for a given language to a given time?

That is not a simple query in MongoDB. This kind of queries can be handled with Map & Reduce. With Map & Reduce you can execute JavaScript on DB Level. Here is the current solution:



border = until_date.at_midnight + 1.day

map = %Q{
  function() {
    if ( this.versions == null || this.versions.count == 0 ) return;

    that_day = new ISODate("#{border.iso8601}");
    for (var version in this.versions){
      created = this.versions[version].created_at
      if (created != null && created.getTime() < that_day.getTime()){
        emit( this.versions[version]._id, { count: 1 } );
      }
    }
  }
}

reduce = %Q{
  function(key, values) {
    var result = { count: 0 };
    values.forEach(function(value) {
      result.count += value.count;
    });
    return result; 
  }
}

Product.where(:language => language, :created_at.lt => border ).map_reduce(map, reduce).out(inline: true)

The tricky part was this line:

that_day = new ISODate("#{border.iso8601}");

To find out how to convert a Ruby Date object into the JavaScript Date object.

Otherwise you have to know that even through you are iterating over a versions collection you can not access the version object through “version”! You have to access it this way:

this.versions[version]

Otherwise it works fine 🙂

How to get notified about out-dated dependencies in build.sbt?

Scala SBT is the build tool for the programming language Scala. You can specify your project dependencies in the “build.sbt” file, similar to the pom.xml file in Maven. A build.sbt file can look like this.

name := "s3crate"

organization := "codexica"

version := "0.0.1-SNAPSHOT"

scalaVersion := "2.10.2"

libraryDependencies += "net.java.dev.jets3t" % "jets3t" % "0.9.0"

libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.3.33"

But how do you get notified about new versions of your dependencies?

VersionEye now supports Scala SBT. VersionEye can monitor your build.sbt file on GitHub / Bitbucket. If you are logged in to VersionEye you just have to navigate to your GitHub / Bitbucket repository and flip the switch beside the build.sbt file.

01-VersionEye-SBT

After fetching and parsing the build.sbt file successfully the file name turns into a link. You can enter the project detail view by clicking on the link. Here you can see which of your dependencies are out-dated and what is the current version.

02-VersionEye-SBT

By clicking on the Link “Visual Dependencies” VersionEye will show you all transitive dependencies, visualized as a super cool JavaScript Graph 🙂

03-VersionEye-SBT

VersionEye will check your build.sbt file once a week and send you email notifications about out-dated dependencies. If everything is up-to-date you will get no email.

Facebook Share Link

Let’s say you have an awesome social network, a dating platform for cars, and you want to share your cars profile on Facebook. It should look like this. How do you do that?

share

First of all you should load Facebooks JavaScript SDK. Insert this code below the “body” tag in your page.

<div id="fb-root"></div>
<script>(function(d, s, id) {
  var js, fjs = d.getElementsByTagName(s)[0];
  if (d.getElementById(id)) return;
  js = d.createElement(s); js.id = id;
  js.src = "//connect.facebook.net/en_US/sdk.js#xfbml=1&appId=99999999999999&version=v2.0";
  fjs.parentNode.insertBefore(js, fjs);
  }(document, 'script', 'facebook-jssdk'));
</script>

Replace 99999999999999 with your Facebook App ID. Next you need a JS function to trigger the share dialog. That can look like this:

function share_my_car(kfz, kfz_id, image_url){
  url_to_share = "http://contactmycar.de/cars/" + kfz_id;
  FB.ui({ method: 'feed',
          name: kfz,
          caption: 'Mein Fahrzeug auf ContactMyCar',
          picture: image_url,
          link: url_to_share }, function(response) {});
}

And finally you need a link/button to trigger the JS function. I used a link.

<a onclick="share_my_car('<%= car.kfz %>', '<%= car.id %>', '<%= car.image_url %>'); return false;" href="">
 Fahrzeug auf Facebook teilen
 </a>

That’s it.

Global Variables with Ansible

Ansible is a great tool for IT automation. It’s similar to Chef and Puppet, just better! 

I wrote here already an introduction to Ansible. In Ansible you have roles and playbooks. A playbook can contain several roles. Inside a role you can have tasks, files, handlers and variables. But sometimes you want to use a variable across multiple roles. Unfortunately Ansible doesn’t have really a concept of global vars, but there is a trick 😉 

You can define group variables for ALL groups, that is indeed like a global variable. I manage my Ansible code in a git repo. The file tree looks like this: 

infrastructure
  \__ group_vars
     \__ all
        \__ vars_file.yml
  \__ roles 
  \__ ansible.cfg
  \__ site.yml
  \__ hosts

 

The variables defined in “group_vars/all/vars.yml” are accessible from ALL roles! 

Geek2Geek at Wikimedia Deutschland

Tomorrow I’m organizing again a Geek2Geek in Berlin. The topic this time is “NoSQL Ring Architecture”. Famous implementation are Cassandra and RIAK. We will have talks to both of them! The event is hosted by Wikimedia Germany in Kreuzberg. 

Screen Shot 2014-08-26 at 17.31.09

We will have free Beer and Pizza, thanks to our sponsors. Check out the MeetUp page and signup for free. You are welcome to join! 

What is maven-compat ???

What is the maven-compat project? I was wondering. So I went to the official page of the project and found this: 

Screen Shot 2014-08-24 at 11.40.23

Maven2 classes maintained as compatibility layer.

 That is not very helpful! I would like to have a little bit more information! So clicked on the “Project Summary” link and got this. 

Screen Shot 2014-08-24 at 11.43.21

All right. Some auto generated site. Nothing I could not lookup in the pom.xml itself. And I still don’t know much more about this project. What does it do exactly? Why I need it? 

Unfortunately many Java projects are documented as bad as this one. Making an SMC public and auto generating a couple HTML pages doesn’t make it to a good Open Source project. Everytime I see something like this I get angry. Because the maintainers don’t give a shit about the community!

And by the way, JavaDocs are very poor docs! Everybody can generate JavaDocs. Why not put the project on GitHub or Bitbucket and provide a nice README with a red line, with a good intro into the project? That makes all the difference between a good and a bad open source project. 

Requirements for the perfect Crowd Software Testing Tool

The StartUp testCloud – a crowd software testing provider based in Berlin – is offering a service fro crowd software testing. 

testcloud_-logo-14

Now they are working on a new Crowd Software Testing Tool, which is completely self-service. They prepared a 3 min survey to get known your needs for the perfect Tool:  

https://www.umfrageonline.com/s/d3d0f6b

Everybody who finishes the survey can use the new tool for free up to 5 Bugs after official launch.  

I finished the survey by myself and I’m very curious to see the results.

PDFKit – invalid byte sequence in US-ASCII

I’m using PDFKit at VersionEye to generate the PDF invoices. It’s a really awesome project. The idea behind PDFKit is that you generate the documents as HTML and CSS and then convert it to PDF. That works really well. Generating a PDF works like this:

kit = PDFKit.new(html, :footer_html => footer_file, :page_size => 'A4')

The first parameter “html” is the HTML as string. In addition to that you can give a separate path to a HTML file as footer. And of course you can choose the output format. In this case DIN A4.

That worked all really well, but sometimes I got a

invalid byte sequence in US-ASCII Exception

I found out that there was some kind of special character in the HTML. That can happen if you fill the HTML template with usernames for example, and one of the users is a French dude or even worst a Chinese dude, then you have some odd characters in your markup 🙂 But luckily there is a solution for that. You can enforce UTF-8 encoding for the string.

This line fixed it for me.

html = html.force_encoding(Encoding::UTF_8)

Reference Badges

VersionEye supports reference badges for open source projects now! A reference badge shows how many other projects depend on a selected software package. This here for example is the reference badge for PHPUnit.

Reference Badge for PHPUnit

It shows immediately that 7704 PHP projects are using PHPUnit. Awesome! Right?

The conclusion is that as more references a project has as more important it is. VersionEyes reference badge can be integrated into Markdown and HTML. For example into a GitHub Readme page. In that way everybody can immediately see the relevance of the project. Read more about it here.

Introduction to Ansible

Ansible is a great tool for IT automation.

I’m using Ansible to manage the whole infrastructure for VersionEye. Currently I have 36 roles and 15 playbooks defined for VersionEye. I can setup the whole infrastructure with 1 single command! Or just parts of it. I even use Ansible for deployments. Deploying the VersionEye crawlers into the Amazon Cloud is 1 single command for me. And I even rebuild the capistrano deployment process for Rails apps with Ansible.

AnsibleLogo_transparent_web

I just wrote an introduction to Ansible on the VersionEye Blog. Check it out here.