I plan to use NoSQL in the not-too-distance future, but not today. When I port the data over I don’t want to have to map “old ids” to “new ids”. By using a uuid from day one I can easily maintain referential integrity when I migrate to a new datastore.

I looked up how this is typically done in Rails using MySQL and all I could find was examples that store the UUID as strings (e.g. here). The UUID is natively just 16 bytes so storing this in a VARCHAR(36) seemed like a poor solution. Ideally we would store the UUID as binary and let the ORM deal with translating it to and from a UUID object.

activeuuid solves this problem. activeuuid is a Rails plugin that will let you use UUIDTools::UUID objects as primary keys for your ActiveRecords and it stores them in MySQL as binary(16).

Here’s how to use it:

Create a Migration

activeuuid adds the uuid type to your migrations. Example:

class CreateEmails < ActiveRecord::Migration
  def self.up
    create_table :emails, :id => false  do |t|
      t.uuid :id, :unique => true
      t.uuid :sender_id  # belongs_to :sender
 
      t.string :subject
      t.text :body
 
      t.timestamp :sent_at
      t.timestamps
    end
    add_index :emails, :id
  end
 
  def self.down
    drop_table :emails
  end
end

include ActiveUUID::UUID in your model

class Email < ActiveRecord::Base
  include ActiveUUID::UUID
  belongs_to :sender
end

use it

Here are some example specs:

require 'spec_helper'
 
describe Email do
 
  context "when using uuid's as keys" do
    before(:each) do
      Email.delete_all
      @guid = "1dd74dd0-d116-11e0-99c7-5ac5d975667e"
      @e = Email.new(:subject => "hello", :body => "world") {|e| e.id = UUIDTools::UUID.parse(@guid) }
      @e.save
    end
 
    it "the id guid should be equal to the uuid" do
      @e.id.to_s.should eql(@guid)
    end
 
    it "should be able to find an email by the uuid" do
      f = Email.find(UUIDTools::UUID.parse(@guid))
      f.id.to_s.should eql(@guid)
    end
 
  end
end

Benefits

There are a number of benefits to using UUIDs as primary keys.:

• no id conflict during multi-master write
• no locking due to auto-increment
• with time-based UUIDs you can store a timestamp within your UUID
• you can create natural keys (based on the SHA of model attributes)

Future work

• more transparent support for natural and composite keys
• support for MySQLs `INSERT … ON DUPLICATE KEY UPDATE` syntax
• support a primary column name other than `id`
• work on other databases (Postgres, etc)
• tests

Installation

Add this to your Gemfile

    gem "activeuuid"

Or get the code here: https://github.com/jashmenn/activeuuid

References

• http://bret.appspot.com/entry/how-friendfeed-uses-mysql
• http://kekoav.com/blog/36-computers/58-uuids-as-primary-keys-in-mysql.html
• https://gist.github.com/937739
• http://www.codinghorror.com/blog/2007/03/primary-keys-ids-versus-guids.html
• http://krow.livejournal.com/497839.html
• https://github.com/jamesgolick/friendly

Dependencies

Note that this depends on Rails 3.1 because it uses the custom column serialization added by Aaron Patterson. (See: https://github.com/rails/rails/commit/ebe485fd8ec80a1a9b86516bc6f74bc5bbba3476)

Share:

One of the great things about state machines is that they often can be visualized. To that end I’ve written a small utility that will load a class that uses state_machine and produce dot output which you can use with graphviz.

This gem is currently used by the spree shopping cart. Let’s look at an example of how this can be used to visualize the Order object in this project:

Save the gist below to visualize.rb and chmod +x visualize.rb. The arguments are 1) the path to RAILS_ROOT of the project using state_machine and 2) the class you want to visualize. Try visualize.rb --help to see all the options.


ruby visualize.rb . Order | dot -Tsvg -Grankdir=LR > graph.svg


Gives us the following image:

Download the raw code (Requires ‘trollop’ gem).

Share:

Example usage:

xml = firm.to_xml :include => [ :account, :clients ]  
firm = Firm.from_xml xml

firm.account and firm.clients will behave as intended (by default there does not seem to be a direct way in rails to unmarshall them.)

Here’s how to get from_xml and from_json defined for your ActiveRecord classes:

Create a file lib/extensions.rb with the following code:

module ActiveRecord
  class Base
 
    def self.from_hash( hash )
      h = hash.dup
      h.each do |key,value|
        case value.class.to_s
        when 'Array'
          h[key].map! { |e| reflect_on_association(
             key.to_sym ).klass.from_hash e }
        when /\AHash(WithIndifferentAccess)?\Z/
          h[key] = reflect_on_association(
             key.to_sym ).klass.from_hash value
        end
      end
      new h
    end
 
    def self.from_json( json )
      from_hash safe_json_decode( json )
    end
 
    # The xml has a surrounding class tag (e.g. ship-to),
    # but the hash has no counterpart (e.g. 'ship_to' => {} )
    def self.from_xml( xml )
      from_hash begin
        Hash.from_xml(xml)[to_s.demodulize.underscore]
      rescue ; {} end
    end
 
  end # class Base
end # module ActiveRecord
 
### Global functions ###
 
# JSON.decode, or return {} if anything goes wrong.
def safe_json_decode( json )
  return {} if !json
  begin
    ActiveSupport::JSON.decode json
  rescue ; {} end
end

At the bottom of config/environments.rb, insert the line:

require 'lib/extensions' # custom class extensions

You can name the “extensions.rb” file anything else you want; just update the environments.rb file accordingly.

This is an improvement over the previous post, in that it can work even when your associations use the :class_name parameter, you’re not editing the vendor/rails files directly, and it can deal with hashes of the rails class HashWithIndifferentAccess.

Share:

Unless I’m missing something, the default rails from_json and from_xml methods don’t work with data that have associated objects. Here are routines that work for some simple examples I’ve tested. I placed this code in vendor/rails/activerecord/lib/active_record/base.rb within the ActiveRecord::Base class, but there might be a better place to put it.

def self.from_hash( hash )
  h = hash.dup
  h.each do |key,value| 
    case value.class.to_s
    when 'Array'
      h[key].map! { |e|
         Object.const_get(key.camelize.singularize).from_hash e }
    when 'Hash'
      h[key] = Object.const_get(key.camelize).from_hash value
    end 
  end     
  self.new h
end   
 
def self.from_json( json )
  hash = ActiveSupport::JSON.decode json
  self.from_hash hash
end   
 
# The xml has a surrounding class tag (e.g. ship-to),
# but the hash has no counterpart (e.g. 'ship_to' => {} )
def self.from_xml( xml )
  hash = Hash.from_xml xml
  self.from_hash hash[self.to_s.underscore]
end

Also available on pastie.

Example usage, using the to_xml example:

xml = firm.to_xml :include => [ :account, :clients ]
firm = Firm.from_xml xml

Note: this only works if the classes of the nested objects are canonically named. If any of your associations use the :class_name option, you will probably need to update this code to discover the proper class names.

I just wrote this a couple hours ago, so please consider this to be an alpha version.

Share:

The 19 22-year-old duo was obviously ecstatic about sharing about what they created but I kept getting the feeling that they were surprised that the crowd didn’t give them a presidential-state-of-the-union-like standing-ovation every few minutes. (If passenger does what they claim maybe they deserve it but respect and appreciation often lose something when they are too eagerly expected.)

So what was it that they claimed? That passenger will not only make Rails deployment dead-simple (think PHP, upload and go) but also crank out better performance while using less memory. It’s a worthy goal and as Kent Beck said in our keynote, “Humility is not a prerequisite to ideas with impact”. I’d like to write up a bit about Passenger and the session they presented at RailsConf. You can download the Keynote slides here [zip] (I’m sure the PDF will appear somewhere soon. If you find it, please feel free to leave a URL in the comments).

Memory Usage and Clustering

Memory Usage

First lets talk about memory usage. When you’re using Mongrel each Mongrel process holds both a full copy of the Rails and application code in memory plus the private memory for the individual process. In this model of N Mongrel processes you have N copies of the application code. With Passenger, each process shares one copy of your Rails/application code. Each process still gets its own chunk of private memory but the shared code greatly reduces the overall memory usage.

The Phusion guys also patched Ruby (and they’ve horribly named it “Ruby Enterprise Edition”). This version has modified garbage collection and causes Ruby to use significantly less memory. They achieve this by doing copy-on-write for memory management. This hasn’t been released so no word yet on how well it works or how stable it is.

Clustering

Another nice feature is that with clustering they use “fair load balancing”. The idea is that you keep track of how many jobs each process in the cluster has and you give the next job to the process with the least amount of work to do.

Competitor Comparison

They compared Passenger’s performance (as an Apache extension) to many competing products (including Nginx and Mongrel) and claimed that it used significantly less memory and was much faster. I won’t repeat all the statistics, you can check out the slides.

mod_rails, RIP

Although they had greatly simplified deployment for Rails they didn’t stop there. Passenger now supports Rack. I see this as probably the coolest thing about Passenger. Now any custom server that you write using Rack can be basically “dropped” into Apache and is effortlessly handled by Passenger. This also means that rails alternatives like Merb and Camping work out of the box. But there is more…

They also added in support for Python’s Django. It was almost comical: when they announced this the crowd nearly boo’ed them. I think it was a bit unfair, but I guess they should have expected it at a Rails conference. Either way, you have to give them props for taking the initiative and pushing the software to its boundaries.

Because Passenger supports frameworks other than Rails they decided to drop the name mod_rails and call it Phusion Passenger. They mentioned that their focus is still going to be developing and perfecting Rails. Passenger will simply not be exclusive to Rails.

Case Studies

Passenger is already being used in production at Dreamhost. It is also being used soocial and ilike.

The Q&A

By the Q&A the crowd was full of skeptics; what they promised seemed too good to be true. However, one gentleman from the crowd sensed this, stood up, and said that he works with a developer/deployer Tom that has been working with production Rails deployment for 4 years. Tom apparently knows the in’s and out’s of Mongrel, Monit, Capistrano, etc. They found out about Passenger two or three weeks ago and he deployed all of their existing applications to Passenger in a single day. His comment was:

Everything I learned [about Rails deployment] over the last few years is moot now, and that is a good thing.Tom the deployer

He said that “[Passenger] is incredibly awesome when it comes to rails deployment.”

A skeptic stood and rightly asked: “Why wasn’t this done five years ago? What was the technological hurdle?” The team answered that they believed the problem was largely social. Developers that had written Rails applications wanted to deploy them as quick as possible. They researched, learned about Capistano, Nginx, and Mongrel etc., and made it work. The Phusion team said that the people that were smart enough to tackle this problem were complacent and choose to deploy applications in this (painful) way.

There is nothing technically preventing [ease of rails deployment]. We’ve shown it’s possible. Why it hasn’t been done is a social or political problem. There is no technical things stopping you. -Hongli Lai (Phusion)

Summary

Time will tell us if Apache/Passenger will live up to the hype and become the new standard. I, for one, am hoping that it really does take hold. If Rails deployment becomes as common and easy as PHP deployment we can spend time solving more interesting problems and that will be a good thing.

Share:

rcov

rcov. rcov is a code coverage tool for Ruby. The idea is that you run it on your tests. Lines that get executed are green and “ok” and lines that are never executed are marked red for being dangerous because they are never tested. This is an easy way for you to clearly see which lines of code are never being touched by any of your test cases.

I think that rcov will significantly change the way I approach testing with my own team and consultants. I’ll have to talk to my team, but it seems reasonable to me that every piece of code that is going to be used in production should have at least 50% code coverage. In growing, changing production systems this seems like a very valuable way to keep the code malleable (because you can refactor with less fear of breaking) which then leads to increased confidence and reduced cost-of-ownership.

flog

flog – a ruby code complexity analyzer. Now the first question you may be asking is “what in the world is a complexity analyzer?” The idea is that flog reads your code and assigns it a score based on criteria which is determined by its author, Ryan Davis. Basically it rates on the type of code Ryan Davis likes to see. This isn’t as terrible as it may sound because Ryan has some very tasteful ideas on what readable code looks like.

You may or may not agree with the way Ryan scores things, but as our speaker pointed out, flog and rcov often agree with each other about where a program’s problem areas are. More often than not the high flog score areas are not tested because they are too bulky to be tested. When one writes tests you often have to design the code in smaller, more testable pieces. Readability and testability often go hand in hand.

When running flog it can be helpful to just grep the output for #. This will give a high level view of the methods and their scores. Good candidates for refactoring are methods with scores between 80-150. Higher than 150 are typically serious problem areas (in terms of readability). Make sure you have them very well tested before you try to refactor them too much.

heckle

From the heckle website: “Heckle is a mutation tester. It modifies your code and runs your tests to make sure they fail. The idea is that if code can be changed and your tests don’t notice, either that code isn’t being covered or it doesn’t anything.” The idea is that hackle tests your tests and makes sure you are actually writing tests that are meaningful.

tarantula

tarantula is a “fuzzy spider”. It crawls your web application and submits tons of garbage data and tries raise exceptions in your rails application. The premise of this tool is that no matter what data is posted, your application should not be returning 500 errors. You should handle bad data gracefully and not let unnecessary exceptions bubble to the surface. (I’m not entirely clear on how this gels with RESTful development where one often uses the return status codes to return meaningful information)

summary

It was pointed out that none of these tools replace thought; they are here to enhance it. flog, for instance, is a tool that helps enhance your sense of code-smell. You may run flog, look at a “complex” method and decide that it is perfectly acceptable for your system. Obviously, you as the programmer know better than an automated tool. However, it is often the case that flog will point out passages of code that are challenging and difficult for anyone but the original author to read. The idea is to encourage code that is simple to test, read, and understand. This makes the code more robust and open to change.

Share:

Ruby Internals

This morning Neal Ford and Patrick Farley gave a great session on Meta-programming Ruby for Fun and Profit. The slides should eventually show up
here.

Particularly interesting was Patrick’s portion about the internals of Ruby’s method dispatch and how it relates to inheritance, mixins, and the object’s eigenclass. The basic idea is that when you call a class-method in Ruby the eigenclass (or metaclass) has a parallel inheritance structure to the actual class. I may write more about this with diagrams after Patrick posts the slides, but the the understanding of the system leads to this koan-like ruby truth:

The super-class of the meta-class is the meta-class of the super-class

Don’t worry if it isn’t clear right away. Reid and I
attended the session and then discussed it over launch and it still took a while to sink in. In the meantime, checkout _why’s article Seeing Metaclasses Clearly.

Use class_eval instead of reopening a class

Patrick gave another very handy tip when dealing with meta-programming. He mentioned that using class_eval is much safer than re-opening a class and defining a method. This is because you don’t always know when files are loaded and when you open a class you may be defining it without realizing it. When you use class_eval you are using an existing constant. For example:

 
# warning! this could be defining ExistingKlass and you don't even know it!
# then the original ExisitingKlass will just overwrite your method
class ExistingKlass
  def change_a_method
    # ...
  end
end
 
# do this instead as it will not try to define ExistingKlass. class_eval
# guarantees the class already exists
ExistingKlass.class_eval do 
  def change_a_method
    # ...
  end
end

A Recorder

Neal shared with us a very interesting class that records the messages sent to it`and can play them back later on. Here’s the code for it:

# recorder.rb
class Recorder
  def initialize
    @messages = []
  end
 
  def method_missing(method, *args, &block)
    @messages << [method, args, block]
  end
 
  def play_back_to(obj)
    @messages.each do |method, args, block|
      obj.send(method, *args, &block)
    end  
  end
end
 
# recorder_test.rb
class TestRecorder < Test::Unit::TestCase
  def test_recorder
    r = Recorder.new
    r.sub!(/Java/) { "Ruby" }
    r.upcase!
    r[11, 5] = "Universe"
    r << "!"
 
    s = "Hello Java World"
    r.play_back_to(s)
    assert_equal "HELLO RUBY Universe!", s
  end
end

I haven’t fully processed the power of this idea yet. However, I feel like it could have some application in genetic programming.

Tabula Rasa, the Recorder, and DSLs

Neal pointed out that one of the issues with Recorder is the following:

# recorder_test.rb
class TestRecorder < Test::Unit::TestCase
  def test_recorder
    r = Recorder.new
    r.upcase!
    r.freeze
 
    s = "hi nate"
    r.play_back_to(s)
    assert_equal "HI NATE", s
 
    s.downcase! # <<< shouldn't work because it is frozen!
    assert_equal "hi name", s
  end
end

The issue is that the #freeze method went to the Recorder and not to the string. This is a problem you are likely to run into with a class like this because a standard ruby Object contains about 40 other methods; methods who’s names may conflict with what you want to delegate or capture with #method_missing (like the recorder). Thankfully Jim Weirich has created a BlankSlate class that you can use that will undefine all of the existing methods. If you have Recorder inherit from BlankSlate it will then work as expected. Neal mentioned that this class is being integrated into ruby 1.9 as SimpleObject.

When the slides become available checkout the section on the Quantifier module. The code is a bit lengthy to reproduce here but worth a look.

Share:

and you want to execute a single query that returns all the data at once within the ActiveRecord tables, with the proper rails associations between them. Wouldn’t it be nice if you could do something like

A.find(:all, :include => :b => :c => :d => :e) #v1

? Though this is not even valid ruby code, it actually comes very close to what you can do in Ruby on Rails. To get this right, let’s take a closer look at the rails associations within the class definitions:

class A < ActiveRecord::Base
  belongs_to :b
end
class B < ActiveRecord::Base
  has_many :as
  has_many :cs
end
class C < ActiveRecord::Base
  belongs_to :b
  has_one :d
end
class D < ActiveRecord::Base
  belongs_to :c
  has_and_belongs_to_many :es
end
class E < ActiveRecord::Base
  has_and_belongs_to_many :ds
end

Let’s try the rails code again, putting an ’s’ after the :c and :e as required by rails in order to denote they are “many”-type associations:

A.find(:all, :include => :b => :cs => :d => :es) #v2

That’s closer, but still not valid ruby code. To fix that, think of the => operator as being right-associative, and instead of putting in parentheses (), put in curly braces {} in order to create nested hashes:

A.find(:all, :include => {:b => {:cs => {:d => :es}}}) #v3

That’s it! Looking in the logs, we see that this only produced a single query, with all the desired SQL joins:

A Load Including Associations (0.001088) SELECT `as`.`id` AS t0_r0, `as`.`b_id` AS t0_r1, `bs`.`id` AS t1_r0, `cs`.`id` AS t2_r0, `cs`.`b_id` AS t2_r1, `cs`.`d_id` AS t2_r2, `ds`.`id` AS t3_r0, `ds`.`c_id` AS t3_r1, `es`.`id` AS t4_r0 FROM `as` LEFT OUTER JOIN `bs` ON `bs`.id = `as`.b_id LEFT OUTER JOIN `cs` ON cs.b_id = bs.id LEFT OUTER JOIN `ds` ON ds.c_id = cs.id LEFT OUTER JOIN `ds_es` ON `ds_es`.d_id = `ds`.id LEFT OUTER JOIN `es` ON `es`.id = `ds_es`.e_id

With this tool in mind, you can use this in any ActiveRecord function that accepts the :include option to reduce the number of times the rails app hits the database, and ultimately speed up your rails application.

Share:

This is a simple rails log analyzer. The initial goal is to
identify SQL queries that could be optimized or find queries that are being
called more often than they should be.

In its current state it is little more than simple regexes. My goal, however is
to build more useful tools on top of it. I’ve added it to a git repository to
allow anyone to make edits and improvements.

It’s not really ready for prime-time (for instance, it doesn’t work well
with log colorization), but I have found it useful so I thought I would share
it. If nothing else it could save someone else 10 minutes to write
a similar script.

What it does

Right now all this does is read a log file and extract SQL “SELECT” statements
and organize them by the amount of time they take. It will also show a summary
of the number of queries called to load each model.

How to get it


git clone \
git://gitorious.org/simple_rails_log_analyzer/mainline.git \
simple_rails_log_analyzer


How to use it

The typical work flow looks like this:

rake log:clear
# hit the URL of the page we want to profile, run the rake task, etc
ruby bin/query_log_analyzer.rb log/development.log


Or visit http://gitorious.org/projects/simple_rails_log_analyzer

TODO

• Deal with the colorization of the logs
• Update the code to not be so SELECT centric. Support other types of queries
• Be smarter about each request’s individual queries

Contributions

I’ve placed this in a git repository, so feel free to submit any patches and
I’ll integrate them into this master.

Similar Projects

• http://rails-analyzer.rubyforge.org/

Share: