On Riding the Latest Rails
December 29, 2011 § Leave a comment
Note: I wrote this up a few weeks a go, but didn’t publish it for some reason. It is a touch out of date since Rails 3.2 RC1 was recently released, but should be close enough to give you a good idea on how to get up and running on Rails master.
I wanted to play with the latest and greatest for Rails, which at the time of writing is 3.2.0.beta (RC 1 is now out). This means building and installing the rails gem locally. It took me a good while to figure out how to get everything working, so I figured I’d throw it down here for future reference… and for anyone else running into the same problems I did.
I am using rvm to isolate my different gem set ups, so as not to get random version conflicts. These are the lists of commands with notes I ran to get everything up and running:
# create a new gemset to use with rails master rvm gemset create rails-master rvm gemset use rails-master # this is where I am going to put all of the git repos cd ~/src # Rails 3.2.0.beta requires the most recent (as in master) # versions of arel and journey git clone git://github.com/rails/arel.git cd arel gem build arel.gemspec gem install arel-*.gem cd .. git clone git://github.com/rails/journey.git cd journey gem build journey.gemspec gem install journey-*.gem # now get Rails cd .. git clone git://github.com/rails/rails.git cd rails # install all of the Rails pieces, # i.e., ActiveRecord, ActiveSupport, etc... cat RAILS_VERSION | xargs ruby install.rb # check to make sure you are riding the correct rails ruby -r 'rails' -e 'puts Rails::VERSION::STRING'
You can get just the commands in a gist too.
On Not Rolling Your Own Verification or Encryption
November 10, 2011 § Leave a comment
When developing code, there are somethings that you should leave to the experts. Encryption is one of them. When I wrote my encrypted cookies and encrypted cookie sessions gems, one of the the things I didn’t want to do was write any sort of encryption routines. Luckily ActiveSupport has some help in the way of ActiveSupport::MessageEncryptor. It’s used much like ActiveSupport::MessageVerifier which is used for signed cookies in Rails. People much smarter than me have put these pieces together, so it just makes sense to use them. Almost nothing good can come from trying to do this stuff yourself. Here are some examples of a verifier and encryptor being used.
ActiveSupport::MessageVerifier
> secret = ActiveSupport::SecureRandom.hex(10)
=> "379af645b8dcce20b607"
> verifier = ActiveSupport::MessageVerifier.new(secret)
=> #<ActiveSupport::MessageVerifier:0x00000103d7ce50 @secret="379af645b8dcce20b607", @digest="SHA1">
> signed_message = verifier.generate("sign this!")
=> "BAhJIg9zaWduIHRoaXMhBjoGRVQ=--af1e810b074b1abd6d9dcd775f71b1fafa53c218"
# this is "<base 64 encoded and serialized string>--<digest of string>"
> verifier.verify(signed_message)
=> "sign this!"
> verifier.verify(signed_message + "alittleextraontheend")
ActiveSupport::MessageVerifier::InvalidSignature
> verifier.verify("alittleextraatthebeginning" + signed_message)
ActiveSupport::MessageVerifier::InvalidSignature
ActiveSupport::MessgeEncryptor
> secret = ActiveSupport::SecureRandom.hex(20)
=> "c1578de6ec2e1789940729dc9d97b335fc7df588"
> encryptor = ActiveSupport::MessageEncryptor.new(secret)
=> #<ActiveSupport::MessageEncryptor:0x000001295337c8 @secret="c1578de6ec2e1789940729dc9d97b335fc7df588", @cipher="aes-256-cbc">
> encrypted_message = encryptor.encrypt_and_sign("Nothing to see here...")
=> "BAhJIl9YbDZkK0czS3o0ZkI0Yml6K05uYzgzM05meDJjWWU4QWh0YzdFeFFrbC85b3BocHFORWtRWXdDVWIxaW45TEQ5LS1yVkxGTURJYzFWb2pva0UrVkkwTkFnPT0GOgZFRg==--e61e02a818960d66c7865f5624fad63b1564283f"
> encryptor.decrypt_and_verify(encrypted_message)
=> "Nothing to see here..."
If your secret is too short you’ll get a OpenSSL::Cipher::CipherError. Make sure your secret is at least 32 bytes. Using ActiveSupport::SecureRandom.hex(16) should satisfy this requirement, but obviously longer is better. You can also pass in a :digest => <digest> option as a second argument to both initializers to specify a different algorithm to use.
One of my thoughts is to submit these two gems into Rails so that people don’t make the mistake of trying to roll their own encryption systems for cookies. We’ll see how it goes.
On New Milestones
November 7, 2011 § Leave a comment
Earlier this year I got the itch to start doing some open source stuff. I hadn’t done anything of the sort before, so I didn’t really have any idea where to begin. There have definitely been some bumps along the way, but I think that I have made some progress. Looking back on my progress I identified a few milestones I hit along the way. They weren’t obvious at the time, and I certainly can’t say that I meant to achieve each one, but I can clearly see them now. Each one helped me a little more on my way to being an open source contributor. Each one also elicited different thoughts and feelings. Now in no particular order and mostly for me to remember what I was thinking…
First Repository Uploaded to GitHub
I signed up for GitHub back at the end of 2008. I forked a few repos here and there just because there was a button to do it. I didn’t really know anything about git and didn’t do anything with repos I had forked. I didn’t actually upload a repo of my own until the end of March 2011. I had been working with subdomains in Rails 3 and experimenting with the new cookie jar chaining. I found a use case for adding another cookie jar that assigned the domain of the cookie. I abstracted it out, gave a poor project name and put it up on GitHub. [tld-cookies]
It was a bit nerve-racking putting something you have made out there for whole world to see. I am sure artists feel the same way when they show their paintings or writing to the general public. I had no expectations that people would look at it or care about it, but still it was out there for people to scoff at if they saw fit to. Nervous as I was, I felt something that I wasn’t expecting… liberation. Even if no one ever used my code, I was now a contributing member of the open source community. It felt great.
Two more repos went up almost immediately.
First Gem Published
When I pushed my first repos, the code was more or less in it’s finalized state, so the gem(s) followed quickly behind. I remember having my RubyGems.org dashboard up all day watching the those first downloads trickle in. I quickly realized that the first dozen or so were mirrors downloading all the new and updated gems. Not going to lie, was a little sad when I realized that. That first gem still sits at 50 downloads, probably because of it’s bad name, but some of the other ones have a few hundred downloads.
It’s awesome. Some of my peers, not my coworkers, are using my code. Building things that people use is awesome, whether it’s a website or a library. As a software developer, I get a thrill when websites I work on get traction and usage. Building successful businesses is my end goal, but with that being said, I get a very different and more personal thrill when I see some of my code being used by other developers. It is a type of validation of your technical skills and it feels great.
First Gist Uploaded
We are a rails shop at work. One of the things that I was getting annoyed with when dealing with debug statements was where to print these statements out to. Do I print them to the log file or to standard out? The log file is the obvious choice, but if you are working in the console, then using puts so you don’t have to switch terminal windows might be preferable. I spent some time and came up with a useful little function. I realized that this might be useful to others so I put it up on GitHub as a gist, and called it a day. Looking back, this was really the first time I just casually threw something out there that I thought might be interesting, but wasn’t a fully functioning library.
First Issue Filed Against Me
Then came the bugs… I had my first GitHub issue opened against me about two months after I published my first gems. Now here is the funny thing, I got excited. Like watching the downloads trickle in on RubyGems.org, someone opening a bug against you means they care enough about what you are doing to want/need it to work right. Obviously I want it to work as well, but I know that I am bound to let bugs slip through. When someone took the time to report a bug and identify the potential source of the problem, I knew that I was doing something right. Well, something was wrong but the situation was right… right? [bug]
First Issue Filed Against Me Fixed
Fixing the issue and pushing out a new version was satisfying. I had the satisfaction of knowing that I cared enough about a little side project and the people using it to fix the issues and rerelease the gem.
Dealing With Future Compatibility
When I first pushed out my encrypted-cookie-store gem it was Rails 3.0 only. I figured I’d update it when 3.1 was released because I wasn’t expecting people to really care that much about it. Turns out I was mistaken. Not long after it was pushed out I got a request to add 3.1 compatibility. Well, one of my customers was asking for it and I was going to have to do it sooner or later, so I fired up rvm and got to work on figuring out what changed between 3.0 and 3.1. It wasn’t that big of a deal, but having different methods defined based on the which gems are installed seems like a less than ideal way to go about things. Thus concluded my first attempt at programing against an unreleased version of a project.
First Pull Request for One of My Projects
Now having someone open an issue against you and pointing you in the direction of the problem is nice and all, but having someone open a pull request against your project because they liked/needed it enough to fix the issue themselves, that is an awesome feeling. My little sparse matrix library was getting some love from across the Atlantic. Couldn’t have been happier.
First Issue Opened Against Another Project
Almost as scary as putting your own code out there is opening an issue against a well established project like Rails. You keep asking yourself, “Am I doing something wrong?” or “Do I just not understand what is going on?” I mean, what if your issue is just a result of you being stupid and not knowing what you are doing? These guys are busy and don’t really need to deal with bugs that probably aren’t actually bugs. So you run every test case you can think of, and then some that have nothing to do with your issue, you know, just in case. Then you hope that everyone is nice to the newbie. PS – They were.
First Pull Request to Another Project
It made me even more nervous to offer up a fix. Opening that Pull Request was nerve racking. I was patching one of the most used methods in ActiveRecord, so I obviously didn’t want to mess up. My pull request went through a couple of iterations, mostly on my test cases, but was finally accepted and merged into master 11 days after I opened the initial issue. Once again everyone was great and very helpful.
Started Blog
And then the blog started. I won’t lie to you, the last bit of motivation I needed to finally get this blog up and running was bit of selfishness. I realized that I needed some way to let people know about the stuff I had put out there that might be useful to them. This obvious thought happened when I came across a blog of someone working on a similar problem with encrypted cookies saying that they were thinking about packing it up and submitting it to rails. Since I liked the way I was doing it better, I figured I needed something other than the GitHub page as a way to promote it. Since then I have started writing on more topics than just the gems I have written. A blog, when properly utilized, is much more than just an advertising platform for your own code.
Being Asked to Contribute to a Project
The most recent development in my open source career is being asked to contribute to SciRuby. They liked the work I was doing on a sparse matrix library I was working on, and asked if they could include it in their offerings. Obviously I was ecstatic. Then I started thinking about the state of the code and how much work was left to do on it. Then I started worrying. It is very much still in an alpha/incomplete state. But now that I know that I have some people that are interested in it, I should be able to get working on it some more. Get that math side of my education some more use.
Conclusions
I have come a long way with my contributions to the open source community. From using to creating to contributing, each of these milestones has been a new challenge and a new experience. This isn’t a how to get a start in OSS, but it a list of things that you can look at for next steps when you are stuck and want to get more involved. It’s easy and most people will be excited that you want to help or are offering something up they can use. Even if you don’t want to write something, use OSS. That is your true first step.
On New Gems
November 1, 2011 § Leave a comment
One doesn’t go through the process of starting a new ruby gem all that often. Even if you have published a few, it is easy to forget how get one started. So here, for my benefit as much as anyone else’s, is a quick easy way to get a gem started and hosted on GitHub.
On the command line navigate to the directory you want to use to work on your different gems.
> cd ~/Projects/gems
I like using bundler, so this is what I do:
> bundle gem <gem name>
create <gem name>/Gemfile
create <gem name>/Rakefile
create <gem name>/.gitignore
create <gem name>/.gemspec
create <gem name>/lib/<gem name>.rb
create <gem name>/lib/<gem name>/version.rb
It will create a directory for you with the gem name and initialize a git repo for you with everything already staged to be committed. Now you need to create a new repo on GitHub. It’s pretty straight forward, so I’ll let GitHub’s on screen instructions take it from there. My only suggestion would be to make the project name the same as your gem name to avoid confusion.
Note: If you use dashes “-” instead of underscores “_” in your gem name, bundler will treat them as module hierarchies and build folders to accomodate.
Now cd into your new gem’s directory.
> cd <gem name>
Hook the git repo up with GitHub. The link should be shown on the empty repo’s main page. Just copy that.
> git remote add origin git@github.com:<GitHub username>/<GitHub Project Name>.git
Now commit your skeleton. Everything is already staged for you.
> git commit -m "Initial commit"
Now push it out to GitHub
> git push origin master
And just like that you are ready to start working on your new ruby gem. You’ll also want to remember the following commands:
> rake build # build <gem name>-<version> into pkg directory > rake install # build and install <gem name>-<version> on local machine > rake release # tags the repo with the version number and pushes new version out to RubyGems.org
Google will be able to tell you more about filling out the .gemspec file than I can. This is just to get you up and running.
On Pure Ruby Sparse Matrices
October 18, 2011 § Leave a comment
A while ago I was working on a web app built on Rails that needed some very basic tag analysis. Having once upon a time obtained a MS in Computational Mathematics I will always view tags as a big matrix. A big, sparse matrix with tag names along one dimension and the objects being tagged along the other. When you think about tags like this all kinds of avenues of analysis open up.
You can think of objects as tag vectors (or tags as object vectors) with non-zero entries representing the tags on an object. By multiplying two tag vectors together you get the dot product (or cosine similarity) of the two vectors. This a nice rough estimate of how similar two objects are with respect to their given tags. Multiplying a tag vector (a.k.a. the query vector) with whole matrix will find you the most similar objects (or tags). Math is fun!
To this end I started poking around trying to find a sparse matrix library written for ruby to make my life easier. Alas, no glory. Now I know that Ruby doesn’t have the same math support that, say python has, but I would have thought someone would have thrown something usable together.
Side note: NumPy and SciPy are awesome.
I ended up hacking together a very basic SparseVector class to take care of my immediate needs, but felt that there was a chance to create something new for the community. I looked at things like LAPACK ruby binding and NArray, but these all dealt with dense matrices. My data was incredibly sparse, so they didn’t really cut it. With that said, I have been working on a sparse matrix gem. It’s a pure ruby implementation, so it should be nice and portable and good enough for small prototypes and datasets. I wanted it to be in Ruby so that I could subclass it off of the standard library Matrix class and have it run anywhere Ruby runs. Not having any bindings to compile is a plus for me, since it means I don’t have to worry about the different environments it might be running in. If you are working with really large datasets then this probably isn’t for you right now.
Check out my progress over on GitHub.
On Cross Subdomain Cookies
July 12, 2011 § 1 Comment
The first Ruby gem I ever wrote was tld-cookies. While it is very poorly named, probably should have been called root-domain-cookies or something like that, it adds a nice little bit of functionality to the Rails 3 cookie jars.
One of the things about Rails 3 that I thought was really cool, was the way cookies were accessed. It’s not a big and fancy piece of code, but to me it is just a slick way to do things. The chaining of the different cookie jars makes it trivial to create the cookies you want and need.
cookies.permanent.signed[:awesome_cookie] = "cookies awesomeness" cookies.signed[:awesome_cookie] => "cookies awesomeness"
At the time I was working on a project at work that required the use of a lot of dynamic subdomains, and we wanted to be able to write cookies across all of the subdomains as well as for individual subdomains. In Rails 3 you could set the domain when you write to the cookie like:
cookies.signed[:awesome_cookie] = { :value => "cookies awesomeness", :domain => "example.com" }
cookies.signed[:awesome_cookie_sub] = { :value => "cookies awesomeness subdomain", :domain => "sub.example.com" }
Now that is a lot of extra work and looks pretty ugly. You could set the default domain for you cookies like this:
Rails.application.config.session_store :cookie_store, :key => '_app_name_session', :domain => :all
But I guess I’d rather explicitly say when a cookie is to be used across all subdomains. To this point I tld-cookies add a tld cookie jar to your Rails 3 app which sets the domain for the cookie to be the root domain, i.e. example.com.
cookies.tld.signed[:tld_cookie] = "ACROSS ALL SUBDOMAINS!!!" cookies.signed[:tld_cookie] => "ACROSS ALL SUBDOMAINS!!!"
As you can see above, you use it similarly to how you would use the permanent cookie jar. The slight difference is when you want to delete the cookie you have to use the tld accessor.
So yeah, first Ruby gem. Poorly named, fun little learning project.
On Selecting A Single Column
July 9, 2011 § 1 Comment
Many times when we are selecting a rows out of the database we just want a single column and have no need for the entire object. There are a number of ways to accomplish this with ActiveRecord. One can get all the records from the database and then collect the attribute needed:
Posts.where(:status => 'published').collect(&:id) => [ 1, 5, 8, 10 ]
This has the benefit of being able to us any overwritten accessors, but has a lot of overhead associated with generating the objects. Another way to do it is to go directly to the database:
ActiveRecord::Base.connection.select_values("SELECT id FROM posts WHERE status = 'published'")
=> [ 1, 5, 8, 10 ]
This is much faster, but requires one to use the direct connection to the database and have the SQL literal prepared. Not particularly user friendly even if you can get the SQL literal using the to_sql:
ActiveRecord::Base.connection.select_values(Posts.where(:status => 'published').select(:id).to_sql) => [ 1, 5, 8, 10 ]
Wouldn’t it be nicer if you could just do the following:
Post.where(:status => 'published').select_column(:id) => [ 1, 5, 8, 10 ]
The select-column gem provides the above functionality above. You can you it in your Rails 3 app or checkout the source code over on github.
Usage
select_column accepts a single optional argument. This is the column that you want to have returned in an array. The returned column can also be specified using the select query method.
If neither a select nor an argument is given, :id is assumed to be the column to be returned. If multiple select query methods are present, the first one defined will be the column returned.
Some examples:
# selects an array of ids Post.select_column # selects an array of titles Post.select_column(:title) # selects an array of ids Post.where(:status => 'published').select_column # selects an array of titles Post.where(:status => 'published').select_column(:title) # selects an array of titles Post.select(:title).where(:status => 'published').select_column
Update (Jan 21, 2012): It’s like they keep looking at my gems and integrating them into Rails. As of Rails 3.2 this gem’s functionality has been replicated by ActiveRecord::Relation#pluck. Check it out in the release notes.
On Encrypted Cookie Sessions
June 16, 2011 § 1 Comment
Once I had finished up with the encrypted-cookie gem, it seemed like a natural extension to convert it into a Rails 3 session store. It operates just like the basic cookie session store, just using an encrypted cookie instead of a signed cookie. It uses the encrypted-cookie gem, so all the encryption is provided by ActiveSupport::MessageEncryptor. To start using it add the following to your Gemfile:
gem 'encrypted-cookie-store'
And change your session store in config/initializers/session_store.rb
AppName::Application.config.session_store :encrypted_cookie_store, :key => '_app_name_session'
The dependencies will include the encrypted-cookie gem for you. Accessing the session is the same as always:
session[:tid_bit] = "of information" session[:tid_bit] # => "of information"
You can check out the source over on github.
Currently this only works with Rails 3.0.*. All of the session code got switched up for Rails 3.1, so it’s going to take some extra work to get it working for the new release of Rails. Update June 18: Got it working with Rails 3.1. Yay conditional method definitions!!! Sigh…
On Encrypted Cookies
June 16, 2011 § 2 Comments
I love cookies. Well, despite obviously loving the cookies of the baked goods variety, I also love Rails 3 cookies. I know it’s weird, but if you are looking for things that aren’t weird, you probably are in the wrong place. The specific thing I really like about the Rails 3 cookie system is that it is chainable. So cool.
They are three types of built-in cookies. Your basic everyday cookie, a permanent cookie and a signed cookie. The basic cookie just saves something to a cookie in the user’s browser. You can add an expiration date if you’d like. The permanent cookie sets the expiration date 20 years from now and the signed cookie cryptographically signs the contents of the cookie so that people can’t tamper with it. The way you interact with them is slick.
cookies[:things] = "stuff" # assign cookies[:things] # read cookies.permanent[:perm] = "Never going away" # write to a permanent cookie cookies[:perm] # read a permanent cookie cookies.signed[:tamper_proof] = "fragile info" # write to a signed cookie cookies.signed[:tamper_proof] # read a signed cookie cookies.permanent.signed[:perm_tamper_proof] = # write to a permanent signed cookie cookies.signed[:perm_tamper_proof] # read the permanent signed cookie
Now it is best practice to not put highly sensitive information in cookies. There is just no telling what is going on with a user’s browser and their computer. But there are times where you might have some pseudo-sensitive info that you want to store in a cookie. Not things like social security number, but maybe some app info that you would rather people not see, but it wouldn’t be the end of the world if they did.
Enter encrypted-cookies.
encrytped-cookies is a ruby gem that provides access to encrypted cookies for Rails 3 apps. It’s built to work exactly like signed cookies, so there isn’t really anything else to say about the usage except to show the two line example:
cookies.encrypted[:secret_cookie] = "nothing to see here" # write to a signed cookie cookies.encrypted[:secret_cookie] # read a signed cookie
Piece of cake.
I should probably mention how I do the encryption, since that is something you are probably curious about. The great thing about is that I don’t do the encryption! ActiveSupport has this nifty little module called ActiveSupport::MessageEncryptor. It does exactly what you think it does. It encrypts and decrypts messages. I didn’t want to take a chance writing the encryption code myself. Waaaaay to many places to make a mistake and really screw over someone using this gem. The signed cookies in Rails 3 use ActiveSupport::MessageVerifier to make sure the cookie payload hasn’t been messed with. In reality, encrypted-cookies does little more than swap out verifier code with the encryptor code. But simple code has fewer places for bugs to pop up.
Happy encrypting.
On Serialized Accessors
June 16, 2011 § 1 Comment
One of the nice things about brand new Rails apps is that all of your database tables are nice, small and manageable. You have relatively few columns in any given table and you are probably querying on most of the columns at some point in the app. As your app grows in size and complexity, so do your tables. A table that once upon a time had 5 or 10 columns now has 20 or 30. Not unreasonable number, but you are starting accumulate a number of columns that aren’t queried against. What should we do with these columns?
One option is to just let them build up. It doesn’t really hurt anything does it? I am sure that some database experts out there can weigh in on how number of columns affects performance. Option two is where I’d to show off something a little more interesting. Check out the data-attributes ruby gem over on github.
The premise is pretty simple. ActiveRecord allows for easy serialization of objects to a text field in the database. data-attributes makes use of this and adds attributes that read from and written to a serialized hash that is stored in a text field in the database. From the developers perspective, once a data attribute is defined, it can be used just like any other attribute. Validations work just like they do on column based attributes. To use this gem just add the following to your Gemfile in your Rails 3 project.
gem 'data-attributes'
Let’s take a look at it in action. We start with a user model with a serialized attribute called data.
class User < ActiveRecord::Base serialize :data end
Now, let’s say we have some piece of information that we want to include in the user record, but it isn’t something that we are going to have every query on. We define a data attribute like so:
data_attribute :favorite_food
Now how do we use this? We can see the results of adding the above to our Userobject.
u = User.new
u.favorite_food = "watermelon"
puts u.favorite_food
=> watermelon
puts u.data.inspect
=> { "favorite_food" => "watermelon" }
Pretty easy. The default field that the gem tries to save everything to is data, but that is just based on personal convention. If you want to change this, it’s as easy as adding the following line to your model:
data_attribute_column :food_preferences
Or if you have multiple serialized attributes and you want to send different data attributes to different serialized attributes you can do the following:
data_attribute :favorite_food, { :serialized_column => :food_preferences }
You can also set the default value to be returned for the data attribute:
data_attribute :favorite_food, { :default => "peanut butter" }
One of the things that makes all of this possible is that fact that ActiveRecord has multiple layers of accessor that happen when you call something like user.name. What happens is that the method name gets called which in turns calls read_attribute(:name). Similarly name=(val) calls write_attribute(:name, val).
data-attributes contains similar under-the-hood method read_data_attribute and write_data_attribute. This way you can have a little more control over the values that are read and written to your object.
Some things to be noted. If your ActiveRecord object has a serialized attribute then that attribute will be saved to database every time you call save. This is because it doesn’t know if the serialized object has been edited in place or not, so it just writes it to the database every time for good measure.
Update 2011-12-26: ActiveRecord 3.2 now has some of this basic functionality built in by way of ActiveRecord::Store.
The On Blog 