SQUARISM

Using a Redis as a Database

Development,Ruby — squarism @ 11:37 am

The Spike

I was spiking on Redis recently. I wanted to use the redis-objects gem to simulate a shopping cart app even though the README specifically says

Just use MySQL, k?

I wanted to see what would happen if I tried it anyway. So the README and examples for the redis-objects gem are great so I’m not going to rehash what’s there. However, I will say though that the example has you hardcode the id field to 1. That detail snuck up on me.

If you don’t set an ID then you can’t work with a redis-object instance. You get an exception: Redis::Objects::NilObjectId: Attempt to address redis-object :name on class User with nil id (unsaved record?)

It’s basically trying to tell you, “hey, save the record first or set an ID”. Well, honestly, I don’t want to set an id myself. This is where the meat of the README is. Redis-objects really fits organically in an existing ActiveRecord model. That means Rails. In this case though, I don’t want an entire Rails app. I can see the value though in a plain old Rails app. Just look at the examples if you want to see more.

Anyway, continuing on with the spiking, I tried to integrate the Supermodel gem with Redis-objects. That sort of worked. You just class User < Supermodel::Base and you can sort of get it to work. This is great because Supermodel gives you finders like User.find_by_email('bob@yahoo.com') to make it act like ActiveRecord but you can't use .create(email: 'bob@yahoo.com') to begin with because of the same errors as I mentioned above. Redis-objects really wants the record to have an ID already. Even using Supermodel's RandomID mixin didn't work. The initialize order and callback hooks don't really work (or at least I couldn't get them to work).

Finally, I tried combining just redis-objects and datamapper redis. That worked. And it's pretty nice. Check it out.

require 'redis-objects'
require 'dm-core'
require 'dm-redis-adapter'

DataMapper.setup(:default, {:adapter  => "redis"})

# you would move this to a common location
Redis.current = Redis.new(:host => '127.0.0.1', :port => 6379)

class User
  include Redis::Objects
  include DataMapper::Resource

  # datamapper fields, just used for .create
  property :id, Serial
  property :email, String

  # use redis-objects fields for everything else
  value :disabled
  value :name
  list :cart, :marshal => true

end

# absolutely need this line for dm-redis
User.finalize

require 'redis-objects'

require 'dm-core'

require 'dm-redis-adapter'

DataMapper.setup(:default, {:adapter => "redis"})

# you would move this to a common location

Redis.current = Redis.new(:host => '127.0.0.1', :port => 6379)

class User

include Redis::Objects

include DataMapper::Resource

# datamapper fields, just used for .create

property :id, Serial

property :email, String

# use redis-objects fields for everything else

value :disabled

value :name

list :cart, :marshal => true

end

# absolutely need this line for dm-redis

User.finalize

So using this is pretty easy.

u = User.create(email: 'test@test.com')
u.name = 'Testy McTesterson'

1 2	u = User.create(email: 'test@test.com') u.name = 'Testy McTesterson'

When you look at Redis, the keys are already composited for you and magic has happened.

redis 127.0.0.1:6379> keys *
user:test@test.com:name

redis 127.0.0.1:6379> get user:test@test.com:name
Testy McTesterson

redis 127.0.0.1:6379> keys *

user:test@test.com:name

redis 127.0.0.1:6379> get user:test@test.com:name

Testy McTesterson

Yay!

The name field is from redis-objects and the create uses datamapper. This is a really odd pairing but I like the fact that I have no sql database in the mix but still have finders similar to an ORM. Something to keep in mind, datamapper's finders are a bit different than the Rails 3 ones (no .where method).

Benchmarking A Million Things

Ok fine. So maybe this works, maybe it doesn't. Maybe it's not the right idea. What about the good stuff? Like, how fast can we load a whole lot of names into MySQL versus Redis using the above code and techniques? Is it even relevant?

Summary
-------------------------------------------------------------------------------
(PL = pipelined redis operation)
 
Loading one million random names (full names) like John Smith, Patty Gerbee Sr)
MySQL:                   06:05
Redis:                   02:45
Redis C ext              01:32
Redis pipelined:         00:56
Redis pipelined C ext:   00:19
Ruby just loading array: 387ms
 
 
Loading 10k ecommerce-style data (orders, users, products)
MySQL:    00:09.40
Redis:    00:14.50
Redis PL: 00:02.72

Summary

-------------------------------------------------------------------------------

(PL = pipelined redis operation)

Loading one million random names (full names) like John Smith, Patty Gerbee Sr)

MySQL: 06:05

Redis: 02:45

Redis C ext 01:32

Redis pipelined: 00:56

Redis pipelined C ext: 00:19

Ruby just loading array: 387ms

Loading 10k ecommerce-style data (orders, users, products)

MySQL: 00:09.40

Redis: 00:14.50

Redis PL: 00:02.72

A gist of these test results is here.

A More Complete Example

If you know the ID and don't need something like an auto-incrementing column outside your code/control then you can greatly simplify the code above by getting rid of Datamapper. You can simply use redis-objects to fake an ORM. I had great success using it as long as you USE NATIVE REDIS TYPES. Listen to the redis-objects author, don't try to force the tool into the use case.

# What if we want to use redis-objects as a database but
# try to stick with native redis objects?
#
# For example, Supermodel is a great gem but using the Redis
# mixin causes Supermodel to serialize to JSON strings in Redis
# which is going to kill performance.  You have to model your
# problem using native Redis objects to keep the speed up.
#
# At the same time, I miss the finders from ActiveModel
# like: Person.find('Joe')
# Supermodel does give you those finders so you will feel right at
# home coming from Rails.  I tried using ActiveModel mixins with
# redis-objects but it didn't work for me.
#
# I found the below a nice compromise but it requires a lot of
# custom methods.  :(


require 'redis-objects'

class Person
  attr_reader :name
  alias :id :name

  include Redis::Objects

  def initialize name
    @name = name
  end

  def self.exists? name
    # Here's a big assumption, if the id attribute exists, the entire
    # object exists.  This might not work for your problem.
    self.redis.exists "name:{#name}:id"
  end

  def self.find name
    # new behaves like find when a record exists, so this works like
    # find_or_create()
    self.new name
  end

  # native redis attributes with redis-objects
  value :age
  list :favorite_foods
end

# example usage

joe = Person.new 'Joe'
joe.age = 34
joe.favorite_foods << ['cake', 'pie']  # it will flatten arrays, don't worry
joe.favorite_foods << 'bacon'          # or you can do this

Person.find('Joe').age = 56

# find and initialize
Person.find('Stan').age = 21

# you cannot just .favorite_foods = ['Steak]' because that's not how native
# Redis objects work
Person.find('Stan').favorite_foods << 'Steak'

# deleting a field
Person.find('Stan').favorite_foods.del  # notice it's .del and not .delete (del is the redis cmd)

# What if we want to use redis-objects as a database but

# try to stick with native redis objects?

# For example, Supermodel is a great gem but using the Redis

# mixin causes Supermodel to serialize to JSON strings in Redis

# which is going to kill performance. You have to model your

# problem using native Redis objects to keep the speed up.

# At the same time, I miss the finders from ActiveModel

# like: Person.find('Joe')

# Supermodel does give you those finders so you will feel right at

# home coming from Rails. I tried using ActiveModel mixins with

# redis-objects but it didn't work for me.

# I found the below a nice compromise but it requires a lot of

# custom methods. :(

require 'redis-objects'

class Person

attr_reader :name

alias :id :name

include Redis::Objects

def initialize name

@name = name

end

def self.exists? name

# Here's a big assumption, if the id attribute exists, the entire

# object exists. This might not work for your problem.

self.redis.exists "name:{#name}:id"

end

def self.find name

# new behaves like find when a record exists, so this works like

# find_or_create()

self.new name

end

# native redis attributes with redis-objects

value :age

list :favorite_foods

end

# example usage

joe = Person.new 'Joe'

joe.age = 34

joe.favorite_foods << ['cake', 'pie'] # it will flatten arrays, don't worry

joe.favorite_foods << 'bacon' # or you can do this

Person.find('Joe').age = 56

# find and initialize

Person.find('Stan').age = 21

# you cannot just .favorite_foods = ['Steak]' because that's not how native

# Redis objects work

Person.find('Stan').favorite_foods << 'Steak'

# deleting a field

Person.find('Stan').favorite_foods.del # notice it's .del and not .delete (del is the redis cmd)

Comments (0)

The Blub Paradox and Delicious Pie

Development — squarism @ 11:59 am

Anything worth doing is worth doing well. “That’s Good Enough” isn’t good enough. There are some cases where you are writing a one-off script but those tasks are few and far between. Most people don’t get paid to write throw-away code. Most developers aren’t sysadmins writing procedural code on the level and complexity of Bash. Even if you are, is this the only style you know? Do you shy away from more complex tasks? Can you not break down an any size problem into smaller pieces?

A musician practices their art constantly. An athlete trains. And yet, we in the tech community have to go out of our way to find people who work on personal projects on the side or have the will/motivation to learn new things outside of “the job”. If you don’t learn on your own then you have to figure it out as you go. And you are already very busy so I know you’re just skimming over this post anyway. So let’s break this down in a bulleted list.

Developers work best when they’re challenged but not overwhelmed. Optimal work is a run, not a jog or a sprint.
Developers will step-up or fill-in a task to make it challenging as stated above.
Your classic boss doesn’t really care how you solve problems.
Culture takes 20 years to change.
The Blub Paradox says (among other things) that you can’t make anyone understand the power of a different language because developers become sedentary.
Good consultants are already busy so no one is going to save you.
Training or bootcamps won’t change your skillset or habits by a large percentage.

Let me put this another way.

My cat will play with a toy on the floor. And after a while, it becomes boring. So she swats the toy under a cabinet. And now it’s exciting! She can barely reach the toy and its a challenge. When she gets the toy out, she’ll bat it under the cabinet again. What is she going to do? Be bored again? She’s good at reaching for the toy and using her claws to hook stuff, tail counter-balance to stay in control and seeing in dark places. She needs to be playing under the cabinet and not given an mouse in a barrel. This is what you must do. You must improve your skills by going beyond “Good enough”. Otherwise, you will stagnate and have limited tools and approaches to problems.

But how do you even know what to learn? When you are looking up and down the power continuum in a problem domain, how can you grow beyond your own knowledge boundaries to see advantages that other people are enjoying? If you were looking at a menu of pies, how would you choose which one you will like?

Here’s the pie menu. Pick the one that you like!

PIE MENU (all pies come with whipped cream on the side) --------------------------------------------- Cranston Ermu Supreme Pie Tagasnackle Mound Blat Pie Rumination Flip Pie Rainclouds and Humility Pie Are You Seeing The Problem Pie

$3.99 per slice

What? This is terrible. This isn’t fair. Certainly technology cannot be like this. This is computer science! How can you choose? How can you know?

Hungry: “Waiter. What’s in humility pie?”
Waiter: “Humility.”
Hungry: “What does that mean?”
Waiter: “Have you ever literally tasted humility?”
Hungry: “No.”
Waiter: “Well it’s kind of like tasting regret but also different than pride. You kind of taste like you should have prepared more or set expectations lower.”
Hungry: “That doesn’t make any sense.”
Waiter: “Of course it doesn’t. It’s an experience. It’s not a definition.”

Given enough questions, the Waiter will quit unless the Hungry customer chooses a pie or leaves.

Now let’s look at some horrifyingly contrived personalities looking down (or not looking down) the pie continuum.

The Eternal Procedural Coder

Let’s say you are a developer who just writes scripts all the time. You piece things together and when they work the first time, you ship it (whatever shipping means to you). If a more complicated task comes along, you don’t change your design, tools or strategy, you just change the amount of time you throw at it. Once it works, you call it done.

If you don’t know how to build something in a variety of ways then you are always going to build it the same way.

If you don’t pre-learn outside work then you have to run to catch up to change your habits.

You can’t train new developers easily on what you are doing because everything you build is “your way”.

You can’t learn new techniques, tools or study good design because you don’t have time at work.

You don’t feel like learning on the weekend or after work.

When you look at another style, it looks like a bunch of crap and you don’t see the value/power/happiness/productivity/peace-of-mind because you haven’t gone through even one example yet.

Exactly what is the exit strategy here? What’s the end game? Who or what is going to save you or change your situation? What would the disruption look like? What would the disruption be called?

Meanwhile, you hate looking at old code. You wonder “what was I thinking?” or “I must have had a lot of coffee this day”. Projects come and go, you hope you never have to go back to old projects. You leave things alone because they are terrible but do the job. You continually start new tasks but they all come out the same. Huge single files, huge methods/functions. One mess after another. No wonder why they call this “work” right? Who would do this for fun or learning?

What’s interesting is that The Blub Paradox applies here too. It’s difficult to explain what development is supposed to be like to someone who has never had a success. Success breeds success. Failure is of course crucial to learning and courage is needed to fail when trying to succeed. But experience really is the success side of trying and learning. You can gain experience through failure but in terms of visualizing the ideal, success is very important.

The It Can’t Be Done Guy

Let’s say there’s a website called Searchbox.com that’s a stripped down version of Google. In a meeting, everyone is talking about how Facebook search and Twitter are all hot right now and you feel like you are behind the curve. So you decide you want to add friends, followers and other social features to Searchbox.com. Your rockstar lead developer says “it can’t be done”. No one questions him. The idea dies, jokes are cracked and the grapes were probably sour anyway. Yay team?

Software is supposed to be soft. So why can certain things be so easy and other things be so hard? I understand certain problems just “are hard” (concurrency, cache invalidation). But not even entertaining ideas or experiments could be a sign of bad design. Even with crappy procedural scripts, simple changes mean omfg-lots-of-lines changed. Design isn’t about avoiding change (and change will happen), it’s about reducing the cost of change.

If you want to learn from a master, read POODR. Sandi says:

…successful but undesigned applications carry the seeds of their own destruction; they are easy to write but gradually become impossible to change.

POODR is full of these insightful observations and Sandi doesn’t just give you advice. She walks you through refactoring code and demonstrates the concepts that are related to these insights.

Anyway, back to our example project. If searchbox.com is broken up into modules then there is a user class or object. Surely this class could be forked into a feature branch and worked on in isolation by a few team members to add Friend and Follower traits/mixins/interfaces? Or is the user really a session cookie combined with a database row and “who knows” how to change it?

If there isn’t really a concept of a user but it’s just a bunch of hacks then yes, no one is going to entertain massive changes to this concept of a “user”. It’s too hard (in your system). However, if a system was explicitly designed thoughtfully (not by default or by accident) then features are easy to add. The software stays soft. This is what Sandi is talking about in POODR.

The Waterfall by Default Guy

Let’s say there’s a guy in charge of a project. He hasn’t ever tried Agile of any sort. He’s read about it and he’s confident that he knows all about it. He thinks it sounds crazy, lazy and chaotic. He’s done plenty of projects in the past that have come out under-budget and on-time so why change anything? On the other hand, he’s not technical so his opinion of past projects is completely dependent on the feedback given to him by developers. If developers ever felt like talking to him was a waste of time then his opinion of how good “waterfall” is could be slightly off. The real question is, can you explain how delicious Agile Pie is to him?

Here’s the wheelhouse of agile type stuff.

Developers aren’t going to want to work with you again if you burn them out.
Project heroes burn out.
You can’t hire your way out of a failing project.
Most people don’t like the term Waterfall because it’s viewed old.
If you don’t declare a methodology it’s Waterfall by default.
You can’t do Agile. Agile is a type of thing. Like Car. You have to buy a type of Car.

So when you try to explain any type of agile techniques to someone who doesn’t have any experience with it, it’s the same thing as the Blub Paradox. The Waterfall Guy is happy enough doing no explicit methodology, why would he change? When asked how to prevent projects from getting off-track he will say, “before you start working on a project, you need to know what you are building”. Even though Agile (and common sense) says that both the customer and the developers are the stupidest at the beginning of a project. Big Design Up Front (BDUF) doesn’t work. BDUF is undesigned project methodology. BDUF is how you put together simple things like grocery lists and legos. BDUF isn’t how you deal with change and complicated systems where developers will upscale the project difficulty until it becomes interesting.

So even if you discuss these items with Waterfall Guy, he’s not going to get a taste of Agile Pie without eating it. You could talk all day long about the ingredients, how people keep eating it, how there’s these case studies showing it’s delicious. “Meh, whatever. I have my dessert already.” At the same time, Waterfall Guy is always asking for visibility as to what developers are doing through status reports. In his heart he knows that he has a few hero developers and if he loses them, he’s screwed. That’s ok, that’s why they call it work right?

The Non-Tester

There’s a team who are running a production app. They have a testing team. When developers are working on “omg the really important thing”, they tell the testing team to run “whatever test”. It usually involves someone manually going down a list with a pencil. When a problem is found then a tester will tell a developer something went wrong. “It showed me this error 0×2123 when I did the thing you told me to do.”

Except a lot of the time there isn’t anything wrong. A lot of times the problem is the testers aren’t developers. Otherwise, they’d be developers. A lot of times the tester doesn’t really understand the system and the developer has to explain the system to them. So while the developer is working on “omg the next really important thing” the tester is asking questions and the developer is getting annoyed. Jokes are made “you only come to me when there’s a problem, haha”. Or, “I bug you so much I owe you a beer! Haha.”.

Looking down the power continuum, a developer looks at testing libraries. They look annoying and time-consuming. Who needs this hassle? They don’t have time to find out if they do. They don’t understand the different levels of testing (from most involved and beneficial):

No testing at all – you have to hit refresh in a browser and you have no idea if you broke anything else. Old bugs pop up.
Some testing – unit tests but you have a team run through end-to-end scenarios.
A lot of testing – you have unit and integration tests and maybe even code coverage reports and CI.
Test-first development – you write your tests before code but that’s the end of that methodology.
Test-driven development (TDD) – you let your tests completely drive the design of the system.
Testing end game – You do red-green-refactor. Your tests run fast. You do UI testing. Your customer requirements map to easily read executable stories. You spend very little time in red. You ping-pong pair with TDD. No code is committed without passing CI. Your testing tools and process are constantly evolving in response to pain points.

The non-tester doesn’t see the value or the constant effort people are making to try to get to the Testing End Game. “I’m not a tester! Why would I test!” Meanwhile, they complain about how little time they have because they have to manually have to see if their code works. Or maybe they are embarrassed / mad / frustrated that they broke the development/production build. In essence, the Blub Paradox is telling us that they can’t see their problems through the lens of Blub because they don’t know Blub. Meanwhile, Blub (TDD) practitioners can’t imagine doing it the way the non-tester does it. In fact, TDD practitioners are honing their craft so much that it is a rare event when they’ll take the time to explain TDD or the End Game to a non-tester. Even if they do, both will be frustrated because any insight or evangelizing will be misunderstood.

The Long Time Ignorant

Take any of the above and put it in the context of time. How long has SQL been around? Does someone on your team not know SQL? Is it because they haven’t needed it? What if they are writing flat database files by hand and do need SQL? Have they just been living under a rock or can no one change their mind? What can you do as an individual to change such a blind eye?

How long has JUnit been around? Is there a Java developer you know that hasn’t written a unit test? Or they have written a few but they don’t anymore because they think they don’t have time to do it?

That Seems Hard

Someone who can only write code in a procedural style can’t break down problems into easily solvable bits. Without concepts like Mixins/Traits, Composing Behavior and TDD; a complex problem is just going to seem too hard. “I don’t do that kind of stuff.” When someone that has OO or Functional experience might say “here’s how I would do it, who wants to help me?” Of course there are things that are just too big but the difference is in experience. A Bash scripter isn’t ever going to understand ncurses events because they haven’t ever written a desktop GUI. So even though ncurses can be great for turning “scripts” into “programs”, they are going to shy away from ncurses because “wtf that seems hard”.

You can’t explain the benefits of a web framework because of the Blub Paradox.
You can’t explain the benefits of an ORM vs raw SQL because of the Blub Paradox.
You can’t explain the advantages of tmux to someone because of the Blub Paradox. Even while they keep losing their ssh sessions over unstable wifi. Given a simple enough problem, you might be able to convince them to try it out.

Tmux is a perfect example. No one understands how great it is until they use it. Then and only then do they never want to go back. Tmux isn’t always great though. Sometimes the terminal gets all weird with certain keyboards or maybe you want the native buffer scrollback to work. So it’s not some silver bullet default. However, once you grok tmux, you know when to use it and miss it when you don’t have it. This is true for many things that people evangelize. However, sometimes you need to understand their world view and take that into consideration.

Tmux pie is delicious. Ask anyone who has tried it.

Answers

I don’t have all the answers. Ok, see you later. *silence* Well listening to Rubyrogues and reading POODR has given me some memorable advice which I’ll repeat here.

Spike on foreign concepts. Ok, you don’t know OOP and breaking down problems. Make up a problem and do it. Don’t try to learn in your main project. Spike on a concept and throw the code away.
Pair with someone who knows this stuff and check your ego at the door. When you are lost, don’t play with your iPhone. Ask questions. Pretend to be stupid. Check your ego.
Learn a different tech stack. When I write Java (poorly these days), I bring Ruby experiences back with me.
Watch conference videos and screencasts. Pre-learn before work. You have no choice. No one is going to pay you to get better.
Read the POODR book even if you’re not a Rubyist.
If you are resistent to an idea, change sides. Let’s say you don’t like soccer. Take the position of a soccer fan and argue with yourself. You might be amazed at the argument you make just looking from another perspective.

Comments (0)

Ruby p385 benchmarks

Ruby — squarism @ 2:14 pm

I was playing around with the falcon p385 patch to see if it’s any faster than some of the more recent MRI rubies.
TL;DR version: looks like p192 is faster than p385 of any type or tweak.

Here’s how to get a p385 Ruby version patched with funny falcon’s performance patches using RVM.
mkdir ~/.rvm/patches/ruby/1.9.3/p385 curl https://github.com/funny-falcon/ruby/compare/p385...p385_falcon.diff > \ $rvm_path/patches/ruby/1.9.3/p385/falcon.patch rvm install 1.9.3-p385 -n perf --patch falcon
Then rvm use 1.9.3-p385-perf or set it as your global ruby.

Test Setup

The following benchmarks were run on an i7 server with a RAID5 array. The disk is slow (lack of large cache) but the benchmarks were run on the same box so it should compare apples-to-apples.

From here on out, here are the definitions for the Ruby versions.
p194 = 1.9.3p194 default
p385 = 1.9.3p385 default
falcon = 1.9.3p385 with the above falcon diff patch applied
gcc_tweak = the falcon patches with GCC compile flags tweaked.

So what are these GCC tweaks? Explicitly setting the CFLAGS for your machine’s CPU type and recompiling ruby with the Falcon patches applied.

Micro and Macro Benchmarks

I used the ruby-benchmark-suite to run these tests.

Here are some example results. I can’t list them all. There are over 100 benchmarks. These are results for the mean times in seconds.

test	1.9.3p194	1.9.3p385	falcon	gcc_tweak
bm_sudoku.rb	1.379112226	1.598182153	1.495923579	1.526717563
bm_open_many_files.rb	0.175602996	0.197096826	0.197673286	0.194135045
… etc etc

Here’s the winner summary for mean times. This is the number of times the ruby version was the fastest for a particular benchmark.
1.9.3p194 – 64 wins
1.9.3p385 – 29 wins
1.9.3p385-falcon with GCC tweaks – 10 wins
1.9.3p385-falcon 5 wins

Boot time and IO

Timing rails boot time is a bit more important to me. If you want to know how to really save “rails boot time” see the DAS screencast on not loading Rails at all.

Even when using domain objects and lib tricks, it’s nice to have Rails and all I/O boot fast. The main thing that funny falcon’s patches do is speed up requires and I/O.

So let’s benchmark booting a Rails app.
$ time bundle exec rake environment

Ruby Version	Seconds
p385 patched with falcon and GCC tweaks	2.481 total
p374 defaults	3.336 total
2.0.0-rc2	2.613 total

In conclusion, p194 looks faster on the macro and micro benchmarks but Falcon patches boot Rails faster.

Comments (0)

Hash Choices

Ruby — squarism @ 11:26 pm

bttf

As I’ve previously talked about, Hashes of Hashes are weird to work with. In the previous post about Captain Planet, I showed how to select, filter and manipulate 2D hashes and arrays but ultimately concluded that a hash of hashes is both weird and unnecessary (most of the time).

If you can control the data, inline your key into the hash data and make an Array of Hashes. It’s really where it belongs. If you don’t, you’ll find yourself doing a few extra iterations or work. Below you’ll see a simple example of the two data structures.

# Hash of Hashes
{
  :dave => { :age => 1, :height => 2 },
  :tony => { :age => 1, :height => 2 }
}

# Hash of Hashes

{

:dave => { :age => 1, :height => 2 },

:tony => { :age => 1, :height => 2 }

}

# Better --> Array of Hashes
[
  { :id => 'dave', :age => 1, :height => 2 },
  { :id => 'tony', :age => 1, :height => 2 }
]

# Better --> Array of Hashes

[

{ :id => 'dave', :age => 1, :height => 2 },

{ :id => 'tony', :age => 1, :height => 2 }

]

In this case of an array of hashes, the data is easier to manipulate with array operations and filters. But before talking about matching on arrays of hashes, I want to talk about matching on dates.

doc = Date.parse "Nov 12 1955"
marty = Date.parse "Oct 27 1985"
future = Date.parse "Jan 13 2094"
doc < marty
=> true

doc = Date.parse "Nov 12 1955"

marty = Date.parse "Oct 27 1985"

future = Date.parse "Jan 13 2094"

doc < marty

=> true

So comparisons with dates work like you might expect. But fuzzy matches do not. Take this example. Is Marty in between Doc Brown and the distant future? The answer should be true.

We use a Range (x..y) object to make a date range. Then we can use === to check if we get a match.

marty === (doc .. future)  # wrong, wrong, wrong
=> nil

1 2	marty === (doc .. future) # wrong, wrong, wrong => nil

But wait, === on Date is different than === on Range. The method === on Date checks to see if it’s the same Date whereas === on Range checks to see if the argument is within the range. So if you flip it, it returns true.

(doc .. future) === marty
=> true

1 2	(doc .. future) === marty => true

require 'date'

holidays = {
  :halloween => { :date => Date.parse("Oct 31 2012"), :presents => false },
  :christmas => { :date => Date.parse("Dec 25 2012"), :presents => true },
  :july_fourth => { :date => Date.parse("July 4 2013"), :presents => false },
  :valentines_day => { :date => Date.parse("Feb 14 2013"), :presents => true },
  :thanksgiving => { :date => Date.parse("Nov 28 2012"), :presents => false }
}

# turn hash of hashes into array of hashes
holiday_array = []
holidays.keys.each do |key|
  holiday_array << { :id => key }.merge(holidays[key])
end

# find all the holidays with presents
puts "yay presents!"
puts holiday_array.select {|holiday| holiday[:presents] == true }

# find all the holidays within a date range
winter = (Date.parse("Dec 21 2012")..Date.parse("Mar 20 2013"))

puts "Winter holidays"
puts holiday_array.select {|h| winter === h[:date] }

require 'date'

holidays = {

:halloween => { :date => Date.parse("Oct 31 2012"), :presents => false },

:christmas => { :date => Date.parse("Dec 25 2012"), :presents => true },

:july_fourth => { :date => Date.parse("July 4 2013"), :presents => false },

:valentines_day => { :date => Date.parse("Feb 14 2013"), :presents => true },

:thanksgiving => { :date => Date.parse("Nov 28 2012"), :presents => false }

}

# turn hash of hashes into array of hashes

holiday_array = []

holidays.keys.each do |key|

holiday_array << { :id => key }.merge(holidays[key])

end

# find all the holidays with presents

puts "yay presents!"

puts holiday_array.select {|holiday| holiday[:presents] == true }

# find all the holidays within a date range

winter = (Date.parse("Dec 21 2012")..Date.parse("Mar 20 2013"))

puts "Winter holidays"

puts holiday_array.select {|h| winter === h[:date] }

So what we did in the middle there was flatten the hash of hashes into an array of hashes by merging the key with the ‘data’ part of the hash. Hopefully that makes sense.

Comments (0)

What a tech stack has to do on our first date

Blog,Development — squarism @ 9:04 pm

Listen to Dave Thomas

Watch this video -> please. Start at 45:50 and watch at least 10 minutes.

I tell a lot of people about this video. I don’t tell 10,000 people up on a stage on some world tour. But whenever I get into a good discussion with geeks, I reference this video and tell them to watch it. So I tell a lot of people in terms of percentage in conversations that I’m enjoying. So in other words, I really like the video personally, it struck a chord (or a brain-thorn) and I want other people to consider his point of view. Specifically, Dave Thomas says you should learn a new language every year. Invariably, I’ll tell people that Dave Thomas writes books and a few times people will dismiss his main idea because they think he’s selling books.

Dismissing Dave Thomas’ opinion that you should learn a new language every year just because he is an author is a crap position to take. First, FUD. Second, Dave Thomas doesn’t sell books about the languages he says to check out in the video. Thirdly, I trust Dave Thomas. This isn’t some fucking Larry Ellison power play.

So in terms of following his advice, I might pick up Scala. It’s different enough and anti-hipster-omg-embrace-the-java-enterprise enough that it will challenge me and/or be different than say continuing with Python (will is really Ruby like Spanish to Portugese) learning. I’ve picked up a Scala book but haven’t started it yet. I looked at the Lift docs and was completely deflated. Chad Fowler tweeted about the Play! framework (thanks for the reminder) and thank god this looks better.

If I get sick of a city, I might move. When I move, I’m looking for something new. Similarly, starting out with Scala is going to be a bit like jumping into a new town. Ok, I need to find the equivalents of the old things that I’m used to. But wait, I moved to a new town to do something new! If I bring my old habits with me, I’ll just be writing Ruby code in Scala. And that’s crap.

Marge: I’ve dug myself into a happy little rut here and I’m not about to
hoist myself out of it.
Homer: Just bring the rut with ya, Honey.
– “You Only Move Twice”

What any technology stack is going to have to do on our first date

I need a list of libraries at least searchable by category and popularity. Popular gems are likely to be used and therefore good. This is great to get to higher level operations and gain productivity. For example, ruby-toolbox.com.

I need screencasts ordered by date so you can learn community trends. I’m going to need a way to keep up with the Jones’ and know what’s falling out of favor in the community because a shiny new library, technique or concept is getting the attention of key people. This also operates as curated content. For example, everyone is talking about CORS. I’m sure there will be a Railscast on CORS even though CORS is not specific to Ruby or Rails.

Related to that last point, the community should be of high quality. I know people like to hate on PHP but if you took a core sample of PHP-land, I think you wouldn’t be too happy. Of course there are exceptions, for example, Facebook is full of brilliance.

Some easy way to install things that’s Internet-enabled. Maven is close. Rubygems is good. NPM is good. Yum beats plain RPM. Pip is fine. CPAN is crap. All these package managers connect to the Internet and resolve dependencies. As a bonus, being able to repackage, bundle, freeze or tar up libs into a lib directory is great for corporate firewalls or making deployment not suck so hard.

I need a way to separate projects from one another. Gemsets with RVM does this. Virtualenv does this with projects. Lib implicitly does this in Java (although CLASSPATH issues suck). Explicit project isolation with libs is great.

Give me videos from the community like Confreaks. Video community talks are necessary. The JavaOne video site sucks and only because of the content. It doesn’t have to be flashy and overly-produced. Oracle publishes 3 minute videos with zero details and zero content. Yay?

I need quick feedback. This means a REPL and/or event-based test suite runners. If I save a file, I want a test to run (Ruby’s Guard gem). If I want to figure out how to do something, I want it to be instant feedback. About the farthest thing I can think of away from this is deploying to Tomcat, reloading the webapp and hitting refresh in a browser (#fml).

I’ll call Scala back

So far, Scala has the REPL thing down (just type $ scala). And the SBT tool does auto-reloading similar to Guard. Of course, it has to compile things so it’s slightly a slower feedback loop but it’s not terrible. SBT also seems to have a manifest thing going on for the equivalent of gemsets. It all goes in project/plugins.sbt which is great. There’s also a concept of a global project in a ~/.sbt folder. This reminds me a lot of NPM.

I haven’t discovered the community side of things yet like screencasts (railscasts) or a news site (rubyflow). On the negative side, there seems to be some Java crosstalk. For example the Play! framework can generate a Scala or a Java project. So if you are trying to get away from Java, this is a good step away but not a clean break. I get a feeling of Java refugee camp (which is fine).

Something I’m working on right now is understanding the IDE landscape. Eclipse was a bit weird with loading the Scala plugins and Intellij is coming out with a new version of IDEA soon. Also I need to grok the sbt-console inside the IDEA plugin vs command line. Are they the same? I had this same challenge when trying RubyMine. At some point you have to get a feel for what the One True Way was intended to be.

Comments (0)

New Hawtnesses

Rails,Ruby — squarism @ 12:04 pm

Wow, how can anyone keep up?

Zeus

Video of Zeus in action. Is it the new spork?

Super fast Rails boot times, but not just booting. Everything! Like DRB on steroids.

FishFish shell

`brew install fishfish`. The new ZSH? A fork of the Fish shell. Featured in the above video link. Be sure to uninstall fish through homebrew if you have already. It’s not the same thing.

YADR

YADR – for me, the new codegram/vimfiles? Impressive documentation, will really change my muscle memory.

Prezto

Included with YADR, looks to be a replacement for oh-my-zsh (my fav). Will have to try it out and update this.

Comments (0)

SVN vs Git Speed

Development — squarism @ 10:00 pm

This is a bit of shooting fish in a barrel but I really wanted to see if for myself. After using SVN for many years, I knew Git was fast. Funny enough, I didn’t know SVN was slow until I saw Git was fast. Hah.

Anyway, I wrote a script that does this:

Creates a repo
Unzips Linux 3.6.1 source
Add all files / commit all files
Branch / Modify / Commit / Merge (3 times)
Cleans up branches, cleans up temp directories

The Linux kernel source is 79MB, ~42,000 files. So this is a bit excessive. You probably won’t be working with a project this large. However, it’s a good way to show speed and performance.

The other interesting thing that I found while doing this test is how much longer my SVN script turned out to be. Both scripts do the same thing but the Git script was much shorter. So in a way, Git seems to be faster and easier (or at least less typing) at the same time.

A note about the benchmarks. The Raid 5 test was done on a web server with a Raid controller that has very little cache. No question that these spinning disks are a bit slow under load. However, the tests were done on the same box so it’s still a fair comparison. The SSD row is a virtual ubuntu machine on an SSD; a different machine all together.

Benchmark Times mm:ss

Disk type	SVN	Git
Raid5	14:56	01:08
SSD	17:12	02:09

Git has a bit of an advantage by doing 3 branches because it does its garbage collection only once on the first merge. So I apologize for the bias. I can say for sure that the one branch test was something like this: SVN 20min, Git 2min.

Here are the links to the scripts I wrote if you want to try them yourself.
svn_stress.sh
git_stress.sh

Run the scripts with the time command:

$ time ./svn_stress.sh /tmp/stress_test
233.78s user 81.81s system 35% cpu 14:56.48 total

1 2	$ time ./svn_stress.sh /tmp/stress_test 233.78s user 81.81s system 35% cpu 14:56.48 total

Comments (0)

Ruby DCamp 2012!

Blog,Ruby — squarism @ 5:47 pm

Great conference! There’s a lot to be said about it. I am not going to post it all, it would take a long of writing. You can find things about #coderetreat meetups out there.

After decompressing and reflecting, I came to a philosophical realization.

Rubyists were not taught Ruby in school or elsewhere, they willed their skill into existence.
Internet access came through the air over cell and we suddenly had dev environments in the middle of the woods.
The wikipedia description of game of life was just some english bullet points that were implemented 100 times

So in these three ways, we willed things out of the ether and that is awesome.

For a less philosophical run-down of what happened, check out Brandon Hays’s blog.

Comments (0)

Changing the TextMate Comment Banner

Blog — squarism @ 10:39 pm

I like to put a little date banner on files that are like a CHANGELOG. The Insert Comment Banner in Textmate is great for this. You just hit Ctrl+Shift+B and this appears:
========= = Banner =========
And the equal signs on top and bottom adjust as you type in “Banner”. It’s pretty cool. But what I wanted is a timestamp in the banner. So I’d leave Banner highlighted and then right click in Textmate and select “Filter through command” and put the Unix command date in there. That’s great except it puts Fri Jan 20 21:46:13 EDT 2034 in there which is a bit long.

=============================== = Fri Jan 1 21:46:13 EDT 2034 ===============================

Too long. So here’s what you can do. Go to Bundles->Bundle Editor->Show Bundle Editor. Then go to the Source category. You can filter at the top for Snippets if it’s easier. See this picture.

Then put this code in:

${TM_COMMENT_START/\s*$//}==${1/(.)|(?m:\n.*)/(?1:=)/g}==${TM_COMMENT_END/^\s*(.+)/ $1/}
${TM_COMMENT_START/\s*$//}= ${1:${TM_SELECTED_TEXT:`date +'%a %b %d %Y'`}} =${TM_COMMENT_END/\s*(.+)/ $1/}
${TM_COMMENT_START/\s*$//}==${1/(.)|(?m:\n.*)/(?1:=)/g}==${TM_COMMENT_END/\s*(.+)/ $1/}

${TM_COMMENT_START/\s*$//}==${1/(.)|(?m:\n.*)/(?1:=)/g}==${TM_COMMENT_END/^\s*(.+)/ $1/}

${TM_COMMENT_START/\s*$//}= ${1:${TM_SELECTED_TEXT:`date +'%a %b %d %Y'`}} =${TM_COMMENT_END/\s*(.+)/ $1/}

${TM_COMMENT_START/\s*$//}==${1/(.)|(?m:\n.*)/(?1:=)/g}==${TM_COMMENT_END/\s*(.+)/ $1/}

This will give you a banner like this:
=================== = Fri Jan 1 2034 = ===================

And it’ll be the right length automatically. Nice.

Comments (0)

Caching an Object in Rails is Easy

Blog — squarism @ 7:11 pm

Caching in Rails is really simple (straight from the docs):

You can set up your application’s default cache store by calling config.cache_store= in the Application definition inside your config/application.rb file or in an Application.configure block in an environment specific configuration file (i.e. config/environments/*.rb). The first argument will be the cache store to use and the rest of the argument will be passed as arguments to the cache store constructor.

Originally, I didn’t even know it was there. So I was thinking I’d just create a hash in memory. So I wrote my own cache class like this (homegrown_cache.rb):

# not exactly a pure cache, implements the lookup or expensive operation too
# probably needs to be decoupled

class Cache
  def initialize
    @cache = {}
  end

  def cache(name, values = {})
    if values = {}
      return @cache[name.to_sym]
    else
      @cache[name.to_sym] ||= values
    end
  end

  def search(name)
    if @cache.keys.include?(name.to_sym)
      puts "getting name from cache instead of expensive operation"
      return @cache[name.to_sym]
    else
      self.cache(name, Expensive::Search(name))
      return @cache[name.to_sym]
    end
  end

  def clear
    cache_size = @cache.length
    @cache = {}
    return cache_size
  end

  def to_s
    "#<Cache:#{self.object_id} Cache Size: #{@cache.length}>"
  end

  def size
    @cache.length
  end

end

# not exactly a pure cache, implements the lookup or expensive operation too

# probably needs to be decoupled

class Cache

def initialize

@cache = {}

end

def cache(name, values = {})

if values = {}

return @cache[name.to_sym]

else

@cache[name.to_sym] ||= values

end

def search(name)

if @cache.keys.include?(name.to_sym)

puts "getting name from cache instead of expensive operation"

return @cache[name.to_sym]

else

self.cache(name, Expensive::Search(name))

return @cache[name.to_sym]

end

def clear

cache_size = @cache.length

@cache = {}

return cache_size

end

def to_s

"#<Cache:#{self.object_id} Cache Size: #{@cache.length}>"

end

def size

@cache.length

end

Then you might want to use it in an implementation like like CacheEnabledSearch or something like that within your app.

But it’s way easier and smarter to just use the ~~goddamn~~ caching that’s built into Rails. Specifically, if you use the in memory store, since Rails 3.0 it’s thread safe. Meaning if two clients hit your cache, you’re not going to have weird bugs pop up. It’s very easy to configure and use.

I’m posting the Cache class anyway though it’s interesting how simple an object it is without being thread-safe. But even then, I’d recommend using the cache class from Rails outside of Rails.

So here’s how you’d use Rails caching “without” Rails. In other words, a console app.

# using the Rails caching class outside of rails
require 'active_support'

# 
cache = ActiveSupport::Cache::MemoryStore.new
cache.write('user', 'bob')
# => true
puts cache.read('user')
# => bob

# using the Rails caching class outside of rails

require 'active_support'

cache = ActiveSupport::Cache::MemoryStore.new

cache.write('user', 'bob')

# => true

puts cache.read('user')

# => bob

Not particularly interesting. This would be way more useful inside of a Rails app where a singleton class called Rails.cache is set if you have something set in your config:
config.cache_store = :memory_store

Let’s avoid the Rails situation for now and create a file cache directory for a file based cache:
$ mkdir /tmp/poro_cache

Here’s a complete example.

# using the Rails caching class outside of rails
require 'active_support'

cache = ActiveSupport::Cache::FileStore.new "/tmp/poro_cache"
if cache.read('user').nil?
  cache.write('user', 'bob')
  puts "wrote to cache, run me again"
else
  puts "Reading from filestore cache: #{cache.read('user')}"
end

# using the Rails caching class outside of rails

require 'active_support'

cache = ActiveSupport::Cache::FileStore.new "/tmp/poro_cache"

if cache.read('user').nil?

cache.write('user', 'bob')

puts "wrote to cache, run me again"

else

puts "Reading from filestore cache: #{cache.read('user')}"

end

If we run this one time, we get this:
ruby rails_cache_poro.rb wrote to cache, run me again

And running it again:
ruby rails_cache_poro.rb Reading from filestore cache: bob

Reset it with rm -rf /tmp/poro_cache

Ok, so that’s ActiveSupport caching outside of Rails. Wiring Rails up to do this inside a controller, lib class, helper or other ruby object is trivial once you have followed the caching guide. You won’t instantiate ActiveSupport::Cache anymore directly but you’ll do it indirectly through the config block for your environment.

And then all the statements will look like this:

>> Rails.cache.write 'foo', 'bar'
=> true

1 2	>> Rails.cache.write 'foo', 'bar' => true

But it works the same. Really easy. Of course, the next subject which is more complicated is when to invalidate your cache.

Comments (0)