Monday, August 16, 2010

When Does “You” Become “We”?

I’ve noticed this every time I start a new job.  How long does it take before you stop referring to the company as “you” (as in, “What did you guys mean to do over here…”) and when do you start using “we”?  Twice now my new bosses have caught me and said “Feel free to start using ‘we’ any time, you’re part of the company now too.”

I think the distinction I make is that when I’m speaking of the efforts that came before me, I say “you”.  After all, it’s not like I want to take credit for their work.  I can’t say “Here’s what we meant to accomplish in version 6.1” if it came out before I was around, right?  Once I’ve got new features added to the product I’ll feel more comfortable saying We.

Wednesday, July 28, 2010

Test Driven Learning

So, I’ve been out of work for about a month.  During my time back in the hunt I learned just how big test-driven development, or TDD, has become.  Quite literally every place I interviewed (for Java and Rails jobs alike) asked me what my experience with it was, my philosophy, and on and on.

And I had to answer, in each case, truthfully.  I get the idea of TDD.  I’m a fan.  But never have I been in a place that’s truly adopted it.  Usually the typical legacy excuses all get in the way – nobody wants to write tests for code that already exists, so there’s no good framework from which to build new tests, and it’ll take too much effort to make that happen when it’s easier to just continue writing code like they’ve always done.  Not a good excuse, just a popular one.

Well, I’ve got a new job now and all the above is still true.  However I think I’ve found a bit of a loophole in that logic that’s going to allow me to force feed TDD into the mix.  I’m a new hire.  There’s an existing code base.  I can read the code all day long, and I can think I understand it, but how do I really know?  Should my first real interaction with the code be when I’m making changes that could potentially introduce bugs? Into production?  Not such a great idea.

So instead, as my first project, I integrate JUnit (it’s a Java shop).  Then, I start writing unit tests.  I read an existing function of an existing object, I think I understand what it’s supposed to do, so I write a test for it.  One of two things is going to happen.  Either the test will work, in which case my understanding of the code was correct, or else it’s going to fail, in which case either my understanding of the code was wrong, or else the code itself was wrong.

The end result is that a) I know the code does what I think it does, b) I potentially uncover bugs that would have otherwise been uncovered the hard way, and c) now unit tests exist for the next guy.  Win win win.

I call it “test driven learning”, and I dub it TDL.  Pass it on.

Monday, May 24, 2010

What Do Drivers Do In Computers?

I've been working on a project recently where I get to look at the kinds of questions people ask of Google, when phrased as a question.  For any given subject I get a glimpse of what people want to know, based on how they ask it.

Today's question is a very popular one on the subject of Windows device drivers :

What do drivers do in computers?
To really understand what drivers do in computers, you need to stop for a second and consider what's really going on under the covers.  You have "hardware".  That's the physical stuff - your memory, your CPU, your hard drive, your DVD player, your microphone.  Then you have "software".  That's the programs and files that live as bits (ones and zeroes) in the memory and on the hard drive.  The magic of how computers work could be summed up very briefly as, "The software tells the hardware what to do."

When you load a DVD, there are some physical steps.  You open the tray, usually by pushing a button.  You put the DVD in, and then close the tray either by pushing the button again or giving the tray a little shove.  And then your movie starts playing.  But that's where software comes in.  How exactly did the computer know that you put a new disk in the drive? The DVD player itself has just enough smarts to know when that tray door opens and closes. So when you close it, a signal is sent down the cable that connects player to motherboard. The CPU, which listens to everything going on around it, says "Ok, I just got an XYZ signal from the PDQ cable, what am I supposed to do with that?"

And that's where your "device driver" comes in.  The driver is a tiny little program whose sole purpose in life is to raise its little digital hand and say "Me!  Ooo!  I know how to handle XYZ signals from the PDQ cable!"  Ok, I'm being a little silly, but that's really what it comes down to.  Signals come through that cable, they go over to this little driver program, and the driver speaks a language that the rest of the software (Windows, typically) can understand.  So that XYZ signal is might translate into something like, "A new DVD has been inserted, pop up that dialog box that asks the user what she wants to do with it."

Before Windows, you actually would have had to do this all yourself.  When you bought a new printer or even a mouse, it would come with a disk that contained these drivers.  You'd copy them over to your hard drive, and then have to modify the startup programs to let the computer know these drivers were available.  One of the reasons that Windows took over the PC world was because, in the words of one famous industry analysis, it was just "a big bag of drivers."  That was the whole point -- now you don't have to deal with drivers anymore, they're just there.

Except when...they're not.  New devices come out all the time, it's impossible for Windows to have every driver for every device.  Worse than that, sometimes these driver programs have bugs in them and need to be updated.  So it's important to recognize the role these drivers play, and keep them updated.  Modern versions of Windows have an Update Service that will let you know when it's time to download new drivers.  Or you could look into a service like which will do all the work for you, scanning your computer to determine which drivers are out of date and then offering you an easy way to download them all in one visit.

Friday, May 21, 2010

Verizon DVR Hack : Deleting Multiple Episodes

If you've got Verizon DVR and you're like me, particularly if you've got the home media option for sharing between boxes, you'll often get 5 or so episodes of a tv show that you've watched but haven't deleted (because you watch them on other tv's where you can't delete).  And perhaps you've noticed a rather annoying issue with this where, once you're down to one episode, what happens is that it'll suddenly kick you back to the list of programs (because now there's no longer a folder for that show) and, because that episode is a few days/weeks old, it's buried at the bottom and you have to go find it.

Know what I'm talking about?  You go into the Grey's Anatomy folder, and delete one.  This is already complicated enough because for Verizon you need to hit like 4 buttons - Right arrow to get into folder, then Enter to select episode, then Down two times (or three, if you're half way through the program and have "Resume Play" option) to get to Delete, and then down for Yes, then enter.  That's alot of keystrokes.  So you have to do that for N shows.  And when there's that last show left, suddenly you have to do even more because now you have to scroll through your shows to go find where in time that last one is hiding.

Ok, enough of that, want to know the trick?  Skip the most recent episode.  Instead of deleting them top down, move to the second most recent one, and then delete them all.  What will happen when you only have the one most recent one left, it'll kick you back to the top level -- but look at that, your cursor is still sitting on the show you want.  Delete.

Monday, May 17, 2010

What We Need Is Psychic Search

Sunday afternoon, out having lunch at a noisy restaurant.  I hear part of a song I recognize, enough to say to my wife “Good song,” but then we can not hear the rest.  For the life of me I can not think of the song, and it drives me nuts all the way home.  Easy enough to Google lyrics, right?  Wrong, the sequence I heard was nothing but harmonizing, just background singers singing “Ahh ahh AHHHH….ahh ahh AHHHHH….ahh ahhh AHHHHHHH” where the third one has gotten higher, and then presumably it moves into the next verse which is what I can’t remember.

Try googling “song opening ahhh ahh ahhh” and see what happens. I got lots of hits for Led Zeppelin’s Immigrant Song, which I think is funny because it is both an accurate answer to the question, and also not even close to what I meant.  There were also lots of hits on modern dance club songs, but I’m old and don’t really know those enough to say one way or the other.

So I post on Facebook, and my friends begin guessing.  One says “At the Hop”, that old 50’s tune that starts “bahh bahhh bahhh bahhh, bahhh bahhh bahh bahhhh At the hop!  Well you can rock and you can roll and you can stomp and you can stroll at the hop….” (or whatever the words are).  Again, an accurate answer to the question pretty much, but not at all what I was looking for.

Undaunted I started googling for “Hum a few bars” hoping to turn up one of those search engines where you can sing into the microphone to guess your song.  I find nothing but hits from the 2006-08 era, none still active, so I’m wondering if I searched wrong.  I do try the iPhone app “Shazam”, but it can not recognize the tune so chances are I’m singing it too poorly.

I do strongly think that it is a 60’s tune, what I described to my friends as “hippie era” :).  Running out of ideas I google “summer of love music” and find a list of hits from that year (1967, I believe?)

And there it is.

Groovin, On A Sunday Afternoon by the Young Rascals.  My “ahh ahh AHHHH'” sequence is right there in the middle.

Now what I need is for somebody make the connection from my original query to the query that ultimately found me the answer.  We’ll make millions. :)

 UPDATE : Adding to my own post because the more I think about this, I think it’s not that far off.  I knew that it was a 60’s tune, right? Imagine if for my music search engine I was able to start there.  Just like the other folks I mentioned knew that there’s was a dance club song, and I knew mine was not.  From there maybe the engine could offer up some guiding questions like the tempo (this one is pretty slow), single or group (group), background harmony (definitely)…and so on.  I know that recommendation engines like Pandora break songs up into N different parameters and then match, saying “If you liked this song with these values then you’ll probably like this other song with similar values.”  But what if instead of doing that you offered up an interface where the user could just select those values on sliders?  “I want strong lead vocals, meaningful lyrics and a fast drum beat.” 

A slow 1960’s song with a harmonizing group certainly wouldn’t guarantee finding my song, but it would at least help narrow the search, and that’s huge. 

Wednesday, May 12, 2010

Variety in User Interfaces

Once upon a time Robert Heinlein wrote a book about a car with a voice synthesizer.  You talked to it, it talked back.  When it started speaking in phrases that the driver had never added, he chalked it up to one of his friends messing around with the database.  Eventually it got to the point where he wasn’t sure that the car wasn’t simply sentient.

I think that there’s lots of room for voice applications in today’s user interfaces.  Take the GPS in your car.  I expect that it talks.  You’re not supposed to read the screen while you’re trying to drive, right?  So let’s think about how it talks.

One way is to have a good solid database of what it can say, including names of all the streets.  That’s very handy, because instead of just “Turn left” it can say “Turn left on Quantum Street.”  But the downside is that you’ll be limited in other ways, and there’ll probably be very few ways that the computer phrases certain things.  It’s strength lies in being able to read you what is effectively a bunch of proper names.

The other approach to take is more generic, and not speak the street names.  “Turn left” and you rely on the layout of the road to know that you’re taking the left it wanted you to take.  Of course there is still an image on the screen, which can have the street name, so it takes half a second to look down and see whether you’re going where you are supposed to.

BUT! Once you substantially drop the size of the database by focusing only on a handful of “Turn left” / “Turn right” types of phrases, now you’ve opened the door to downloading your choice of voice.  You could have a celebrity read the 100 or so phrases, zip it up, and there you go.

Even better, you could be given instructions for how to do it yourself.  My TomTom has this, but I’ve never taken advantage of it.  I do have John Cleese from Monty Python doing my directions, but I’ve never sat down to record the kids and wife doing it (“Turn left here, turn left, here, turn left! You’ll kill us all!”).  The instructions were fairly complicated, explaining how you had to get each soundclip exactly the right length for it not to sound broken.  Forget that.

What I’d love to see, though, and this gets be back to my original story, is more variety.  The TomTom worked by saying “Ok, provide a folder consisting of 59 sound files labelled as follows.”  There’s a single sound clip for “end of trip”, so no matter how many times I use it, John Cleese is always there saying “You have reached your destination.  You may get out now, but I’m not going to carry your bags.”

What if instead of 59 sound files I was told to provide 59 directories, and in each directory there could be as many files as I wanted?  And whenever the device needed to play something it would instead just grab a random file from the appropriate directory?

That would be AWESOME.  For the most common ones you could sit down and record a good couple of dozen different ways to say it.  Maybe set something up that is user contributed so you just plain don’t know how many different sounds there are?  Or set a priority on them so that most of the time you get “Turn right” but once in a hundred you get “Turn right, moron.”

This approach, of course, would thus work for any device that talks back to you.  Your answering machine, maybe?

Type Type Revolution?

So, anybody know how to hack a “dance mat”?  I’ve  seen a bunch of games with this type of input device where you jump up and down on it – my kids even had a simple little “Strawberry Shortcake And Friends Dance Party” one if I recall.

Take one of those and build a game up around it where instead of dancing to music, you’re typing like you would on a cell phone. 

Improve your spelling, your accuracy…your appreciation of classic literature.  A game could be made out of it, where you have to answer questions but instead of jumping to A-D you have to spell out the answer.

I would totally buy that.  Somebody, if they haven’t already, needs to make a generic “dance mat” controller for Wii or something (balance board does not count!) that games can be written to.  Or maybe even a USB one so you can use it on the PC?

Tuesday, May 11, 2010

MongoDB Ruby Driver : nToReturn=100?

The other day I noticed in my logs that I kept getting my very large (6 million rows) result set back 100 rows at a time.  The log kept saying getmore, getmore, getmore… with nToReturn=100 set as a parameter.  So it was doing the right thing, but where was that 100 coming from?

A post on the user group gave me the answer – the default “batch_size” option for find is set at 100 in my version of the Ruby driver, when it should be set at o (which is the signal for the driver to just figure out the optimal size).  So you can override this by forcing batch_size:

@results = @objects.find(@conditions, {:batch_size=>0})

You should notice an improvement in performance.

Monday, May 03, 2010

Mongo Good.

Been working on an interesting project lately where I was handed some 6 million XML files with no meaningful schema, and told to make a warehouse out of them.  Each XML file also happens to have several hundred elements in it.  With no schema it would be a nightmare to try and make one, never fully sure whether you'd allocated enough space, whether just because your sample of this field yielded integers means that it always has to be an integer .. stuff like that.

I tried a very simple "Elements and Attributes" schema that is capable of loading any XML file.  Basically each Element and each Attribute get a row in the database.  This works fine for smallish data sets, but mine quickly blew up into billions of rows and was no longer manageable.

Then I stumbled over the "No SQL" movement, where we quite literally throw out decades of knowledge about how to set up a SQL schema of tables and rows, joins and where clauses .. and replace it with something more document-driven, no schema. This sounded perfect, since that's exactly what I had.

Enter Mongo DB.  With a little help from Ruby, my loader script for 6 million XML files now looked like this:

Foreach XML file:
  Load it via REST web service
  Parse it into a JSON object
  mongo.insert(object) Done!

Done!  Now if I'm hiding the name of a software vendor somewhere in the middle of one of those objects, like say {"machine"=>{"software=>[ "instance"=>{"name"=>"Microsoft something" ... }]}}  and I want to find them?  All I need to say is  find("" => /Microsoft/) and presto, I've got back my filtered list.  I can add a count() to get the total number of matches, or an each(...) or map(...) to process the list, or basically anything else I might want to do.  Of course I can also add an index on it to keep the performance high, too.

I'm finding this NoSQL stuff fascinating.  Takes a little while to mentally get around everything you've learned about normalizing a schema.  Instead of a dozen tables all crosslinked with each other, I've got one "collection" that is nested in a variety of ways.  But I quite literally spent 0 effort on schema design. 

I haven't even begun to touch on things like the built in sharding, which enables you to scale across multiple instances without having to worry about which server hold which portion of your data. Another player in this space, Cassandra (from the Facebook people), is most well known for being great at that. We looked at Cassandra but it had some schema setup required that I was not able to complete.  With Mongo I just inserted everything and now I'm free to query at will, no matter how complex the original key structure was.

I hope to write more here as I come up to speed on what is obviously the next big thing in the world of database design.

Friday, March 12, 2010

How Geeks Pick Battles

My son’s got a birthday coming up, and we’re having it at one of those local gymnastic places.  On their website is a “print your invitations” link, which turns out to be a 2 page PDF.  More specifically it is 4 rectangles – two front, two back – intended to be printed out double sided on card stock, then cut out and mailed.  Cool enough.

I do not have a double sided printer, though, so I print out both pages and take it down to the local copy place, where I realize that…they don’t line up.  Damnit.    It’s clear what’s gone wrong, the people who cobbled this thing together put a 1” border from the left on both pages – but since page 2 is on the back they should have measured 1” from the right.

The wife tells me no worries, she will just buy some invitations at the store.  I assure her that this is not a problem, I just need to muck with the margins on page 2.

After discovering that your typical PDF reader does not let you play with the margins on a per-page level like that, I switch over to a PDF editor – InkScape, to be specific.  Technically, though, InkScape is a graphic editor with import/export PDF, so I can only edit one page at a time.  Fair enough.

I will save you lots of cussing, but fast forward a half dozen sample printouts when I realize that it’s not that they are not lined up properly (they are not), but that one of the front rectangles is a different size than the others! This is throwing everything off.    That’s seriously annoying.

But!  I’m in an editor after all.  Select all.  Ungroup.  Select background rectangle.  Resize.  Done. :)

I spent maybe an hour and a half on that.  At any time I could have just skipped the whole thing and let the wife buy some new ones, but part of being a geek is taking the battle for supremacy very seriously, and not letting the machines win.  If the only thing standing between a problem and a solution is something involving technology, I’m not about to give up easily.  And it has been known to drive me occasionally crazy.  But I wouldn’t change it. >:) 

Wednesday, January 27, 2010

Thoughts on iPad

On the one hand, Apple’s new gadget is a big fat iTouch.  After all, what’s the real difference between an iPhone and an iTouch to begin with? Voice calls via 3G, for one.  Take those out and the only real difference is the video camera, which is neglible since they can put one in at any time.

Here is why the best ebook reader in the world will still not get me to buy one, and it’s not because I’m a book nut who loves the smell of paper.  Ready?  You can’t rip a book.

Go back to when iTunes was new, and you could buy music from Apple.  Not all music, sure.  But what were your options at that point?  If you already had the music in a different digital format then maybe you could port it.  Fine.  But the big killer idea was that you could get the CD and you could rip it.  Convert physical into digital.  And then you win.

What’s the comparable example for books? I don’t read a great deal of brand new bestseller types.  When I pick up a book it’s either a non-fiction thingie out of the Shakespeare section, or its a classic scifi that I missed the first time around.  Sometimes, to be fair, a new scifi tome or Shakespeare novel will come out and I do pick those up off the “New Releases” shelf, but those are only a portion of my purchases.

So, what am I to do?  The book I want is not available in iPad format.  Is it available in some other digital format because somebody else did the work for me?  Maybe.  Sometimes.  Not often.

So then what?  I can’t meaningfully rip a book.  I am well aware that there is technology that can do it, and Google’s pioneering some work in that area.  I’m talking about a single home user trying to turn his physical book into a digital version.  Can’t be done.  And that kills me.  If you told me that you were installing a book ripping machine in every Apple retail store, and that I could walk in and get a genius to rip my books into iPad format for me? I’d be all over it.  I’d systematically rip my whole collection (depending on much Apple charged me for the service :)). 

Where I do see a future for these devices is a few years down the line when my kids go to college.  I see them carrying a single device, instead of a backpack full of text books.  All their text books are right there in a tablet.  All the updates, too.  And homework and study guides, and question and answer forums.  It’ll have a keyboard for speedy word note taking, but you can also draw on it for diagrams.  It’ll have net so you’ve always got Google and Wikipedia, but it’ll also have instant message and SMS because you can’t live without those.

The big question will be how you keep the difference between “study machine” and “cheating machine”.  If you’re allowed to have it, you’ll be able to cheat with it.  But if you’re not allowed to have it, you’ll lose all your reference materials.  I’m not sure how that will play out yet.  I’d love to imagine some sort of signal blocking device so that once you’re in the classroom your teacher could assure that all your devices switched to local mode, but I don’t see that happening.

Monday, January 11, 2010

Rails : Passenger Is Deleting my Sub / RailsBaseURI Prefix

Ok, this was a weird one.  I’m not going to jump into a lesson in how to make Phusion Passenger work to deploy Rails applications, but suffice to say that I had a situation working where we have multiple controller entry points all in to the same application, so I had this:

RailsBaseURI   /foo
RailsBaseURI  /bar

/foo and /bar, then, are soft links from the docroot over to the Rails application’s public folder.  The dispatcher is smart enough to say “Ok, based on the public folder I know where the Rails app lives, and once I know that I can go invoke the appropriate controller.”  So a request for /foo would expect there to be a foo_controller.rb file.

That’s been working fine for me in production for months.

But this week I got the most annoying problem – I added /baz, and kept everything else the same – the soft link, the RailsBaseURI, the baz_controller file.  But now all of a sudden I’m getting  :

No route matches "" with {:method=>:get}

And, let me tell you, it is very hard to Google for empty strings.  I got plenty of examples of people having trouble with No route matches “/foo” or “/approot”, but in each case it was at least telling them their route – mine was being eaten!

On a hunch I would try /baz/baz – and it worked fine.  I tried /baz/ (note the trailing slash) and suddenly my “No route matches” error would turn into “/”.  Aha, I had it – somehow my prefix, which I’m relying on to handoff to my controller, is being eaten.  So by the time Passenger spots “/baz” and hands off to Rails, it just gives “” – and Rails in this case doesn’t know what to do with “” because I don’t have a default route set up.

The problem appears related to an update to Rails, in conjunction with Passenger.  Many people pointed me to this fix:


I dismissed this out of hand, though, because of my multiple controllers.  What value would I put in there?  I did not want my app to default to one particular entry point.

Turns out I dismissed it too soon.  What I did was add this:


And guess what?  Success!  The best way I can take to interpret this is that it’s telling Phusion “Hey, the value in this variable?  I need this, don’t kill it.”  For instance when I made it a “/” value I suddenly started getting errors saying that I had no “baz” route (see how no slash?)  So by saying that I have no relative url root, I’m saying “It’s not safe to delete anything that came in as if it were extra baggage, just leave it all in place please.”

That may be a painful explanation, particularly to anybody who understands the guts of the Passenger/Rails handoff, but it’s working for me now and I’m abiding by my “If I couldn’t find the answer by googling that means other people have the same problem” rule and posting for the next guy.