Thursday, December 22, 2005

AWK Tricks

I think my favorite go-to tool for Unix scripting work is "awk". After all, how often do you want to split a line of text into columns and then do something based on those columns? Pretty darned often.

Here's a trick I use frequently. You've just created a CSV or otherly delimited file, and you want to validate that there's no extraneous characters in there to break your parsing:

awk -F, '{print NF}' file.csv | sort | uniq -c

if all of your fields have the same number of columns (as they likely should), then you'll just see something like

17 1500

and you know you're all set. But if you see:

17 1498
18 2

you know you've got a problem. How to find it? Easily:

awk -F, '{if (NF!=17) print $0;}' file.csv

Done. Then you can go back in and edit those rows to fix them, if you like. Or if you're in a situation where you can just chuck them out, do the opposite:

awk -F, '{if (NF==17) print $0;}' file.csv > cleanfile.csv

And there you go, cleanfile.csv contains only the rows with the proper number of columns.

Technorati Tags: , ,

Monday, December 12, 2005

If you send them away, they will come back.

I just love this post of Dave Winer's about how to many money on the Internet. In short, the best thing you can do for your visitors is send them away. What does Google do? Sends people away from Google, more efficiently than anybody else. How about Yahoo? What did adding RSS support do for Yahoo? Drastically increased their share of the online news market. Why go to CNN, MSNBC, etc... when you can go to Yahoo and then Yahoo will send you to those places? Who are the most popular blogs? Those that link the most. Even basically puts a little value add on top of what is just a link to somebody else.

I think the underlying lesson is in knowing your strengths, and not fearing competition that is not really there. Yahoo is not a source of news, after all. Nor is Google a source of answers. Both are merely pointers. That is their value. So in order to be the most valuable pointer to things, you have to be willing to point to everything in an unbiased manner. If you pick and choose what you point to, or how you provide your results, then you'll lose value because you're basically saying that you don't trust the customer to come back to you. You're trying to force them into something. They'll spot it in a heartbeat, and you've already lost them.

The new business model appears to be "Look, here's the value I offer. I realize that other sites are valuable to you in a different way. They are not my competitors, I am not worried if you go over there. In fact, I encourage you to do so. Come back when you need me." Certainly you want to add value on top and not just be a list of links -- those have been around since the dawn of the net. Anybody and everybody can do that. You still need differentiators. You need a reason for your customers to pick you over somebody else that does exactly what you do. Maybe if you're a blogger it is in your personal writeup style. Or maybe the way you pick and choose your links for quality. In the case of something like a Google, it's the efficiency and relevancy with which they provide your results. And delicious' value lies in the wisdom of crowds.

I can only daydream about getting myself into a position where I can make money like this. But it's fun to think about. :)

Technorati Tags: , , , ,

Thursday, December 08, 2005

Why Podcast Alley ruins podcasts

Please, save me from podcasts that every month get on the "Hey, go vote for us at Podcast Alley" bandwagon. Every time I hear this, one of a few things is going to happen:

  • I'm going to unsubscribe.

  • I'm going to hit Next and be done with that episode.

  • I'm going to fast forward past it.

You'll notice I didn't say "I'll go vote for you." Why? Because not only is there no value in it for me (I already subscribe!), there's negative value -- because whether I vote for you or not I have to listen to you waste show time with this nonsense. If I like your show, I will talk to my friends about it, I will talk about it on my blogs, and in general I will promote it the best I can. But I will not vote for you on podcast alley, or any other directory. Never have, don't plan on starting.

Why is it important to vote? What do podcasters win? There are no prizes - they win traffic. The higher up you are, the better the traffic you get. Fair enough assessment? My point is that we should be coming up with more creative ways to drive up traffic. I think it's funny that on the one hand, podcasts aren't competitive with each other - after all, they play each other's promos all the time -- but yet they all whore for votes, when we all know for you to be on the top ten someplace means that someone else is not. So you are competing with your brethren using the "vote for me" approach. There's only room at the top for a few.

Here's what I want. I want a feed that periodically tells me about podcasts I don't already know about. Maybe they are brand new and worth a listen, or maybe they've been around since the beginning and I've just never heard of them. For some I will be able to tell if I want it or not just by a description. For some, I'll need to download a sample. I don't plan on voting for anybody that I find this way. I don't mind reviewing them, and saying why I like or don't like them. I'll even go so far as to give a Tivo-style thumbs up or down for them so that my rotating sampler knows which ones it has already given me. But I will not vote for you. Especially not every month.

You want traffic, learn to promote your show in new ways. Right now it seems like every podcast under the sun only has two moves -- whore for votes, or send feedback to Adam Curry pretending to have something interesting to say but really just to get your name out there. Take some public relations lessons. Go hang out on some freelance writers groups, because they all have the same issues. How do you guerilla market yourself? Call up a local radio station and see if you can get an interview spot. Podcasting is the next big thing, everybody knows it, and if you call around you're going to find people who want to talk to you. They don't just want to talk to Dawn and Drew. Especially if you've got something of value to say.

Technorati Tags: ,

Monday, November 21, 2005

Ruby on Rails : Yet Another Getting Started Tutorial

When something as big and buzzy as Ruby on Rails comes out, it's very hard to evaluate. Are you saying you love it just because everybody else does? Or do you hate it because everybody loves it and therefore it is so obviously just the next big buzzword? What if it is the next big thing, are you doing yourself and your career (and projects) a disservice by dismissing it out of hand?

I'm a Java coder by trade, have been since Java existed, pretty much. But lately I've been in need of some small, one-off tools that I just can't motivate myself to write because of Java's overhead. Say that I want a simple system to create new press releases and put them in a database. I want to publish them as individual HTML files, but also with a table of contents file and an RSS feed. Should be easy, but I have enough work on my plate already with other projects, so unless I can make it happen in a matter of hours instead of days, it's never going to see the light of day. So I decided to give Ruby on Rails a shot.

Quick terminology check. "Ruby" is a language. Been around for about 10 years now, so I'm told. Started in Japan. "Rails" is a sort of code generating framework for Ruby. For instance you might start down the road to a web application called "blog" by going into your Rails directory and running "script/generate controller Blog". Hit the web server and you're good to go - you have a web application you can start playing with.

Over the last few days I worked on getting Ruby on Rails up and running on my Powerbook. That's always the hard part, getting it up and running in the first place. Here are some tips I came across that made it easier for me:

* Start by grabbing the tutorial video. Don't just stream it - download it. That way you can play it back at will, pausing and rewinding. You're going to want to do this as you fire up your own copy of Rails and do the same thing the speaker does.

* The editor he's using, by the way, is TextMate. It is shareware, not free, in case you were curious.

* If you are on a Mac OS X machine, like I am, then skip the whole connecting to the database part and go get yourself Locomotive, which packages up everything you need to deploy Ruby on Rails in one nice application. If all you're doing is evaluating the technology, this gives you an easy way to trash it when you're done. But seriously -- integrating database support into Ruby on the Mac seems to be a royal pain in the neck according to my googling, and I spent way too much time on it. I prefer to use Postgres over MySql (since that's why my company uses and I have the installations all ready to go) which only made it that much harder. Locomotive also contains its own HTTP server.

* Locomotive will not necessarily help you get your database configured. RadRails will, however. It is an IDE based on Eclipse that does a number of things via GUI instead of code, including the database configuration. However, because I could not get Ruby compiled with database support, I never actually got the server to come up. So what I did was to take the code that RadRails had made for me and "Add Existing" to Locomotive. Once I ran my project under the Locomotive application everything worked fine.

The "scaffold" trick is cool. That's where you just say one word, like "scaffold post", and then magically you get a web interface that provides you the ability to create/list/edit/delete objects of type "post" based entirely on the database schema. In other words, you don't write any Ruby code at all, you just go into the database and create a table called "Posts" that has id, title, date, body and presto, you have your interface. It's even dynamic, so that you can muck with the schema and without even rebooting your server watch your interface change. Now that's highly cool as far as rapid prototyping goes.

BUT, that's what it is - a rapid prototyping trick. Once you say "I want to modify the actual HTML code that the interface produces", now you have to switch over to a static version of the scaffold (by "/script/generate scaffold Post") and after you do that it's not going to be dynamically pinging the database anymore. So you have to work back and forth -- suffer with a generic user interface until you get your schema right, and then generate the files so that you can muck with them.

At the end of the day, don't forget that this is a Ruby tool that will produce Ruby code. I had at least one person agree with me that the Rails concept is very cool until I pointed that out - he thought the language was pluggable. Eventually you will have to write some Ruby code. So if you weren't planning on learning a new language (and maintaining the code you write in it!) you might want to think twice. Projects like this that promise huge gains on the front end are often appealing to folks who didn't want to write much code in the first place. So there's an interesting tradeoff there -- you're going to write less code, absolutely, but it's going to be in a new language to you, so those lines you do write are going to take you more time. For example when working on the view you can say things like "link_to post.title" but how are you supposed to know that link_to existed in the first place? Sure you can read books on the language but now you're taking the fun out of the rapid prototyping - you can do that with any language. Of course, this only applies to your first app. Once you know the language and the library, you can crank out the code.

Thus far I can say that Rails does what it claims to do - I can build myself a database schema and have the interface dynamically keep up. That alone is outstanding, and saves a great deal of work. Whether or not I write something meaningful in it, I'm not sure. At least I can say that I got it up and running and now I have the option of exploring more. I really did want to write what I said above, a press release manager. I'll try to followup with updates on whether I made it happen and whether I get my company to use it.

Technorati Tags: , , ,

Tuesday, November 15, 2005

Coupons by RSS

This weekend I was up in North Conway with the family and inlaws doing our yearly tradition of getting a jump on the Christmas shopping at all the outlet malls. Without fail, I watched every cashier at every register ask "Would you like to recieve coupons via email?" I did not see anyone give out an email address. It really demonstrates the performance of the whole email marketing model -- sure, if you ask 100 people and get 2 to sign up, hey, you got 2 people you didn't have before...but you lost 98 people!

Imagine instead of these cashiers didn't ask anything, they simply flashed you a little card and said, "And if you're interested, here's some information on how you can use a computer to subscribe to our coupons. There's a web site address and directions on it, what happens is you just add that to your internet browser and then whenever we have new coupons they show up on your browser." I can only use the metric that I'm most familiar with -- myself. I would sign up for such a service. Why not? What does it cost me? Nothing.

The technology business opportunity comes in hosting coupon feeds for a bunch of companies, who otherwise are going to have no clue how to do it. They want to track this information as best they can and get an idea of who is using the coupons. At the end of the day the answer is easy -- you put some sort of identifying mark on the coupons and then you track which coupons come back into the store. But maybe you could experiment with regional feeds if you are a national chain. Or allowing users to sign up to personalize the feed to get coupons that are relevant just to them.

The real hurdle here is not technology, it is faith. When a marketer comes back with 2 email addresses, he knows for a fact that he can send out 2 emails. They may not be read, or converted, or heck, even opened. But at least it's something that can be reported upon. It's an implied "one valid email address per person". With RSS you have to rebuild all of this. What constitutes a set of eyeballs? If somebody has their reader set to ping your feed every 15 minutes, how do you determine whether the user is actually reading it? But all that will come with time. None of that is a reason not to give it a try. I find it hard to believe that getting that 2% of response on "Can I have your email address?" is really more valuable than at least attempting something a little new and different in the coupon biz.

Technorati Tags: , , ,

Monday, October 31, 2005

Micro Persuasion: Travelocity Syndicates Fare Alerts with RSS

Micro Persuasion: Travelocity Syndicates Fare Alerts with RSS

Kind of interesting the way they did this. You setup the flight you want, and then if the "cost of the flight drops by at least 20%" you get a message. So it's basically an alert service.

The first thing that comes to my mind is how long the price is good for. If you only have one block of time per day when you watch your RSS feeds, and you've got a few dozen or hundred new messages, you may not get to the Travelocity alert for a little while. But maybe you segregate off these "hot deals" feeds into a different list so that you can spot them more quickly? That would make sense.

Technorati Tags:

Friday, October 28, 2005

Screw the Outliers

I had an idea today. It doesn't matter what it was, for the moment. It was something where I thought, "Hey, that would be useful to me. Therefore it's likely that it might be useful to someone else, too."

And here's what I thought next: "But I can imagine people for whom it would be useful, but the way I'm thinking about implementing it, it wouldn't work."

Such thoughts can just kill your motivation. You immediately shift gears and now you're thinking about why something won't work, instead of why it will. Even if there are customers for the product that you're thinking about, you're focused on the people who aren't your customer. Sometimes it's good to think this way. If you're trying to engineer the solution to a particular problem in software or hardware, for instance, it's good to stop and think about whether your solution will indeed solve the problem in all cases. It's almost natural for engineers to do that.

Screw that. Those are the outliers. This isn't designing in a vacuum where you can come up with a perfect solution for the universal audience. If you've found any audience at all for your idea, then worry about them. Don't distract yourself by saying "Gee, it won't work for people who do it this way instead of that way." Fine, you're right, it won't. But it will work for people who do it your way, and there probably are people like that. Make your product and get it to them, and then maybe if you're lucky you'll find yourself with the resources to expand what your product does.

You'll often see this illustrated when a technical person with an idea talks to a business person. The techie explains the idea in the most modest way possible, "It only does this, it will only work in the following cases, it doesn't do this that and the other thing." The business person doesn't even hear the second half, because they know that's not where the money is. They say, "It will help people? They'll buy it." I've had that conversation with business people, and it took awhile for me to get it into my brain. I'd say, "But it won't work for everybody," and the business guy would say, "So what?"

Once you grasp that concept and can relax and let it go, you've made a giant step toward being able to make something real out of your ideas instead of keeping them in your head and wishing that they'd get to see the light of day.

Technorati Tags: , , ,

Wednesday, October 26, 2005

Your Own Personal Queue Podcast

The blogging revolution didn't start because of what was being said or who was saying it. The most important thing to come out of blogging was that it lowered the publishing bar so that anybody who had an opinion could voice it. No longer did you have to create your own web site just to say something. Now with just the click of a mouse you could get a text box, type some words, and instant publish. Often you could go one better and simply leave an audio file or a picture from your mobile phone.

Likewise, the revolution behind podcasting is not in the content, nor in the "lowering the production bar". Because let's face it, actually recording a tolerable audio file, producing and hosting it all take more work than they should.

No, the genius of podcasting lies in the delivery mechanism. Podcasting is what it is because the files just automatically show up on your ipod. And by extension, if we assume that you take your ipod with you when you walk away from the computer, that the files go with you to wherever you want them. That's the brilliant bit. "Audio blogging" has been around forever. But you either had to go to each individual site and get the file, or, even if you were using an RSS aggregator, you'd have to either listen at your computer or manually move files down. Podcasting makes that all invisible. Subscribe, and content is magically just there.

What I find most intriguing right now is that everybody only talks about podcasting one way -- you produce some content, and then you make it available, and people subscribe. In the messaging terms this is known as the "publish/subscribe" (or pub/sub) model. One publisher, many subscribers - usually anonymous. The goal seems to be to publish a feed and then build up a huge audience so that you can preach gospel or sell ads or do whatever you want to do with those eyeballs.

But there is another model known as the "message queue" (or sometimes "private or personal" message queue) which is very different. In the queue model, a message is intended for one recipient, to be received once, and read once. Email is a good example. You don't subscribe to email - email is sent to your address. If a message goes to many people it is because it was addressed that way.

As I sit in my car listening to the current crop of podcasts I wonder how this model could work to my advantage. There's no rule that says all my content has to be of the pub/sub variety. What if there was a way that I could take all of my voice mail, turn that into mp3, and drop that into my own personal queue podcast? On the way to work I could catch up on my messages. What about my appointments for the day? There's plenty of podcasts out there now that take a generic text news feed and pipe it through a text to speech converter. So why not do the same thing with my daily appointments and todo's? What better way to make use of the time driving to the office than to take care of all these tasks before you ever get there?
True, you could use the cell phone and call in to your voice mail, and I'm sure lots of people do. I also know what happens when everybody on their morning commute is too busy talking on the phone to pay attention to the road.

What else works in audio format ... actors and singers could use the time to rehearse. Go to rehearsal, record what you sound like, then listen back to it at your leisure. You could practice for business presentations. Plenty of people do that into tape recorders now. Why not into a personal podcast for yourself so that you can take advantage of the "invisibly delivered wherever you want to go" model?

As a programmer myself I wish I had the resources to get something like this working so that I could point to a prototype instead of a blogpost. But in lieu of that, better to throw the idea out there and see if maybe somebody else can run with it and I can be a beta tester.

I was thinking of this this morning as Adam Curry went through his audio comments for the Daily Source Code. Why doesn't he just pipe those as a personal podcast to himself? Does he really sit at a computer with gmail open and double click on the little attachments? Does he see the irony in that, I wonder? :)


The Path to Podcasting

Podcasting is not just the next buzzword, it's the logical evolution of what people listen to, and how they listen to it. I've been a raging zealot for podcasting ever since I heard about it last year sometime. Now that podcasts are right there in iTunes, anybody with an ipod is potential audience.

Here's the sales pitch I've been using:

Ok, here's how it happens. You tell me where you're at. You get a new ipod. The first thing you do is rip all your cds. Check. Now you create some funky playlists, because you can. Workout music, mellow music, whatever you want. Next you burn some cds with those remixes. Maybe give some to friends who haven't gotten the religion yet.

That was fun. What else can it do? Aha, you find the Music Store. Now you can buy songs one at a time. That's pretty cool. Feels a bit more legal than just finding them on the P2P networks. So you buy some songs. Eventually you start looking for ideas for new things you might like, instead of always searching for songs you already knew.

Eventually, especially if you're listening to your ipod for any long stretch of time (like the morning commute), you get bored with the music and want something more...productive? Audio books, you say! Great idea. Pick up Harry Potter, or the Da Vinci Code. On CD. Realize that a new book on CD runs a good $35-$50. Put it back down. It's a great idea, I've done it, and it gives you lots of content to listen to for a long time. But it's not the sort of thing you want to make a hobby. You going to buy a few audio books every month at that price? And the worst part is the price basically never goes down. Hardcovers turn paperback, and before you know it you can get the original $25 book for $5. But that audio book is always $35 or more. Annoying.

So you're bored with music and starved for some "content".

Welcome to podcasting.

Pick your topic. You like sports? How about ESPN on your ipod? Maybe you prefer news. Take your pick of sources -- CNN, ABC, NPR, Newsweek... which spin is your favorite? There's also plenty of talk shows that are not bound by the FCC (in other words not safe for little ears, although many are also family friendly). How about full fledged audio books? A number of projects out there are podcasting full length novels. So much for that $35 Harry Potter!

The greatest thing about podcasting that you probably don't yet realize is that you're subscribing to a feed of new content. It's not just going to the ABC site and downloading the MP3 of today's news. It's linking in and saying "Whenever new stuff is available, just go ahead and put it on my ipod." You can do this with as many sources as you want. Your choice. iTunes and your ipod are smart enough to get all the new content and put it on the player for you when you sync. It doesn't matter if new content comes out every day, every couple of days, or every couple of weeks. It's not up to you to go and get it. The computer's going to do all that for you.

So go into the Music Store and hit that Podcast button. Sign up for some stuff. Then sync your ipod and take it with you to your workout or your drive to work or wherever. Instant content. As much as you want. On any topic you want. All free, all the time.

Not bad, huh?