Early Access Survey Results

To help us iron out some issues before the initial launch, we let about 400 people access rubyproblems.com a week early. They were given a free solution download and were encouraged to explore the site and let us know what they thought.

Because we realized that it'd be hard to deal with a ton of individual replies, we invited our early access members to take a 15 question survey in exchange for another free solution. We got about 25% response rate, with 101 users submitting answers. What follows is a summarized feedback report, along with some of our plans on how to address the concerns our users had.

We started with some questions about OpenID, because we felt our decision not to offer a traditional login option was a controversial one, and wanted to see what our users thought about it.

Q1: Did you have any difficulties using OpenID to sign up for our site?

RESULTS: 17 YES / 84 NO

One thing we recognize is that there is significant selection bias here, given that anyone responding to this survey had to have already successfully completed an OpenID authentication on our site. But we were still quite interested in what our users experiences were. Looks like the 80/20 rule pretty much prevails here.

Q2: Do you think providing a traditional login option is important, for those who can't or won't use OpenID?

RESULTS: 60 YES / 41 NO

It's clear from these numbers that even some folks who didn't have issues with their OpenID login thought we were being a bit ambitious expecting everyone to use it. But this may have to do with our initial implementation, which was a simple text box asking for a URL. We'll comment more on this a little later.

Q3: Were you able to successfully navigate around our site and find what you were looking for?

RESULTS: 91 YES / 8 NO

Q4: If not, let us know any technical problems you ran into:

Fortunately, it looks like despite some initial hurdles, most people found what they were looking for.

Most of the NO responses corresponded with OpenID issues, a couple people were unable to log in at all. We suspect this is user error combined with our very ugly implementation of OpenID that shipped with our early access release. Many folks suggesting taking ideas from StackOverflow by adding some provider buttons, and we've started to do that. Hopefully, things weren't as confusing at the official launch as they were during the pre-launch. If there is a particular provider you want us to support on our login page, please let us know in the comments.

Some other folks were confused by our design for solution downloads. It was originally very hard to see where you needed to go to download a solution once you purchased it. We have since updated the design a bit before launch, and we hope this makes things a lot easier.

We then went on to ask some questions about our problems and solutions. Luckily, it seems that we've done a better job here than we did with our login system.

Q5: What do you think of the size and scope of our solution writeups?

RESULTS: LESS INFORMATION IS BETTER (6) / MORE INFORMATION IS BETTER (12) / GOOD SIZE AND SCOPE AS-IS (79)

We were a bit worried about this one, as our solution articles have evolved into something very different than what we originally envisioned. With The Traveling Hacker coming in at 15 pages of content and Sudoku Service at a whopping 17 pages, we were afraid that these articles might offend the tl;dr; crowd. However, it appears that the vast majority of our users like what we've put together so far. I suppose there is still room for the Steve Yegge's of the world, after all. :)

The reason why these articles tend to be long is that they're not really just about the solution. Instead, they're about the skills and techniques that are needed to tackle the sorts of problems we've been presenting. As we stated from the very beginning, the reason why we think rubyproblems.com might be worth your money is that it doesn't just show you the answer to something, but the hows and whys as well. It's still very experimental, but I've been very happy with the results so far.

The only concern I have is that these articles are extremely time consuming to write. Because we don't expect rubyproblems.com to ever in a million years let us quit our day jobs, I'm going to need to figure out a way to do this sustainably. But we've got some ideas rolling around about that, so no need to panic just yet.

Q6: What do you think about the difficulty level of the problems we've proposed so far?

RESULTS: TRIVIAL (2) / VERY EASY (3) / MODERATELY COMPLEX (66) / CHALLENGING (26) / TOO HARD (0)

It looks like we hit the sweet spot here. I realize now that I should have followed up with a question about what difficulty people wanted to have, but personally, I think the area between moderately complex and challenging is where we want to be. We want people to be able to complete the problems on their own, ideally speaking. But we also want the problems to be interesting enough so that our users would benefit from our perspective in the solution writeups.

While the complexity of building software doesn't fit well to a linear scale, I think that we're on the right track with our problems based on these responses. That said, I certainly would like to talk to the two people who found these problems trivial, as I can probably find good work for you.

Q7: Do you feel like the topics we've proposed are interesting and relevant?

RESULTS: 92 YES, 4 NO

I'll let this one speak for itself. ;)

Q8: Any other suggestions on how we could improve our content?

This was my favorite question, just because so many people had great ideas here. I'll run through just a few of them to give you a sense of what we've got on our radar.

DIFFICULTY RATINGS FOR PROBLEMS:

This came up a lot but is a bit tricky because I doubt that I'm the best judge of how difficult a problem will be to our users, since they come from from a wide range of backgrounds. But one possibility is that we can open up a vote for those who've purchased solutions to vote on the complexity of the problem. We could display this difficulty rating to everyone who visits the problem, and possibly make some navigation features based on problem difficulty. This is the first idea that comes to mind, if anyone can think of some other ones, let us know.

TEAM UP WITH PEEPCODE:

The idea here was that we could offer exercises that complement specific screencasts. Hmm...
that's a fantastic idea, I should be talking to Geoffrey.

GROUP PROBLEMS INTO RELEVANT SKILLSETS:

Yes, we will definitely do this once we have a larger set of problems. This will be especially easy because every problem has a couple key points it tries to cover.

ALLOW USERS TO ADD TWITTER HANDLE TO THEIR ACCOUNTS:

The idea here was that when people comment on the problems, it'd be nice if their username could click through to something like twitter. This is easy for us to build, and a good idea, so maybe we'll add this in the near future.

RATING AND VOTING ON USER SUBMITTED SOLUTIONS (LIKE STACKOVERFLOW):

Not a bad idea, but not exactly what we're going for. Maybe somewhere way down the road.
That said, people are welcome to post the problem definitions wherever they'd like, so this
can happen on a third party site as of right now.

SHORTER PROBLEMS FOR FREE:

We liked this idea a lot and decided to start working on it right away.

We're going to do this in two ways: Mini-sprints and Micro-quizzes. The Micro-quizzes are going to be one line questions posted to the @rubypractices twitter daily, so follow there if you want really quick little problems.

Mini-sprints are closer to our full featured problems, but about half the size and complexity. These are a good way to try out our system before buying a solution, because they allow you to access the solution articles and discussion system for free. We plan to alternate between mini-sprints and full-scale problems, so there will be a new mini-sprint roughly every other week.

LESS WEB-ORIENTED PROBLEMS:

No worries, the problems will cover a lot of ground, and we will spend just as much time in other areas of Ruby as we do on web topics (if not more).

The next set of questions were about our business model. These were the most sensitive issues on the survey, given that we're trying to come up with a hybrid commercial / community service, and it's hard to figure out where to strike the right balance.

Q9: Based on what you've seen of our solutions so far, do you think that $3.00 an article is too cheap/reasonable/too expensive?

RESULTS: TOO CHEAP (3) / REASONABLE (73) / TOO EXPENSIVE (21)

The commentary on this in a followup question was really interesting, and we'll discuss that in just a moment. But this is about what I expected to see, if not a bit better.

Q10: Do you plan to purchase articles from us once we begin accepting payments?

RESULTS: YES (40) / NO (7) / UNDECIDED (51)

We really should have followed up with an "If undecided, let us know what we can do". If you are reading this and responded undecided, please let us know what we can do in the comments. But again, this is as good or better than what I could expect, the early access folks didn't know what to expect, and since we're trying to data-drive rubyproblems.com, it'll be a little while before the overall vision comes together.

Q11: Any other comments or suggestions about our business model?

For those who found the pricing of our service too expensive, I was relieved to see this wasn't an indicator that they were cheapskates ;)

There were basically two major categories, one relating what we're offering on rubyproblems.com to other services, and another reflecting the global economic inequalities that we're faced with. I'll tackle the second one first, because I admittedly didn't give it as much foresight as I wish I did.

GLOBAL ECONOMIC INEQUALITIES:

There were a number of responses (from India in particular), suggesting that we should charge $1 or less per solution, given that what we're asking for is way more than the average cost of an Indian meal. I don't have a good answer to this problem, because unfortunately, the folks running this service live in Chicago, IL and New Haven, CT in the US. It's embarrassing how expensive things can be here, and I'd just like to point out that we're very excited about the Indian restaurant nearby my apartment in which we can get a "cheap lunch" at $10 a person.

Someone had suggested doing regional pricing, and if this were a large scale project with a lot of cashflow coming in the door, I think this would be a great idea. Unfortunately, at the scales we're looking at now, I rather just open source our solutions after a few months, and cross the divide that way. We will definitely do something like that, we just haven't worked out specifics yet.

My sincere apologies to those folks who honestly would like to pay for our solutions but find $3 prohibitively expensive. Hopefully the mixture of free content will help make up for it.

PRICE COMPARED TO OTHER SERVICES:

As for those who compare what we're offering to what is available at PeepCode or what you can find in a book, I think that you've got the wrong idea about this service. Watching a screencast or reading a book is a great way to learn, but it's typically a passive activity. Rubyproblems.com is by nature, an active form of learning. Our solution writeups provide some context that helps sharpen your skills, but you're only going to draw the most value out of it if you attempt the solutions yourself, and only read ahead as far as you need to in order to get past the next hurdle.

Also, just to put the whole thing into perspective, we've sunk about 100-150 hours so far into getting this service up and running, and our weekly cost is looking like it'll be able 15-20 hours of work. If you're worried about us getting rich off of this project, there is no danger of that happening any time soon. This is a labor of love, but one that's costing us a lot of our time, so it would pretty much be impossible for it to be run as a purely volunteer project for us. If we can make enough to offset that somewhat, I think we can keep producing good content sustainably.

The next set of questions were meant to help us get a sense of the sort of audience we are working with, so that we know where to focus our energies.

Q12: How long have you been working with Ruby?

RESULTS: LESS THAN 1 YEAR (27) / 1-2 YEARS (34) / 3-4 YEARS (33) / 5+ YEARS (5)

This distribution looks promising, and indicates that we should write problems that will be accessible to a wide range of skillsets. It makes me feel a little old when I see only 5 responses for 5+ years of experience, but I only barely make it into that category myself, so it's not that surprising.

Q13: Where would you rate your experience level with Ruby?

RESULTS: BEGINNER (28) / INTERMEDIATE (45) / ADVANCED (23) / EXPERT (3)

Another even distribution. While it makes our job harder, we're going to try to make sure that the problems are written in such a way that they are solvable by those on the beginner-intermediate end of things, while still having some subtle twists that should appeal to our more advanced audience.

Q14: Anything we can do to make your experience with rubyproblems.com more enjoyable?

Lots of good feedback here, but a lot of it was reiterating things that were said in the technical difficulties and suggestions to improve our content sections.

One suggestion that came up here that didn't in the other sections was to add RSS feeds for the problem definitions. That is on our radar and is something you can expect to see soon.

Some folks would like to see the layout of the site spruced up by a designer. If there are any volunteers who want to create some mockup for us, I'm sure Jordan would be happy to implement. Right now most of the focus is on the content itself, and on getting the necessary features in place in our app.

IN SUMMARY

This survey was enormously educational for us. We got great feedback that we're busy acting on, and we hope that you'll see changes for the better at rubyproblems.com in the near future. Given the high response rate, we're hoping that the findings apply well enough over the population that what we learned will help us improve the service not just for those who responded to the survey, but for everyone who is using it as well.

We want to keep an open ear about how to make rubyproblems.com as awesome as we possibly can make it, so if you have ideas, just let us know!

We're Live!

Well, it's been a wild couple of weeks, but rubyproblems.com is now officially launched. Our initial offering includes three problems, along with their detailed solution writeups. Here's a quick overview of what content is already available.

OUR INITIAL PROBLEM SET

The first problem is The Traveling Hacker, which is a testing oriented problem. You're given some basic domain logic, but in order to implement it, you need to mock out some calls to a web service. The goal is end up with fully tested code that can just be dropped into production, even though you haven't run it against the live service before.

The second problem is Sudoku Service, and it is a bit inverted from the first one. The idea is that you need to find an open source project that handles Sudoku solving for you, and then write a service and reference client around it. Our solution ended up using Sinatra and REST Client, but there are many ways to go about this. During the course of writing up the solution, I also spent a good deal of time talking about how to quickly search through free software projects and make some decisions on which ones might be suited for your needs. If you don't already have a decent process for auditing open source projects, you might find what I outlined to come in handy.

The third problem, Geography Lesson, is a different sort of problem, which we'll mix in between our traditional ones. It is what we're calling a mini-sprint, and should be solvable in a relatively short amount of time IF you can find the key tricks that make it easy. We've decided to make problems, solutions, and discussion freely available for all mini-sprints, to make rubyproblems.com available to a wider audience.

WHAT TO EXPECT

At this point, we have everything we need to get this party started. Here's what you can expect right now if you head over to rubyproblems.com:

• The three above mentioned problem definitions, all of which are licensed under CC NC-SA,
  which means you can repost them or share them with your friends pretty much anywhere.
• Three nice PDF solutions, all of which were typeset using Prawn and Jambalaya.
• A simple discussion system that is open to the public on all mini-sprints, and is available
  to those who purchase solutions to our traditional problems.
• A Tender support system, so that you can get in touch with us about any problems with
  the service.
  

We expect to release new content roughly every week, targeting two long-form problems and two mini-sprints a month. However, this is going to depend a bit on our ability to produce quality content in that short time frame, and we'll generally prefer quality over quantity. We'll just have to see how it goes over time.

"BETA" STATUS

We do have some rough edges that we still plan to iron out. We'd like to find even better fonts for our typesetted solutions, and while our content has reached the copyeditor, it hasn't made it back to us yet. The materials we've posted so far should still be highly readable, so don't let this stop you from checking them out.

During the early access period, we put out a survey asking about a number of issues related to the site, and got some excellent feedback. We'll follow up very soon with our responses to that feedback, to let you know what we've already done, and what we plan to do in the coming weeks.

We hope you enjoy the site, and that you'll learn something from it. Please let us know if there are ways that we can make things better.

Enjoy!