Topic: stackoverflow

While we decided a long time ago that we wouldn’t be open-sourcing the core Stack Exchange Q&A engine, we do try very hard to open source as many useful parts of our code as we can.

As Stack Overflow is one of the most prominent Microsoft .NET-created sites for software developers in the world, we feel that it’s part of our mission to help lead fellow .NET developers — and the most effective way to do that is by contributing some of the code that we use to build Stack Overflow and Stack Exchange back to the greater .NET developer community as reusable open source packages. This isn’t just a fundamental part of every Stack Exchange developer’s “be more awesome” plan, it’s an explicit goal embedded in the very DNA of the company.

In fact, over the last few years, we’ve contributed a number of useful open source projects back to the world:

Dapper

Dapper is, quite simply, the world’s most elegant .NET micro-ORM™. We created Dapper out of frustration with all the existing .NET ORMs that were out there. It is the simplest and fastest thing that works, the thinnest sensible layer you can put over your database without getting all Enterprisey© on you like, uh, some other ORMs. It is a shining example of the KISS (Keep It Simple) and YAGNI (You Ain’t Gonna Need It) principles in action. If you need to access a SQL database from .NET, try it out. You just might fall in love with it. Read a bunch more about Dapper over on Sam’s blog.

StackID

On Stack Exchange, you can log in with any OpenID provider and any OAuth 2.0 provider, including Google, Facebook, Yahoo, and so on. But as of May 2011, we also issue our own credentials for those people who want to have a traditional username/password arrangement. StackID is the .NET OpenID provider we created, so we can be both an OpenID consumer (we accept all OpenIDs, as well as OAuth 2.0 where available) and an OpenID provider — that is, we issue our own OpenIDs that are valid on any website that accepts an OpenID. This in turn is based on the excellent work of the open source dotNetOpenAuth library.

MiniProfiler

Here at Stack Exchange, Performance is a Feature, and we found the absolute best way to emphasize our shared family value of performance is to keep webapp performance numbers front and center in every .NET developer’s web browser. Yes, even in production. If you are a developer you’ll see a little number in the upper right hand corner on every single Stack Exchange page you load — that’s how long it took to render the page. And it’s a one click operation to drill down, two clicks to take those performance numbers and share them with blame them on someone else on the team. It’s a wonderful system that I can’t recommend highly enough to every .NET developer who works on a webapp. If that’s you, go download it. Now. Remember, we use .NET partly because it really is blindingly fast, but all it takes is a few lines of errant code to throw all those performance benefits (and more) in the toilet. So download and use MiniProfiler to make sure your fast code stays fast!

MarkdownSharp and PageDown

Markdown is, of course, the simple plain-text-alike markup language that we use throughout Stack Exchange for formatting. But we have to convert it from “raw” Markdown to “cooked” HTML and that takes code. We provide two implementations, for server-side C# Markdown use MarkdownSharp and for client-side JavaScript real time Markdown preview use PageDown. I did a lot of the work on MarkdownSharp, but Ben did most everything on PageDown and is now officially maintaining both of these.

Since this gets asked all the time, yes, it is legal to mix HTML of any kind within Markdown. MarkdownSharp and PageDown don’t do any cleanup of the HTML, they only guarantee that valid Markdown will be converted to valid HTML for display purposes. You must bolt on your own HTML sanitization to taste. If you’re looking for basics, start with this C# sanitization routine and this tag balancing routine. They are mostly loops and regular expressions, so trivially translatable to most languages.

Booksleeve

Redis is our in-memory key-value store of choice. We started out using it just a little, but now it’s become an absolutely critical and totally indispensable part of our infrastructure, much like HAProxy. We use Booksleeve for pipelined, asynchronous, multiplexed and thread-safe access to Redis via our C# code. Performance here is beyond critical, as we talk to Redis from every web server constantly, and we’ve been through several passes of refinements and improvements already. If you, too, need an in-memory key-value store for your .NET webapp, consider the Redis and Booksleeve combination we use. Works great for us!

Data Explorer

Everything contributed to Stack Exchange is under a Creative Commons license. Stack Exchange Data Explorer is the open source .NET tool that we built so anyone can browse and analyze our creative commons data via standard SQL, at data.stackexchange.com. So if you’re looking for a highly flexible, general front end to a bunch of SQL data, SEDE is your huckleberry. For more, see the blog entries we wrote about it.

Protobuf-net

This is technically something Marc Gravell created before he joined Stack Exchange, but we use protobuf-net extensively (and AFAIK exclusively) for high performance, compact serialization of .NET objects before storing them — and I daresay that our heavy use has driven the current version of Protobuf-net to be at least 3x as awesome as it would otherwise be. I don’t think you’ll find a faster and more elegant .NET serialization library in the world.

ServiceStack.Text

This is part of Demis Bellot’s excellent open source Service Stack REST web service framework. And again, something that he created before he joined us at Stack Exchange. We switched to ServiceStack.Text for all our .NET JSON serialization duties a while ago because it was blindingly fast, much faster than any other JSON serializer we could find for .NET. All of Demis’ open source work is of similarly high quality.

And that’s not all! There are a few more awesome bits of our infrastructure we’ll be open sourcing later this year, and someone (sadly, not me) will update this post to include them too.

If any of this looks useful or interesting, please check it out! And if you have the time or inclination, contribute patches and forks back to the greater community. I know we will!

A few months ago we had James Portnow of Extra Credits on the podcast.  We’re huge fans of everything that they’re doing over at Extra Credits, so when they asked us to help them write an episode on programming, we jumped at the opportunity.  Here’s the link:

Extra Credits: So You Want To Be a Developer (Part 1)

If you’ve never heard of Extra Credits, it’s a weekly show about the video game industry from the perspective of people who actually work in it.  It’s a fascinating look into game craftsmanship — not just how video games are made, but what makes them work, what games do well and not so well, and the effects that they have on people.  They talk a lot about about game elements and using them for good and not evil, which is something that we think a lot about with Stack Exchange.

The episode we helped them write is called So You Want To Be a Developer.  It’s part of a loose series of episodes on the jobs involved in making games and how to be awesome at them.  They’ve previously done episodes on game designers and producers, and wanted to do one on developers but didn’t have any direct experience with development.  That’s where we came in.

The episode ended up being a two-parter, because apparently we gave them way too much material to fit into one episode.  You can check out part 1 now, and come back next week for part 2.

In December we launched the 3rd ever Stack Overflow Annual User Survey to measure changes in user demographics and trends from last year.  First, a big thank you to everyone who participated, and now on to the results!

View the survey results

Demographics

Let’s start off with some basic demographics: the majority (50.3%) of users are between ages 25-34 and very experienced (64.3% of users have 6+ years of programming experience). So, where do all of these developers work?  The percentage of developers working at a start-up remained strong with 30.7% of respondents.  We then took a look at salary by company size, and, not surprisingly, users who work at larger companies tend to make more money.

Knowledge

The top 5 languages users reported knowing were: SQL, JavaScript, CSS, C#, Java.  Coincidentally, the most common projects were “Web Platform” at 37.1%, followed by “Enterprise” at 24.4%.  We dug a little deeper to see if language knowledge impacted salary, and it certainly does.  There is a strong correlation between number of languages known and a higher salary.

Mobile

On the mobile front, smart phone usage showed a dramatic increase.  In particular, Android usage actually surpassed iPhone usage with 48.0% of respondents saying they own an Android phone as compared to 34.1% who own an iPhone.  Last year, the iPhone was the most popular device (34.3%) and Android trailed in second place (30.4%).  RIM’s Blackberry continued to fall out of favor with only 6% of respondents owning one.

Reputation

Last year, many of you asked to segment the data by reputation, so here goes.  First we segmented all of the respondents into three reputation groups and then we cross tabulated the results to see if there were any differences in these groups.  You can draw your own conclusions from the table below, but higher rep users tend to be older and more likely to be happy in their jobs than those with little or no rep (however we can’t quite go so far as to say that the WAY to be happier in your job is to spend more time earning rep on Stack Overflow).

If you’d like to receive the entire data set, please email me [email protected].

After much anticipation, we are happy to announce that our friends at Nerd Merit Badges have created a Stack Overflow GOLD Badge. That’s right:

If you’ve earned one of our virtual gold badges on Stack Overflow, you’ve put a lot of work into doing something awesome. We give you flair so you can direct people around the Internet to your profile and show off all the badges you’ve earned, but we wanted to enable you to show off your accomplishment in real life, too.

It’s not all fun and games, though; John and Randy point out that this badge comes with responsibility:

…this badge comes with a job: if you see someone wearing this badge, it’s your duty to ask them how they earned it…And if they can’t answer you, you’re standing face to face with EVIL. Evil that didn’t EARN the badge, and has no right to wear it.

So go ahead: treat yourself to a little extra something this holiday. You look good on Stack Overflow, and now Nerd Merit Badges have made another way that Stack Overflow looks good on you.

(Haven’t earned a gold badge on Stack Overflow yet, or are not a Stack Overflow user? Fear not! Head over to Stack Exchange’s online shop for SE fan patches. Or, if patches aren’t your thing, check out t-shirts, stickers, mugs and more!)