We’ve been publishing an anonymized dump of all user-contributed Stack Exchange content since 2009. Unfortunately, at the end of last year our former host, ClearBits, permanently shut down. So we set out to look for a new home for our data dumps, and today we’re happy to announce that the Internet Archive has agreed to host them:

The Stack Exchange Data Dump at the Internet Archive

We’ve been big fans of the Internet Archive for a long time, and we’re really happy to be working with them on this.

Wait, what’s this data dump?

All community-contributed content on Stack Exchange is licensed under the Creative Commons BY-SA 3.0 license. As part of our commitment to that, we release a quarterly dump of all user-contributed data (after carefully sanitizing it to protect user private data, of course).

Each site can be downloaded individually, and includes an archive with Posts, Users, Votes, Comments, Badges, PostHistory, and PostLinks (new). You’re free (and encouraged!) to share, remix, analyze and build on top of this data any way you want, as long as you follow the attribution requirements.

What are the attribution requirements?

In keeping with the spirit of sharing and proper attribution, and as the “attribution” part of the CC BY-SA license, we require that you do the following when you use the data:

1. Visually indicate that the content is from the Stack Exchange network

2. Link back to the original source question or answer

3. Display the author names for each question and answer you show

4. Link back to the author’s user page

Those links should be ordinary hyperlinks directly to the Stack Exchange site, without “nofollow” or any obfuscation or redirection tricks plainly visible on the page (we’re looking at you, content farms).

I’m too lazy to download this giant zip file. Can’t I just play with it online?

You’re in luck! We also make the data available through the Stack Exchange Data Explorer (an open-source project maintained by community member Tim Stone) which lets you run SQL queries directly against a copy of the data. It’s updated weekly, and includes some data that’s not in the data dumps in order to keep the size of the downloads reasonable.

If you want to access the data programmatically, we also have a pretty expansive JSON API that returns similar data to the dumps (but updated in real time with the websites). If you need help, we have a whole site for people developing apps on top of the API.

So take our data for a spin! We love seeing what people create with it, from apps to research papers or even machine learning contests. Making this data easily accessible is just our way of giving back to the community that has made Stack Exchange so successful.

9 comments

Another holiday season has drawn to a close. We’ve had three glorious weeks with our beloved hats. Now as we pack away the tinsel and the party horns, it’s time to put the hats back in their boxes for another year. Before we move on to 2014 with our bare heads (and our full hearts), let’s take a few moments to reminisce.

76,586 users from all over the network earned 214,172 hats this year – that’s just about twice the number of hats they earned last year. 95 sites opted to participate in Winter Bash, which is more than the total number of sites that simply existed during last year’s event.

The most commonly earned hat was the Old Hat, earned 74,631 times (by 35,589 distinct users). The least commonly earned public hat was Oh the Horror, earned just 46 times. And the rarest hat of all was the top-secret Don Draper, earned only 14 times across the whole network.

Something new we did this year was keeping the secret hats’ triggers… well, secret. Since the community asked so nicely, it’s now time to reveal the mysteries of the secret hats! In ascending order of rarity:

• Chuck Yeager was the most commonly earned secret hat, awarded first to Óscar López - the very first user to discover a secret hat. This hat was awarded to users who answered a question within an hour of it being posted, with their answer scoring 2 or more.
• With Great Power was awarded to moderators (elected or pro tem), former moderators, and Stack Exchange employees.
• Those who earned three hats in a single day earned Johnny Three-hats for their trouble.
• The Ghost of Winter Bash Past appeared only to those who earned a Necromancer badge.
• IG-88 was a less well-known bounty hunter, and the hat that bears his name went to users who tried for a bounty, but didn’t win it.
• I’m Not Listening was awarded to users who rejected a suggested edit on their own post.
• For I See Your Point, users had to leave 5 comments on a site meta, each comment scoring 2 or more.
• Before It Was Cool was awarded to forward-thinking users who asked a question with a brand new tag (that was not deleted or removed).
• Eureka! was awarded manually by SE staff to users who correctly determined (or guessed) the trigger for any of the secret hats.
• Don Draper, in homage to everyone’s favorite smooth-talking ad man, went to users who posted a community ad that received enough upvotes to be displayed on the site (usually 6). 

And finally, we need to send a special shout-out to the top hat earner across the entire Stack Exchange network. This user earned a whopping 44 hats – all of the hats they were eligible for, missing only With Great Power due to not being a moderator. Please join me in giving the eminent Logan M a hearty round of applause!

Honorable mention is due to Manishearth, who held the network-wide lead for almost the entire duration of Winter Bash and was only edged out in the final hours by Logan M’s 44th hat. Well done to you both!

Lastly, we send our gratitude to each and every one of our users for the tireless and high-quality work you do throughout the year, even when there aren’t any hats to earn. Winter Bash is our chance to kick off our shoes and have some fun during the holiday season, and we hope you enjoyed it! The whole Stack Exchange team wishes everyone a happy and healthy 2014. That’s all, folks!

30 comments

This is a time of year of traditions and celebrations — and we have a tradition at Stack Exchange where we set this time aside each year to give back to the groups and organizations that need our help. Each year, we reach out to our moderators and offer to make a $100 donation to charity on behalf of each moderator for their Stack Exchange community. It’s just a small gesture of thanks for the tremendous amount of work every community has contributed to make this entire thing possible.

This “giving back” program actually goes waaay back to the beginning when we started with only 18 moderators and three sites. As our ranks grew, so did the donations. So on behalf of the 375 moderators this year, we have made the following donations to charity:

It is also important to remember and support the tools and organizations that make what we do possible, so we also made the following donations:

• HAProxy — $1,000.00
• jQuery Foundation — $1,000.00
• Linux Foundation — $1,000.00
• OpenSTV, the voting engine that drives our elections — $1,000.00

In addition, we continue to be a MathJax Partner with a donation of $20,000 in our commitment to helping math and science communities on the web.

As we approach 2014, I think a lot about what we have built here together. I think about the fact that this is all made possible by people who DONATE their time and GIVE freely their knowledge to benefit future readers who come here seeking help. It’s your contributions here that make all this possible. It’s what keeps the lights on and the wheels turning… and what makes this small gesture of giving back possible.

I take a lot of pride in what we do — and you should, too. This giving back program is just one of the many things we as a community do that is easy to feel pretty good about. It makes me delighted to be a part of this organization and part of a community that works so hard to help people they will probably never meet.

Take care, and see you in 2014!

42 comments

Hi folks, just a quick note here to remind y’all that the yearly Stack Overflow User Survey is live!

We’ve edited the questions/answers based on your feedback from last year’s survey and suggestions posted on Meta.

As a small token of our appreciation, we’ll be donating $1 for each completed survey to your choice of one of this year’s Stack Exchange Gives Back charities (on top of what we’re already donating on behalf of each site).

As always, we’ll be posting the results here on the blog once the survey is completed.

Take the survey!

38 comments

Ahh, the wintry season…

The gatherings of family and friends, the giving and receiving of gifts, the making and/or breaking of New Year’s resolutions – however you and yours celebrate, the end of a calendar year heralds many traditions.

Here at Stack Exchange, we wanted to get each of you an awesome, personal gift, and mail it to you as our way of saying “thanks.” But our accountant pointed out that there are 4.5 million of you, which promptly reminded us that the holidays aren’t about gifts. The real spirit of the holidays can only be captured with…

Hats!

That’s right: Winter Bash is back for another three weeks of millinery-related holiday fun.

What’s new, you ask?

New hats:  There are over 30 new hats to earn this year (with many thanks to contract designer Elias Stein). And by “hats,” we of course mean, “things you can stick on your avatar’s face.”

And it’s possible that there just might be a couple of secret ones, too. (By “it’s possible,” we mean  ”there definitely are, because we made them, like with computer code and everything, so there’s not really much doubt whatsoever.”)

Hats are transferrable: What? No, you can’t sell them to each other. Hats are transferrable across sites! You read that correctly: this year, if you earn a hat on any site, you can wear it on any participating Stack Exchange site. This was one of our most asked-for feature requests after last year’s event, and it’s a great way for everyone to highlight their achievements on their favorite site across the network.

Hat position is adjustable on your face: You remember how crushed you were after finally earning a mustache “hat,” only to discover that on your avatar, it was basically an extremely dapper unibrow? NEVER AGAIN.

You can finally reposition hats in the box until Don Draper’s suit fits as well it fits him. (I know, I know… “it’s not a suit; it’s a carousel.” Give it a rest, Don. Not everything is a carousel.)

Winter Bash 2013 will run from Monday 16 December 2013 through Friday 3 January 2014. During that time, participate on any Stack Exchange site to earn awesome hats (and other accessories!) Each hat has a different activity to trigger it. You can see all the hats and their triggers on the Winter Bash 2013 homepage. Still have questions? Of the kind that get asked… frequently?  Check out the  Winter Bash FAQ

All the hats will go back into storage at the end of Winter Bash, so get out there, earn some hats, and show them off while you can! Just be careful. We paid a deposit on them.

48 comments