As the post-Web 2.0 notion of “Web Squared” (a smarter web extending into the real world through sensor applications) becomes reality, we will see a proliferation of data. Figures such as Google Chief Economist Hal Varian are predicting that statisticians will be the hot profession as their skills are increasingly in demand.

As suggested by Daniel Pink’s assertions on the rise of a right-brained working elite, the ability to extract stories from a world of increasing and abundant data will be increasingly critical to many industries. Indeed, the opening of U.S. federal government data at data.gov (and the appointment of Sir Tim Berners-Lee to similarly open the UK’s data archives) implies a new societal and cultural importance for data wranglers.

Consider this: IBM is preparing to expand its data analysis employee base from 200 to 4,000 — a staggering twenty-fold increase. You can be certain that a significant portion of this new work force will be untethered, distributed widely across the globe, implying that one of the core skills for a new generation of web workers will be analysis.

So, if you’re looking to sharpen up your data analysis skills, where do you start?

1. The recently published book “Beautiful Data” brings together essays some of the world’s most cutting-edge data practitioners — such as Stamen Design — on subjects as diverse as DNA analysis, crime maps and crowdsourcing.
2. Ben Fry’s PhD thesis “Computational Information Design,” which outlines the need for a new field based on multiple disciplines.
3. The post “Three Sexy Skills Of Data Geeks,” which explains statistics, data munging and visualization — or studying, suffering and storytelling, as the author jokingly suggests.
4. Blogs such as Dataspora and Flowing Data.

For all of us working in and around digital media, computing, or data of any form, it looks like there will be a whole new vocabulary and skillset that we’ll need to be prepared for. The resources above should help get us all started.

How’re your data analysis skills?

I thought I’d share some tips gleaned from my latest move to a new desktop. My way isn’t necessary the best way, but it may give you some ideas when it comes time for you to make the switch. Here are the steps that I took.

1. Back up your data. Use an online backup service like BackBlaze or Mozy. (Solo web workers should have an offsite backup solution, anyway) I also have an external drive that does nothing but back up my computer’s data. The free built-in Microsoft Windows Synctoy took care of my syncing. Make sure you synchronize everything, including the data on any mobile devices.
2. Take a computer inventory. Run System Information for Windows (SIW), Magical Jelly Bean Keyfinder and Belarc Advisor.  Save the output files in Gmail or someplace where you can access it from any computer. These free applications create a list of installed software, license keys, hardware inventory, network information and other details. These apps are all free.
3. Put the computers near each other. This makes the move easier and allows you to check what’s on the old computer and install it on the new PC and compare the setups. It’s also worth checking which apps are used frequently. While the inventory software lists all the installed applications, you might not need to exactly duplicate your setup. The key is to get the important and most frequently used tools and software installed so you can get up and running on the new machine. Then, if you find you need one of the apps you didn’t move over to the new machine later, you can install it. Don’t pressure yourself to get everything installed.
4. Install the applications and tools. Dig up all your software, download the ones that don’t have a CD/DVD and download updated software. Some of my software (Palm Desktop, for example) is so old that I didn’t bother using the original CD/DVD to install it. Instead, I went to the companies’ web sites to download the latest versions.
5. Share folders over the network. Turn on network file-sharing by opening Explorer. Find the folders you want to copy to your new computer, right-click the folder and select “Share.” Look for the option to share the folder (it’s different in Windows XP and Vista). Doing this, I shared the folder with all my work documents so that I had instant access to the documents I needed without waiting for the online backup to do its job. The online backup application then restored the rest of the files.
6. Copy the data from old to new.
7. Start using the new computer. I plan to keep the old computer nearby for a little while so if I run into something I need to customize or verify, I can look at it and update the new computer accordingly.
8. Set up a backup system. Whether you used one before or not, put a backup system in place. Even new computers mess up and you never know when some disaster decides to make life harder for your home. It can happen. I was in my grandparents’ house when it caught fire.

Having many of my applications and data in the cloud made this the easiest and fastest desktop transition ever.

What other ways can you ease and speed up the transition from old computer to new?

Thanks to my smartphone, I don’t carry my laptop very much, unless I’m doing a presentation or going on an extended trip. But it’s nice to have my desktop sync with my laptop, so that it’s ready when I need it. And that’s where things start getting complicated.

My desktop’s a Mac mini, and my laptop’s a Toshiba Libretto U105 (which was a netbook before they started calling them that) running Windows XP. So how do I synchronize my data? It can be done, but it’s harder than it should be.

• Contacts. I use the Apple Address Book, and sync it to my Google Contacts with Googaby. On the laptop, I use Thunderbird with the gContactSync add-on.
• Calendar. I set up several calendars through Google Apps. I’ve subscribed to the calendars in both Apple iCal (on the Mac) and Mozilla Sunbird (on the PC). Alas, the future of Sunbird is unclear at the moment.
• Passwords. I use SplashID because it syncs flawlessly with my Treo, although I’ve also been experimenting with LastPass, which is better at filling in some web forms, but is a bit obtrusive for my taste.
• Bookmarks. For those of us who use Firefox, Weave will probably be a great system someday. But right now, it’s too experimental: It only supports the upcoming Firefox 3.5, and it has a tendency to stop working at random intervals. In the meantime, I’m reluctantly using Xmarks — which strikes me as trying to do too many things. If you turn off all of the options except bookmark syncing, the Firefox add-on seems to work fine. But the Safari version is hard to install and obtrusive. (I mostly use Firefox anyway, so I turned Xmarks for Safari off.)
• Files. Dropbox. It’s easy. It just works.

If you’re wondering how all of these programs get along, well, they do, mostly. Googaby slows my Mac to a crawl when it’s updating, but it doesn’t update very often. Dropbox and Missing Sync work perfectly, and without getting in the way.

So why don’t I just use the web interfaces of all of these programs, and not worry about syncing?

It’s nice to have local backups in case Google goes down (which it does occasionally) or if my Internet connection goes south (which is does more often than I would like). Google Gears helps, but it’s not as smooth as having local copies of everything.

Both Apple Mail and Thunderbird make it much easier to manage and transfer messages between multiple accounts. Gmail Manager is a great way of accessing multiple accounts through Firefox, but it doesn’t let me drag and drop messages between accounts, like I can in Apple Mail.

One of these days, we might be able to use any computer, any browser, and get to our own “mobile desktop.” Glide is working on such a system, as are Xcerion, although neither system is complete.

In the meantime, syncing basic data between computers is much too complex. As long as desktop applications are more robust than their web counterparts, sync will be needed, so let’s hope easier and more user-friendly systems are on the way.

How do you manage data in multiple locations?

Image by stock.xchng user flaivoloka

According to some new Twitter research published on the Harvard Business Blog, my usage is unusual:

A typical Twitter user contributes very rarely. Among Twitter users, the median number of lifetime tweets per user is one. This translates into over half of Twitter users tweeting less than once every 74 days. At the same time there is a small contingent of users who are very active. Specifically, the top 10% of prolific Twitter users accounted for over 90% of tweets. On a typical online social network, the top 10% of users account for 30% of all production. To put Twitter in perspective, consider an unlikely analogue — Wikipedia. There, the top 15% of the most prolific editors account for 90% of Wikipedia’s edits. In other words, the pattern of contributions on Twitter is more concentrated among the few top users than is the case on Wikipedia, even though Wikipedia is clearly not a communications tool. This implies that Twitter resembles more of a one-way, one-to-many publishing service more than a two-way, peer-to-peer communication network.

I suspect than many of us fit into that top 10 percent of users who contribute the majority of the content, but I’m a bit concerned about the conclusion that Twitter might be more of a one-to-many publishing platform rather than a community of peers interacting with each other. I’m not sure that the data shown in the blog post leads to that conclusion. I do worry about all of the brands jumping on Twitter to broadcast their marketing messages and tell people about their products and services without really engaging in the conversation. However, there are also many people and companies who engage effectively in the conversation by becoming a part of the community. @replies and retweets are part of the community-focused culture of sharing content that is also a part of Twitter.

As a freelance consultant, clients often ask me about using Twitter, and it can be difficult to get them to understand the conversational nature of Twitter to shift them from thinking of Twitter as a broadcast medium to Twitter as a community of people holding conversations. It’s also important to remember that most people are not likely to be using Twitter as obsessively as I do, which makes techniques for mining and monitoring Twitter even more important for the average Twitter user.

(As an aside, the Harvard Business blog post also contains some interesting observations about gender and Twitter usage that are outside of the scope of this post, but are definitely worth a read.)

What are your thoughts about Twitter as a conversation vs. a way to broadcast content?

Twitter Search

Twitter’s built-in search engine is probably my favorite way to find information in Twitter. It is great for quick searches to find specific pieces of information, watching trending topics, and persistent vanity searches for your name or company. While I do use the search engine to type in queries, the real power is in using RSS feeds for searches and running them through Yahoo Pipes for additional filtering. In many cases, I use Yahoo Pipes to loop through a series of keywords from a CSV file to search Twitter for each of those keywords and monitor the results as an RSS feed in my reader. I recently did a 2 minute video explaining exactly how to search twitter using Yahoo Pipes and a CSV file of keywords, so I won’t cover it in any more detail here.

Twitter Stream Graphs

I’m a big fan of getting data via RSS, but sometimes you just need a picture. Twitter Stream Graphs are a great way to show your boss how much buzz you generated on Twitter with a particular tweet or a keyword. In this example, I searched for legionoftech, an organization here in Portland than organizes our local BarCampPortland event, and you can see that we did a call for volunteers over Twitter on April 21 and had some conversation leading up to the event on May 1 and 2.

TweetVolume

TweetVolume is a great tool for comparing the number of mentions across several keywords. With the recent Mother’s Day holiday, I’m happy to report that mothers are more important than vacations, the beach and Star Trek. However, sadly, bacon seems to be a more popular topic on Twitter than mothers (shame on you!)

What are your favorite tools for mining Twitter for information?