Neoformix - Discovering and Illustrating Patterns in Data

Six Ways to Find Value in Twitter's Noise

By: Jeff Clark Date: Mon, 07 Jun 2010

The June 2010 Issue of the Harvard Business Review contains a small data visualization piece by myself and Scott Berinato. It's called Six Ways to Find Value in Twitter's Noise and has a StreamGraph showing tweets about the iPad during the launch weekend. I collected and analyzed the data and created the StreamGraph. Scott did a great job picking out some interesting features and explaining what it all means. It was a fun project and it's great to see my work in such a prestigious print magazine. Thanks for the opportunity Scott!

StreamGraph for Makers

By: Jeff Clark Date: Sat, 15 May 2010

A few weeks ago I had the pleasure of reading Makers, a novel by Cory Doctorow. It's an interesting story, well told, and filled with stimulating ideas related to technology, creative culture, and intellectual property.

Cory makes his work available for free download so I was able to create a Document StreamGraph based on the text of the book. The document is split up into 24 equal sized segments and the word counts are done within each segment. These segments are used in place of time along the horizontal axis of the StreamGraph. I chose to show capitolized words and the resulting image does a reasonable job of illustrating the ebb and flow of the various characters within the narrative.

Click for larger version

Some Thoughts on Flash

By: Jeff Clark Date: Thu, 29 Apr 2010

Steve Jobs published some thoughts today about why Apple isn't supporting Flash on their mobile platforms. The Shaped Word Cloud below was created from the text.

Just for fun I made a Clustered Word Cloud as well.

The Art of Tatiana Plakhova

By: Jeff Clark Date: Wed, 21 Apr 2010

I really like the work of Tatiana Plakhova and have been following her Flickr stream since last year. Some of her images make me think of alien life forms or cities of the distant future. The one on the top right here below reminds me of Cerenkov Radiation.

Using her image from the top left above as inspiration I created a simple animation that tries to recreate her style. This video isn't great quality but seems to get the idea across.

Blue Flow 1 from Jeff Clark on Vimeo.

NHL Points Over Career

By: Jeff Clark Date: Mon, 19 Apr 2010

One charting technique that I really like is to take time series for related data that occured over different time periods and align them to a common starting point so they can more easily be compared. One good example is this graph comparing this recession to the last five in terms of employment decline. Another one, this time interactive, is from the NYT and depicts Paths to the Top of the Home Run Charts.

I have created a couple of simple line charts showing cumulative point production (goals + assists) for selected NHL players over their careers. I'm actually using Adjusted Points which try to control for the fact that teams played fewer games in the past and rule changes and other factors impact the ease of scoring goals over time. Data is from Hockey-Reference.com.

This first chart shows many of the top players from the past. I only showed data up until age 43. Gordie Howe did get points in the NHL at age 51 but they were negligible in the overall results other than to illustrate his amazing longevity as a player. The graph clearly shows why Wayne Gretzky is called the 'Great One'. You can also see the various plateaus due to injury for Lemieux, and early career end for Bobby Orr (who is also the only defenceman shown here).

The second graph keeps Gretzky and Richard for comparison but highlights many of today's top stars. Crosby appears to have a legitimate chance to match Gretzky but has a long way to go...

Tweets Containing Love

By: Jeff Clark Date: Fri, 09 Apr 2010

I have been collecting tweets containing the word 'love' for more than a year now and just analyzed a sample to see what other words are being used in conjunction with 'love'. I naively assumed I'd see lots of company or product names as the top non-generic terms. There were a few near the top - iphone, ipod, and starbucks for example. The most commonly used non-generic terms were actually almost all Twitter accounts for singers. The person with the most references was @justinbieber. Note that I analyzed 1 out of every 50 tweets so the counts shown here are ~50 times smaller than the real totals for the year.

During the last few months the total for @justinbieber exceeded the next top 14 combined. The streamgraph also shows a strong decrease for @mileycyrus and @ddlovato. References to @jonasbrothers seem to have split into separate streams for both @nickjonas and @joejonas.

Here is a PDF version of the streamgraph.

Inline Images for Twitter Clients

By: Jeff Clark Date: Wed, 03 Mar 2010

Wouldn't it be cool if your twitter client could directly show tweets with small embedded images? Things like stock charts, graphical weather reports, server status, traffic reports, graphical emoticons expressing emotional state of your friends, mini-graphical movie ratings with thumbs up/down or stars, sports record summaries, or a million others that I haven't though of? Perhaps something like this?

This shouldn't be very hard. In fact, I think all that's required is the following:

Somebody create a new URL shortener that by convention is only for links to images of dimensions 234x60 pixels or smaller. It should verify at the time of link creation that images fulfill the size constraint. I'll call it inpic for now but any short name would work.

Twitter clients that want to support inline images in tweets are modified to recognize tweets with links to http://inpic.com/ABCD and display the image inline rather than the text link. Twitter clients that don't support inlining would show the text link and people could see the image with a click as they do now.

Step 1 is easy. There are hundreds of URL shorteners already in existence. We just need to adopt one that indicates by its' name that it points to a small embeddable image. An alternative that would avoid having to get different companies to adopt the same convention would be to use a special hashcode to indicate the same thing. Have all tweets with any link and the tag #inlinedimage handled by showing the image inline. If the link is invalid or doesn't point to a small image then the twitter client should revert to showing the text form.

Step 2 is easy as well since Twitter clients already show images in tweets - the user avatar images. I chose the size constraint by measuring the space used by TweetDeck to show the text of a tweet - I got about 237x62 pixels. This is just slightly bigger than the standard half banner size of 234x60 used for online advertising so I chose that instead.

Here are a few more things that could be added to make this even more useful:

The URL shortener service (inpic or whatever it gets called) would host images in a manner similar to twitpic.com

Twitter clients would support letting people easily embed graphical emoticons.

If a second link in an inline image tweet is provided it would act as a browser target link if the inline image is clicked on. So an inline image in a tweet would give summary information and when clicked on the user would see more details inside a browser window.

Twitter clients that support this might have an option to turn it off for anyone who prefers to always see text.

I think many people would find this valuable and it seems quite simple to accomplish. Come on TweetDeck, Twhirl, and other Twitter Client companies - get to work!

Where this idea came from

This morning I came across the interesting post Visualizing time series data embedded in tweets by Chris McDowall. The basic idea he discusses is to send time series data in tweets and have twitter clients recognize the format and present it as a small graph ( or Sparkline ) embedded in the tweet stream rather than just text. Chris seemes to have been inspired by the Twitter Data proposal.

It's an intriguing idea and Chris created a proof of concept twitter client called the Twitter Sparkline Visualizer.

One problem I see is that a twitter client that doesn't recognize the special data format would show the cryptic form which would probably be undesirable in most cases. Also, the 140 character limit of a tweet would put a fairly tight boundary on how much could be encoded. In a comment on the post, Tom Carden suggested looking at the Google Charts API as a "good example of a concise vocabulary for passing chart data around using URLs".

Tom's suggestion triggered an idea for me: Use any RESTful api like Google Charts to encode small charts in a URL, then use a URL shortener to construct a tweetable link representing the chart. Furthermore, we can use a specially named URL shortener that indicates to a twitter client that all of its' links point to small inline charts. This lets a twitter client determine efficiently that a given link can be rendered inline.

It makes sense to generalize the idea further to support use of any small image rather than charts in particular.

Profile in Harvard Business Review

By: Jeff Clark Date: Sat, 27 Feb 2010

About ten days ago I was contacted by Scott Berinato, an editor at the Harvard Business Review, who was interested in writing up some of my visualization work for the HBR Research blog. We had a nice chat and he subsequently published Four Ways of Looking at Twitter which profiled my four twitter visualization tools.

He did a wonderful job and the article got lots of attention on Twitter. I've been tracking many of the tweets about the article and there have been at least 1500 tweets sent by various people telling their friends to read it. All the extra attention has made this the busiest week on Neoformix over the past year. Thank you to Scott for creating the article and thanks also to everybody who passed it along to all their friends!

Apple Logo from Products

By: Jeff Clark Date: Tue, 02 Feb 2010

I was looking for pictures of the new Apple iPad and stumbled across this image of Apple Form Factor Evolution. It's got lots of images of Apple products on a nice simple white background and was perfect fodder to use with the Image Foam Technique so I made this version of the Apple logo from the product sub-images.

SOTU 2010 Word Cloud Map

By: Jeff Clark Date: Thu, 28 Jan 2010

Last night President Obama delivered the State of the Union Address. The Shaped Word Cloud below was created from the text.

This second image is from a chart showing references in Beatles songs to earlier songs. There are full images and several other interesting graphics on his site.

Top 20 Data Visualizations Mentioned on Twitter

By: Jeff Clark Date: Mon, 18 Jan 2010

For many people Twitter has become the best place for discovering the latest and most interesting work in a variety of fields. In my twitter client I keep a search column open that gets constantly updated with the latest tweets pertaining to data visualization or infographics and I see lots of beautiful content flow by. I've been collecting these tweets for quite a while and thought it would be interesting to analyze them and see which visualizations were shared through twitter the most often.

Many of the top links in the domain were articles containing collections of visualizations chosen to be the 'Top NNN' by some panel of experts. For example, the top most shared link was 50 Great Examples of Data Visualization by Web Designer Depot. I will have another post in the near future that lists the most popular of these types of links as well as separate lists for products/frameworks and news/analysis. For this list I chose to focus instead on references to individual data visualizations or infographics.

Here are the top 20 ordered by popularity. Click on either the link or image to go to the original article.

1. Historical Browser Statistics - Axiis

2. Stunning data visualization in the AlloSphere - Video on TED.com

3. Worldwide Real-Time Firefox Downloads

4. The Geography of Jobs - TIP Strategies

5. Realtime Downloads from the App Store - Michael Lebowitz

6. Manhattan's Population By Day vs Manhattan's Population By Night - Manhattan population - Gizmodo

7. Take a new look at health - GE

8. The Billion Dollar Gram - Information Is Beautiful

9. Death and Taxes 2009 - WallStats

10. Turning a Corner? - NYTimes.com

Note that the link made popular on Twitter for #9 Death and Taxes was actually a link to an image on imageshack and I have used instead a link to the original source of the material.

The tweets for this entire analysis were collected from March 24, 2009 until December 31, 2009. Only the first link to a specific item from each Twitter ID was counted so that one person did not unfairly impact the results by tweeting frequently about the same thing.

Items 11-20 are listed below.

(More...)

Twitter Word Map for Android

By: Jeff Clark Date: Sat, 16 Jan 2010

Here is a Shaped Word Cloud for tweets containing 'android' from 2009. I removed the tokens 'android' and '#android' from the analysis. You can click on the words to jump to Twitter Search and see the matching tweets. It's pretty clear that android is a 'google' 'phone' and is related to 'iphone' and 'htc'.

Obama 2009 Tweets and #tcot

By: Jeff Clark Date: Mon, 11 Jan 2010

I've taken another look at the set of tweets from 2009 that contain 'Obama'. This time I started by focusing on the most popular hashtags that were used. This graph shows the top 10 hashtags, their distribution over the course of 2009, and the total references to them. The top hashtag by far was #tcot which stands for 'Top Conservatives on Twitter'.

How do tweets that contain #tcot differ from those that don't have it? What words seem especially associated with the tag? What topics do people using the tag seem to be focusing on?

I've done an analysis on the word frequency inside tweets containing the tag versus tweets without it. This chart below shows the words that are used much more frequently in the #tcot tweets compared to the baseline. Words on the left like 'CARE' and 'BUSH' are used at a rate of around 100-120% of the baseline rate. Words on the right like 'BHO' (shorthand for Barack Hussein Obama) and 'RASMUSSEN' are used around 500% of the baseline rate - or, in other words, they occur around five times as often in #tcot tweets as they do in non-#tcot tweets.

The chart is an interesting collection of terms and is an attempt at distilling what the people who use the tag #tcot are saying in relation to Obama. Some notable words in the set are 'DANGEROUS', 'SOCIALIZED', 'EXPOSE', 'RADICALS', 'ARROGANT', 'MARXIST', 'COMMUNIST', 'CLIMATEGATE'.

Tweets About Obama in 2009

By: Jeff Clark Date: Thu, 07 Jan 2010

I collected all the public tweets containing 'Obama' during 2009. There were over 5 million recorded during the course of the year. I've done some analysis on a sample containing every 20th tweet. This first graph simply shows the distribution over the course of the year of the number of times the name 'Obama' was used. The curve has a big peak during the inauguration, a few smaller ones in February and March and is then remarkably level for the rest of the year.

This set of graphs shows other words that were used frequently in the tweets about Obama and that had distributions with a high concentration near specific dates during the year. When ordered by the peak date for each graph they give an interesting graphical narrative of Obama-related events during 2009.

Snow Doves

By: Jeff Clark Date: Tue, 05 Jan 2010

It's been snowing where I live for the last month or so and I've been playing around with generating a dove image from snowflake constituents. This first image is constructed from smaller snowflakes built using the Text Snowflake Creator based on the words PEACE, LOVE, and TRUTH. The dove image is from Wikimedia Commons.

This second version uses the three unicode snowflake characters in the font Arial Unicode MS. I've also applied a small variation in color.

Neoformix Review 2009

By: Jeff Clark Date: Mon, 04 Jan 2010

Thank you everybody for your interest in Neoformix over the past year. I wish you all a Wonderful and Happy 2010!

These are the 20 most popular posts published on Neoformix during 2009 ordered by their popularity. There are a large number of popular posts based on the Shaped Word Cloud concept and a few more on the related Image Foam Technique.

1. Iran Election Word Cloud

2. September 11 Pager Data Visualization

3. Butterfly Plane

4. Oscar Chatter on Twitter

5. Hudson River Landing

6. Fish Tank

7. Butterfly Falcon

8. Shaped Word Clouds

9. TED Shaped Word Cloud

10. The Raven

11. Apple Twitter Word Map

12. Obama Twitter Word Map

13. Earth Day Twitter Map

14. Peace Dove

15. World News Clustered Word Cloud

16. Word Portrait: Michael Jackson

17. Obama Inauguration Speech

18. Twitter List Profile Clouds

19. Toronto Twitter Community

20. Temporal Correlation for Words in Tweets

Note that many of the most popular parts of Neoformix visited during the past year were for projects published prior to 2009 and include Twitter StreamGraphs, Twitter Venn, Big Small, and Word Hearts.

Twitter Venn Birthday

By: Jeff Clark Date: Thu, 17 Dec 2009

One year ago today I launched Twitter Venn. Those of you who have not used it before or have forgotten about it might want to check it out. The image below is an example of what it produces.

Launch Twitter Venn

ACM Crossroads Cover

By: Jeff Clark Date: Tue, 15 Dec 2009

I'm very pleased to announce that an image from my Twitter StreamGraphs tool was chosen as the cover for the current issue of ACM Crossroads - the Student Journal of the Association for Computing Machinery. There is also a small writeup inside about the image. It depicts the streamgraph for the phrase 'data visualization' and suits the issue well since it is dedicated to the Social Web. The entire issue is available online.

Thanks to Chris Harrison, the editor-in-chief, for inviting me to contribute the image and to Senior Editor Jill Duffy for sending me some copies of the issue.

By: Jeff Clark Date: Mon, 07 Jun 2010

By: Jeff Clark Date: Sat, 15 May 2010

By: Jeff Clark Date: Thu, 29 Apr 2010

By: Jeff Clark Date: Wed, 21 Apr 2010

By: Jeff Clark Date: Mon, 19 Apr 2010

By: Jeff Clark Date: Fri, 09 Apr 2010

By: Jeff Clark Date: Wed, 03 Mar 2010

Where this idea came from

By: Jeff Clark Date: Sat, 27 Feb 2010

By: Jeff Clark Date: Tue, 02 Feb 2010

By: Jeff Clark Date: Thu, 28 Jan 2010

By: Jeff Clark Date: Sat, 23 Jan 2010

Top Collections of Data Visualization Links

Top Data Visualization Product Links Mentioned on Twitter

Top Data Visualization Websites Mentioned on Twitter

By: Jeff Clark Date: Mon, 18 Jan 2010

By: Jeff Clark Date: Mon, 18 Jan 2010

By: Jeff Clark Date: Sat, 16 Jan 2010

By: Jeff Clark Date: Mon, 11 Jan 2010

By: Jeff Clark Date: Thu, 07 Jan 2010

By: Jeff Clark Date: Tue, 05 Jan 2010

By: Jeff Clark Date: Mon, 04 Jan 2010

By: Jeff Clark Date: Thu, 17 Dec 2009

By: Jeff Clark Date: Tue, 15 Dec 2009