Matching sketches with photos

by Eszter Hargittai on January 3, 2006

Retrievr is an interesting experiment in matching up people’s sketches with photos on the photo-sharing site Flickr (granted, for now limited to a small segment of images on the service based on interestingness). There is definitely room for improvement, but it’s an interesting idea even if a bit off the mark regarding the matches for now.

I tried the service by creating some sketches of Christmas trees thinking there should be plenty in the recent Flickr pool.

retrievrtrees

Of these four, only the last one yielded any Christmas trees as results (only one of which was not itself just a drawing). The other hits were pretty random both in terms of shape and color depicting anything from cats and birds to close-up shots of flowers. One architecture image did make sense since it really does look like a tree (as noted by its creator in the photo’s title as well).

I played around with the system a bit more and realized that it may be most interesting for retrieving pictures based on color distribution. A blue-orange square yielded photos dominated by related colors.


retrievrblueorange

It’s easy to find photos on Flickr based on topic (e.g. using tags or groups), but less obvious to find images based on color combination (there are exceptions, but these modes are less widespread). It is perhaps in that realm that Retrievr holds the most promise for now.

{ 6 comments }

1

lago 01.03.06 at 2:46 pm

Someone will put in a sketch of Bush’s face and get Hitler as a match in 3…2…1…

2

John Quiggin 01.03.06 at 2:55 pm

If successful, Retrievr would be a challenge to my view that (hyper)text is the natural medium of the Internet. But obviously a long way to go yet.

3

Daniel 01.03.06 at 4:57 pm

I can report that it recognises faces and crudely drawn penises but not shoes or guitars.

4

Eszter 01.03.06 at 6:23 pm

I thought this combo was pretty neat (sketch on left, retrieved image on right):

Eye on Retrievr

5

Tim 01.03.06 at 7:07 pm

I was thinking of this very same issue yesterday, while watching football of all things: a slow-motion close-up view of a quarterback’s eyes as he read the field, breaking the situation down into recognizable chunks (a zone defense, four men rushing, so-and-so is double-covered…). The social and long-term processes by which this knowledge is formed are the adult analogues to the process (during early childhood) by which we, for example understand that a vertical line with branches sticking out of it represents a tree.

Any guesses on how long before Flickr can abstract from a sketch and then recontexutalize as a photo? or even cheat and say, hey, I’ll produce all the images tagged as “trees” too?

6

Luke 01.04.06 at 5:23 pm

I don’t think he has ever cared about civil liberties – he sees his job as protecting us, not protecting our liberties.

Comments on this entry are closed.