I’m looking to outsource some work and could use some advice on:
1. what to call the work I need done
2. what service to use
3. what pitfalls to avoid
First, I’m having a hard time figuring out the correct name and classification for the work with which I need assistance. It is important to find a good match for the job (both in terms of optimal pricing and optimal output) so it seems important to categorize and describe the job well. I want to make sure I find the right person/people for the job. I don’t want to have to pay too much, because the person doing the job has skills this task doesn’t even require. But I also don’t want to be stuck without any leads, because of misclassifying my ad.
The work entails looking at video screenshots of people’s online behavior and writing down, on a spreadsheet, what sites the user visited, what search terms the user entered, what text/image the user clicked on and the timing of each such action. Basically, I have a spreadsheet with items that needs to be filled in. What would one call this job? My first thought was coding, but coding usually refers to programming on these service sites and that’s not what this is. Is it classification? Is it content analysis? What would be the most accurate description? (For anyone curious as to where I’m going with all this, this article describes the process in more detail.)
Second, I am still looking for recommendations for such a matching service. From an article in Esquire I learned about Brickwork India. I had already heard about RentACoder before, but the folks there may be too technical for my needs. Additionally, thanks to suggestions through illumio, I have gotten a few other leads: oDesk, CyberTemp and eLance. Anyone know of others I should consider?
It doesn’t matter where the person resides. The goal is to get reliable work at a reasonable price (which I realize is relative). I would hire locally, but it doesn’t seem any easier to find someone where I am.
Third, if anyone has experiences in this realm (that is, working with people from afar), I would be curious to hear. What do you wish you had known before hiring someone? How could it have gone more smoothly? How might you have found a better match or found a good match quicker?
{ 1 trackback }
{ 23 comments }
Barry 08.08.06 at 12:28 pm
Eszter, this isn’t a technical thing at all; it’s data entry. My suggestions:
1) Don’t use a programmer; that’s a waste of his/her time, and your money. Ask around for somebody who’s done clerical work like this, and has proven to be accurate and intelligent.
2) Think very carefully about how the spreadsheet is structured, for two aspects: data structure and entry efficiency.
(a) Assuming that the spreadsheet will be imported to a stat package, go over that process ahead of time, preferrably with the statistician involved. You don’t want to have to recode stuff which could have been entered in a better way.
(b) From the viewpoint of entry efficiency, work with the data entry person, to get a design which they will be able to use with speed and accuracy. For example, use default values, look up tables, and conditional formatting for data validation. This can make a large difference in both speed and the error rate.
luci 08.08.06 at 12:29 pm
No experience with outsourcing, but your description kinda looks like standard “data entry” work. I did such grunt work for a few years – much of it involved looking at images (like medical insurance claim forms that had been scanned/imaged) and making (not always perfectly objective) judgements about how the data should fit into categories.
The incentives usually favor quantity over quality, so I’d make sure you stress the importance of the latter.
SamChevre 08.08.06 at 12:42 pm
Agreeing with the above posters: this is pretty much routine data entry.
Your institution almost certainly has someone who does data entry sometimes. Pay them $50 or whatever seems fair for a couple hours of consulting, and have them help you figure out how to set up your input sheet and how to set up error detection. That set up is key to getting good data entry done.
Andrew 08.08.06 at 12:44 pm
Go through a coding agency, but get a tester. You don’t want an actual programmer, but a QA person who is going to do a good job of paying attention to these details.
matt 08.08.06 at 12:48 pm
In language processing, we generally call this sort of work ‘annotation,’ which strikes me as a better fit that ‘data entry’ as it includes the notion of adding information to raw data, rather than just transferring it to a different medium. If I were writing the ad, I’d call it ‘HCI data annotation’ (human-computer interaction) or something along those lines.
Apart from having a well-defined annotation scheme, the input method can have a big influence on results. Having a decent tool/input method can make such work go much faster, particularly if it’s able to catch errors such as event 2 coming before event 1 because of a typo in the time field, though of course there’s the overhead of making such a tool.
Kimmitt 08.08.06 at 12:49 pm
Wouldn’t Mechanical Turk possibly work for this?
Chris 08.08.06 at 12:53 pm
I wonder if you had a clear set of instructions describing how to code the data and what you expect in terms of time and quantity, if this task wouldn’t be well-suited for Amazon’s Mechanical Turk.
Might be worth a shot.
Chris 08.08.06 at 12:53 pm
Oy. Kimmit beat me to the Mechanical Turk punch. :-)
Ryan Miller 08.08.06 at 2:55 pm
I also thought of Mechanical Turk, but the upfront cost there to get your stuff interfaced to their library probably isn’t worth it for you. In lieu of that, why not just use Craigslist? Posting in the right category (data entry, or clerical) will eliminate the overqualified people because they don’t look in that category to begin with, regardless of the search terms. I think just a Craigslist entry in the appropriate ‘Gigs’ category, saying just what you said here, would be more than adequate. Do specify some kind of salary range, so that people know you’re for-real. And if you don’t have any bits in Chicago, just cross-post it to the same section in other cities to increase your reach.
Rob St. Amant 08.08.06 at 3:30 pm
I think Matt’s suggestions of using “HCI” and “annotation” are good; “tagging” is also a plausible term: “video annotation” or “video tagging”. This is not an unusual task in the area of usability analysis; there are tools to help people do it (I think that Noldus is one of the best-known companies out there), but unfortunately, as one might imagine, the process is hard to automate completely, even for video captured from computer screens. (It’s obviously too late to suggest that logging software would have been a good thing to have.)
I’m sure that this has been considered, but I’ll mention anyway that more than one annotator is a good thing to have if there are any judgment calls to be made.
Finally, if I were doing this, I’d advertise for undergraduates, at a rate comparable to what work study students get.
rented mule 08.08.06 at 4:29 pm
I’m (supposed to be doing) this very kind of work right now. I’m on a temp assignment at Wells Fargo doing annotation and data entry. It’s grueling (sp?) work — might I suggest that you pay whoever you hire at least $10 an hour? I know I’m not the best advertisement, surfing when I should be entering…
Stuart 08.08.06 at 5:21 pm
I think data entry is the most common term for this job, at least in the areas I work adjacent to (a lot of data added to the software I create is first written and then needs to be added to a database, it is usually transcribed into excel or equivalent).
As has been suggested take care in setting up your data entry form, whatever way you do it (asp, excel template, etc) its good to think about the terms you use being as simple and generic as possible as the person usually that will do most of the work isnt likely to be highly skilled or have niche knowledge or they wont be doing this job. Especially consider fields that need to be filled in where there is some level of judgement about the result, and try to either eliminate or clarify them. And of course try to minimise the free text fields in the form except where absolutely necessary, the entry should be from a list (or be numeric) wherever possible, or you are likely to end up with lots of variants of the same term.
After that you have to think about quality – and this obviously depends on how critical accuracy is for the task, the simplest way being to get a third party (or yourself) to check a sample of the data to estimate the level of accuracy, or the other extreme is to find a way to uniquely identify all rows of data that would be entered from the videos and then get the entire list done twice (by two completely unrelated people/organisations), and then have someone resolve all rows with discrepancies between the two entered lists.
I am sure there are plenty of places to find people to do this sort of work, dont know any offhand as I all the companies I work with have their own internal staff to do it.
Stuart 08.08.06 at 5:25 pm
Just read the link you provided to the area you are studying, sounds a lot like the things Jakob Nielsen talks about on his blog/website over at useit.com, you may well know his work but just in case you arent aware it could be useful to check it out briefly.
Alex H. 08.08.06 at 9:31 pm
FWIW, I just finished bidding on something similar–well, not really similar, but human-based coding–over on RentACoder. There are lots of people on there who are interested (I got 15 bids). I plan on blogging how it turns out, but I think it’s safe to assume there are people who are capable of coding (in the non-programmatic sense) and interested in it on RentACoder.
C. L. Ball 08.08.06 at 10:37 pm
Let me get this straight: you video-captured the subjects actions rather than using a key-stroke recording program? Why didn’t you (or hire someone to) modify one to produce the very data that you seek?
Doesn’t Northwestern have undergraduates? You’re not asking them to do any interpretation, so they aren’t really “coding.” Why “outsource”?
Vijay 08.09.06 at 12:09 am
I maintain a detailed blog on IT and India and have written several articles on Outsourcing and India. You can either visit the India IT Pulse or directly search for Outsourcing posts on the blog by clicking here
nb 08.09.06 at 8:29 am
My wife went through a similar problem last year when she needed to have survey data coded for her dissertation. She ended up using a “friend of a friend” who had lots of data entry experience. He did a great job and the price was pretty good–drop me a line if you want more info.
Zephania 08.09.06 at 8:45 am
Data Entry
Have a look at http://www.getafreelancer.com
Eszter 08.09.06 at 1:16 pm
I posted this a few hours ago, but it’s nowhere to be seen so here it goes again.
First, thanks for the various helpful suggestions.
Second, please note that I’ve already done work of this sort and I will be the one using the output so yes, I am aware of what is the optimal format for later purposes.
Third, I have looked into automating the process, I have looked into it several times, in fact. But what I need is not that obvious to automate especially given the parameters of the project. So yes, while I do have keylogs of what people typed in, that is a TINY fraction of the data I need in the end. Of course, I continue to look for automated solutions and welcome feedback. But please don’t assume that I have no idea what I’m doing here.
Fourth, I do hire undergrads, but wanted to look into some alternatives. Undergrads often have a lot of other things going on in their lives and aren’t always as efficient as I’d like. So while I’ve had some great undergrad research assistants and will continue to work with them, I wanted to see what else may be available.
Jason Chicola 08.09.06 at 4:44 pm
Hi Eszter,
Your project sounds like Data Entry. There are many marketplaces that offer Data Entry workers, but here is why oDesk is the site you should try:
(Full disclosure: I am oDesk’s Director of Marketing)
Talent. oDesk has over 3,000 providers with skills ranging from programming to tech writing to data entry. Over 200 oDesk providers offer data entry services. You can choose from individual freelancers or a dedicated co-located team. Data entry workers typically charge $5-10/hour. For example, click here to see a co-located firm that specializes data entry. Check their oDesk Feedback score to see the ratings they received from their past clients.
Hourly Flexibility. Most of the sites you listed offer fixed bid projects. Fixed bid is problematic for ongoing data entry tasks – do you really know exactly how long your project will take to complete? The vendors don’t either. oDesk lets you hire workers on an hourly basis. You can give them work for 1 hour, 100 hours, or 1,000 hours. It’s totally flexible, and you can stop at any time. There are no minimum commitments.
Tools. oDesk is the only service that lets you see what the workers are doing. Buyers can see screenshots of the workers activity. This ensures that you only pay for time worked on your project, and also lets you spot check the work as it is performed, to ensure that your requirements are being met.
To learn more, watch our video tour.
To get started, first create an account, and then post a job description.
See what buyers have to say about oDesk.
What do you think? Is this what you’re looking for? Is there anything about our site or service that you don’t like? Let us know. You can email me at jason[at]odesk[.]com
Regards,
Jason Chicola
Director of Marketing
http://www.odesk.com
daelm 08.11.06 at 5:21 am
data capture for HCI studies, is what it was called when i was organising it. usability anthropology :)
mitchell porter 08.12.06 at 12:38 am
I’ve got some advice: outsource to me! :-) I need the money, I’ve done data entry, and I live in Brisbane, so you can check me out in person next month.
Eszter 08.12.06 at 7:29 am
Jason, thanks for the information about oDesk. I did check it out when someone recommended it, but found hte 30% commission a bit high. I understand the upsides of the tool, but if I’m not going to save in costs then I will likely opt for hiring someone locally. And given the prices, it doesn’t look like I’d save in costs.
Mitchell, again, the problem is that I doubt paying someone in Australia is going to save me money, but I appreciate the offer.
All of which is not to say that I’m only interested in saving money. I’m also curious to know what the options are in case I don’t find enough people locally. (I guess at some level that’s still cost-related. If I had an unlimited budget then I guess I’d likely find people locally no matter what.)
Comments on this entry are closed.