Author Topic: Yep, the review process has changed this year (Read 16336 times)

Shelma1 · « **on:** May 07, 2015, 14:59 »

"In the past year, we have further upgraded the ways that we review content by introducing innovative data-driven auditing to monitor and manage reviewer performance. Despite the diligent review procedures we have in place and the immense number of the content submissions, the efficiency of our review team enabled us to get the vast majority of images up on our site in less than 24 hours."

http://seekingalpha.com/article/3154256-shutterstocks-sstk-ceo-jonathan-oringer-on-q1-2015-results-earnings-call-transcript?page=6&p=qanda&l=last

Semmick Photo · « **Reply #1 on:** May 07, 2015, 15:04 »

Just for my own understanding, how does that indicate or explain the rejections? Clearly they still use reviewers. The quote says they reviewers are monitored not that the technology reviews the image.

Shelma1 · « **Reply #2 on:** May 07, 2015, 15:11 »

Quote from: Semmick Photo on May 07, 2015, 15:04

Just for my own understanding, how does that indicate or explain the rejections? Clearly they still use reviewers. The quote says they reviewers are monitored not that the technology reviews the image.

It's pretty much what I hypothesized several days go. They're auditing reviewers because they have a certain ratio of rejections, and I think the reviewers are now under pressure to reject a certain percentage of submissions because they know they're being audited. The easiest way to do that is t reject a batch or most of a batch. Also, it sounds like they're under pressure to get reviews done as quickly as possible, which leads back to quick batch rejections.

Pauws99 · « **Reply #3 on:** May 07, 2015, 15:11 »

It probably means once you are on the naughty list its hard to get off it and if a reviewer is (un)lucky enough to get a run of good or poor images they will be under scrutiny. If its a statistical method the best strategy for a reviewer would be to accept/reject images randomly to keep within statistically acceptable parameters.......or am I being too cynical. Most (all) performance targets have unintended consequences in this case rejecting ALL my images!!!

Surely review turnround time is the WRONG priority.

Semmick Photo · « **Reply #4 on:** May 07, 2015, 16:22 »

Quote from: Shelma1 on May 07, 2015, 15:11

Quote from: Semmick Photo on May 07, 2015, 15:04
Just for my own understanding, how does that indicate or explain the rejections? Clearly they still use reviewers. The quote says they reviewers are monitored not that the technology reviews the image.

It's pretty much what I hypothesized several days go. They're auditing reviewers because they have a certain ratio of rejections, and I think the reviewers are now under pressure to reject a certain percentage of submissions because they know they're being audited. The easiest way to do that is t reject a batch or most of a batch. Also, it sounds like they're under pressure to get reviews done as quickly as possible, which leads back to quick batch rejections.

I dont understand this. Are you implying they need to reject i.e. 20% of the total amount of submitted images as a target? Why on earth would that be a target? Shouldnt they want all images that are good accepted? And why would an admin agree with a bogus rejection?

Shelma1 · « **Reply #5 on:** May 07, 2015, 16:32 »

All I can do is guess. Knowing SS they've taken a look at the percentage of rejections averaged over the years. If they're using an automated system to audit reviewers, I would think reviewers who have substantially more or less than that percentage would be scrutinized. Reviewers are paid by volume, I believe, so they need to get through images quickly. The people who review the reviewers also need to work quickly. It's easier to just say "we agree with the reviewer" than to really take a close look. I've had them say they agree, then when I ask them to point out to me specifically where in the image the problem is, they're forced to actually look and usually overturn the rejection.

Also, as competition increases higher and higher-quality images are being submitted, which raises the bar across the board. So you need to submit better and better work to be in the percentage that's accepted. It makes the collection more attractive to buyers and results in more sales and a better collection to sell to large enterprises, which now make up a larger percentage of SS revenues.

Again, just an educated guess.

heywoody · « **Reply #6 on:** May 07, 2015, 16:37 »

Obviously they would use software to monitor reviewers in the same way that service desk organisations monitor their people. Most likely they would be looking at activity outside statistical norms rather than specifying some random required rejection ratio.

Semmick Photo · « **Reply #7 on:** May 07, 2015, 16:42 »

So a reviewer accidentally ending up reviewing batches of Andresr, Sean, Lise Gagne, Monkey Business, Yuri, etc would at some point have to reject 20% because other wise it gets suspicious to let 100% through? Come on.

Shelma1 · « **Reply #8 on:** May 07, 2015, 16:46 »

I would think the chances that one reviewer would end up reviewing only stellar contributors is pretty slim.

Semmick Photo · « **Reply #9 on:** May 07, 2015, 16:48 »

Thats not the point. Its basically what you and Mike are implying if I understand correctly. If I am missing the point, I am sorry, please explain differently, because sometimes that happens.

ShadySue · « **Reply #10 on:** May 07, 2015, 16:54 »

Quote from: Semmick Photo on May 07, 2015, 16:42

So a reviewer accidentally ending up reviewing batches of Andresr, Sean, Lise Gagne, Monkey Business, Yuri, etc would at some point have to reject 20% because other wise it gets suspicious to let 100% through? Come on.

Didn't Sean report a rejected batch fairly recently?

Shelma1 · « **Reply #11 on:** May 07, 2015, 16:59 »

Quote from: Semmick Photo on May 07, 2015, 16:48

Thats not the point. Its basically what you and Mike are implying if I understand correctly. If I am missing the point, I am sorry, please explain differently, because sometimes that happens.

No, I think what we're saying is that they use software to keep track of reviewers. If a reviewer rejects substantially more or less than the historical statistical norm, someone will say hey, you're rejecting too many good images or hey, you're letting too many bad images through. After a while you'd get an idea of about what percentage of image rejections would be acceptable to SS. If you have a handle onn that the easiest way to get there is by rejecting a certain percentage of batches rather than image by image. If you review Lisa Gagne you see she's a real pro, so you let all her images through, and then you have to make up for that somewhere else. Or you check one image in a batch, see some noise, and reject the whole bunch because you figure there's probably noise in all of them, so why bother really looking?

Clearly the auditing put pressure on reviewers when it comes to rejection rates.

gbalex · « **Reply #12 on:** May 07, 2015, 17:49 »

Quote from: Shelma1 on May 07, 2015, 16:32

All I can do is guess. Knowing SS they've taken a look at the percentage of rejections averaged over the years.

In the end we are left with the above oxymoron.

Without actually being involved in shutterstock's review or content screening process; we are left to guess.

Based on shutterstocks statement all we actually know for sure, is that the process involves storing undefined data or attributes for each review in a database.

We could take a leap and also assume that they use data driven database queries to analyze that data, combined with review statistic's which they have already collected .

Shelma1 · « **Reply #13 on:** May 07, 2015, 18:01 »

How is the statement an oxymoron, exactly?

gbalex · « **Reply #14 on:** May 07, 2015, 19:15 »

Quote from: Shelma1 on May 07, 2015, 18:01

How is the statement an oxymoron, exactly?

If you work for shutterstock and are familiar with the process as well as the review attributes which they collect and utilize while compiling review data for this database; you would not need to guess.

If you are guessing in regard to shutterstock's review processes, are not intimately part of the process and have no knowledge of the details or review attributes collected and monitored.

Then your statement "Knowing SS they've taken a look at the percentage of rejections averaged over the years" is an oxymoron to your first statement "All I can do is guess."

Rinderart · « **Reply #15 on:** May 07, 2015, 20:02 »

SO....Robots are reviewing.lol probably just want to automate everything. Trouble is they can't automate us.

Mantis · « **Reply #16 on:** May 07, 2015, 21:16 »

Whatever is happening is a result of a policy shift integrated with some form of automation. And it doesnt work and they dont care. With a bazillion images they are in a position of sustainability so to cut costs through inspection labor only adds to the bottom line. And they can simply afford the defection of uploads as a consequence of their failed system.

Its all about shareholder wealth.

gbalex · « **Reply #17 on:** May 07, 2015, 22:25 »

Quote from: Mantis on May 07, 2015, 21:16

Whatever is happening is a result of a policy shift integrated with some form of automation. And it doesnt work and they dont care. With a bazillion images they are in a position of sustainability so to cut costs through inspection labor only adds to the bottom line. And they can simply afford the defection of uploads as a consequence of their failed system.

Its all about shareholder wealth.

We all judge the process by our experiences. To date I have not experienced an increase in rejections.

That does not mean that others are not, they could be testing segments of our population and have not rolled this out for everyone. When I experienced a downturn in sales a small segment of my friends experienced the same, over time that changed and more of us have been hit.

In any case I agree that they do not care on any level how their actions affect our workflows or income. I am more convinced everyday that my passion for stock is counter productive to my overall goals.

Pauws99 · « **Reply #18 on:** May 08, 2015, 00:40 »

Quote from: Rinderart on May 07, 2015, 20:02

SO....Robots are reviewing.lol probably just want to automate everything. Trouble is they can't automate us.

I don't think they are saying Robots are reviewing rather they are saying they are using statistically based sampling methods to audit quality.

gbalex · « **Reply #19 on:** May 08, 2015, 01:24 »

Quote from: Pauws99 on May 08, 2015, 00:40

Quote from: Rinderart on May 07, 2015, 20:02
SO....Robots are reviewing.lol probably just want to automate everything. Trouble is they can't automate us.

I don't think they are saying Robots are reviewing rather they are saying they are using statistically based sampling methods to audit quality.

I agree they seem to be talking about a data-driven application which audits, monitors and manages reviewer performance.

"we have further upgraded the ways that we review content by introducing innovative data-driven auditing to monitor and manage reviewer performance'

Semmick Photo · « **Reply #20 on:** May 08, 2015, 01:37 »

They should monitor and manage reviewer performance based on the tsunami of complaints from real people who supported SS to get where they are now, instead of relying on data-driven drivel.

gbalex · « **Reply #21 on:** May 08, 2015, 08:12 »

Quote from: Semmick Photo on May 08, 2015, 01:37

They should monitor and manage reviewer performance based on the tsunami of complaints from real people who supported SS to get where they are now, instead of relying on data-driven drivel.

Yes their new application seems to have blatant flaws as evidenced by contributor real time experience.

heywoody · « **Reply #22 on:** May 08, 2015, 14:29 »

Quote from: Semmick Photo on May 07, 2015, 16:48

Thats not the point. Its basically what you and Mike are implying if I understand correctly. If I am missing the point, I am sorry, please explain differently, because sometimes that happens.

I think Shelma and I are implying totally different things. My point is that this kind of high volume activity would be subjected to some form of statistical monitoring to achieve a consistent standard (as far as possible). There may or may not be some automated QC of images but, frankly, some of the folks complaining would never submit images with the kind of technical issues that would be picked up by an automated review. Again, logically, these must be human decisions based on company guidelines.

etudiante_rapide · « **Reply #23 on:** May 08, 2015, 15:49 »

Quote from: ShadySue on May 07, 2015, 16:54

Quote from: Semmick Photo on May 07, 2015, 16:42
So a reviewer accidentally ending up reviewing batches of Andresr, Sean, Lise Gagne, Monkey Business, Yuri, etc would at some point have to reject 20% because other wise it gets suspicious to let 100% through? Come on.
Didn't Sean report a rejected batch fairly recently?

i don't think rejecting Sean was a case of avoiding suspicion;
more a case of conflict of interest . reviewer is a contributor too.
how else would an experienced top seller be getting rejections

not unless Sean is also giving his kid , or pet monkey ...to do his latest batches

PixelBytes · « **Reply #24 on:** May 08, 2015, 17:36 »

Quote from: gbalex on May 07, 2015, 22:25

I am more convinced everyday that my passion for stock is counter productive to my overall goals.

Have you managed to maintain a passion for stock in the face of all the disasters we contributors have faced in the past couple of years? If so, my hat is off to you. My passion for stock has gradually morphed into an intense dislike.

And before some smarta*s asks "then why are you still doing it". The answer is FOR THE MONEY. But if the money continues to evaporate, the answer to that same question will eventually be I'M DONE.

MicrostockGroup Sponsors

Author Topic: Yep, the review process has changed this year (Read 16336 times)

Shelma1

Semmick Photo

Shelma1

Pauws99

Semmick Photo

Shelma1

heywoody

Semmick Photo

Shelma1

Semmick Photo

ShadySue

Shelma1

gbalex

Shelma1

gbalex

Rinderart

Mantis

gbalex

Pauws99

gbalex

Semmick Photo

gbalex

heywoody

etudiante_rapide

PixelBytes

Related Topics

Sponsors

Microstock Poll Results

Sponsors