Author Topic: Shutterstock using some kind of software A.I. to review images? (Read 28718 times)

StockBoy · « **on:** September 18, 2012, 16:44 »

Judging by the last couple of batches I have submitted (images with motion-blur that is clearly supposed to be there and images with shallow depth of field) that have been rejected for "focus" or other suspect reasons, I am kind of thinking that SS is using some sort of software A.I. to review images.

Has anyone else seen this type of review of their images? What are your thoughts on companies using some type of software to review your images?

stockastic · « **Reply #1 on:** September 18, 2012, 16:54 »

I have a couple of motion-blur shots of toys, on which I lavished a lot of time and which I think are quite good - and I won't even bother submitting them to the micros.

Jo Ann Snover · « **Reply #2 on:** September 18, 2012, 17:11 »

I think they have acknowledged (in their IPO filing?) that they use automated tools.

I've had shots that sell really well elsewhere and which I know are in focus rejected for focus. I suspect that something about those images runs afoul of the automated software they use.

I'd just move along - I doubt they are interested in doing anything about it (most unfortunately).

Equus · « **Reply #3 on:** September 18, 2012, 17:14 »

Quote from: StockBoy on September 18, 2012, 16:44

Judging by the last couple of batches I have submitted (images with motion-blur that is clearly supposed to be there and images with shallow depth of field) that have been rejected for "focus" or other suspect reasons, I am kind of thinking that SS is using some sort of software A.I. to review images.

Has anyone else seen this type of review of their images? What are your thoughts on companies using some type of software to review your images?

It's been talked about for years. Judging by the number of forum posts on both this and their own forum, SS is aware of what's going on, but are quite happy with the results they are getting. So, nothing you can do, except submit elsewhere.

leaf · « **Reply #4 on:** September 19, 2012, 00:53 »

Here's where Shuterstock talks about reviews in their IPO

Quote

The content we provide to our users is created by a community of contributors from around the world and is vetted by our specialized team of image and video reviewers. Whether photographers, videographers, illustrators or designers, our community of more than 35,000 approved contributors range from part-time enthusiasts to full-time professionals, and all of them must meet high standards in order to work with Shutterstock.

In order to become a contributor, an individual must submit an application that includes a portfolio of images or videos. Of more than 375,000 contributor accounts that have been created, less than 40,000 contributors have been approved. Once accepted by Shutterstock's review team, contributors can upload as many images as they would like; however, every submitted image is reviewed and either accepted or rejected by our team to ensure that images in our library meet certain standards of aesthetic and technical quality. Approximately 38 million images have been submitted to our review team by approved contributors and, of those, only 20 million, or approximately 50%, were approved and made available in our marketplace. Each image that is rejected by our review team is tagged with at least one rejection reason that is communicated to the submitting contributor to help him or her to improve and to give insight into our review standards. Such rejection reasons include focus, composition, poor lighting, trademark infringement and limited commercial value. We combine proprietary technology and highly trained content review staff to deliver sophisticated yet efficient image review—we typically process images within 36 hours of upload.

Contributors are required to associate keywords with each image they submit in order to make their images more easily found using our search algorithms. Keywords usually contain both descriptive terms that literally identify the content of an image (e.g., "padlock") and conceptual terms that describe what an image might convey (e.g., "security"). We have over 650 million contributor generated keywords in our database, an average of approximately 30 keywords per image.

All images accepted into our collection are added to our website where they are available for search, selection, license and download. Contributors are paid monthly based on how many times their images have been licensed in the previous month. Contributors may choose to remove their images from our library at any time. Due to our large number of contributors, we do not have any material content supply concentration; the content contributed by our five highest-earning contributors was together responsible for less than 4% of downloads in 2011.

Shutterstock provides different earnings structures for photographs, illustrations and vector art, and for video footage:

http://secfilings.nasdaq.com/edgar_conv_html/2012/08/30/0001047469-12-008610.html#A2210439ZS-1A_HTM_BG47301A_MAIN_TOC

They do refer to software in this sentanct "We combine proprietary technology and highly trained content review staff " but I think they are talking about the proprietary tech the review staff use, and not some tech to review the images automatically. In the rest of the write up it talks about each image being reviewed by a person.

StockBoy · « **Reply #5 on:** September 19, 2012, 01:24 »

Don't get me wrong, I'm not complaining, only mentioning that I think it is taking place more often than we might think (images being reviewed by a computer rather than an actual person).

I can see both sides of the argument, with the massive number of images that need to be reviewed, how can SS be more efficient, etc.

However, just like anything else if an image doesn't pass the software review because it has a very shallow depth of field, then the image buyers are the ones that don't get to see and decide for themselves if the image has "real" value.

As always, some good images will be rejected and some bad images will be approved — doesn't matter who, or what reviews the images, it is what is is.

microstockphoto.co.uk · « **Reply #6 on:** September 19, 2012, 01:26 »

my grandma - which is highly untrained in photography - finds that photos with dull "perfect" lighting and flat dof are better than more dynamic pictures with proper depth of field and some shadows;

so, if they are not using software, they are probably using their grandma

that's fine: their site, their choice

BaldricksTrousers · « **Reply #7 on:** September 19, 2012, 01:55 »

I've seen numerous threads where people say they have had a rejection, waited a bit and then resubmitted and got the image through. If the process was entirely automated it would always deliver the same result, therefore humans must be making the final decision.
I suspect their software flashes for stuff like blown highlights or digital noise. The software we use can spot this sort of thing and has been around for more than a decade.

nicku · « **Reply #8 on:** September 19, 2012, 04:08 »

Yesterday were reviewed 4 particular pictures... two of them were accepted and two declined. All four pictures are at the same resolution, object, quality. the reason for declined ones was: Trademark. not even one has a trademark sign/logo on them.

This experience can mean two thinks:

1. SS is using some sort of software as a filter before review , and the ''filtered'' images end up for a second ''human'' review. In this case the soft decided that two of my pics have trademark logos/brands and rejects them..... and the human reviewer accepted the other 2.

2. same bench of images are reviewed by multiple reviewers and one person receive two pics and another one the other two.

I can't find other explanation in my case.

heywoody · « **Reply #9 on:** September 19, 2012, 05:38 »

Quote from: nicku on September 19, 2012, 04:08

Yesterday were reviewed 4 particular pictures... two of them were accepted and two declined. All four pictures are at the same resolution, object, quality. the reason for declined ones was: Trademark. not even one has a trademark sign/logo on them.

This experience can mean two thinks:

1. SS is using some sort of software as a filter before review , and the ''filtered'' images end up for a second ''human'' review. In this case the soft decided that two of my pics have trademark logos/brands and rejects them..... and the human reviewer accepted the other 2.

2. same bench of images are reviewed by multiple reviewers and one person receive two pics and another one the other two.

I can't find other explanation in my case.

Or, the automated filter accepted 2 and the human reviewer rejected 2

OM · « **Reply #10 on:** September 19, 2012, 05:45 »

I had a strange one in a recent submission. I got an email with most of the batch approved but one shot had been rejected with no reason given. When I looked at the submission on the site, the 'rejected' image had actually been approved and started selling almost immediately. The only thing that I can think of is that in a previous submission, I had already 2 images that were similar and that maybe a machine had rejected the third shot based on either keywords or optical similarity but that decision had been over-ridden by a senior (human) reviewer. Thought the discrepancy a little strange but as I'm fairly new to SS, I don't know how things are done there.

leaf · « **Reply #11 on:** September 19, 2012, 07:17 »

To be on the safe side, I would suggest wearing a tinfoil helmet whenever you consider uploading to Shutterstock.

ruxpriencdiam · « **Reply #12 on:** September 19, 2012, 08:36 »

Anthony says he stands by his reviewers and that they are qualified.

Shallow DOF can, does and will get accepted as long as it is done right and sometimes it is best to be safe and add a note with an explanation.

I doubt it is software because there are some things getting accepted that if I or anyone else was to try and get them approved they would be rejected.

And also as stated by someone else if you resubmit they usually pass without a problem.

So it is Attila that is at work.

And I wont be editing in 2 minutes either!

stockastic · « **Reply #13 on:** September 19, 2012, 09:13 »

It's inevitable that software is going to be used to 'review' [cough] images, and used increasingly. Human reviewers are a big cost item and an obvious target for the new generation bean-counters now flooding into these agencies as they go public, are sold, or take on new investors. And it's also inevitable that this software is often going to fail and misfire, in ways that will be frustrating, or hilarious, or occasionally spectacular.

Like any industry, they want to automate and get rid of as many of those problematic and expensive employees as possible, and they'll try to do it before the necessary technology is really ready.

In other words this is only going to get worse.

gostwyck · « **Reply #14 on:** September 19, 2012, 09:26 »

Quote from: stockastic on September 19, 2012, 09:13

It's inevitable that software is going to be used to 'review' [cough] images, and used increasingly. Human reviewers are a big cost item and an obvious target for the new generation bean-counters now flooding into these agencies as they go public, are sold, or take on new investors. And it's also inevitable that this software is often going to fail and misfire, in ways that will be frustrating, or hilarious, or occasionally spectacular.
Like any industry, they want to automate and get rid of as many of those problematic and expensive employees as possible, and they'll try to do it before the necessary technology is really ready.

In other words this is only going to get worse.

Generally speaking machines, computers, software, technology, etc are vastly better than humans at performing complex tasks. That's why we have them.

In the unlikely event that your image was rejected by an automated process it will be because it did not meet the set specification. It may be that the specification is wrong ... but the specification would have been defined and set by a human.

RacePhoto · « **Reply #15 on:** September 19, 2012, 09:45 »

I'd agree, it's not "AI bots" doing reviews. You can check things like size and format with a computer program and save the reviewer time, so they don't have to look. I don't believe that SS uses image recognition for reviews. There's nothing but forum rumors and suspicion that claims they do. One would think that by this stage of the game, someone on the inside would have told us the facts? Like yes they do, or no they don't.

As for soft focus and blurred or depth of field. I don't bother anymore. Soft in back can pass, anything soft in front = rejection. It's their standards and anything like that with any artistic exposure or motion blur, is more likely to get refused. Same goes for shadows. Even useful and intentional shadows! Frustrating.

My favorite part, that answers an often asked question: "Of more than 375,000 contributor accounts that have been created, less than 40,000 contributors have been approved." What's that? Oh about 10% pass the first review. And of those? 13,065 authors have portfolio with less than 50 works, is 38.86% from the total amount.

Roughly 30% of the total approved contributors (10,000 people) have over 250 images on SS.

40% less than 50 images, 30% with more than 250, 5% with more than 2000 images. (people here are much of that 5%) This can also be read as 70% of the people who joined and passed the test have less than 250 images on SS!

Quote from: leaf on September 19, 2012, 00:53

Here's where Shuterstock talks about reviews in their IPO

Quote
The content we provide to our users is created by a community of contributors from around the world and is vetted by our specialized team of image and video reviewers. Whether photographers, videographers, illustrators or designers, our community of more than 35,000 approved contributors range from part-time enthusiasts to full-time professionals, and all of them must meet high standards in order to work with Shutterstock.

In order to become a contributor, an individual must submit an application that includes a portfolio of images or videos. Of more than 375,000 contributor accounts that have been created, less than 40,000 contributors have been approved. Once accepted by Shutterstock's review team, contributors can upload as many images as they would like; however, every submitted image is reviewed and either accepted or rejected by our team to ensure that images in our library meet certain standards of aesthetic and technical quality. Approximately 38 million images have been submitted to our review team by approved contributors and, of those, only 20 million, or approximately 50%, were approved and made available in our marketplace. Each image that is rejected by our review team is tagged with at least one rejection reason that is communicated to the submitting contributor to help him or her to improve and to give insight into our review standards. Such rejection reasons include focus, composition, poor lighting, trademark infringement and limited commercial value. We combine proprietary technology and highly trained content review staff to deliver sophisticated yet efficient image review—we typically process images within 36 hours of upload.

Contributors are required to associate keywords with each image they submit in order to make their images more easily found using our search algorithms. Keywords usually contain both descriptive terms that literally identify the content of an image (e.g., "padlock") and conceptual terms that describe what an image might convey (e.g., "security"). We have over 650 million contributor generated keywords in our database, an average of approximately 30 keywords per image.

All images accepted into our collection are added to our website where they are available for search, selection, license and download. Contributors are paid monthly based on how many times their images have been licensed in the previous month. Contributors may choose to remove their images from our library at any time. Due to our large number of contributors, we do not have any material content supply concentration; the content contributed by our five highest-earning contributors was together responsible for less than 4% of downloads in 2011.

Shutterstock provides different earnings structures for photographs, illustrations and vector art, and for video footage:

http://secfilings.nasdaq.com/edgar_conv_html/2012/08/30/0001047469-12-008610.html#A2210439ZS-1A_HTM_BG47301A_MAIN_TOC

They do refer to software in this sentanct "We combine proprietary technology and highly trained content review staff " but I think they are talking about the proprietary tech the review staff use, and not some tech to review the images automatically. In the rest of the write up it talks about each image being reviewed by a person.

Sean Locke Photography · « **Reply #16 on:** September 19, 2012, 10:10 »

Quote from: StockBoy on September 18, 2012, 16:44

Judging by the last couple of batches I have submitted (images with motion-blur that is clearly supposed to be there and images with shallow depth of field) that have been rejected for "focus" or other suspect reasons, I am kind of thinking that SS is using some sort of software A.I. to review images.

Post the examples instead of dreaming up conspiracies.

Jo Ann Snover · « **Reply #17 on:** September 19, 2012, 10:42 »

Here's an example that SS rejected twice - second time I downsized and I did note it was a resubmit. You can see the same image at DT here (as IS's zoom is busted)

I've had a lot of beach (tropical, colorful, shots that sell elsewhere) shots rejected by SS. It's possible that they treat beaches as an oversupplied subject and thus are picky, but the ones they do accept sell well there too. I really have no idea what they do or how they do it, but I do get focus rejections for images that are clearly in focus and mostly on the beach shots. I've also had "poor framing" rejections from them from images that sell well elsewhere - so perhaps buyers love poor framing?

I just move on, but it does irk me that the rejection reason is rubbish - they're entitled to say they don't want it but I wish they had a formal appeals process (and don't suggest the forum mosh pit, it's not the same thing).

heywoody · « **Reply #18 on:** September 19, 2012, 11:40 »

Quote from: jsnover on September 19, 2012, 10:42

.... It's possible that they treat beaches as an oversupplied subject and thus are picky, but the ones they do accept sell well there too.....

I believe they are much more picky about "oversupplied subject matter" and if even the ones that sell well weren't there, a buyer has plenty more to pick from so no sale lost (for SS). I know this Ps off a lot of contributors but it makes sound business sense (again for SS).

ruxpriencdiam · « **Reply #19 on:** September 19, 2012, 12:28 »

You have way to much time on your hands!

BTW where did you find this info? Post it on the SS forums for them to actually see how many submitters there
really are!

Quote from: RacePhoto on September 19, 2012, 09:45

I'd agree, it's not "AI bots" doing reviews. You can check things like size and format with a computer program and save the reviewer time, so they don't have to look. I don't believe that SS uses image recognition for reviews. There's nothing but forum rumors and suspicion that claims they do. One would think that by this stage of the game, someone on the inside would have told us the facts? Like yes they do, or no they don't.

As for soft focus and blurred or depth of field. I don't bother anymore. Soft in back can pass, anything soft in front = rejection. It's their standards and anything like that with any artistic exposure or motion blur, is more likely to get refused. Same goes for shadows. Even useful and intentional shadows! Frustrating.

My favorite part, that answers an often asked question: "Of more than 375,000 contributor accounts that have been created, less than 40,000 contributors have been approved." What's that? Oh about 10% pass the first review. And of those? 13,065 authors have portfolio with less than 50 works, is 38.86% from the total amount.

Roughly 30% of the total approved contributors (10,000 people) have over 250 images on SS.

40% less than 50 images, 30% with more than 250, 5% with more than 2000 images. (people here are much of that 5%) This can also be read as 70% of the people who joined and passed the test have less than 250 images on SS!

Quote from: leaf on September 19, 2012, 00:53
Here's where Shuterstock talks about reviews in their IPO

Quote
The content we provide to our users is created by a community of contributors from around the world and is vetted by our specialized team of image and video reviewers. Whether photographers, videographers, illustrators or designers, our community of more than 35,000 approved contributors range from part-time enthusiasts to full-time professionals, and all of them must meet high standards in order to work with Shutterstock.

In order to become a contributor, an individual must submit an application that includes a portfolio of images or videos. Of more than 375,000 contributor accounts that have been created, less than 40,000 contributors have been approved. Once accepted by Shutterstock's review team, contributors can upload as many images as they would like; however, every submitted image is reviewed and either accepted or rejected by our team to ensure that images in our library meet certain standards of aesthetic and technical quality. Approximately 38 million images have been submitted to our review team by approved contributors and, of those, only 20 million, or approximately 50%, were approved and made available in our marketplace. Each image that is rejected by our review team is tagged with at least one rejection reason that is communicated to the submitting contributor to help him or her to improve and to give insight into our review standards. Such rejection reasons include focus, composition, poor lighting, trademark infringement and limited commercial value. We combine proprietary technology and highly trained content review staff to deliver sophisticated yet efficient image review—we typically process images within 36 hours of upload.

Contributors are required to associate keywords with each image they submit in order to make their images more easily found using our search algorithms. Keywords usually contain both descriptive terms that literally identify the content of an image (e.g., "padlock") and conceptual terms that describe what an image might convey (e.g., "security"). We have over 650 million contributor generated keywords in our database, an average of approximately 30 keywords per image.

All images accepted into our collection are added to our website where they are available for search, selection, license and download. Contributors are paid monthly based on how many times their images have been licensed in the previous month. Contributors may choose to remove their images from our library at any time. Due to our large number of contributors, we do not have any material content supply concentration; the content contributed by our five highest-earning contributors was together responsible for less than 4% of downloads in 2011.

Shutterstock provides different earnings structures for photographs, illustrations and vector art, and for video footage:

http://secfilings.nasdaq.com/edgar_conv_html/2012/08/30/0001047469-12-008610.html#A2210439ZS-1A_HTM_BG47301A_MAIN_TOC

They do refer to software in this sentanct "We combine proprietary technology and highly trained content review staff " but I think they are talking about the proprietary tech the review staff use, and not some tech to review the images automatically. In the rest of the write up it talks about each image being reviewed by a person.

RT · « **Reply #20 on:** September 19, 2012, 13:22 »

Bit OT but how long are people waiting for reviews these day's, I haven't uploaded there for a few months but have had some in the queue for 5 days now - is that the current norm?

ruxpriencdiam · « **Reply #21 on:** September 19, 2012, 13:31 »

SS reviews are running up to 10 days right now.

RT · « **Reply #22 on:** September 19, 2012, 13:32 »

^ Thanks

fotografer · « **Reply #23 on:** September 19, 2012, 13:34 »

I've never known them to take so long at reviews as they do at the moment.

stockastic · « **Reply #24 on:** September 19, 2012, 14:11 »

They're not using "AI to review images" because AI doesn't exist - it's just a computer science term that's been a speculation, a dream, a prediction, and now a marketing buzzword, without ever becoming a reality.

To cut their (human) reviewing costs these agencies will be investing in software that can reject as a first pass, reducing the number of images that have to be seen by a human. For example, it's quite possible today to use software to determine what percentage of an image is in focus. Not perfectly, of course, but good enough to weed out a lot of photos that look ok at reduced size but actually aren't sufficiently sharp at 100%. This could save an agency a lot of money.

Other sorts of screening are possible, and some might not be obvious. For example, the agencies obviously prefer high-key light toned images, nice pastels, and they don't want anything that's dark overall - easy enough to filter on this criteria. Fair? No, but when you have images pouring in like they're coming out of a fire hose, why care?

Software can pick out faces, and reject images of people without model releases.

These companies won't care if the software is "fair" or it rejects some "perfectly good images". It only has to increase profitability, i.e. pay for itself.

MicrostockGroup Sponsors

Author Topic: Shutterstock using some kind of software A.I. to review images? (Read 28718 times)

StockBoy

stockastic

Jo Ann Snover

Equus

leaf

StockBoy

microstockphoto.co.uk

BaldricksTrousers

nicku

heywoody

OM

leaf

ruxpriencdiam

stockastic

gostwyck

RacePhoto

Sean Locke Photography

Jo Ann Snover

heywoody

ruxpriencdiam

RT

ruxpriencdiam

RT

fotografer

stockastic

Related Topics

Sponsors

Microstock Poll Results

Sponsors