Support UNICEF's relief efforts in Myanmar
                                  Google

Monday, December 19, 2005

Riya - Alpha Release: So alpha it hurts!

After 3 weeks in my draft folder, I finally got around to this today. I was one of the lucky 1000 selected to try the alpha release of Riya. There had been so much buzz about this on Slashdot, Digg and TechCrunch, that I was talking about this application's capabilities to friends and colleagues long before I got to see the application. And I learnt a big lesson. No matter what the experts say, hold off talking about an app, till you have actually test-driven it.

My experience with Riya's alpha release has been very disappointing. Well, to start with, it is probably my fault because I had high expectations, especially after I read this on Wired "Currently in alpha testing, the software has proven sensitive enough to tell the difference between twins and recognize members of the same family.". Forget twins, Riya failed to recognize the exact same snap of the same person. In my testing, Riya failed at some very basic image recognition test cases.

Test approach:
I did not want to upload any of my personal pictures. So I chose to use pictures that are already widely available on the web - celebrities, Bollywood celebrities to be precise! The folks at coolbuddy were kind enough to give me permission to use pictures from their website. 569 photos of Aishwarya Rai, Amisha Patel, Bipasha Basu, Preity Zinta, Sonali Bendre, Tabu, Urmila, etc made their way to my Riya account. And I got started.

Identification:
Riya automatically recognizes areas of the photograph that appear like a face and marks it for identification. This is however not consistent and many faces were not auto-marked. I had over 50 Aishwarya's pictures and so went about identifying her snaps. The more snaps recognized by the user, the better Riya is trained to recognize the rest of the snaps. I started with recognizing 10 of Ash's snaps and some of Amisha, Amrita, Bipasha and Preity. I then let Riya recognize the rest. The results were rather poor with some of Aishwarya's snaps being recognized as Amrita and Bipasha. So I went through another round of recognizing some more Aishwarya's snaps to better train Riya; but there was little improvement in the auto-identification by Riya. Next, I recognized 31 of Ash's snaps (one more than what Riya considers great for auto-recognition), but this did not change the end results.

Test Case Summary for identification:
1. Identify some images and then let Riya do its magic. Test Result: Poor
2. Use mirror images of the same person. Test Result: Failed
3. Tiled images of the same person. Test Result: Failed
4. Use different size images of the the same photograph. Test Result: Failed
5. Use the same photographs, but with different sharpness. Test Result: Failed

I expect an image recognition application to positively identify the same people with different expressions and also at different ages. But Riya fails at this. In fact, a scaled down version of the exact same snap of Aishwarya was not recognized by Riya even though I identified Aishwarya in the bigger image. Even text recognition works only when the text is crisp and normal.

So IMHO, this alpha release of Riya is ways away from where the Riya team would like it to be.

Here are some actual examples where Riya failed:
  • In this picture, Amisha Patel's picture is tiled multiple times. (i) Riya wrongly identified Amisha as Esha Doel. To be fair, I trained Riya with only 8 Amisha snaps. (ii) Even if this was correct, it should have identified all instances in the image as Esha, but it did not.

  • In this picture, we have mirror images of Twinkle Khanna. Riya identified one of the images as Preity Zinta and the other as Amrita Arora.

  • Same images of Aishwarya - one is a mirror image and the other is brighter. I trained Riya with these 2. But it wrongly identified Asihwarya in this picture as Amrita.

  • Mirror images of Sonali Bendre. I trained Riya by recognizing one and expected Riya to identify the other. But it didn't.

  • Faces are not always recognized. Here are some examples (1 2 3 4). Riya did not think any of these were faces!

Market Perception:
A lot of hype has been generated around Riya by veterans such as Michael Arrington at Tech Crunch and Robert Scoble at Scobleizer. It would be interesting to see some of their test images that went through Riya. In the current form, the only gain for anyone acquiring Riya is getting them for a lot less than when the application actually works! It appears like Google is not interested for now.

The business case for Riya is very strong. There are a lot of uses for a technology such as Riya's - when it works. Here is hoping the next release does better.

Related links:

54 Comments:

At Mon Dec 19, 11:24:00 AM, Anonymous Sunder said...

Hey I like the new layout!! Its very easy on the eye. I have heard a lot about Riya too but didnt get a chance to test it out. I think your review the most informative of all of them out there. It appears that their photo recognition technology is nascent and probably will need tons of refining. I think their approach is to split up the picture into pixels and then look for patterns. I believe Microsoft's R&D team have Something similar

 
At Tue Dec 20, 04:07:00 PM, Anonymous Emilio said...

With all due respect, I don’t think this is fair review because you are not testing it for the type of content that the application was developed for: consumer images. For example: there are no repeated images of the same person or faces at different faded levels in consumer pictures. Reviewing Riya based on how it performs with professional images is like reviewing TurboTax based on how well it completes a corporation tax filing! I agree that it’s alpha, but suspect that “it doesn’t hurt” as bad as your paint it.

 
At Tue Dec 20, 04:40:00 PM, Blogger Balakumar said...

Emilio, if Riya cannot identify the exact same images, how will it successfully identify the same person with different emotions. And it doesn't matter if the image is repeated in the same snap or across 2 or more snaps - an image recognition application HAS to identify the person.

Look at these examples:
1. Riya did not identify the other faces marked on this picture though its the exact same one as the one identified.
2. The person in this snap was identified by me 31 times (i.e., I trained Riya 31 times with pictures of this person) and yet, Riya failed to recognize this person with different expressions.

See more examples here.

 
At Tue Dec 20, 05:58:00 PM, Anonymous emilio said...

But it DOES matter if the person appears multiple times in the snap! It matters if your test plan breaks a basic assumption for the problem space that the system was developed for.

An application is just a solution developed for a SPECIFIC problem. Limiting the scope of a problem allows the problem solver to make assumptions that facilitate a solution. Both the images you link to here break what I believe to be a basic valid assumption in consumer photography: “a person appears only once in each picture”. I can perfectly understand how you may identify her face +30 times, and get results all over the place!

Its been my impression that Riya has been explicit about not portraying themselves as “an image recognition application”, but specifically targeting consumer photography. It would be nice if it did work on marketing material, but this is not the problem they set to solve and therefore seems unfair to evaluate them on such.

 
At Tue Dec 20, 09:25:00 PM, Blogger Balakumar said...

Emilio,
Riya claims that they are trying to "look into every photo to automatically identify people/text" and tag them. That right there is image recognition. So only fair to evaluate them on those claims. More on their website: http://riya.com/corp/about.jsp

As for your argument about why Riya will not identify the same person multiple times in the same image; it doesnt really matter if the the same person is present multiple times in the same image or across multiple pictures. Riya fails to find them in both cases. I have many of those examples as well.

Feel free to email me (email ID at the top of my blog) and we can chat about this as we look at more examples.

 
At Fri Jan 06, 01:11:00 AM, Blogger Random Access said...

Sorry for the late wishes as I wasnt around. Wish you a very Happy New Year!

Random Access
The search has just begun !!!

 
At Tue Jan 10, 10:52:00 PM, Anonymous Munjal Shah said...

Bala,

1) Riya is not designed to work on random photos it is designed to work on photos you took with multiple photos of the same person in the set. Please don't confuse the vision of the company with were we are today. I have blogged about this several times on my blog.

2) It uses time stamp and other clues which are not in photos on the web (typically the Exif in the jpeg has been striped).

3) It also needs hi-res photos - most web ones are very low res

It is alpha maybe you found a bug on the identical photos maybe you didn't we can find out more if you send us the photos - zip them up an send them all and we'll look into it.

 
At Tue Jan 10, 11:25:00 PM, Anonymous Munjal Shah said...

Bala... I wrote my thoughts here... but also please contact me at mshah at ojos-inc.com I want to make sure we don't have a bug by getting your examples and reproducing it ourselves. You did a very methodical test. Impressive!

 
At Tue Jan 10, 11:59:00 PM, Blogger Balakumar said...

Munjal, thanks for your response. I will send my test images to you so you can try and run them locally. I had plenty of images of the same person in my test set. So there was really no random photos in my test set.

Most of the literature I see on Riya (including on riya.com) talks about photo search on the web and says Riya's goal is to find every photo of mine on the web. The only problem is that not all of them would be high resolution. Because of bandwidth/storage challenges, most folks typically upload low resolution pictures.

I didn't realize EXIF information was critical to help Riya find people in images. The alpha invite didn't say anything about that. For the next round of testing (when Riya's next release becomes available), I will use images with EXIF.

Thanks again for stopping by.

 
At Wed Jan 11, 10:39:00 AM, Anonymous Anonymous said...

One other fine point which may inform your further testing:

IMHO, a mirror image is not the same face. Few people have symetrical faces (try yours by putting a mirror up to your nose and view your "faces" of two right sides and then two left sides).

I know a mirror is not two left or two right sides, but the point is that depending upon how the face recognition works, it may be sensitive to this difference.

Also, the claim of recognizing twins was not likely from Riya folks.

Of course one also needs to see both eyes (and nose and mouth) and too severe of an angle is a problem, I believe in most face recognition.

That being said, the test was a great example of thoroughness and certainly stands as a measure of how well web pictures can be recognized.

 
At Mon Jan 23, 09:14:00 PM, Anonymous Anonymous said...

Hello All:

I haven't read all that was written there, but I could gather that this so-hyped-not-yet-released-tool riya is certainly not reliable. I wouldn't want to analyse my images first and then argue if riya is meant to analyse those or not. I completely agree with Balakumar, and I believe I would never find time to evaluate this tool myself. I dont know about others but to me its just another tool which I dont really need, and even if I needed it, it wouldn't serve the purpose because of its inaccuracies.

Last comment: If it makes one mistake, its not worth the time, forget about arguing here.

Thanks for reading this. Have fun !!

 
At Tue Jan 24, 05:29:00 AM, Anonymous Ravi said...

I point this blog to the 'anonymous' guy who has given a comment above.
Man if you were brave enough to challenge the technology, and BUSY enough to tell that testing it is not worth you time, you could have at least givin your IDENTITY. I need to bow to you!! And by the way it took Edison more than 1200 trials/experiments to finally present what we know as 'incandescent light bulb'. Alas! some people are so impatient.

 
At Thu Feb 02, 10:28:00 PM, Blogger Random Access said...

Image recognition and VOIP are taited to be the technologies of the next decade, and the beginning is always arduous. Lets hope they make things better. I guess the next gen Blog spammers will be mulling over it to overcome word verification. Im thinkin of joining their gang too :P

Random Access
The search has just begun !!!

 
At Fri Feb 03, 03:29:00 PM, Blogger Piyush said...

I guess what Munjal psoted was correct. Though I agree that this is a very comprehensive testing of the product, still it overlooked basic assumptions Riya makes. It is meant to recognize images taken from a digital camera. Also you must realize that image recognition by any means (anywhere in this world) is far from perfect. Infact we may even say it's still in it's infancy. The picture of Aishwarya that you tested it with in whihc ash has a ghostly image of Ash is not going to be recognized by image Riya for long time because Riya searches for eyes to be recognize a picture and there is no eye visible (to the code ).
Also in a more technical term what these guys are doing is commendable. These are a bunch of brave geeks who are trying to venture into an area of technology never tried before. Technically even text recognition of a camera image is difficult due to lighting conditions. Even binarization of such images proves to be a monster! And these guys are so near to launching a workign beta with alost 75% recognition.
Okay I am blabbering a lot so in the end I'd like you to rememeber the story of tesla(if I am not mistaken he was the one) who first discovered electricity using magnets and a coil and demonstrated it using a simple experiment as an entertainment show on a ship. One day after the show was over, a woman with a child in her hand came to him and told him that his experiment was useless. She said that she travelled so far for observing his experiment and all he had to show was a lousy needle moving when he did something! He said "Ma'am like your kid my invention is very young...let it grow and I am sure it'll light up the world" ...it did ...didn't it ?

 
At Fri Feb 03, 05:18:00 PM, Blogger Balakumar said...

Piyush,

My test was based on the initial definition of Riya as an application that can find people in digital images. And that helped define the test cases in my post.

Subsequent to my test, Munjal clarified that Riya works only with high-res images (2MB photos and above).

Also to your point, digital camera images are a subset of all digital images. And Riya claims to work on all digital images.

To live up to its goal of identifying a person in every digital image in the world, I believe the application has to significantly improve. And must support beyond the 2MB limit. Remember, most people host low-res photos and also photos from camera phones. With the 2MB threshold, a majority of these images will not be properly processed.

Hopefully the upcoming beta will address some of these issues.

 
At Fri Oct 13, 11:46:00 AM, Anonymous Lindsay Lohan said...

best

 
At Fri Oct 13, 11:47:00 AM, Anonymous Lingeries said...

best

 
At Fri Oct 13, 11:47:00 AM, Anonymous Lingeries said...

best

 
At Fri Oct 13, 11:50:00 AM, Anonymous Cat said...

best

 
At Fri Oct 13, 11:51:00 AM, Anonymous Cruise said...

best

 
At Tue Oct 17, 11:21:00 PM, Anonymous Movies said...

best site
http://www.moviesmania.org

 
At Tue Oct 17, 11:22:00 PM, Anonymous Computers said...

nice site

 
At Wed Nov 01, 11:52:00 PM, Anonymous Lindsay Lohan said...

best

 
At Wed Nov 01, 11:52:00 PM, Anonymous Lingeries said...

best

 
At Wed Nov 01, 11:54:00 PM, Anonymous Cat said...

best

 
At Thu Nov 02, 12:04:00 AM, Anonymous Cruise said...

best

 
At Wed Dec 13, 09:53:00 AM, Anonymous aishwarya rai said...

good site
http://www.aishwaryaraiworld.org/

 
At Wed Dec 13, 09:54:00 AM, Anonymous fitness said...

good site
http://www.fitness-world.org/

 
At Wed Dec 13, 09:56:00 AM, Anonymous language said...

nice site
http://www.info-language.com/

 
At Wed Dec 13, 09:56:00 AM, Anonymous outsourcing world said...

nice site
http://www.outsourcing-world.org/

 
At Thu Dec 14, 06:55:00 PM, Anonymous B2b said...

B2b,
B2b Business,business to business industry professional
B2b Marketplace,nternational multilingual B2B marketplace
B2b Marketing,marketing content site giving B2B
B2b Site,Web site with B2B
B2b E-Commerce,benefited from B2B e-commerce include
B2b Portal,Online searchable business
B2b B2c,Information for C-Level
B2b Trust, B2B Trust offers
B2b Sales,usiness to business b2b sales
B2b Trade, b2g import export trade
B2b Directory, comprehensive directory of b2b
Indymac B2b, career potential,
B2b Contractors, The B2B contractors
B2b supply chain, integrating a B2B supply chain is not magic
B2b Trade Leads,trade leads for import export
Yahoo B2b,Yahoo! reviewed these sites and found them related to Business to Business (B2B)
B2b Commerce,Build powerful e-commerce
B2b Magazine,A magazine for business-to-business
B2b Websites,Designing B2B Websites
Taiwan B2b, Global Taiwan China Asian B2B Commerce

 
At Thu Dec 14, 06:56:00 PM, Anonymous Flights said...

Flights,
British Airways Flights,
Domestic Flights,
Economic Flights,
Flight Tracker,
Flights Plans,
Flights Schools,
Slights Security,
Flights Offers,
International Flights,
Military Flights,
Private Pilot Certificate,

 
At Thu Dec 14, 06:57:00 PM, Anonymous night vision said...

Night Vision,
Night Vision Binoculars,With the proper night-vision equipment
Night Vision Scopes,Offers night vision binoculars
Night Vision Cameras,Night Vision Googles Night vision
Itt Night Vision,Night Vision Yahoo (NVG)
Night Vision Glasses,High spec night vision technology
Night Vision Camcorder,Modules for Camcorders
Night Vision Devices,High quality optical devices
Night Vision Security Cameras,security camera equipment
Ant Night Vision,A lot of us would like
Bushnell Night Vision,monoculars
Night Owl Night Vision,Complete Darkness with Night Owl
Yukon Night Vision,manufacturers of Yukon
Night Vision Optics,variety of day-time optics
Night Vision Rifle Scopes,Supplier of night vision device
Night Vision Equipment, the most complete selection
Us Night Vision,US Army and Electronic Sensors Directorate
Cheap Night Vision,Technology products
Night Vision Gear,Hard to find spy cameras
Russian Night Vision,Russian made night vision
Litton Night Visionfeaturing unrivaled Litton Night Vision

 
At Thu Dec 14, 06:58:00 PM, Anonymous Flooring said...

Flooring,
Armstrong flooring, Welcome to the colorful world of Flooring
Bamboo flooring, Bamboo is botanically classified as grass
Brick flooring,The brick flooring is one of the best unmatched alternative for flooring devices.
Concrete Flooring,it is cost effective, durable & high strength
Cork Flooring, warm & quite with high durability
Wood floor, Wood floors are excellent choice
Kitchen flooring,
Laminate Floors, Laminate flooring is the best alternate for real hardwood flooring Laminate flooring is the best.more info avaleble on google ,yahoo & msn
Marble flooring,One of which is hard stone product
Rubber flooring, his one is a revolutionary product in flooring phenomena
State flooring,Welcome to State Flooring
Flooring Industry,

 
At Thu Dec 14, 07:00:00 PM, Anonymous Cat said...

Cat,
Cat families,nformation about cats and kittens
Cat health, best way to keep your cat healthy
Cats behavior, All about Cats behavior
Cat picture,yahoo Shopping is the best Crystal Orange Cat Picture
Cat Sound, Control of Sound Localization in the Cat
Cat Name, Name for the Domestic Cat. Category provided on google
Cat Home, About Cat. Home
Cat Power, Cat Power find on msn Encarta
Funny Cat, Do you have a funny Cat Picture
Cat Eyes, view the world through the eyes of a cat
Cat Tattoos, Cat Tattoos Shop
Cat Women,

 
At Tue Dec 26, 09:27:00 AM, Anonymous Health Insurance said...

http://www.healthinsurance.net.in/

 
At Tue Dec 26, 09:28:00 AM, Anonymous online datinge said...

http://www.onlinedating.net.in/

 
At Tue Dec 26, 09:29:00 AM, Anonymous Movies Point said...

http://www.moviespoint.org/

 
At Tue Dec 26, 09:30:00 AM, Anonymous Movies Point said...

http://www.moviespoint.org/

 
At Thu Dec 28, 11:09:00 AM, Anonymous furnitures said...

best site
http://www.furnitures.org.in

 
At Thu Dec 28, 11:09:00 AM, Anonymous Poker said...

http://www.pokkers.org

the most popular of a class of games called vying games, in which players with
fully or partially concealed cards make wagers into a central pot, which is awarded
to the player or players with the best combination of cards or to the player who makes
an uncalled bet. Poker can also refer to video poker, a single-player game seen in
casinos much like a slot machine, or to other games that use poker hand rankings.

 
At Thu Dec 28, 11:10:00 AM, Anonymous pregnancy said...

http://www.pregnancy.net.in

The period during which a developing fetus is carried within the uterus. In humans, pregnancy
averages 266 days (38 weeks) from conception to childbirth. Traditionally, pregnancy duration
is counted from the woman's last menstrual period, which adds roughly 2 weeks to gestational
age. This is how physicians arrive at a pregnancy length of 40 weeks (280 days).

 
At Thu Dec 28, 11:11:00 AM, Anonymous flowers said...

http://www.flowers-shop.org

In modern times, people have sought ways to cultivate, buy, wear, or just be around
flowers and blooming plants, partly because of their agreeable smell. Around the world,
people use flowers for a wide range of events and functions that, cumulatively, encompass
one's lifetime

 
At Thu Dec 28, 11:12:00 AM, Anonymous Women said...

http://www.women1.org/

a woman, or the feminine in men and women, seeks to share deep awareness of the world
in a sacralized communion. the presence of soft candle light, wild flowers, and the
rituals of dressing for the occasion are simply metaphors acknowledged and “lived out”
in honor of the moment. in honor of life. in honor of shared awareness of the infinite
in a moment.

 
At Fri Jan 12, 09:04:00 PM, Anonymous Health Insurance said...

best site
http://www.healthinsurance.net.in

 
At Sun Jan 14, 10:55:00 PM, Anonymous moreaboutdogs said...

nice site
http://www.moreaboutdogs.org/

 
At Fri Jan 19, 06:14:00 PM, Anonymous bestofbeers said...

well site
bestofbeers,
Beer Girl,
Beer Costume Girl,
Beer Garden Girl,
Beer Girl Miller,
Beer German Girl,
Beer Costume Garden Girl,
Beer Cart Girl,
Party Beer Girl,
Beer Game Girl,
German Beer Girl Costume,


Beer Making,
Beer Making kit,
Beer Making Supply,
Home Beer Making,
Beer And Wine Making,
Making Root Beer,
Beer Making Recipe,
Beer Making Process,
Ginger Beer Making,
Beer Homemade Making,
Beer And Wine Making Supply,


Beer Sign,
Neon Beer Sign,
Hamms Beer Sign,
Beer Tin Sign,
Antique Beer Sign,
Cheap Neon Beer Sign,
Old Beer Sign,
Vintage Beer Sign,
Metal Beer Sign,
Lighted Beer Sign,
Miller Beer Sign,

 
At Sun Jan 21, 08:34:00 PM, Anonymous businesscardsshop said...

best site
http://www.businesscardsshop.com/

 
At Sun Jan 21, 08:35:00 PM, Anonymous depression said...

best site
http://www.depression-care.org/

 
At Sun Jan 21, 08:36:00 PM, Anonymous wifiplanets said...

nice site
http://www.wifiplanets.org/

 
At Sun Jan 21, 08:37:00 PM, Anonymous furnitures said...

nice site
http://www.furnitures.org.in/

 
At Sun Jan 21, 08:40:00 PM, Anonymous clickpoint said...

good site
http://www.clickpoint.org/

 
At Tue Feb 27, 07:20:00 AM, Anonymous mycitymarket said...

ActressThe story begins with an elderly actress who recounts her life and careerSwimsuitSwimsuits offers online shoppers a fashionable selection of cute bikini swimsuits and accessoriesAishwaryaThe official Aishwarya Rai website offers photographs wallpapers and screensavers of the former Miss WorldLohanThe sexiest pictures of beautiful Lindsay LohanMarketComplete financial market coverage with breaking news analysis stock quotes before & after hours market data research and earningsNight VisionSupplier of night vision device including goggles binoculars rifle scopes and other daytime and night vision opticsWifiLearn how WiFi networks work build your own wireless network and find the best prices on wireless routersMoviesThe best online source for movies movie times movie trailers entertainment news celebrity photos box office celebrity interviews and movie reviewsFlooringFlooring is the general term for a permanent covering of a floorlingeriesThe best lingeries underwear nightwear bras lingeries briefs panties pajamas sleepwear loungewear shorts beach wearevideosView thousands of music videos and create your own video channels based on your favorite artistshot moviesFree milf clips screen mates animali free xxxrated porn granny fuck movies flowers of hentai black teens fucking with the hentai lesbian movies fuc

 
At Wed Aug 04, 01:52:00 PM, Blogger babon said...

http://www.bestpenisproducts.com is an all-natural Penis Enlargement, safe, and guaranteed alternative to painful and dangerous Penis enlargement methods such as surgery, straps, or rings.
penis enlargement pills or male enhancement pills will immediately boost your performance, improve your orgasms, and increase the size of your penis within just a few weeks!
VISIT OUR BEST PENIS ENLARGEMENT PILLS PRODUCTS:

VIGRX PLUS PILLS
VIMAX PILLS
MALE EXTRA PILLS
MALE ENHANCEMENT PILLS

 

Post a Comment

Links to this post:

Create a Link

<< Home


   Copyright 2005
Subscribe to Passing Clouds on your cell phone