Riya - Alpha Release: So alpha it hurts!

After 3 weeks in my draft folder, I finally got around to this today. I was one of the lucky 1000 selected to try the alpha release of Riya. There had been so much buzz about this on Slashdot, Digg and TechCrunch, that I was talking about this application's capabilities to friends and colleagues long before I got to see the application. And I learnt a big lesson. No matter what the experts say, hold off talking about an app, till you have actually test-driven it.

My experience with Riya's alpha release has been very disappointing. Well, to start with, it is probably my fault because I had high expectations, especially after I read this on Wired "Currently in alpha testing, the software has proven sensitive enough to tell the difference between twins and recognize members of the same family.". Forget twins, Riya failed to recognize the exact same snap of the same person. In my testing, Riya failed at some very basic image recognition test cases.

Test approach:
I did not want to upload any of my personal pictures. So I chose to use pictures that are already widely available on the web - celebrities, Bollywood celebrities to be precise! The folks at coolbuddy were kind enough to give me permission to use pictures from their website. 569 photos of Aishwarya Rai, Amisha Patel, Bipasha Basu, Preity Zinta, Sonali Bendre, Tabu, Urmila, etc made their way to my Riya account. And I got started.

Identification:
Riya automatically recognizes areas of the photograph that appear like a face and marks it for identification. This is however not consistent and many faces were not auto-marked. I had over 50 Aishwarya's pictures and so went about identifying her snaps. The more snaps recognized by the user, the better Riya is trained to recognize the rest of the snaps. I started with recognizing 10 of Ash's snaps and some of Amisha, Amrita, Bipasha and Preity. I then let Riya recognize the rest. The results were rather poor with some of Aishwarya's snaps being recognized as Amrita and Bipasha. So I went through another round of recognizing some more Aishwarya's snaps to better train Riya; but there was little improvement in the auto-identification by Riya. Next, I recognized 31 of Ash's snaps (one more than what Riya considers great for auto-recognition), but this did not change the end results.

Test Case Summary for identification:
1. Identify some images and then let Riya do its magic. Test Result: Poor
2. Use mirror images of the same person. Test Result: Failed
3. Tiled images of the same person. Test Result: Failed
4. Use different size images of the the same photograph. Test Result: Failed
5. Use the same photographs, but with different sharpness. Test Result: Failed

I expect an image recognition application to positively identify the same people with different expressions and also at different ages. But Riya fails at this. In fact, a scaled down version of the exact same snap of Aishwarya was not recognized by Riya even though I identified Aishwarya in the bigger image. Even text recognition works only when the text is crisp and normal.

So IMHO, this alpha release of Riya is ways away from where the Riya team would like it to be.

Here are some actual examples where Riya failed:

• In this picture, Amisha Patel's picture is tiled multiple times. (i) Riya wrongly identified Amisha as Esha Doel. To be fair, I trained Riya with only 8 Amisha snaps. (ii) Even if this was correct, it should have identified all instances in the image as Esha, but it did not.


• In this picture, we have mirror images of Twinkle Khanna. Riya identified one of the images as Preity Zinta and the other as Amrita Arora.


• Same images of Aishwarya - one is a mirror image and the other is brighter. I trained Riya with these 2. But it wrongly identified Asihwarya in this picture as Amrita.


• Mirror images of Sonali Bendre. I trained Riya by recognizing one and expected Riya to identify the other. But it didn't.


• Faces are not always recognized. Here are some examples (1 2 3 4). Riya did not think any of these were faces!

Market Perception:
A lot of hype has been generated around Riya by veterans such as Michael Arrington at Tech Crunch and Robert Scoble at Scobleizer. It would be interesting to see some of their test images that went through Riya. In the current form, the only gain for anyone acquiring Riya is getting them for a lot less than when the application actually works! It appears like Google is not interested for now.

The business case for Riya is very strong. There are a lot of uses for a technology such as Riya's - when it works. Here is hoping the next release does better.

Related links:

• More test photos on flickr
.

• Technorati Tags: riya, photosearch, soalphaithurts