The weekend of the 14th December I attended the London Startup Weekend Fashion Hackathon. This was a much larger event than the previous hackathon I attended and was more geared towards creating a viable business as well as the tech to support it.
The format was fun, on the first day a number of people would pitch ideas, we would all vote for them, then form teams to begin on the Saturday morning. I attended in order to build something new and fun, so just stood back and listened for some interesting pitches.
There were two super interesting pitches: A smart bag which worked out what was in your bag and alerted you if things were missing; and an automatic garment detector which would allow you to take a picture, and then buy the clothes from the picture.
I ended up picking the image recognition project as it sounded the most fun and I didn’t think we would be able to source an RFID reader (or similar) over the weekend. (it turned out that this team didn’t pitch, so maybe they pivoted or disbanded?)
The mini-startup we made was called LookSnap, and it was fun and quite gratifying to see that my business instincts were reinforced by the actions of the rest of the group. Over the day and a half that it was worked on, I think the business model ended up fairly solid.
My main job for the weekend was getting the image recognition working. In terms of the technology and with the very short time-scale in mind I decided to limit the acceptable inputs as much as possible. As such, I designed an algorithm that would be able to extract the clothing (top, bottoms, shoes) from a picture of someone who was facing forward and had their arms down.
The algorithm works as follows:
- Use OpenCV to detect a face
- With the face position, composite a “clothing mask” (see images) onto the original photo using graphicsmagick
- This than gives you a fairly decent cut out of just that persons clothes. Apply different masks for top, bottom, and shoes.
Once I had these images, the idea was to use reverse image search on the lyst.com domain to always return something relevant.
However, there was a slight hitch with this plan. Google reverse image search, which worked well manually, had no API in which to pass an image…
So the stopgap method was to extract the average colour from the garment by averaging all the pixel colours that were in the appropriate garment masks, and then mapping these colour to their more broader hue. This turned out to be incredibly hard and would have been impossible if not for reverse engineering a very good hue detector at http://www.color-blindness.com/color-name-hue/
Once this was working I packaged it all up in a FLASK api where an image file was posted to the endpoint, the above magic happened, and a json file was returned giving the X,Y of the garment in the photo, and information on the product name, description, image, and a buy link.
Unfortunately there was not enough time to integrate the service into our POC app, which would have made persuading the judges that we have actually done basic image detection much easier!
Overall, the team did an excellent job, and even though we didn’t win I feel the weekend was very well spent.