Conversational Interfaces

How I built a conversational interface for the Auckland Art Gallery

I thought I’d tell the story of how I led a sprint team at Pixel Fusion – where I work – and share our learnings developing a chatbot for the Auckland Art Gallery.

A few months ago, inspired by the success of Send Me SFMOMA — a natural language processing system that allows the public to search and discover artworks listed in the San Francisco Museum of Modern Art’s collection by sending a SMS to a particular number — we started asking ourselves what something similar might look like for the Auckland Art Gallery.

At Pixel Fusion, we value validation, prototyping and lean agile. On the back of that, we run hack nights every Thursday so it felt right to feed some of our engineers with pizza and beer to get a quick prototype up and running. The challenge was set:

  • Receive SMS messages
  • Leverage Natural Language Understanding to make sense out of the message
  • Surface artworks from Auckland Art Gallery’s collection using the same API that fuels the Gallery’s website
  • Reply via SMS with what — hopefully — is an appropriate response.

Off we went, one of the engineers started making the necessary changes to the API, the other one started exploring SMS systems and I started looking at the natural language understanding side of things.

In under 3 hours we had a working prototype and the first stage of validation was completed: We can actually build this thing.

We presented it back to the Gallery team who were just as jaw opened as we were. We were onto something really cool.

We got it out there, users started testing and we came across the first bump on the road; the actual queryable data was very limited.

You see, the Auckland Art Gallery has a large collection built over a 130 years of history — the importance of their catalogue to the history of Auckland and to New Zealand as a country is unquestionable — but when it comes to cataloging art 10, 20, 50 or 100 years ago, no art curator was thinking “One day we will have a chatbot that will need to make sense of this data to allow the community to see our entire collection without leaving the comfort of their homes”.

At this point we are deep in the woods, trying to find a way out. The obvious thing was to ask the Gallery’s curators to make their catalogue more “machine readable”, but the Gallery didn’t immediately have the human resources to go through its entire collection of more than 16,000 artworks to increase the queryable data.

Luckily, they didn’t have to.

By leveraging modern computer vision techniques —a machine learning branch that teaches a computer to “look” at an image, and describe what’s in it — we were able to produce queryable tags for the entire collection, but we were not done just yet…

Surfacing artworks was great, but we figured we could do much more. At this point we had already identified that Gallery visitors preferred Facebook Messenger to SMS, and we had collated the most frequently asked questions via Gallery’s Facebook page and website contact form and found that it would be useful if a bot was able to answer to those.

Furthermore, we identified that it would be valuable to surface exhibitions and events through a chatbot too, but, again, none of that data was available in a format that was queryable by a bot.

We could not ask the website content editor to go back over historical data to make changes to it. But we could leverage Natural Language Processing to discover insights and find text relationships to make it more queryable.

At this point, with the decision to use Facebook Messenger in place of SMS and to surface artworks from the Gallery’s collection, as well as details of exhibitions and events and the answers to frequently asked questions, the project had grown.

Users, though, would have no idea of the abilities and limitations of the bot and would be free to ask it just about anything from “What time does the Gallery open?” to “Will I get married one day?”. We wanted to keep users entertained whilst focusing on Gallery specific questions. So our next challenge was how to keep users engaged while making sure they’re aware of the types of questions to which they would receive meaningful responses.

We solved this by presenting the user with options every time there was a problem with understanding what they were saying or when their query wasn’t valid.

My key takeaways from this experience are:

  • Before building a chatbot, make sure your data can be surfaced
  • If it can’t, building a chatbot will certainly improve the quality of your data

And that’s how building a conversational interface allowed us to bring hundreds of years of historical data of the Auckland Art Gallery collection to the 21st century, and to share New Zealand art with the world.