Pasky’s Log

Guest post by Petr Marek (source)

Everybody driving a car needs the navigation to get to the destination fast and avoid traffic jam. One of the biggest problems is how to enter fast the destination and how to find where are the congestions, what is the traffic situation. YodaQA Traffic is a project attempting to answer the traffic related questions quickly and efficiently. Drivers may ask questions in natural language like: “What is the traffic situation in the Evropská street?” or “What is the fastest route from Opletalova street to Kafkova street?” You can try out the prototype (demo available only for limited time) – try to ask for example “traffic situation in the Wilsonova street” .

YodaQA Traffic still has some limitations. Currently we only have a browser version not suitable for smart phones. It is answering traffic questions for Prague’s streets only.

But as usual, this whole technology demo is open source – you can find it in the branch f/traffic-flow of our Hub project.

How does it work and where we get the data from?

All YodaQA are first analyzed to recognize and select traffic questions. We do it in two steps. The first step is to recognize the question topic. We use six topics like traffic situation, traffic incident or fastest route. The topic is determined by comparing semantic similarity of the user’s question with a set of reference questions. We estimate the similarity with our Dataset-STS Scoring API. Each reference question is labeled by a “topic”. The Sentence Pair Similarity algorithm selects the reference question “topic” with the highest similarity to the question.

Next we need to recognize the location, i.e. to recognize the street name. This is handled by another tool called the Label-lookup which we normally use for entity linking in YodaQA. It compares questions words with a list of all street names in the Prague. We exported the list of streets names in Prague from OpenStreetMap. We do not do exact match, we try to select the closest street name from the list.

The last step is to decide whether the question is really the traffic question, because the Dataset-STS API and Label-lookup can find topic and street name even in a pure movie question like “When was the Nightmare on Elm Street released?”. The Dataset-STS and Label-lookup return not only topic or street name but also the score, fortunately. We created dataset of over 70 traffic questions and over 300 movies questions and founded the minimal score thresholds, with which the recognition makes the lowest classification error on this dataset.

Once we know the type of question and the location we start a small script accessing the traffic situation data from HERE Maps. The only complication is that the the API doesn’t return traffic situation for particular street, but bounding box only. To overcome this problem we have to find a bounding box for a desired location, using an algorithm we developed for this purpose. Then we call the traffic flow API to acquire the information for all streets in the bounding box. Finally, we filter out the traffic situation for the desired street.

It was great fun to work on this application, it is not perfect but it shows how to create intelligent assistants helping people solving various everyday situations. We are also excited to see, how the users will use the new functionality of YodaQA and how it will help them.

With kind support of the Medialab foundation and Jan Šedivý, we are looking hard for students in Prague to work with us on a summer internship! We actually have two options for you:

• eClub Summer Camp (main option) – we have some ambitious projects and ideas for you to try out if you are excited by machine learning, big data and artificial intelligence. Exploratory, exciting, state-of-art research without required previous in-depth knowledge! (Just good basic math and programming.)
• Summer Job (auxiliary option, full-time coder) – we need help polishing the edges of some of our projects, seeking students that are skilled programmers.

We are mainly affiliated with FEL CVUT, but we also have students from MFF UK and we’ll welcome students from other Czech universities too. As long as you are a competent programmer, want to do something more than yet another Android game, and willing to come in person three times a week – let’s do something groundbreaking together!

Recently, the YodaQA team is collaborating with Falk Pollok from RWTH Aachen who is interested in using Question Answering in education to help people digest what they have learned better and to generally assist with studying. To this end, he has created PalmQA – a QA application that multiplexes between many question answering backends, ensembling them together to a more accurate system.

Falk has built backends for IBM Watson’s DeepQA among others (Google, Evi, Kngine and MIT’s Start), but in the end, the combination of YodaQA and Wolfram Alpha is “a match made in heaven,” as Falk said in an email a short while ago.

As a finishing touch to his work (being submitted as a diploma thesis), Falk made a Grand Challenge – letting an independent third party make a list of 30 factoid questions of varying difficulty, and pitching the PalmQA against a wide variety of humans. Perhaps not quite as dramatic or grandiose as IBM Watson’s Jeopardy participation, but still a nice showcase of where we are now.

Well, PalmQA did great! 26 people competed, and typically could get about 15 out of the 30 right. The best human answered 24 questions correctly. But no matter – PalmQA managed to answer 25 out of 30 questions right!

So, in this challenge, Falk’s ensemble-enhanced YodaQA beats the best human!

As mentioned above, PalmQA offers integration of YodaQA with Wolfram Alpha, Google QA, MIT’s Start, Amazon’s Evi and Kngine. We hope to merge this ensembling system into the YodaQA project in the future!

I also entered just plain YodaQA into the Grand Challenge, in the configuration that’s running at live.ailao.eu right now. It got 18 questions right, still better than an average human! If we also included purely “computational” questions (algebra, unit conversions) that YodaQA just isn’t designed to answer (it’s still essentially a search engine), that’d make 24 questions out of 30. Pretty good!

See the Grand Challenge Github issue for more info. We should get the complete details of the challenge, comparisons to other public QA engines (like Google) etc. in Falk’s upcoming thesis.

This is how the plain YodaQA fared:

Guest post by Petr Marek (source)

YodaQA learned how to use twitter during easter holidays. You can ask it by sending tweet with question to @askYodaQA . YodaQA will answer you shortly. How is it possible? I created app in the Google’s App Script, which handles receiving question from twitter and answering them.

Why did I create it? YodaQA can reach more users in the new interesting form thanks to it. I believe they will help us to find even more ways how YodaQA can help them. It is pretty symbiosis. YodaQA will help twitter users, and they will help it back. Let’s look how it is made.

The two important tools

The most important ingredient was Google’s App Script. It is basically JavaScript with the connection to Google services. You can make your own App Script apps in Google Drive. The best thing is that you can make triggers run the app every minute for example. And it’s for free.

The second thing you need is to create twitter app on account, which your bot will use to communicate with its followers. It will grant you access tokens, which you need to connect to twitter API. I used Twitter Lib for Google Apps Script to simplify the communication with API. It allowed me to tweet and get tweets with questions easily. You just need to call the right function with some arguments.

General idea behind

That was the tools that I used. But how did I make it work? I will describe the general idea now. I set App Script project to run my code every minute. The code does basically two things.

The first step is to obtain answers from twitter and to ask YodaQA. Bot searches all tweets with @askYodaQA. It saves the users that tweeted them and the time when it found the tweets. Then it sends the text of the tweet to YodaQA. YodaQA replies with dialog id and question id, which it saves to the list of questions.

The second step is to go through list of questions and to ask YodaQA for answers to these questions. Bot sends questions to users as soon as the answers are finished. You can even rely on features of the Hub, such as dialogs and coreference resolution. Two questions are connected to dialog when they are asked within five minutes interval.

I said that I save some information. Where? I used spreadsheet as memory. I use one sheet as “user memory”, the second as “asked question list” and the last as memory for the id of the last served tweet. I even log some information into a Google Docs text document. It may sound simple (and it is simple), but it works.

You can try it right now. Just tweet question with @askYodaQA and answer will arrive within few minutes. You can even use hashtags or mention other users. They will also receive the answer.

You can see the whole code on GitHub. You can use it and modify it for your own twitter bots too, maybe on your own data?

One of our great student interns at eClub/Ailao Petr Marek who also made the current YodaQA web interface is now working on adding a new element to our ecosystem – the Hub. This is an interface between the web app and the YodaQA system which takes care of various tasks that don’t fit a “pure question answering” system well. For example, it tracks dialog context or allows domain-specific question handling (if you want to add support for retrieving current traffic information, TV schedules or custom question transformations).

We see voice (and chat) as the perfect fit for question answering systems like YodaQA, and together with this, dialog comes naturally. This is why Petr M. has recently transformed our live QA interface to the dialog format (and it now goes through the Hub). The dialog tracking is internally still relatively simplistic from a scientific point of view, but it’s more than enough to already create a great impression. And right now, in a simple, tongue-in-cheek informal test, YodaQA does great compared to “competition”!

See the complete presentation!

I’m still working on YodaQA and there is quite some interest in it in my mailbox. One thing leads to another and our startup Ailao already has a few first customers, we work together on various related semantic NLP / search projects.

In YodaQA, we have a much neater web interface as well as a mobile app as the natural way to interact with a QA system is using your voice. Plus, on a limited domain (movies), we are getting pretty close to crossing the 80% mark for accuracy on simpler questions, entering the “magic zone” where people might start really trusting the system. A few essential blocks for that are still in the pipeline, though.

I’ll try to post a bit more about YodaQA and other work we are doing in the coming weeks / months (as well as some of my hobby projects, of course).

For a course of Jan Šedivý, I prepared a presentation on building apps around the semantic web and linked data. See it here for an intro to the tech, it also includes two silly web mashups that might be inspiring.

I was working on Question Answering last year. Guess what, I’m still on it!

I threw away my first prototype BlanQA and started building a second system, YodaQA. It currently has reasonable performance of answering about a third of trivia questions properly and listing the correct answer in top five candidates for half of the questions – without doing any googling or binging.

A few weeks ago, I published the first paper on YodaQA. With a few fellow scientists, we also re-started the qa-oss Google Group on open source question answering systems.

Today, I finally made a proper homepage for YodaQA and launched a live demo of the system. It’s pretty primitive, but hopefully will serve as a proof of concept.