Posts Tagged ‘students’

YodaQA Grand Challenge!

April 7th, 2016 1 comment

Recently, the YodaQA team is collaborating with Falk Pollok from RWTH Aachen who is interested in using Question Answering in education to help people digest what they have learned better and to generally assist with studying. To this end, he has created PalmQA – a QA application that multiplexes between many question answering backends, ensembling them together to a more accurate system.

Falk has built backends for IBM Watson’s DeepQA among others (Google, Evi, Kngine and MIT’s Start), but in the end, the combination of YodaQA and Wolfram Alpha is “a match made in heaven,” as Falk said in an email a short while ago.

As a finishing touch to his work (being submitted as a diploma thesis), Falk made a Grand Challenge – letting an independent third party make a list of 30 factoid questions of varying difficulty, and pitching the PalmQA against a wide variety of humans. Perhaps not quite as dramatic or grandiose as IBM Watson’s Jeopardy participation, but still a nice showcase of where we are now.

Well, PalmQA did great! 26 people competed, and typically could get about 15 out of the 30 right. The best human answered 24 questions correctly. But no matter – PalmQA managed to answer 25 out of 30 questions right!

So, in this challenge, Falk’s ensemble-enhanced YodaQA beats the best human!

As mentioned above, PalmQA offers integration of YodaQA with Wolfram Alpha, Google QA, MIT’s Start, Amazon’s Evi and Kngine. We hope to merge this ensembling system into the YodaQA project in the future!

I also entered just plain YodaQA into the Grand Challenge, in the configuration that’s running at right now. It got 18 questions right, still better than an average human! If we also included purely “computational” questions (algebra, unit conversions) that YodaQA just isn’t designed to answer (it’s still essentially a search engine), that’d make 24 questions out of 30. Pretty good!

See the Grand Challenge Github issue for more info. We should get the complete details of the challenge, comparisons to other public QA engines (like Google) etc. in Falk’s upcoming thesis.

This is how the plain YodaQA fared:

Question Text correct found
What is the capital of Zimbabwe? ✓ Harare Harare
Who invented the Otto engine? ✓ Nikolaus Otto Nikolaus Otto
When was Pablo Picasso born? ✓ 1881 1881
What is 7*158 + 72 – 72 + 9? ✗ 1115 78
Who wrote the novel The Light Fantastic? ✓ Terry Pratchett Terry Pratchett
In which city was Woody Allen born? ✓ New York New York
Who is the current prime minister of Italy? ✓ Matteo Renzi Matteo Renzi
What is the equatorial radius of Earth’s moon? ✗ 1738 the Moon and Su
When did the Soviet Union dissolve? ✓ 1991 1991
What is the core body temperature of a human? ✗ 37 Bio 42 and cour
Who is the current Dalai Lama? ✓ Tenzin Gyatso Tenzin Gyatso
What is 2^23? ✗ 8388608 the Gregorian c
Who is the creator of Star Trek? ✓ Gene Roddenberr Gene Roddenberr
In which city is the Eiffel Tower? ✓ Paris Paris
12 metric tonnes in kilograms? ✗ 12 *000 SI
Where is the mouth of the river Rhine? ✓ the Netherlands the Netherlands
Where is Buckingham Palace located? ✓ London London
Who directed the movie The Green Mile? ✓ Frank Darabont Frank Darabont
When did Franklin D. Roosevelt die? ✓ 1945 1945
Who was the first man in space? ✓ Yuri Gagarin Yuri Gagarin
Where was the Peace of Westphalia signed? ✗ Osnabrück France
Who was the first woman to be awarded a Nobel Priz ✗ Marie Curie Elinor Ostrom
12.1147 inches to yards? ✗ 0.3365194444 CUX 570 17 577
What is the atomic number of potassium? ✓ 19 19
Where is the Tiananmen Square? ✓ China China
What is the binomial name of horseradish? ✓ Armoracia Rusti Armoracia Rusti
How long did Albert Einstein live? ✗ 76 Germany
Who earned the most Academy Awards? . Walt Disney Jimmy Stewart
How many lines does the London Underground have? ✗ 11 Soho Revue Bar
When is the next planned German Federal Convention ✗ 1850
Categories: ailao, software Tags: , , ,

YodaQA learned to tweet

April 4th, 2016 No comments

Guest post by Petr Marek (source)

YodaQA learned how to use twitter during easter holidays. You can ask it by sending tweet with question to @askYodaQA . YodaQA will answer you shortly. How is it possible? I created app in the Google’s App Script, which handles receiving question from twitter and answering them.

Why did I create it? YodaQA can reach more users in the new interesting form thanks to it. I believe they will help us to find even more ways how YodaQA can help them. It is pretty symbiosis. YodaQA will help twitter users, and they will help it back. Let’s look how it is made.

YodaQA twitter

The two important tools

The most important ingredient was Google’s App Script. It is basically JavaScript with the connection to Google services. You can make your own App Script apps in Google Drive. The best thing is that you can make triggers run the app every minute for example. And it’s for free.

The second thing you need is to create twitter app on account, which your bot will use to communicate with its followers. It will grant you access tokens, which you need to connect to twitter API. I used Twitter Lib for Google Apps Script to simplify the communication with API. It allowed me to tweet and get tweets with questions easily. You just need to call the right function with some arguments.

General idea behind

That was the tools that I used. But how did I make it work? I will describe the general idea now. I set App Script project to run my code every minute. The code does basically two things.

The first step is to obtain answers from twitter and to ask YodaQA. Bot searches all tweets with @askYodaQA. It saves the users that tweeted them and the time when it found the tweets. Then it sends the text of the tweet to YodaQA. YodaQA replies with dialog id and question id, which it saves to the list of questions.

YodaQA twitter

The second step is to go through list of questions and to ask YodaQA for answers to these questions. Bot sends questions to users as soon as the answers are finished. You can even rely on features of the Hub, such as dialogs and coreference resolution. Two questions are connected to dialog when they are asked within five minutes interval.

I said that I save some information. Where? I used spreadsheet as memory. I use one sheet as “user memory”, the second as “asked question list” and the last as memory for the id of the last served tweet. I even log some information into a Google Docs text document. It may sound simple (and it is simple), but it works.

You can try it right now. Just tweet question with @askYodaQA and answer will arrive within few minutes. You can even use hashtags or mention other users. They will also receive the answer.

You can see the whole code on GitHub. You can use it and modify it for your own twitter bots too, maybe on your own data?

Categories: ailao, software Tags: , , , ,

Dialog for YodaQA!

March 31st, 2016 No comments

One of our great student interns at eClub/Ailao Petr Marek who also made the current YodaQA web interface is now working on adding a new element to our ecosystem – the Hub. This is an interface between the web app and the YodaQA system which takes care of various tasks that don’t fit a “pure question answering” system well. For example, it tracks dialog context or allows domain-specific question handling (if you want to add support for retrieving current traffic information, TV schedules or custom question transformations).

We see voice (and chat) as the perfect fit for question answering systems like YodaQA, and together with this, dialog comes naturally. This is why Petr M. has recently transformed our live QA interface to the dialog format (and it now goes through the Hub). The dialog tracking is internally still relatively simplistic from a scientific point of view, but it’s more than enough to already create a great impression. And right now, in a simple, tongue-in-cheek informal test, YodaQA does great compared to “competition”!

See the complete presentation!

Categories: ailao, software Tags: , , ,