[Most-ai-contest] The performance of the current integrated system on the dataset ver. 1.5
謝尊安
pups003c於gmail.com
Fri 12月 20 13:56:03 CST 2019
Dear all, attached please find the new output formatting module.
dev set:
*QID QTEXT
ATEXT ATEXT_
(formatted) --*
D108Q04 「協助單身及鼓勵婚育租金補貼試辦方案」補助的單身青年最高年齡是幾歲?
40歲 40歲
no QID D108904, right answer in D108Q04
D241Q07 世界上最大的教堂高度是幾公尺?
138米
138公尺
D299Q03 文中提及，賴教授認為二十一世紀的時候，哪兩種動能取代了大部分的石油燃料，減少了空氣的污染？ 「太陽能」和「風能」
太陽能及風能 和、與、及、以及 are valid according to the regulation
test set:
D260Q06 于伯伯等了四十幾年，終於可以回大陸哪裡探親了？
南京
年 fixed (鄰居)
D283Q01 文中提及，作者共被退了幾次稿？
二
二次
D296Q01 此篇內容描述的是去台灣哪裡畢業旅行二日遊？
東臺灣
東臺灣 ATEXT_ is 東臺灣 but 臺灣 in the original
test, check this with new formatter
train set:
D015Q03 高雄流行文化中心在哪裡?
新堀江 新堀江
D243Q02 聖伯多祿大殿至少可容納幾萬人?
60,000人
6萬人
D314Q06 戲水事故發生率最高的時段是否在晚上?
否
否
D314Q07 戲水事故發生率最高的時段是否在白天?
是
是
D314Q08 在國家公園發生登山的事故是否多於中級山域?
否
否
D314Q09 在海邊發生戲水意外的機率是否多於河邊?
否
否
D281Q03 黃果樹大瀑布在中國的哪一省？
貴州
貴州省 貴州省 is the
official name in Wikidata, so I suppose that 貴州 is not correct.
D292Q02 白帝城在中國哪一省？
四川 四川省
D304Q06 從文中公告得知，『校樹選拔』之老榕樹的得票樹有幾票？
一五六四 三票
D304Q07 從文中公告得知，『校樹選拔』的總票樹有幾票？
三八一四 三八一四
票
kysu <kysu於iis.sinica.edu.tw> 於 2019年12月20日 週五 下午12:40寫道：
> For 郭家銍,
>
> In the training-set:
>
> >ANSWER MODULE STATS (WHEN CORRECTLY ACTIVATED):
> >Date-Duration 7 / 26 = 0.2692
>
> The performance of your module seems too low even in the training-set.
> Please check the problem. Thanks.
>
> KY
>
> -----Original Message-----
> From: 張光瑜 [mailto:simonc於iis.sinica.edu.tw]
> Sent: Friday, December 20, 2019 11:47 AM
> To: kysu <kysu於iis.sinica.edu.tw>
> Cc: 'Most-ai Contest' <Most-ai-contest於iis.sinica.edu.tw>
> Subject: Re: [Most-ai-contest] The performance of the current integrated
> system on the dataset ver. 1.5
>
> I just forgot it.
> The attached files contain stats with the number of activations, correct
> answers.
>
>
> 張光瑜 於 2019-12-20 11:30 寫到:
> > For the YesNo module:
> > We are still looking into this situation.
> >
> > For the date-duration module:
> > Since it also outputs single-span answers, it has a high probability
> > to get the correct answer when the correct answer mode is
> > single-span-extraction.
> >
> > For the output formatter:
> > It actually has 3 errors for the dev set and 3 errors for the test set.
> > DEV SET:
> > D108904: Output formatter was not activated.
> > D241Q07: Question is "世界上最大的教堂高度是幾公尺?", the official answer is "138米",
> > our formatted answer is "138公尺".
> > D299Q03: The official answer is "太陽能和風能", our formatted answer is
> > "太陽能及風能".
> > TEST SET:
> > D260Q06: "南京" was formatted into "年", it seems that it is caused by
> > ATYPE outputting "Date-Duration".
> > D283Q01: The question is "文中提及，作者共被退了幾次稿？", the official answer is "二"
> > or "2", our formatted answer is "二次".
> > D296Q01: The question is "此篇內容描述的是去台灣哪裡畢業旅行二日遊？", our answer "东台湾" was
> > formatted into "臺灣".
> >
> >
> >
> > kysu 於 2019-12-19 22:10 寫到:
> >> Simon,
> >>
> >> Thanks. Could you please also associate each module with its
> >> #Activation (i.e., the number of times that it is correctly/wrongly
> >> activated) and its %Activation (i.e., its relative percentage among
> >> the total number of activation-times)? Thanks.
> >>
> >> The performance of “Date-Duration” module seems strange. Also,
> >> “Output-Formatter” is unable to keep two answers in either Dev or
> >> Test data-sets. Please take a look.
> >>
> >> Last, the performance of “YES/NO” module is surprisingly good. Is
> >> its associated setting correct?
> >>
> >> KY
> >>
> >> FROM: most-ai-contest-bounces於iis.sinica.edu.tw
> >> [mailto:most-ai-contest-bounces於iis.sinica.edu.tw] ON BEHALF OF
> >> 張光瑜
> >> SENT: Thursday, December 19, 2019 9:50 PM
> >> TO: 范正忠 <jjfan於iis.sinica.edu.tw>
> >> CC: Most-ai Contest <Most-ai-contest於iis.sinica.edu.tw>
> >> SUBJECT: Re: [Most-ai-contest] The performance of the current
> >> integrated system on the dataset ver. 1.5
> >>
> >> Dear all,
> >>
> >> The attached files contain the statistics and errors for our results.
> >>
> >> 張光瑜
> >>
> >> -------------------------
> >>
> >> 寄件者: "范正忠" <jjfan於iis.sinica.edu.tw>
> >> 收件者: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> >> 寄件備份: 2019 12 月 19 星期四 上午 9:58:38
> >> 主旨: Re: [Most-ai-contest] The performance of the current
> >> integrated system on the dataset ver. 1.5
> >>
> >> Dear all,
> >>
> >> Enclosed please find train/dev/test dataset (1.5) performance using
> >>
> >> 1. 廖沛俊 bestBERTFGC_full_em Bert Model
> >>
> >> 2. Dr. Chang aggregrator
> >>
> >> 3. Dr. Chang single-span-multi-hops enable
> >>
> >> 4. Smolka kinship_module4
> >>
> >> train: 0.65, dev: 0.55, test: 0.53
> >>
> >> Best,
> >>
> >> jjfan
> >>
> >> -------------------------
> >>
> >> FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> >> TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> >> SENT: Wednesday, December 18, 2019 3:56:18 PM
> >> SUBJECT: Re: [Most-ai-contest] The performance of the current
> >> integrated system on the dataset ver. 1.5
> >>
> >> Dear all,
> >>
> >> Enclosed please find train/dev/test dataset (1.5) performance using
> >>
> >> 1. 廖沛俊 bestBERTFGC_full_em Bert Model
> >>
> >> 2. Dr. Chang aggregrator
> >>
> >> 3. Dr. Change single-span-multi-hops enable
> >>
> >> train: 0.65, dev: 0.55, test: 0.52
> >>
> >> Best,
> >>
> >> jjfan
> >>
> >> -------------------------
> >>
> >> FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> >> TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> >> SENT: Wednesday, December 18, 2019 11:30:36 AM
> >> SUBJECT: Re: [Most-ai-contest] The performance of the current
> >> integrated system on the dataset ver. 1.5
> >>
> >> Dear all,
> >>
> >> Enclosed please find train/dev/test dataset (1.5) performance using
> >>
> >> 1. 廖沛俊 bestBERTFGC_full_em Bert Model
> >>
> >> 2. Dr. Chang aggregrator
> >>
> >> Best,
> >>
> >> jjfan
> >>
> >> -------------------------
> >>
> >> FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> >> TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> >> SENT: Saturday, December 14, 2019 4:47:37 PM
> >> SUBJECT: [Most-ai-contest] The performance of the current integrated
> >> system on the dataset ver. 1.5
> >>
> >> Dear all,
> >>
> >> Enclosed please find the performance of our current integrated system
> >> on train / dev dataset.
> >>
> >> Please refer to the AMODE fields in each question for each module's
> >> performance.
> >>
> >> Hope this can help the error analysis of each module and please share
> >> your finding in the next week's meeting.
> >>
> >> I am still waiting for 'multi-spans-extraction' & 'supporting
> >> evidence' modules to integrate. Please anyone has improved version
> >> just send to me.
> >>
> >> Best,
> >>
> >> jjfan
> >>
> >> -------------------------
> >>
> >> FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> >> TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> >> SENT: Monday, November 25, 2019 9:39:07 AM
> >> SUBJECT: Re: [Most-ai-contest] It's time to start system integration
> >>
> >> Dear all,
> >>
> >> Please don't forget give me a readme.txt or I really don't know what
> >> is inside.
> >>
> >> Suggest readme.txt contains at least
> >>
> >> 1. Brief API description / document, function list at minimum
> >>
> >> 2. System requirements
> >>
> >> Thanks.
> >>
> >> jjfan
> >>
> >> -------------------------
> >>
> >> FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> >> TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> >> SENT: Monday, November 25, 2019 9:18:05 AM
> >> SUBJECT: [Most-ai-contest] It's time to start system integration
> >>
> >> Dear all,
> >>
> >> Good Morning!
> >>
> >> It's time to start integration of the system. Please all module owners
> >> send me your Python Code. Thanks.
> >>
> >> Best,
> >>
> >> jjfan
> >>
> >> -------------------------
> >>
> >> FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> >> TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> >> SENT: Thursday, November 14, 2019 8:58:50 AM
> >> SUBJECT: Re: [Most-ai-contest] 科技大擂台討論會(Today,
> >> 12:30-15:00)
> >>
> >> Dear all,
> >>
> >> Note the following action items should be finished today.
> >>
> >> 3. FGC_Questions_S7.json son 包含系統進行答題的過程中,
> >> 各個 module 預計輸出.
> >> 今日會議上討論要修改的部分, 請各位 module owner
> >> 儘快提出各自修改的部分給我,
> >> 一起整理完後再寄送給大家 (10/14 以前完成)
> >> 4. 提供個自 module 所需的 Python library 及使用的 Python
> >> version. (10/14 以前完成)
> >> 建議 pip list
> >> 7. 為了確保大家沒有誤會各自負責的 module function,
> >> 煩請簡單描述一下 (input, out, function) 為何. (10/14
> >> 以前完成)
> >>
> >> ITEM 4. PLEASE ALSO LIST ADDITIONAL INSTALLATIONS IF PIP IS NOT
> >> ENOUGH, SUCH AS DATABASE, FILES, ...
> >>
> >> Best,
> >>
> >> jjfan
> >>
> >> -------------------------
> >>
> >> FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> >> TO: "Po-Fen Lin" <pofen於iis.sinica.edu.tw>
> >> CC: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> >> SENT: Tuesday, November 12, 2019 5:29:03 PM
> >> SUBJECT: Re: [Most-ai-contest] 科技大擂台討論會(Today,
> >> 12:30-15:00)
> >>
> >> Dear all,
> >>
> >> 謝謝大家的參與討論.
> >>
> >> 附上今天討論的內容.
> >> 1. schedule.ppt 包含每個參與者的 job items
> >> 及未來計畫進行 milestones
> >> 2. FGC_release_A_all.json 包含 answer type, answer mode, ...
> >> 的基礎的 annotation. 提供大家參考
> >> 若發現 answer type 或 answer mode 有錯的地方, 請 email
> >> 通知我, 有我這邊統一修正, 在再寄送給大家更新版
> >> (隨時)
> >> 請廖沛俊儘快完成 answer keyword spans 的 annotation,
> >> 並將完成後的 json 檔案提供給我一起整理. (10/15
> >> 以前完成)
> >> 3. FGC_Questions_S7.json son 包含系統進行答題的過程中,
> >> 各個 module 預計輸出.
> >> 今日會議上討論要修改的部分, 請各位 module owner
> >> 儘快提出各自修改的部分給我,
> >> 一起整理完後再寄送給大家 (10/14 以前完成)
> >> 4. 提供個自 module 所需的 Python library 及使用的 Python
> >> version. (10/14 以前完成)
> >> 建議 pip list
> >> 5. 1007說明會簡報V9, 國網機器遠端連線操作說明:
> >> 請郭家鉦幫忙看一下如何處理"自動化介接"及"程式上傳/執行"
> >> 6. 附件二科技大擂台簡答題之答題規範,
> >> 科技大擂台簡答題之答題規範v2: 提供給謝尊安參考.
> >> 7. 為了確保大家沒有誤會各自負責的 module function,
> >> 煩請簡單描述一下 (input, out, function) 為何. (10/14
> >> 以前完成)
> >>
> >> 有任何問題, 請隨時讓我知道(email, line, skype, tel, ...).
> >> 我會一起來協助如何處理.
> >> 謝謝的家的幫忙!
> >>
> >> Best,
> >> jjfan
> >>
> >> _______________________________________________
> >> Most-ai-contest mailing list
> >> Most-ai-contest於iis.sinica.edu.tw
> >> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >>
> >> _______________________________________________
> >> Most-ai-contest mailing list
> >> Most-ai-contest於iis.sinica.edu.tw
> >> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >>
> >> _______________________________________________
> >> Most-ai-contest mailing list
> >> Most-ai-contest於iis.sinica.edu.tw
> >> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >>
> >> _______________________________________________
> >> Most-ai-contest mailing list
> >> Most-ai-contest於iis.sinica.edu.tw
> >> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >>
> >> _______________________________________________
> >> Most-ai-contest mailing list
> >> Most-ai-contest於iis.sinica.edu.tw
> >> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >>
> >> _______________________________________________
> >> Most-ai-contest mailing list
> >> Most-ai-contest於iis.sinica.edu.tw
> >> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >>
> >> _______________________________________________
> >> Most-ai-contest mailing list
> >> Most-ai-contest於iis.sinica.edu.tw
> >> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >
> > _______________________________________________
> > Most-ai-contest mailing list
> > Most-ai-contest於iis.sinica.edu.tw
> > https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
>
>
> _______________________________________________
> Most-ai-contest mailing list
> Most-ai-contest於iis.sinica.edu.tw
> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
>
--
Tsun-An Hsieh
tel: +886 975 329907 <+886%20975%20329%20907>
email: pups003c於gmail.com
Institute of Networking and Multimedia, CSIE, National Taiwan University,
Taiwan
-------------- 下一部份 --------------
抹去了一個 HTML 附加檔...
URL: <http://www.iis.sinica.edu.tw/pipermail/most-ai-contest/attachments/20191220/c106aa24/attachment-0001.html>
-------------- 下一部份 --------------
A non-text attachment was scrubbed...
Name: output_format.py
Type: text/x-python
Size: 14447 bytes
Desc: 〔無法取得〕
URL: <http://www.iis.sinica.edu.tw/pipermail/most-ai-contest/attachments/20191220/c106aa24/attachment-0001.py>
More information about the Most-ai-contest
mailing list