[Most-ai-contest] The performance of the current integrated system on the dataset ver. 1.5

Fri 12月 20 22:16:00 CST 2019

The training-set performance of a reasonable model usually falls within 90% to 95%. If the training-set performance of your module is less than 90%, it frequently implies that the prediction/discrimination power of your adopted model is not enough, and you might need to add more adjustable parameters. Currently, our overall training-set performance is only 0.65, which is too low. Please check if the training-set performance of your model is under 90%. If the answer is YES, please check if the learning process converge, and also try to enhance your model via adding more discriminative features.

KY

-----Original Message-----
From: 張光瑜 [mailto:simonc at iis.sinica.edu.tw] 
Sent: Friday, December 20, 2019 11:47 AM
To: kysu <kysu at iis.sinica.edu.tw>
Cc: 'Most-ai Contest' <Most-ai-contest at iis.sinica.edu.tw>
Subject: Re: [Most-ai-contest] The performance of the current integrated system on the dataset ver. 1.5

I just forgot it.
The attached files contain stats with the number of activations, correct answers.

張光瑜 於 2019-12-20 11:30 寫到:
> For the YesNo module:
> We are still looking into this situation.
> 
> For the date-duration module:
> Since it also outputs single-span answers, it has a high probability
> to get the correct answer when the correct answer mode is
> single-span-extraction.
> 
> For the output formatter:
> It actually has 3 errors for the dev set and 3 errors for the test set.
> DEV SET:
> D108904: Output formatter was not activated.
> D241Q07: Question is "世界上最大的教堂高度是幾公尺?", the official answer is "138米",
> our formatted answer is "138公尺".
> D299Q03: The official answer is "太陽能和風能", our formatted answer is 
> "太陽能及風能".
> TEST SET:
> D260Q06: "南京" was formatted into "年", it seems that it is caused by
> ATYPE outputting "Date-Duration".
> D283Q01: The question is "文中提及，作者共被退了幾次稿？", the official answer is "二"
> or "2", our formatted answer is "二次".
> D296Q01: The question is "此篇內容描述的是去台灣哪裡畢業旅行二日遊？", our answer "东台湾" was
> formatted into "臺灣".
> 
> 
> 
> kysu 於 2019-12-19 22:10 寫到:
>> Simon,
>> 
>> Thanks. Could you please also associate each module with its
>> #Activation (i.e., the number of times that it is correctly/wrongly
>> activated) and its %Activation (i.e., its relative percentage among
>> the total number of activation-times)? Thanks.
>> 
>> The performance of “Date-Duration” module seems strange. Also,
>> “Output-Formatter” is unable to keep two answers in either Dev or
>> Test data-sets. Please take a look.
>> 
>> Last, the performance of “YES/NO” module is surprisingly good. Is
>> its associated setting correct?
>> 
>> KY
>> 
>> FROM: most-ai-contest-bounces at iis.sinica.edu.tw
>> [mailto:most-ai-contest-bounces at iis.sinica.edu.tw] ON BEHALF OF
>> 張光瑜
>> SENT: Thursday, December 19, 2019 9:50 PM
>> TO: 范正忠 <jjfan at iis.sinica.edu.tw>
>> CC: Most-ai Contest <Most-ai-contest at iis.sinica.edu.tw>
>> SUBJECT: Re: [Most-ai-contest] The performance of the current
>> integrated system on the dataset ver. 1.5
>> 
>> Dear all,
>> 
>> The attached files contain the statistics and errors for our results.
>> 
>> 張光瑜
>> 
>> -------------------------
>> 
>> 寄件者: "范正忠" <jjfan at iis.sinica.edu.tw>
>> 收件者: "Most-ai Contest" <Most-ai-contest at iis.sinica.edu.tw>
>> 寄件備份: 2019 12 月 19 星期四 上午 9:58:38
>> 主旨: Re: [Most-ai-contest] The performance of the current
>> integrated system on the dataset ver. 1.5
>> 
>> Dear all,
>> 
>> Enclosed please find train/dev/test dataset (1.5) performance using
>> 
>> 1.  廖沛俊 bestBERTFGC_full_em Bert Model
>> 
>> 2. Dr. Chang aggregrator
>> 
>> 3. Dr. Chang single-span-multi-hops enable
>> 
>> 4. Smolka kinship_module4
>> 
>> train: 0.65, dev: 0.55, test: 0.53
>> 
>> Best,
>> 
>> jjfan
>> 
>> -------------------------
>> 
>> FROM: "范正忠" <jjfan at iis.sinica.edu.tw>
>> TO: "Most-ai Contest" <Most-ai-contest at iis.sinica.edu.tw>
>> SENT: Wednesday, December 18, 2019 3:56:18 PM
>> SUBJECT: Re: [Most-ai-contest] The performance of the current
>> integrated system on the dataset ver. 1.5
>> 
>> Dear all,
>> 
>> Enclosed please find train/dev/test dataset (1.5) performance using
>> 
>> 1.  廖沛俊 bestBERTFGC_full_em Bert Model
>> 
>> 2. Dr. Chang aggregrator
>> 
>> 3. Dr. Change single-span-multi-hops enable
>> 
>> train: 0.65, dev: 0.55, test: 0.52
>> 
>> Best,
>> 
>> jjfan
>> 
>> -------------------------
>> 
>> FROM: "范正忠" <jjfan at iis.sinica.edu.tw>
>> TO: "Most-ai Contest" <Most-ai-contest at iis.sinica.edu.tw>
>> SENT: Wednesday, December 18, 2019 11:30:36 AM
>> SUBJECT: Re: [Most-ai-contest] The performance of the current
>> integrated system on the dataset ver. 1.5
>> 
>> Dear all,
>> 
>> Enclosed please find train/dev/test dataset (1.5) performance using
>> 
>> 1.  廖沛俊 bestBERTFGC_full_em Bert Model
>> 
>> 2. Dr. Chang aggregrator
>> 
>> Best,
>> 
>> jjfan
>> 
>> -------------------------
>> 
>> FROM: "范正忠" <jjfan at iis.sinica.edu.tw>
>> TO: "Most-ai Contest" <Most-ai-contest at iis.sinica.edu.tw>
>> SENT: Saturday, December 14, 2019 4:47:37 PM
>> SUBJECT: [Most-ai-contest] The performance of the current integrated
>> system on the dataset ver. 1.5
>> 
>> Dear all,
>> 
>> Enclosed please find the performance of our current integrated system
>> on train / dev dataset.
>> 
>> Please refer to the AMODE fields in each question for each module's
>> performance.
>> 
>> Hope this can help the error analysis of each module and please share
>> your finding in the next week's meeting.
>> 
>> I am still waiting for 'multi-spans-extraction' & 'supporting
>> evidence' modules to integrate. Please anyone has improved version
>> just send to me.
>> 
>> Best,
>> 
>> jjfan
>> 
>> -------------------------
>> 
>> FROM: "范正忠" <jjfan at iis.sinica.edu.tw>
>> TO: "Most-ai Contest" <Most-ai-contest at iis.sinica.edu.tw>
>> SENT: Monday, November 25, 2019 9:39:07 AM
>> SUBJECT: Re: [Most-ai-contest] It's time to start system integration
>> 
>> Dear all,
>> 
>> Please don't forget give me a readme.txt or I really don't know what
>> is inside.
>> 
>> Suggest readme.txt contains at least
>> 
>> 1. Brief API description / document, function list at minimum
>> 
>> 2. System requirements
>> 
>> Thanks.
>> 
>> jjfan
>> 
>> -------------------------
>> 
>> FROM: "范正忠" <jjfan at iis.sinica.edu.tw>
>> TO: "Most-ai Contest" <Most-ai-contest at iis.sinica.edu.tw>
>> SENT: Monday, November 25, 2019 9:18:05 AM
>> SUBJECT: [Most-ai-contest] It's time to start system integration
>> 
>> Dear all,
>> 
>> Good Morning!
>> 
>> It's time to start integration of the system. Please all module owners
>> send me your Python Code. Thanks.
>> 
>> Best,
>> 
>> jjfan
>> 
>> -------------------------
>> 
>> FROM: "范正忠" <jjfan at iis.sinica.edu.tw>
>> TO: "Most-ai Contest" <Most-ai-contest at iis.sinica.edu.tw>
>> SENT: Thursday, November 14, 2019 8:58:50 AM
>> SUBJECT: Re: [Most-ai-contest] 科技大擂台討論會(Today,
>> 12:30-15:00)
>> 
>> Dear all,
>> 
>> Note the following action items should be finished today.
>> 
>> 3. FGC_Questions_S7.json son 包含系統進行答題的過程中,
>> 各個 module 預計輸出.
>>    今日會議上討論要修改的部分, 請各位 module owner
>> 儘快提出各自修改的部分給我,
>> 一起整理完後再寄送給大家 (10/14 以前完成)
>> 4. 提供個自 module 所需的 Python library 及使用的 Python
>> version. (10/14 以前完成)
>>    建議 pip list
>> 7. 為了確保大家沒有誤會各自負責的 module function,
>> 煩請簡單描述一下 (input, out, function) 為何. (10/14
>> 以前完成)
>> 
>> ITEM 4. PLEASE ALSO LIST ADDITIONAL INSTALLATIONS IF PIP IS NOT
>> ENOUGH, SUCH AS DATABASE, FILES, ...
>> 
>> Best,
>> 
>> jjfan
>> 
>> -------------------------
>> 
>> FROM: "范正忠" <jjfan at iis.sinica.edu.tw>
>> TO: "Po-Fen Lin" <pofen at iis.sinica.edu.tw>
>> CC: "Most-ai Contest" <Most-ai-contest at iis.sinica.edu.tw>
>> SENT: Tuesday, November 12, 2019 5:29:03 PM
>> SUBJECT: Re: [Most-ai-contest] 科技大擂台討論會(Today,
>> 12:30-15:00)
>> 
>> Dear all,
>> 
>> 謝謝大家的參與討論.
>> 
>> 附上今天討論的內容.
>> 1. schedule.ppt 包含每個參與者的 job items
>> 及未來計畫進行 milestones
>> 2. FGC_release_A_all.json 包含 answer type, answer mode, ...
>> 的基礎的 annotation. 提供大家參考
>>    若發現 answer type 或 answer mode 有錯的地方, 請 email
>> 通知我, 有我這邊統一修正, 在再寄送給大家更新版
>> (隨時)
>>    請廖沛俊儘快完成 answer keyword spans 的 annotation,
>> 並將完成後的 json 檔案提供給我一起整理. (10/15
>> 以前完成)
>> 3. FGC_Questions_S7.json son 包含系統進行答題的過程中,
>> 各個 module 預計輸出.
>>    今日會議上討論要修改的部分, 請各位 module owner
>> 儘快提出各自修改的部分給我,
>> 一起整理完後再寄送給大家 (10/14 以前完成)
>> 4. 提供個自 module 所需的 Python library 及使用的 Python
>> version. (10/14 以前完成)
>>    建議 pip list
>> 5. 1007說明會簡報V9, 國網機器遠端連線操作說明:
>> 請郭家鉦幫忙看一下如何處理"自動化介接"及"程式上傳/執行"
>> 6. 附件二科技大擂台簡答題之答題規範,
>> 科技大擂台簡答題之答題規範v2: 提供給謝尊安參考.
>> 7. 為了確保大家沒有誤會各自負責的 module function,
>> 煩請簡單描述一下 (input, out, function) 為何. (10/14
>> 以前完成)
>> 
>> 有任何問題, 請隨時讓我知道(email, line, skype, tel, ...).
>> 我會一起來協助如何處理.
>> 謝謝的家的幫忙!
>> 
>> Best,
>> jjfan
>> 
>> _______________________________________________
>> Most-ai-contest mailing list
>> Most-ai-contest at iis.sinica.edu.tw
>> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
>> 
>> _______________________________________________
>> Most-ai-contest mailing list
>> Most-ai-contest at iis.sinica.edu.tw
>> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
>> 
>> _______________________________________________
>> Most-ai-contest mailing list
>> Most-ai-contest at iis.sinica.edu.tw
>> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
>> 
>> _______________________________________________
>> Most-ai-contest mailing list
>> Most-ai-contest at iis.sinica.edu.tw
>> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
>> 
>> _______________________________________________
>> Most-ai-contest mailing list
>> Most-ai-contest at iis.sinica.edu.tw
>> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
>> 
>> _______________________________________________
>> Most-ai-contest mailing list
>> Most-ai-contest at iis.sinica.edu.tw
>> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
>> 
>> _______________________________________________
>> Most-ai-contest mailing list
>> Most-ai-contest at iis.sinica.edu.tw
>> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> 
> _______________________________________________
> Most-ai-contest mailing list
> Most-ai-contest at iis.sinica.edu.tw
> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest