[Most-ai-contest] The performance of the current integrated system on the dataset ver. 1.5

Wed 12月 25 14:17:03 CST 2019

Dear all,

Some empty answers were found in the week1 data, and all of these errors
are caused by the wrong classification of AMODE (not multi-spans-extraction
-> multi-spans-extraction).

It is solved by moving both "Date-Duration" and "Num-Measure" functions
ahead of "Multi-Spans-Extraction" since there is no question which is
multi-spans-extraction and num-measure/date-duration simultaneously in
previous data.

Please find the revised output formatter in the attached file.

張光瑜 <simonc於iis.sinica.edu.tw> 於 2019年12月25日 週三 下午12:43寫道：

> Dear all,
>
> Please see the attached files for the errors/stats.
>
> Cheers,
> 張光瑜
>
>
> 范正忠 於 2019-12-25 11:23 寫到:
> > Dear all,
> >
> > Enclosed please find the last performance of the last integrated
> > system for your analysis.
> >
> > 'bestBERTFGC10_full_em.pth'
> > 'FGCBERT_Chinese'
> > 'hotpot_mix_fgc'
> > MSPE_v3
> > FGC Server
> >
> > week_1
> > total: 50, correct: 22, accuracy: 0.44
> >
> > week_2
> > total: 50, correct: 23, accuracy: 0.46
> >
> > week_3
> > total: 50, correct: 20, accuracy: 0.4
> >
> > Best,
> > jjfan
> >
> > -------------------------
> >
> > FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> > TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> > SENT: Tuesday, December 24, 2019 10:30:19 PM
> > SUBJECT: Re: [Most-ai-contest] The performance of the current
> > integrated system on the dataset ver. 1.5
> >
> > Dear all,
> > There are three performances of three different model parameters
> > combination.
> > 2019.12.24.2201:
> >
> > 'bestBERTFGC3_full_em.pth'
> > 'FGCBERT_Chinese'
> > 'hotpot_mix_fgc'
> > FGC server
> >
> > week_1
> > total: 50, correct: 21, accuracy: 0.42
> > week_2
> > total: 50, correct: 23, accuracy: 0.46
> > week_3
> > total: 50, correct: 17, accuracy: 0.34
> > 2019.12.24.2208
> >
> > 'bestBERTFGC6_full_em.pth'
> > 'FGCBERT_Chinese'
> > 'hotpot_mix_fgc'
> > FGC server
> > week_1
> > total: 50, correct: 19, accuracy: 0.38
> > week_2
> > total: 50, correct: 21, accuracy: 0.42
> > week_3
> > total: 50, correct: 19, accuracy: 0.38
> > 2019.12.24.2210
> >
> > 'bestBERTFGC8_3_full_em.pth'
> > 'FGCBERT_Chinese'
> > 'hotpot_mix_fgc'
> > FGC server
> > week_1
> > total: 50, correct: 19, accuracy: 0.38
> > week_2
> > total: 50, correct: 21, accuracy: 0.42
> > week_3
> > total: 50, correct: 18, accuracy: 0.36
> >
> > You could find that the test ' 2019.12.24.2201' seems the best
> > performance.
> > Please all you help to conduct error analysis to see if the
> > performance is miss-matched with your own module test.
> >
> > Thanks.
> >
> > jjfan
> >
> > -------------------------
> >
> > FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> > TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> > SENT: Tuesday, December 24, 2019 10:34:44 AM
> > SUBJECT: Re: [Most-ai-contest] The performance of the current
> > integrated system on the dataset ver. 1.5
> >
> > Dear all,
> > Enclosed please find the last performance on 3 weeks.
> > with 'hotpot_mix_fgc' model in single-span-multi-hop module &
> > 'bestBERTFGC6_full_em.pth' in single-span-extraction module
> >
> > week_1
> > total: 50, correct: 18, accuracy: 0.36
> > week_2
> > total: 50, correct: 18, accuracy: 0.36
> > week_3
> > total: 50, correct: 18, accuracy: 0.36
> > Best,
> > jjfan
> >
> > -------------------------
> >
> > FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> > TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> > SENT: Monday, December 23, 2019 7:38:40 PM
> > SUBJECT: Re: [Most-ai-contest] The performance of the current
> > integrated system on the dataset ver. 1.5
> >
> > Dear all,
> > After Smolka's refinement of her kinship module, the last performance
> > of 3 weeks as follows for your reference
> >
> > week_1
> > total: 50, correct: 17, accuracy: 0.34
> > week_2
> > total: 50, correct: 18, accuracy: 0.36
> > week_3
> > total: 50, correct: 15, accuracy: 0.3
> > Best,
> > jjfan
> >
> > -------------------------
> >
> > FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> > TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> > SENT: Monday, December 23, 2019 1:16:56 PM
> > SUBJECT: Re: [Most-ai-contest] The performance of the current
> > integrated system on the dataset ver. 1.5
> >
> > Dear all,
> > Enclosed please find the week_1 ~ week_3 performance based on
> > annotated answers.
> >
> > week_1
> > total: 50, correct: 17, accuracy: 0.34
> > week_2
> > total: 50, correct: 18, accuracy: 0.36
> > week_3
> > total: 50, correct: 14, accuracy: 0.28
> > jjfan
> >
> > -------------------------
> >
> > FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> > TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> > SENT: Sunday, December 22, 2019 4:48:39 PM
> > SUBJECT: Re: [Most-ai-contest] The performance of the current
> > integrated system on the dataset ver. 1.5
> >
> > Dear all,
> > You can first check the performance of week_1
> > (question_1_result.json), since it has only 8 questions not in
> > FGC_release_dataset in question_1 (there are total 50 questions).
> > the performance will be 18/50 = 0.36, 18/42 = 0.43 (ignore 8 questions
> > without given correct answers)
> > jjfan
> >
> > -------------------------
> >
> > FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> > TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> > SENT: Sunday, December 22, 2019 2:43:57 PM
> > SUBJECT: Re: [Most-ai-contest] The performance of the current
> > integrated system on the dataset ver. 1.5
> >
> > Dear all,
> > Enclosed find the performance of  FGC past three weeks testing result.
> >
> > week_1: 0.28, week_2: ? week_3: 0.28
> > Please check your modue's performance. Also note that the most of the
> > questions in week_2 and week_3 are not in FGC_release_dataset.
> > The correct answer of week_3 are as follows
> > D014Q02,  D040Q03,  D042Q04,  D064Q04,  D072Q06,  D087Q07,  D116Q06,
> > D238Q04,  D242Q06,  D244Q04,  D247Q02,  D247Q05
> > Please conduct error analysis and try some possible improvements
> > Best,
> > jjfan
> >
> > -------------------------
> >
> > FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> > TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> > SENT: Thursday, December 19, 2019 9:58:38 AM
> > SUBJECT: Re: [Most-ai-contest] The performance of the current
> > integrated system on the dataset ver. 1.5
> >
> > Dear all,
> > Enclosed please find train/dev/test dataset (1.5) performance using
> > 1.  廖沛俊 bestBERTFGC_full_em Bert Model
> > 2. Dr. Chang aggregrator
> > 3. Dr. Chang single-span-multi-hops enable
> > 4. Smolka kinship_module4
> >
> > train: 0.65, dev: 0.55, test: 0.53
> >
> > Best,
> > jjfan
> >
> > -------------------------
> >
> > FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> > TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> > SENT: Wednesday, December 18, 2019 3:56:18 PM
> > SUBJECT: Re: [Most-ai-contest] The performance of the current
> > integrated system on the dataset ver. 1.5
> >
> > Dear all,
> > Enclosed please find train/dev/test dataset (1.5) performance using
> > 1.  廖沛俊 bestBERTFGC_full_em Bert Model
> > 2. Dr. Chang aggregrator
> > 3. Dr. Change single-span-multi-hops enable
> >
> > train: 0.65, dev: 0.55, test: 0.52
> >
> > Best,
> > jjfan
> >
> > -------------------------
> >
> > FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> > TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> > SENT: Wednesday, December 18, 2019 11:30:36 AM
> > SUBJECT: Re: [Most-ai-contest] The performance of the current
> > integrated system on the dataset ver. 1.5
> >
> > Dear all,
> > Enclosed please find train/dev/test dataset (1.5) performance using
> > 1.  廖沛俊 bestBERTFGC_full_em Bert Model
> > 2. Dr. Chang aggregrator
> >
> > Best,
> > jjfan
> >
> > -------------------------
> >
> > FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> > TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> > SENT: Saturday, December 14, 2019 4:47:37 PM
> > SUBJECT: [Most-ai-contest] The performance of the current integrated
> > system on the dataset ver. 1.5
> >
> > Dear all,
> > Enclosed please find the performance of our current integrated system
> > on train / dev dataset.
> > Please refer to the AMODE fields in each question for each module's
> > performance.
> > Hope this can help the error analysis of each module and please share
> > your finding in the next week's meeting.
> > I am still waiting for 'multi-spans-extraction' & 'supporting
> > evidence' modules to integrate. Please anyone has improved version
> > just send to me.
> > Best,
> > jjfan
> >
> > -------------------------
> >
> > FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> > TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> > SENT: Monday, November 25, 2019 9:39:07 AM
> > SUBJECT: Re: [Most-ai-contest] It's time to start system integration
> >
> > Dear all,
> > Please don't forget give me a readme.txt or I really don't know what
> > is inside.
> > Suggest readme.txt contains at least
> > 1. Brief API description / document, function list at minimum
> > 2. System requirements
> > Thanks.
> > jjfan
> >
> > -------------------------
> >
> > FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> > TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> > SENT: Monday, November 25, 2019 9:18:05 AM
> > SUBJECT: [Most-ai-contest] It's time to start system integration
> >
> > Dear all,
> > Good Morning!
> > It's time to start integration of the system. Please all module owners
> > send me your Python Code. Thanks.
> > Best,
> > jjfan
> >
> > -------------------------
> >
> > FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> > TO: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> > SENT: Thursday, November 14, 2019 8:58:50 AM
> > SUBJECT: Re: [Most-ai-contest] 科技大擂台討論會(Today,
> > 12:30-15:00)
> >
> > Dear all,
> >
> > Note the following action items should be finished today.
> >
> > 3. FGC_Questions_S7.json son 包含系統進行答題的過程中,
> > 各個 module 預計輸出.
> >    今日會議上討論要修改的部分, 請各位 module owner
> > 儘快提出各自修改的部分給我,
> > 一起整理完後再寄送給大家 (10/14 以前完成)
> > 4. 提供個自 module 所需的 Python library 及使用的 Python
> > version. (10/14 以前完成)
> >    建議 pip list
> > 7. 為了確保大家沒有誤會各自負責的 module function,
> > 煩請簡單描述一下 (input, out, function) 為何. (10/14
> > 以前完成)
> >
> > ITEM 4. PLEASE ALSO LIST ADDITIONAL INSTALLATIONS IF PIP IS NOT
> > ENOUGH, SUCH AS DATABASE, FILES, ...
> > Best,
> > jjfan
> >
> > -------------------------
> >
> > FROM: "范正忠" <jjfan於iis.sinica.edu.tw>
> > TO: "Po-Fen Lin" <pofen於iis.sinica.edu.tw>
> > CC: "Most-ai Contest" <Most-ai-contest於iis.sinica.edu.tw>
> > SENT: Tuesday, November 12, 2019 5:29:03 PM
> > SUBJECT: Re: [Most-ai-contest] 科技大擂台討論會(Today,
> > 12:30-15:00)
> >
> > Dear all,
> >
> > 謝謝大家的參與討論.
> >
> > 附上今天討論的內容.
> > 1. schedule.ppt 包含每個參與者的 job items
> > 及未來計畫進行 milestones
> > 2. FGC_release_A_all.json 包含 answer type, answer mode, ...
> > 的基礎的 annotation. 提供大家參考
> >    若發現 answer type 或 answer mode 有錯的地方, 請 email
> > 通知我, 有我這邊統一修正, 在再寄送給大家更新版
> > (隨時)
> >    請廖沛俊儘快完成 answer keyword spans 的 annotation,
> > 並將完成後的 json 檔案提供給我一起整理. (10/15
> > 以前完成)
> > 3. FGC_Questions_S7.json son 包含系統進行答題的過程中,
> > 各個 module 預計輸出.
> >    今日會議上討論要修改的部分, 請各位 module owner
> > 儘快提出各自修改的部分給我,
> > 一起整理完後再寄送給大家 (10/14 以前完成)
> > 4. 提供個自 module 所需的 Python library 及使用的 Python
> > version. (10/14 以前完成)
> >    建議 pip list
> > 5. 1007說明會簡報V9, 國網機器遠端連線操作說明:
> > 請郭家鉦幫忙看一下如何處理"自動化介接"及"程式上傳/執行"
> > 6. 附件二科技大擂台簡答題之答題規範,
> > 科技大擂台簡答題之答題規範v2: 提供給謝尊安參考.
> > 7. 為了確保大家沒有誤會各自負責的 module function,
> > 煩請簡單描述一下 (input, out, function) 為何. (10/14
> > 以前完成)
> >
> > 有任何問題, 請隨時讓我知道(email, line, skype, tel, ...).
> > 我會一起來協助如何處理.
> > 謝謝的家的幫忙!
> >
> > Best,
> > jjfan
> >
> > _______________________________________________
> > Most-ai-contest mailing list
> > Most-ai-contest於iis.sinica.edu.tw
> > https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >
> > _______________________________________________
> > Most-ai-contest mailing list
> > Most-ai-contest於iis.sinica.edu.tw
> > https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >
> > _______________________________________________
> > Most-ai-contest mailing list
> > Most-ai-contest於iis.sinica.edu.tw
> > https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >
> > _______________________________________________
> > Most-ai-contest mailing list
> > Most-ai-contest於iis.sinica.edu.tw
> > https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >
> > _______________________________________________
> > Most-ai-contest mailing list
> > Most-ai-contest於iis.sinica.edu.tw
> > https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >
> > _______________________________________________
> > Most-ai-contest mailing list
> > Most-ai-contest於iis.sinica.edu.tw
> > https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >
> > _______________________________________________
> > Most-ai-contest mailing list
> > Most-ai-contest於iis.sinica.edu.tw
> > https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >
> > _______________________________________________
> > Most-ai-contest mailing list
> > Most-ai-contest於iis.sinica.edu.tw
> > https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >
> > _______________________________________________
> > Most-ai-contest mailing list
> > Most-ai-contest於iis.sinica.edu.tw
> > https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >
> > _______________________________________________
> > Most-ai-contest mailing list
> > Most-ai-contest於iis.sinica.edu.tw
> > https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >
> > _______________________________________________
> > Most-ai-contest mailing list
> > Most-ai-contest於iis.sinica.edu.tw
> > https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >
> > _______________________________________________
> > Most-ai-contest mailing list
> > Most-ai-contest於iis.sinica.edu.tw
> > https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> >
> > _______________________________________________
> > Most-ai-contest mailing list
> > Most-ai-contest於iis.sinica.edu.tw
> > https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> > _______________________________________________
> > Most-ai-contest mailing list
> > Most-ai-contest於iis.sinica.edu.tw
> > https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
> _______________________________________________
> Most-ai-contest mailing list
> Most-ai-contest於iis.sinica.edu.tw
> https://www.iis.sinica.edu.tw/mailman/listinfo/most-ai-contest
>

-- 
Tsun-An Hsieh
tel:      +886 975 329907 <+886%20975%20329%20907>
email:  pups003c於gmail.com
Institute of Networking and Multimedia, CSIE, National Taiwan University,
Taiwan
-------------- 下一部份 --------------
抹去了一個 HTML 附加檔...
URL: <http://www.iis.sinica.edu.tw/pipermail/most-ai-contest/attachments/20191225/1209f8b1/attachment-0001.html>
-------------- 下一部份 --------------
A non-text attachment was scrubbed...
Name: output_format.py
Type: text/x-python-script
Size: 14447 bytes
Desc: 〔無法取得〕
URL: <http://www.iis.sinica.edu.tw/pipermail/most-ai-contest/attachments/20191225/1209f8b1/attachment-0001.bin>