MT EVALUATION:   BASIS FOR FUTURE DIRECTIONS

AN NSF-SPONSORED WORKSHOP

(San Diego, 2-3 November 1992)

PROGRAM  and Proceedings

 

                                PrefaceM.Vasconcellos

Monday, 2 November

       Announcements and Greetings

Association for Machine Translation in the AmericasM. Vasconcellos International Association for Machine Translation—M. Nagao

European Association for Machine Translation—M. King

National Science Foundation—Y.T. Chien

      Traditions in the Evaluation of MTY. Wilks

      International Coordination

International Working Group on MT EvaluationM. King

EC Evaluation ActivitiesL. Rolling

     Evaluation Surveys

The JEIDA Methodology and SurveyH. Nomura, H. Isahara

The MITRE Methodology and SurveyP. Jordan, J. Benoit, B. Dorr

      The DARPA Methodology

ObjectiveG.Doddington

PresentationJ. White

Discussion:

Participants—P. Brown, L. Carlson, G. Doddington, D. Farwell, E. Hovy, E. Lange,

M. Leon, M. Mandel, S. Nirenburg

Outside commentators—R. Berwick, L. Rolling, M. Vasconcellos

      New Directions

It’s Lucky If Concatenation Does AnythingR. Berwick

Automatic Evaluation of Translation QualityH. Thompson

Evaluation of the Machine-Aided Voice Translation SystemC.A. Montgomery

      Evaluation of Related NLP Research

Message Understanding: The MUC Methodology—B. Sundheim

Semi-Automated Evaluation of Japanese and English Output—L. Carlson

The Murasaki Project: Multilingual Natural Language Understanding—R. Doerr

Measurement of Human Performance for Information Extraction—C. Will, B. Onyshkevych

 

Tuesday, 3 November

      Discussion: "Apples, Oranges, or Kiwis?" Criteria for the Comparison of Systems—

M. Vasconcellos (moderator), R. Dessau, E. Hovy, V. Lawson, C. Miller, M. Nagao, B. Scott, M.Tacelosky

      Internal Evaluation

Practical Use and Effect of the JICST Japanese-English Machine Translation System—K. Tamura, I. Tominaga

LOGOS— B. Scott

Methodology for the Computing Research Laboratory’s Multilingual MT System—D. Farwell

Quality Analysis, an Internal Evaluation Tool at SYSTRAN—E. Lange, J. Homer, L. Gerber

METAL—W. S. Bennett

    Approaches to the Evaluation of Japanese-English MT

Criteria for Evaluating the Linguistic Quality of Japanese-English MT—S. Ikehara

Application of the TOEFL Test to the Evaluation of Japanese-English MTM. Tomita [abstract]

      Evaluation of Lexical CoverageG. Magnusdottir

      Discussion: Assessing the Labor-intensive Elements of MT—M. León (moderator), D. Albisser, M. Baena, L. Childs, G. Magnusdottir

      Results of the Warmup ExercisesM. León

      Discussion: Future Directions in MT—S. Nirenburg (moderator), W.S. Bennett, B. Dorr, E. Hovy, S. Ikehara, M. King, A. Melby, M. Nagao, M. Tomita, V. Teller

Closing Messages

IAMT—M. Nagao

AMTA—M. Vasconcellos

 

Annex A:

        Machine Translation Evaluation CriteriaR.Dessau

Annex B:

      Warmup exercises [introduction]

      Photocopy of: Machine Translation Evaluation Methodology [Washington, DC: DARPA, June 1992]—J.Baker, P.Brown, L.Carlson, E.Hovy, C.Wayne, J.White

      Photocopy of: Appendix 10 of the ALPAC report: An experiment in evaluating the quality of translations

      [Texts, translations, questions, instructions]