Loomuliku keele töötlus 2024/25 kevad

Project

For the project work, you will choose a NLP-related problem and attempt to solve it. The projects are done in groups of up to three people but can also be done individually. Selection of the project group as well as the report submission are in Moodle.

How to choose a project topic?

There are various options for that.

• It can be a problem that is of interest to you.
• It can be a replication or improvement of an existing paper.
• It can be based on a Kaggle competition (or some other benchmark data).

If you have a problem with choosing a topic, you can consult the teachers and we can help you to find a topic of interest.

Keep in mind that this is a relatively small project, expected to be executed in max half a semester and thus, it is extremely important that the data would be available or easily obtained. The project can also center on collecting and annotating some new data, with less emphasis on experimental work.

The criteria of a successful project

It is hard to give exact criteria of what constitutes a successful project. In a very broad sense, the criteria are:

• The project topic should be related to NLP (although occasionally students have successfully worked also on projects related to another study area using the methods of NLP).
• The project should have a reasonable goal that can be stated as a hypothesis, research question or simply a goal.
• For achieving the stated goals, the project should implement appropriate methods. These methods can vary widely depending on the project goal, but in most projects the method involves training some model on a dataset and evaluating this model on the test set. Nowadays, the selected methods can be less centered on training a model, but instead utilizing LLM's in appropriate ways. Nevertheless, the method from prompting an LLM should be still systematically developed. Choosing appropriate methods for your problem is an important skill that this project task is designed to allow you to practice.
• The project should assess somehow whether the goal was achieved or not. Typically this means using some quantitive evaluation measures to evaluate results. For some project topics, using quantitative evaluation measures is not so feasible, so these projects need to consider other ways of evaluation (human evaluation, qualitative error analysis). Every project should do some qualitative error analysis.

Project assessment

The project gives a maximum of 35 points in total, of which up to 30 point will be given out based on the final project report and up to 5 points based on peer review.

The project assessment includes five activities/deliverables:

• Presenting the project method in the seminar on 24.04.2025, 28.04.2025 or 30.04.2025 (instructions in Moodle).
• Submitting the first draft for peer review by 20.05.2025.
• Presenting the project results in the seminar on 19.05.2024, 21.05.2025 or 26.05.2024.
• Sumitting a written review for another project by 27.05.2025.
• Submitting the final project report (due date after the final project seminars, to be agreed).

The seminar presentations will not give points per se, but not presenting your project in seminar will deduct 5 points per seminar. That means, if your group does not present your project in either seminar, you can still get max 25 points just based on the peer review and the project final report.

However, we highly enourage attending and presenting on both seminars, because:

• In the method seminar we can make sure that your project is methodologically on track and guide you, if necessary.
• In the project results seminar, it will be interesting for other to see what you have accomplished and it will be interesting for you to see what others have done.
• During the project results seminar, you can get some useful feedback you will still have time to incorporate in the final project report.

Project presentations

The allocated time for each project presentation depends on the number of projects we will have per each seminar and will be specified later.

Presentation in the project method seminar should convey the following information:

1. Project title and group members.
2. Little bit of background on the topic and motivation for addressing it.
3. Problem formulation: the stated goal or hypothesis or research question.
4. The plan for the methodological approach, including:
   1. What data will be used and some characteristics of that data (size etc)
   2. What methods will be used (models, algorithms, experiments planned)
   3. Plan for evaluating the results (evaluation measures or procecures)

The final project presentations should convey the following information:

1. Project title and group members.
2. Little bit of background on the topic and motivation for addressing it.
3. Problem formulation: the stated goal or hypothesis or research question.
4. The methodological approach adopted:
   1. What data was used used and some characteristics of that data (size etc)
   2. What methods were used (models, algorithms, experiments)
   3. How the results were evaluated (evaluation measures or procecures)
5. The main results (if you have them) or the preliminary results (if not all the work has been done yet).
6. Conclusions based on the results.
7. Contributions of the group members so far.

Peer review

• Peer review involves giving feedback to a report draft of another project.
• The prerequisite for being able to participate in peer review process and getting points for it is submitting your own report that can be peer reviewed.
• The submitted draft must be complete, which means that it should be the version you would be willing to submit to final assessment. The only aspect that can be incomplete are the results. It is acceptable for you to present only preliminary results in the first draft, if you haven't been able to obtain the final results yet.
• If the submitted draft is not complete, ie it is missing relevant parts or the writing is so preliminary that it is not readable then it will not be allowed to the peer review process, i.e., you will not get the opportunity to review someone else's project and get points for that.

Project report

The instructions and the submission of the project final report will be given in Moodle. The due date will be agreed during the semester.