Competancies, Intendend learning outcomes (ILOs) and levels of achievements

Here are some of the learning objectives suitable for a course in LLMs. The ILOs descibe what a student should be able to do after completing a course in using LLMs.

ILOs related to understanding LLMs

A1: Technical knowledge of how LLMs work

ILO	Explain and distinguish between the concepts of artificial intelligence, language models, neural networks, and training data, and discuss how training data can lead to biases in the responses given by language models.
Low achievement	The student can explain what AI is in a simple way.
Medium achievement	The student can explain the difference between AI and language models and can contextualize the concepts of neural networks and training data.
High achievement	The student can precisely explain the concepts of AI, language model, neural networks, and training data.

A2: Understanding of data and privacy issues

ILO	Find out where data is stored and how it is used by the language model, and discuss the privacy implications of using large language models.
Low achievement	The student is able to find out where data is stored.
Medium achievement	The student is able to find out where data is stored and how it is used by the language model. They discuss some implications of that.
High achievement	The student is able to find out where data is stored, how it is used by the language model, and discuss the privacy implications.

A3: Understanding of environmental impact of LLMs

ILO	Discuss the environmental impact of training and using large language models.
Low achievement	The student expresses some notions of how training large language models affects the environnement. (To be refined).
Medium achievement	The student discusses how training large language models affects the environment. (To be refined).
High achievement	The student discusses how training and using large language models affects the environment, and can evaluate the impact. (To be refined).

A4: Understanding biases in LLMs

ILO	Discuss how biases can arise in language models and how they can impact society.
Low achievement	The student can explain how biases can arise in language models.
Medium achievement	The student can explain how biases can arise in language models and give examples of how they can impact society.
High achievement	The student can explain how biases can arise in language models, give examples of how they can impact society, and discuss how to mitigate these biases.

A5: Understanding intellectual property rights

ILO	Discuss the intellectual property rights of the language model and the generated text.
Low achievement	The student can explain what intellectual property rights are.
Medium achievement	The student can explain the intellectual property rights of the language model and the generated text.
High achievement	The student can explain the intellectual property rights of the language model and the generated text, and discuss the implications of these rights.

A6: Understanding the broader societal impact of LLMs

ILO	Discuss the broader societal impact of using large language models.
Low achievement	The student can explain some of the societal impacts of using large language models.
Medium achievement	The student can explain some of the societal impacts, detailing how bad use can lead to negative consequences.
High achievement	The student can explain the societal impacts of using large language models, detailing how bad use can lead to negative consequences, and discuss how to mitigate these consequences. Detailing notions of accessibility, inequality, transparency, accountability, misue potential, the role of media, and the role of the individual.

ILOs related to using LLMs

B1: Aligned use

ILO	Verbalize the intended goal of their task and judge whether the use of LLMs is aligned with that goal.
Low achievement	The student expresses opinions on what is productive use of generative AI, but without justification.
Medium achievement	The student distinguishes between productive and unproductive use of generative AI, with relevant examples.
High achievement	The student defines precisely what distinguishes productive vs. unproductive use of generative AI, with relevant examples, and assesses the consequences of unproductive use.

B2: Prompt engineering

ILO	Write effective prompts that include, if necessary, assigning a role to the language model, providing relevant context, examples, and a specific task.
Low achievement	The student breaks complex tasks down to some degree and asks the language model to solve these.
Medium achievement	The student provides the language model with context, examples, and specific tasks tailored to the topic. The student adjusts based on the responses from the language model and assigns new tasks accordingly.
High achievement	The student assigns roles, context, and examples to the language model where relevant, and is able to formulate the task clearly and precisely. The student effectively adjusts these factors according to the responses from the language model.

B3: Evaluating the output from the language model

ILO	Critically evaluate the response given by the language model by cross-checking with other sources.
Low achievement	The student simply accounts for why the response from the language model is not necessarily trustworthy.
Medium achievement	The student accounts for cases in which one can trust the language model and cross-checks the answers with other sources.
High achievement	The student evaluates the response given by the language model based on context, task, and purpose, and cross-checks the relevant parts of the answer with other sources.

B4: Engaging in a productive dialogue with the language model

ILO	Engage in a productive dialogue with the language model by asking follow-up questions, clarifying the task, and adjusting the prompt based on the response.
Low achievement	The student asks follow-up questions to the language model.
Medium achievement	The student asks follow-up questions, clarifies the task, and adjusts the prompt based on the response.
High achievement	The student asks follow-up questions, clarifies the task, adjusts the prompt based on the response, and evaluates the response given by the language model.

B5: Knowing good use cases for LLMs

ILO	Identify and discuss good use cases for LLMs.
Low achievement	The student can name some use cases for LLMs.
Medium achievement	The student can name and discuss some use cases for LLMs, and can explain why they are good use cases.
High achievement	The student can identify, discuss, and evaluate use cases for LLMs, and can explain why they are good use cases.

Subject specific ILOs

Ph1: Physics

ILO	Use the language model to implement physical equations and verify the solution against known cases.
Low achievement	The student formulates a physical problem to the language model and gets the code to run in, for example, Colab.
Medium achievement	The student formulates a physical problem, gets the code to run, and compares the solution with known cases.
High achievement	The student formulates a physical problem, gets the code to run, compares the solution with known cases, and discusses the limitations of the language model in solving physical problems.

Maths