# Task\_type parameter of LoraConfig

**URL:** <https://discuss.huggingface.co/t/task-type-parameter-of-loraconfig/52879>\
**Category:** 🤗Transformers\
**Created:** [August 30, 2023, 9:28pm UTC](https://discuss.huggingface.co/t/task-type-parameter-of-loraconfig/52879 "2023-08-30T21:28:51Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![thenatefisher](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thenatefisher/32/17153_2.png) [@thenatefisher](https://discuss.huggingface.co/u/thenatefisher)\
**Post date:** [August 30, 2023, 9:28pm UTC](https://discuss.huggingface.co/t/task-type-parameter-of-loraconfig/52879/1 "2023-08-30T21:28:51Z")

</div>

I am training a fine-tune of codellama using PEFT but not sure how to use the task\_type parameter of LoraConfig. Should it be CAUSAL\_LM or SEQ\_2\_SEQ\_LM or something else? Does it have any affect?

The goal of my model is to parse an input for independent clauses in a sentence. For example, it would insert a delimiter, such as in this sentence: “the tea was on the stove and was at high temperature” , separating the independent clause from the subordinate clause. My training data is all in a single col and each row looks like this (where the → and are custom tokens I add to the tokenizer vocab and the is the EOS token):

“the tea was on the stove and was at high temperature → the tea was on the stove and was at high temperature ”

---

<div class="post-metadata">

**Author:** ![ndvb](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndvb/32/17239_2.png) [@ndvb](https://discuss.huggingface.co/u/ndvb)\
**Post date:** [September 11, 2023, 4:06pm UTC](https://discuss.huggingface.co/t/task-type-parameter-of-loraconfig/52879/2 "2023-09-11T16:06:09Z")

</div>

Does task\_type matters in LoraConfig, and if so, in what way?

---

<div class="post-metadata">

**Author:** ![youcefjd](https://avatars.discourse-cdn.com/v4/letter/y/e274bd/32.png) [@youcefjd](https://discuss.huggingface.co/u/youcefjd)\
**Post date:** [September 17, 2023, 10:05pm UTC](https://discuss.huggingface.co/t/task-type-parameter-of-loraconfig/52879/3 "2023-09-17T22:05:07Z")

</div>

I have the same question. Does task\_type matter?

---

<div class="post-metadata">

**Author:** ![raptorkwok](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/raptorkwok/32/26481_2.png) [@raptorkwok](https://discuss.huggingface.co/u/raptorkwok)\
**Post date:** [October 17, 2023, 8:54am UTC](https://discuss.huggingface.co/t/task-type-parameter-of-loraconfig/52879/4 "2023-10-17T08:54:44Z")

</div>

In the source code, `task_type` is not even there:

> <https://github.com/huggingface/peft/blob/15a013af5ff5660b9377af24d3eee358213d72d4/src/peft/tuners/lora/config.py#L24>

---

<div class="post-metadata">

**Author:** ![pursuitYP](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pursuityp/32/27106_2.png) [@pursuitYP](https://discuss.huggingface.co/u/pursuitYP)\
**Post date:** [October 20, 2023, 3:11am UTC](https://discuss.huggingface.co/t/task-type-parameter-of-loraconfig/52879/5 "2023-10-20T03:11:15Z")

</div>

The `task_type` parameter is used in the superclass `PeftConfig`

---

<div class="post-metadata">

**Author:** ![braunagn](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/braunagn/32/24737_2.png) [@braunagn](https://discuss.huggingface.co/u/braunagn)\
**Post date:** [February 25, 2024, 6:02pm UTC](https://discuss.huggingface.co/t/task-type-parameter-of-loraconfig/52879/6 "2024-02-25T18:02:17Z")

</div>

```auto
    SEQ_CLS = "SEQ_CLS"
    SEQ_2_SEQ_LM = "SEQ_2_SEQ_LM"
    CAUSAL_LM = "CAUSAL_LM"
    TOKEN_CLS = "TOKEN_CLS"
    QUESTION_ANS = "QUESTION_ANS"
    FEATURE_EXTRACTION = "FEATURE_EXTRACTION"

```

```auto
    Overview of the supported task types:
    - SEQ_CLS: Text classification.
    - SEQ_2_SEQ_LM: Sequence-to-sequence language modeling.
    - Causal LM: Causal language modeling.
    - TOKEN_CLS: Token classification.
    - QUESTION_ANS: Question answering.
    - FEATURE_EXTRACTION: Feature extraction. Provides the hidden states which can be used as embeddings or features
      for downstream tasks.

```

See here for source: [peft/src/peft/utils/peft\_types.py at v0.8.2 · huggingface/peft · GitHub](https://github.com/huggingface/peft/blob/v0.8.2/src/peft/utils/peft_types.py#L68-L73)

---

<div class="post-metadata">

**Author:** ![Ldwdz5](https://avatars.discourse-cdn.com/v4/letter/l/9de053/32.png) [@Ldwdz5](https://discuss.huggingface.co/u/Ldwdz5)\
**Post date:** [November 28, 2024, 3:44pm UTC](https://discuss.huggingface.co/t/task-type-parameter-of-loraconfig/52879/7 "2024-11-28T15:44:39Z")

</div>

Late reply but I was wondering the same and could not find details in the doc so I put an answer here.

Based on Peft git repo, it seems that the model class returned by `get_peft_model(peft_config)`  
depends on this value :  
[peft/src/peft/mapping](https://github.com/huggingface/peft/blob/131efba5d48753a3355ecd4f3833ae010a0510d6/src/peft/mapping.py#L82) , with the following mapping Line 81 :

```auto
MODEL_TYPE_TO_PEFT_MODEL_MAPPING: dict[str, type[PeftModel]] = {
    "SEQ_CLS": PeftModelForSequenceClassification,
    "SEQ_2_SEQ_LM": PeftModelForSeq2SeqLM,
    "CAUSAL_LM": PeftModelForCausalLM,
    "TOKEN_CLS": PeftModelForTokenClassification,
    "QUESTION_ANS": PeftModelForQuestionAnswering,
    "FEATURE_EXTRACTION": PeftModelForFeatureExtraction,
}

```

If you do not specify a value, you get a default PeftModel (Lines 210-220)

```auto
    if peft_config.task_type not in MODEL_TYPE_TO_PEFT_MODEL_MAPPING.keys() and not peft_config.is_prompt_learning:
        return PeftModel(
            model,
            peft_config,
            adapter_name=adapter_name,
            autocast_adapter_dtype=autocast_adapter_dtype,
            low_cpu_mem_usage=low_cpu_mem_usage,
        )

    if peft_config.is_prompt_learning:
        peft_config = _prepare_prompt_learning_config(peft_config, model_config)
    return MODEL_TYPE_TO_PEFT_MODEL_MAPPING[peft_config.task_type](
        model, peft_config, adapter_name=adapter_name, autocast_adapter_dtype=autocast_adapter_dtype
    )

```

Finally it seems that in the case where `task_type=SEQ_CLS` , the classifier heads are excluded from LoRA : in peft/src/peft/tuners/tuner\_utils:

```auto
    if output_emb is not None:
        # ignore the last classification head for text generation models
        last_module_name = [name for name, module in model.named_modules() if module is output_emb][0]
        module_names_to_exclude.add(last_module_name)
    elif peft_config.task_type == TaskType.SEQ_CLS:
        # ignore classifier head for classification models (issue 2027)
        # there is no fix name for the classifier head, so check the common ones
        for name in SEQ_CLS_HEAD_NAMES:
            cls_head = getattr(model, name, None)
            if cls_head is not None:
                last_module_name = [name for name, module in model.named_modules() if module is cls_head][0]
                module_names_to_exclude.add(last_module_name)
                break

```

and also the number of transformer\_submodules increases from 1 to 2 in the case of SEQ\_2\_SEQ\_LM (in peft/src/peft/peft\_model.py):

```auto
        if config.num_transformer_submodules is None:
            config.num_transformer_submodules = 2 if config.task_type == TaskType.SEQ_2_SEQ_LM else 1

```