Spaces:

---

Naphula

/

model_tools

like 7

Running

@grimjim

ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! occurs when trying to run measure.py on the base Nemo.

(ZeroDivisionError: float division by zero occurs for analyze.py if you swap the tokenizer_config.json with the one from instruct version.)

The issue is that measure.py assumes all models have chat templates configured, but base models like Mistral-Nemo-Base-2407 don't include them since they're not instruction-tuned. The format_chats() function calls apply_chat_template() unconditionally, which fails for base models.

Solution

Modify the format_chats() function in measure.py to check if a chat template exists before applying it. If no template is present, return the prompts as-is:

def format_chats(
    tokenizer: PreTrainedTokenizer | PreTrainedTokenizerFast,
    prompt_list: list[str],
    processor = None,
):
    # Use processor's tokenizer if available, otherwise use tokenizer directly
    actual_tokenizer = processor.tokenizer if processor is not None else tokenizer
    
    # Check if chat template exists
    if actual_tokenizer.chat_template is None:
        # For base models without chat templates, return prompts as-is
        return prompt_list
    
    result_formatted = [
        actual_tokenizer.apply_chat_template(
            conversation=[{"role": "user", "content": inst}],
            add_generation_prompt=True,
            add_special_tokens=False,
            tokenize=False,
        )
        for inst in prompt_list
    ]
    return result_formatted

This change allows the script to work with base models by skipping chat template formatting when it's not available.

Notes

The ZeroDivisionError in analyze.py occurs because your workaround (swapping tokenizer configs) produced corrupted measurement data where both harmful and harmless norms were zero. The proper fix is to handle base models correctly in measure.py rather than modifying the tokenizer configuration.