Elfrino/PlasmaPearl-20B

merge

This is a merge of pre-trained language models created using mergekit.

Merge Details

Merge Method

This model was merged using the SLERP merge method.

Models Merged

The following models were included in the merge:

• Undi95/PsyMedRP-v1-20B
• Undi95/MXLewd-L2-20B

Configuration

The following YAML configuration was used to produce this model:

models:
  - model: Undi95/PsyMedRP-v1-20B
  - model: Undi95/MXLewd-L2-20B

merge_method: slerp
base_model: Undi95/PsyMedRP-v1-20B

parameters:
  t:
    - filter: model.embed_tokens.weight
      value: 0.5
    - value: 0.5   # Default fallback t for all other parameters

layer_weights:
  transformer.h.0-6.mlp.*: 0.2
  transformer.h.6-18.attn.*: 0.65
  transformer.h.18-24.*: 0.3
  transformer.ln_f.*: 0.7
  transformer.wte: 0.5
  transformer.wpe: 0.5
  transformer.h.*.attn.c_proj: 0.75
  lm_head: 0.3