ml-sensei-lora-tinyllama-1.1b

This model is a fine-tuned version of TinyLlama/TinyLlama-1.1B-Chat-v1.0 on an unknown dataset. It achieves the following results on the evaluation set:

• Loss: 1.6145

Model description

More information needed

Intended uses & limitations

More information needed

Training and evaluation data

More information needed

Training procedure

Training hyperparameters

The following hyperparameters were used during training:

• learning_rate: 0.0002
• train_batch_size: 4
• eval_batch_size: 8
• seed: 42
• gradient_accumulation_steps: 4
• total_train_batch_size: 16
• optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
• lr_scheduler_type: linear
• lr_scheduler_warmup_ratio: 0.03
• num_epochs: 3
• mixed_precision_training: Native AMP

Training results

Training Loss	Epoch	Step	Validation Loss
2.8588	0.0393	15	1.9034
2.4688	0.0787	30	1.7235
2.0414	0.1180	45	1.7096
1.8122	0.1574	60	1.6616
1.5248	0.1967	75	1.6241
1.506	0.2361	90	1.6213
1.4022	0.2754	105	1.6158
1.3042	0.3148	120	1.6117
1.3063	0.3541	135	1.6116
1.2091	0.3934	150	1.6203
1.1199	0.4328	165	1.6349
1.1519	0.4721	180	1.6278
1.0676	0.5115	195	1.6301
1.1187	0.5508	210	1.6374
0.9942	0.5902	225	1.6369
1.0554	0.6295	240	1.6471
0.9429	0.6689	255	1.6403
0.6697	0.7082	270	1.6430
0.9911	0.7475	285	1.6541
0.9834	0.7869	300	1.6385
0.7859	0.8262	315	1.6415
0.8774	0.8656	330	1.6360
1.0287	0.9049	345	1.6209
0.8125	0.9443	360	1.6235
0.8073	0.9836	375	1.6142
0.7979	1.0210	390	1.6231
0.7986	1.0603	405	1.6200
0.6892	1.0997	420	1.6193
0.6407	1.1390	435	1.6245
0.9794	1.1784	450	1.6155
0.6886	1.2177	465	1.6136
0.7396	1.2570	480	1.6079
0.7691	1.2964	495	1.6158
0.6276	1.3357	510	1.6327
0.6803	1.3751	525	1.6095
0.7745	1.4144	540	1.6127
0.8967	1.4538	555	1.6103
0.66	1.4931	570	1.6161
0.7305	1.5325	585	1.6162
0.6281	1.5718	600	1.6135
0.6539	1.6111	615	1.6076
0.5903	1.6505	630	1.6170
0.826	1.6898	645	1.6120
0.7834	1.7292	660	1.6078
0.7071	1.7685	675	1.6026
0.9263	1.8079	690	1.6083
0.6286	1.8472	705	1.6093
0.6588	1.8866	720	1.6032
0.7084	1.9259	735	1.6019
0.6823	1.9652	750	1.5994
0.6274	2.0026	765	1.6038
0.5752	2.0420	780	1.6128
0.5912	2.0813	795	1.6282
0.7835	2.1207	810	1.6093
0.69	2.16	825	1.6101
0.5452	2.1993	840	1.6134
0.6807	2.2387	855	1.6268
0.4333	2.2780	870	1.6227
0.628	2.3174	885	1.6227
0.7805	2.3567	900	1.6200
0.8267	2.3961	915	1.6158
0.6766	2.4354	930	1.6081
0.729	2.4748	945	1.6090
0.7099	2.5141	960	1.6190
0.6708	2.5534	975	1.6145
0.6874	2.5928	990	1.6115
0.7857	2.6321	1005	1.6161
0.7838	2.6715	1020	1.6081
0.6171	2.7108	1035	1.6118
0.5603	2.7502	1050	1.6132
0.6887	2.7895	1065	1.6191
0.618	2.8289	1080	1.6188
0.6477	2.8682	1095	1.6182
0.6099	2.9075	1110	1.6177
0.7329	2.9469	1125	1.6147
0.8107	2.9862	1140	1.6145

Framework versions

• PEFT 0.18.0
• Transformers 4.57.1
• Pytorch 2.6.0+cu124
• Datasets 4.4.1
• Tokenizers 0.22.1