| --- |
| language: |
| - code |
| extra_gated_prompt: >- |
| ## Model License Agreement |
| |
| Please read the BigCode [OpenRAIL-M |
| license](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement) |
| agreement before accepting it. |
| |
| extra_gated_fields: |
| I accept the above license agreement, and will use the Model complying with the set of use restrictions and sharing requirements: checkbox |
| --- |
| # StarEnCoder |
|
|
| ## Table of Contents |
|
|
| 1. [Model Summary](##model-summary) |
| 3. [Training](##training) |
| 4. [Use](##use) |
| 5. [Limitations](##limitations) |
| 6. [License](##license) |
|
|
| ## Model Summary |
|
|
| This is an encoder-only model (i.e., bi-directionally self-attentive Transformers) trained on [The Stack](https://huggingface.co/datasets/bigcode/the-stack) dataset. |
|
|
| - **Project Website:** [bigcode-project.org](https://www.bigcode-project.org) |
| - **Point of Contact:** [contact@bigcode-project.org](mailto:contact@bigcode-project.org) |
| - **Languages:** 80+ Programming languages |
|
|
|
|
| We leveraged the : |
| - Masked Language Modelling (MLM) and Next Sentence Prediction (NSP) objectives from [BERT](https://arxiv.org/abs/1810.04805). |
| - Predicted masked-out tokens from an input sentence and whether a pair of sentences occur as neighbors in a document. |
|
|
| ## Training |
|
|
| We train for 100,000 steps with a global batch size of 4,096 sequences of a maximum length of 1,024 so that approximately 400B~tokens are observed. This takes roughly two days using 64 NVIDIA A100 GPUs. |
| Details about the model architecture are reported in the table below. |
|
|
| | Hyperparameter | Value | |
| |--------------------------|-----------| |
| | Hidden size | 768 | |
| | Intermediate size | 3072 | |
| | Max. position embeddings | 1024 | |
| | Num. of attention heads | 12 | |
| | Num. of hidden layers | 12 | |
| | Attention | Multi-head| |
| | Num. of parameters | ≈125M | |
|
|
|
|
| ## Use |
|
|
| This model is trained on 86 programming languages from GitHub code including GitHub issues and Git Commits, and can be efficiently fine-tuned for both code- and text-related tasks. |
| We fine-tuned on a token classification task to detect PII and have released [StaPII](https://huggingface.co/bigcode/starpii) model. |
|
|
|
|
| ## Limitations |
| There are limitations to consider when using StarEncoder. It is an encoder-only model, which limits its flexibility in certain code generation or completion tasks, |
| and it was trained on data containing PII, which could pose privacy concerns. Performance may vary across the 80+ supported programming languages, |
| particularly for less common ones, and the model might struggle with understanding domains outside programming languages. |
|
|
| ## License |
|
|
| The model is licensed under the BigCode OpenRAIL-M v1 license agreement. You can find the full agreement [here](https://huggingface.co/spaces/bigcode/bigcode-model-license-agreement). |
