CodeShell
CodeShellๆฏๅไบฌๅคงๅญฆ็ฅ่ฏ่ฎก็ฎๅฎ้ชๅฎค่ๅๅๅทๅคฉๅบ้ถ่กAIๅข้็ ๅ็ๅค่ฏญ่จไปฃ็ ๅคงๆจกๅๅบๅบงใCodeShellๅ ทๆ70ไบฟๅๆฐ๏ผๅจไบๅไบฟTokens่ฟ่กไบ่ฎญ็ป๏ผไธไธๆ็ชๅฃ้ฟๅบฆไธบ8194ใๅจๆๅจ็ไปฃ็ ่ฏไผฐBenchmark๏ผHumanEvalไธMBPP๏ผไธ๏ผCodeShellๅๅพๅ็ญ่งๆจกๆๅฅฝ็ๆง่ฝใไธๆญคๅๆถ๏ผๆไปฌๆไพไบไธCodeShell้ ๅฅ็้จ็ฝฒๆนๆกไธIDEๆไปถ๏ผ่ฏทๅ่ไปฃ็ ๅบCodeShellใๅๆถ๏ผไธบไบๆนไพฟไธญๅฝ็จๆทไธ่ฝฝ๏ผๆไปฌๅจModelscopeๅWisemodelไธญไนไธไผ ไบๅฏนๅบ็ๆฌ๏ผๅฝๅ ็จๆทๅฏไปฅ่ฎฟ้ฎใๆฌไปๅบไธบCodeShell-7B้ข่ฎญ็ปๆจกๅไปๅบใ
CodeShell is a multi-language code LLM developed by the Knowledge Computing Lab of Peking University. CodeShell has 7 billion parameters and was trained on 500 billion tokens with a context window length of 8194. On authoritative code evaluation benchmarks (HumanEval and MBPP), CodeShell achieves the best performance of its scale. Meanwhile, we provide deployment solutions and IDE plugins that complement CodeShell. Please refer to the CodeShell code repository for more details. This repository is for the CodeShell-7B base model.
Main Characteristics of CodeShell
ๅผบๅคง็ๆง่ฝ๏ผCodelShellๅจHumanEvalๅMBPPไธ่พพๅฐไบ7Bไปฃ็ ๅบๅบงๅคงๆจกๅ็ๆไผๆง่ฝ
ๅฎๆด็ไฝ็ณป๏ผ้คไบไปฃ็ ๅคงๆจกๅ๏ผๅๆถๅผๆบIDE๏ผVS CodeไธJetBrains๏ผๆไปถ๏ผๅฝขๆๅผๆบ็ๅ จๆ ๆๆฏไฝ็ณป
่ฝป้ๅ้จ็ฝฒ๏ผๆฏๆๆฌๅฐC++้จ็ฝฒ๏ผๆไพ่ฝป้ๅฟซ้็ๆฌๅฐๅ่ฝฏไปถๅผๅๅฉๆ่งฃๅณๆนๆก
ๅ จ้ข็่ฏๆต๏ผๆไพๆฏๆๅฎๆด้กน็ฎไธไธๆใ่ฆ็ไปฃ็ ็ๆใไปฃ็ ็ผบ้ทๆฃๆตไธไฟฎๅคใๆต่ฏ็จไพ็ๆ็ญๅธธ่ง่ฝฏไปถๅผๅๆดปๅจ็ๅคไปปๅก่ฏๆตไฝ็ณป๏ผๅณๅฐๅผๆบ๏ผ
้ซๆ็่ฎญ็ป๏ผๅบไบ้ซๆ็ๆฐๆฎๆฒป็ไฝ็ณป๏ผCodeShellๅจๅฎๅ จๅทๅฏๅจๆ ๅตไธ๏ผๅช่ฎญ็ปไบไบๅไบฟTokenๅณ่ทๅพไบไผๅผ็ๆง่ฝ
Powerful Performance: CodeShell achieves optimal performance for a 7B code base model on HumanEval and MBPP.
Complete Ecosystem: In addition to the mega code model, open-source IDE plugins (for VS Code and JetBrains) are also available, forming a comprehensive open-source full-stack technology system.
Lightweight Deployment: Supports local C++ deployment, offering a lightweight and fast localized software development assistant solution.
Comprehensive Evaluation: Provides a multi-task evaluation system that supports full project context, covering code generation, code defect detection and repair, test case generation, and other common software development activities (to be open-sourced soon).
Efficient Training: Based on an efficient data governance system, CodeShell, even when starting from scratch, achieved outstanding performance with training on just 500 trillion tokens.
Quickstart
Code Generation
Codeshell ๆไพไบHugging Faceๆ ผๅผ็ๆจกๅ๏ผๅผๅ่ ๅฏไปฅ้่ฟไธๅไปฃ็ ๅ ่ฝฝๅนถไฝฟ็จใ
Codeshell offers a model in the Hugging Face format. Developers can load and use it with the following code.
import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("WisdomShell/CodeShell-7B", trust_remote_code=True)
model = AutoModelForCausalLM.from_pretrained("WisdomShell/CodeShell-7B", trust_remote_code=True).cuda()
inputs = tokenizer('def print_hello_world():', return_tensors='pt').cuda()
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))
Fill in the Moddle
CodeShell ๆฏๆFill-in-the-Middleๆจกๅผ๏ผไป่ๆดๅฅฝ็ๆฏๆ่ฝฏไปถๅผๅ่ฟ็จใ
CodeShell supports the Fill-in-the-Middle mode, thereby better facilitating the software development process.
input_text = "<fim_prefix>def print_hello_world():\n <fim_suffix>\n print('Hello world!')<fim_middle>"
inputs = tokenizer(input_text, return_tensors='pt').cuda()
outputs = model.generate(inputs)
print(tokenizer.decode(outputs[0]))
Model Details
Code Shellไฝฟ็จGPT-2ไฝไธบๅบ็กๆถๆ๏ผ้็จGrouped-Query AttentionใRoPE็ธๅฏนไฝ็ฝฎ็ผ็ ็ญๆๆฏใ
Code Shell uses GPT-2 as its foundational architecture and incorporates technologies such as Grouped-Query Attention and RoPE relative position encoding.
| Hyper-parameter | Value |
|---|---|
| n_layer | 42 |
| n_embd | 4096 |
| n_inner | 16384 |
| n_head | 32 |
| num_query_groups | 8 |
| seq-length | 8192 |
| vocab_size | 70144 |
Evaluation
ๆไปฌ้ๅไบ็ฎๅๆๆต่ก็ไธคไธชไปฃ็ ่ฏๆตๆฐๆฎ้๏ผHumanEvalไธMBPP๏ผๅฏนๆจกๅ่ฟ่ก่ฏไผฐ๏ผไธ็ฎๅๆๅ ่ฟ็ไธคไธช7bไปฃ็ ๅคงๆจกๅCodeLllamaไธStarcoder็ธๆฏ๏ผCodeshell ๅๅพไบๆไผ็ๆ็ปฉใๅ ทไฝ่ฏๆต็ปๆๅฆไธใ
We selected the two most popular code evaluation datasets currently available (HumanEval and MBPP) to assess the model. Compared to the two most advanced 7b LLM for code, CodeLllama and Starcoder, Codeshell achieved the best results. The specific evaluation results are as follows.
Pass@1
| ไปปๅก | CodeShell-7b | CodeLlama-7b | Starcoder-7b |
|---|---|---|---|
| humaneval | 34.32 | 29.44 | 27.80 |
| mbpp | 38.65 | 37.60 | 34.16 |
| multiple-js | 33.17 | 31.30 | 27.02 |
| multiple-java | 30.43 | 29.24 | 24.30 |
| multiple-cpp | 28.21 | 27.33 | 23.04 |
| multiple-swift | 24.30 | 25.32 | 15.70 |
| multiple-php | 30.87 | 25.96 | 22.11 |
| multiple-d | 8.85 | 11.60 | 8.08 |
| multiple-jl | 22.08 | 25.28 | 22.96 |
| multiple-lua | 22.39 | 30.50 | 22.92 |
| multiple-r | 20.52 | 18.57 | 14.29 |
| multiple-rkt | 17.20 | 12.55 | 10.43 |
| multiple-rs | 24.55 | 25.90 | 22.82 |
Statement
ๆไปฌ้้ๅฃฐๆ๏ผๆไปฌๅผๅๅข้ๅบไบCodeShellๆจกๅๅผๅไบๅบไบvscodeๅintellij็ๆบ่ฝ็ผ็ ๅฉๆๆไปถๅนถๅๅทฒๅผๆบใ้คๆญคไปฅๅค๏ผๆ ่ฎบๆฏ้ๅฏนiOSใAndroidใHarmonyOSใWeb๏ผ่ฟๆฏๅ ถไปไปปไฝๅนณๅฐ๏ผๆไปฌ็ๅผๅๅข้ๅๆชๅผๅไปปไฝๅบไบCodeShellๆจกๅ็ๅบ็จ็จๅบใๆไปฌๅผบ็ๆฆไฟๆๆ็จๆทไธ่ฆๅฉ็จCodeShellๆจกๅไปไบๅฑๅฎณๅฝๅฎถๅ็คพไผๅฎๅ จๆ่ฟๆณๆดปๅจใๅๆถ๏ผๆไปฌ่ฆๆฑ็จๆทไธ่ฆๅจๆช็ป้ๅฝ็ๅฎๅ จๅฎกๆฅๅๅคๆก็ไบ่็ฝๆๅกไธญไฝฟ็จCodeShellๆจกๅใๆไปฌๅธๆๆๆ็จๆท้ฝ่ฝ้ตๅฎ่ฟไธๅๅ๏ผไปฅ็กฎไฟๅจๅ่งๅๅๆณ็็ฏๅขไธๅๅฑ็งๆใ
ๅฐฝ็ฎกๆไปฌๅจ็กฎไฟๆจกๅ่ฎญ็ป่ฟ็จไธญไฝฟ็จๆฐๆฎๅ่งๆงๆน้ขๅทฒไปๅบๅทจๅคงๅชๅ๏ผไฝ็ฑไบๆจกๅๅๆฐๆฎ็ๅคๆๆง๏ผๅฏ่ฝไผๅบ็ฐ้พไปฅ้ขๆ็้ฎ้ขใๅ ๆญค๏ผๅฏนไบไฝฟ็จCodeShellๅผๆบๆจกๅๅฏผ่ด็ไปปไฝ้ฎ้ข๏ผๅ ๆฌไฝไธ้ไบๆฐๆฎๅฎๅ จ้ฎ้ขใๅ ฌๅ ฑ่่ฎบ้ฃ้ฉ๏ผๆๆจกๅ่ขซ่ฏฏ็จใๆปฅ็จใไผ ๆญๆไธๅฝๅฉ็จ็ญ้ฃ้ฉๅ้ฎ้ข๏ผๆไปฌๆฆไธ่ด่ดฃใ
We hereby declare that our development team has developed intelligent coding assistant plugins for vscode and intellij based on the CodeShell model, both of which have been open-sourced. Beyond this, whether for iOS, Android, HarmonyOS, Web, or any other platform, our development team has not developed any applications based on the CodeShell model. We strongly urge all users not to use the CodeShell model for activities that endanger national and social security or are illegal. At the same time, we request users not to use the CodeShell model in internet services that have not undergone proper security reviews and registration. We hope all users will adhere to this principle to ensure the development of technology in a compliant and legal environment.
Despite our significant efforts to ensure compliance in the data used during the model training process, unforeseen issues may arise due to the complexity of the models and data. Therefore, we are not responsible for any issues arising from the use of the open-sourced CodeShell model, including but not limited to data security issues, public opinion risks, or risks and problems related to the model being misused, abused, disseminated, or exploited improperly.
License
็คพๅบไฝฟ็จCodeShellๆจกๅ้่ฆ้ตๅพชCodeShellๆจกๅ่ฎธๅฏๅ่ฎฎๅApache 2.0 ่ฎธๅฏ่ฏใCodeShellๆจกๅๅ ่ฎธ็จไบๅไธ็จ้๏ผไฝๅฆๆๆจ่ฎกๅๅฐCodeShellๆจกๅๆๅ ถๆดพ็ไบงๅ็จไบๅไธ็จ้๏ผ้่ฆๆจ็กฎ่ฎคไธปไฝ็ฌฆๅไปฅไธๆกไปถ๏ผ
- ๅ ณ่ๆน็ๆๅกๆไบงๅ็ๆฏๆฅๅนณๅๆดป่ท็จๆทๆฐ๏ผDAU๏ผๅๅไธไธ่ฝ่ถ ่ฟ100ไธใ
- ๅ ณ่ๆนไธๅพๆฏ้ขๅไธชไบบ็จๆท็่ฝฏไปถๆๅกๆไพๅๆไบๆๅกๆไพๅใ
- ๅ ณ่ๆนไธๅญๅจๅฐ่ทๅพๆไบ็ๅไธ่ฎธๅฏ๏ผๅจๆช็ป่ฎธๅฏ็ๅๆไธๅฐๅ ถๅๆๆ็ปๅ ถไป็ฌฌไธๆน็ๅฏ่ฝๆงใ
ๅจๆปก่ถณไธ่ฟฐๆกไปถ็ๅๆไธ๏ผๆจ้่ฆ้่ฟๅcodeshell.opensource@gmail.comๅ้็ตๅญ้ฎไปถ๏ผๆไบคใCodeShellๆจกๅ่ฎธๅฏๅ่ฎฎใ่ฆๆฑ็็ณ่ฏทๆๆใ็ปๅฎกๆ ธ้่ฟๅ๏ผๅฐๆไบๆจไธไธชๅ จ็็ใ้ๆไป็ใไธๅฏ่ฝฌ่ฎฉ็ใไธๅฏๅๆๆ็ๅไธ็ๆ่ฎธๅฏใ
Community use of the CodeShell model requires adherence to the "CodeShell License Agreement" and the Apache 2.0 License. The CodeShell model is allowed for commercial use, but if you plan to use the CodeShell model or its derivatives for commercial purposes, you need to ensure that the entity meets the following conditions:
- The Daily Active Users (DAU) of your or your affiliate's service or product is less than 1 million.
- You and your affiliates must not be a software service provider or cloud service provider targeting individual users.
- You and your affiliates should not have the possibility of sub-licensing to other third parties without obtaining the commercial license granted.
Under the aforementioned conditions, you need to submit the application materials required by the "CodeShell License Agreement" by sending an email to codeshell.opensource@gmail.com. After approval, you will be granted a global, non-exclusive, non-transferable, non-sublicensable commercial copyright license.
- Downloads last month
- 160