LLM

[testpppppp]

行业角度看LLM

通向AGI之路：大型语言模型（LLM）技术精要

大模型有哪些

https://zhuanlan.zhihu.com/p/611403556

模型结构

为什么现在的LLM都是Decoder only的架构？

lowrank角度

如何训练

Ladder Side-Tuning：预训练模型的“过墙梯”
LoRA: Low-Rank Adaptation of Large Language Models 简读
https://huggingface.co/blog/peft
https://github.com/tloen/alpaca-lora
如何评价 LLaMA 模型泄露？ - 苏洋的回答 - 知乎
https://www.zhihu.com/question/587479829/answer/2925378135

如何部署推理

量化

Pytorch Lightning 完全攻略
https://github.com/Shivanandroy/simpleT5

资源需要多少

参考

t5 finetune

https://www.kaggle.com/code/evilmage93/t5-finetuning-on-sentiment-classification
https://discuss.huggingface.co/t/how-to-fine-tune-t5-base-model/8478
https://shivanandroy.com/fine-tune-t5-transformer-with-pytorch/
https://www.kaggle.com/code/nulldata/training-t5-models-made-easy-with-simplet5/

[testpppppp]

候选大模型

https://github.com/LianjiaTech/BELLE 开源中文对话大模型
https://github.com/THUDM/ChatGLM-6B
https://abuqader.substack.com/p/releasing-alpaca-30b
https://huggingface.co/EleutherAI/gpt-j-6B

[testpppppp]

T5 finetune

https://www.kaggle.com/code/rizdelhi/fine-tuning-t5-on-text-summarization-task/notebook
https://pytorch.org/text/0.14.0/tutorials/t5_demo.html#generate-summaries

https://tsmatz.wordpress.com/2022/11/25/huggingface-japanese-summarization/
https://github.com/Shivanandroy/T5-Finetuning-PyTorch
https://www.philschmid.de/fine-tune-flan-t5
https://github.com/abhimishra91/transformers-tutorials/blob/master/transformers_summarization_wandb.ipynb

[testpppppp]

LLM leadboard
https://accubits.com/large-language-models-leaderboard/

[testpppppp]

how to debug

debug on small model, or create a demo model
Then you can inspect the model structure and run very fast to debug dmodel.model.embed_tokens

from transformers import LlamaModel, LlamaConfig,LlamaForCausalLM
configuration = LlamaConfig(hidden_size=64,intermediate_size=128,num_hidden_layers=3)
model = LlamaForCausalLM(configuration)
model.save_pretrained('exp_model_size/llama_demo')
tokenizer = LlamaTokenizer.from_pretrained('decapoda-research/llama-7b-hf')
tokenizer.save_pretrained('./exp_model_size/llama_demo')

dmodel = LlamaForCausalLM.from_pretrained('exp_model_size/llama_demo')

tokenizer

in stanford code, add special token by pad and then unk, if you reorder, then fail(this line https://github.com/tatsu-lab/stanford_alpaca/blob/aa65c492bb788e144712daab42bc5d11c2761591/train.py#L208) . why?

this line https://github.com/huggingface/transformers/blob/656e869a4523f6a0ce90b3aacbb05cc8fb5794bb/src/transformers/tokenization_utils.py#L421

if unk and then pad, has already change the unk word to , so the condition don't meet, so don't resize embed.

[testpppppp]

how to evaluate the FLOPs and params

FNN 2IO

https://github.com/mli/transformers-benchmarks/blob/main/micro_bench.ipynb
https://stackoverflow.com/questions/64485777/how-is-the-number-of-parameters-be-calculated-in-bert-model
https://www.jianshu.com/p/677fc8be5e14

b,s,h,n_layer,cross_attention = [1,512,4096,32,True]
mm = 4
ffn = 4*b*s*h*mm*h / 1e12  # TFLOPS for the Feed-Forward Network, 2 layer of FFN,mm*h,
atten = (4*b*h*s*s + 8*b*s*h*h) / 1e12  # TFLOPS for attention            
forward = (ffn + (2 if cross_attention else 1) * atten)*n_layer
forward,ffn/atten

[testpppppp]

decoder-only ⚠️

inference: padding left
don't confuse the role of attention_mask (let token own different perceptions to other tokens, not ignore when decode next word)
https://blog.csdn.net/qq_51750957/article/details/128856264

[testpppppp]

deploy

meta-llama/llama#79
https://news.ycombinator.com/item?id=35029766
ggml-org/llama.cpp#39 #Hows the inference speed and mem usage?
https://github.com/tloen/alpaca-lora/blob/main/generate.py

https://docs.google.com/spreadsheets/d/1EsRZcsvbITBCfb5N1toir5huomZxIq4lDNwQW3NfPRE/edit#gid=0 #LLama Memory Requirements v2023.03.03

tatsu-lab/stanford_alpaca#48 #How to inference after finetuning ?

[testpppppp]

apply

大模型指导标签体系设计，训练1/2epoch然后看结果，更合理设计标签。
新的类似任务和现在的兼容问题扩展预测。

visualization 不归属BI/algo，职责划分方式

新加词表，预训练只改embed，再进行finetune得到最后模型。
ymcui/Chinese-LLaMA-Alpaca#43
ymcui/Chinese-LLaMA-Alpaca#128
ymcui/Chinese-LLaMA-Alpaca#10

[testpppppp]

wiki数据使用

https://dumps.wikimedia.your.org/backup-index.html
https://dumps.wikimedia.org/backup-index.html
https://github.com/NTT123/viwik18

[testpppppp]

LORA
huggingface/peft#262