ICLR
#Analysis#Pocket#NLP#LanguageModel#Chain-of-Thought
Issue Date: 2025-04-30 When More is Less: Understanding Chain-of-Thought Length in LLMs, Yuyang Wu+, ICLR25 CommentICLR 2025 Best Paper Runner Up Award元ポスト:https://x.com/yifeiwang77/status/1916873981979660436?s=46&t=Y6UuIHB0Lv0IpmFAjlc2-Q ... #Pocket#NLP#LanguageModel#KnowledgeEditing
Issue Date: 2025-04-30 AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models, Junfeng Fang+, ICLR25 Comment元ポスト:https://x.com/hillbig/status/1917343444810489925?s=46&t=Y6UuIHB0Lv0IpmFAjlc2-QOpenReview:https://openreview.net/forum?id=HvSytvg3JhMLPに新たな知識を直接注入 ... #Pocket#NLP#Transformer#Chain-of-Thought#In-ContextLearning#SSM (StateSpaceModel)
Issue Date: 2025-04-26 RNNs are not Transformers （Yet）: The Key Bottleneck on In-context Retrieval, Kaiyue Wen+, ICLR25 Comment元ポスト:https://x.com/yuma_1_or/status/1915968478735130713?s=46&t=Y6UuIHB0Lv0IpmFAjlc2-Q関連:#1210↑とはどういう関係があるだろうか？ ...
Issue Date: 2025-04-30 When More is Less: Understanding Chain-of-Thought Length in LLMs, Yuyang Wu+, ICLR25 CommentICLR 2025 Best Paper Runner Up Award元ポスト:https://x.com/yifeiwang77/status/1916873981979660436?s=46&t=Y6UuIHB0Lv0IpmFAjlc2-Q ... #Pocket#NLP#LanguageModel#KnowledgeEditing
Issue Date: 2025-04-30 AlphaEdit: Null-Space Constrained Knowledge Editing for Language Models, Junfeng Fang+, ICLR25 Comment元ポスト:https://x.com/hillbig/status/1917343444810489925?s=46&t=Y6UuIHB0Lv0IpmFAjlc2-QOpenReview:https://openreview.net/forum?id=HvSytvg3JhMLPに新たな知識を直接注入 ... #Pocket#NLP#Transformer#Chain-of-Thought#In-ContextLearning#SSM (StateSpaceModel)
Issue Date: 2025-04-26 RNNs are not Transformers （Yet）: The Key Bottleneck on In-context Retrieval, Kaiyue Wen+, ICLR25 Comment元ポスト:https://x.com/yuma_1_or/status/1915968478735130713?s=46&t=Y6UuIHB0Lv0IpmFAjlc2-Q関連:#1210↑とはどういう関係があるだろうか？ ...
#ComputerVision#Pocket#NLP#Dataset#LanguageModel#Evaluation#MulltiModal#x-Use
Issue Date: 2025-04-18 AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents, Christopher Rawles+, ICLR25 CommentAndroid環境でのPhone Useのベンチマーク ... #Analysis#MachineLearning#Pocket#NLP#LanguageModel#Alignment#Hallucination#DPO#Repetition
Issue Date: 2025-04-18 Learning Dynamics of LLM Finetuning, Yi Ren+, ICLR25 Comment元ポスト:https://x.com/joshuarenyi/status/1913033476275925414?s=46&t=Y6UuIHB0Lv0IpmFAjlc2-Q解説ポスト:https://x.com/hillbig/status/1917189793588613299?s=46&t=Y ... #Pocket#NLP#RewardHacking
Issue Date: 2025-04-06 CREAM: Consistency Regularized Self-Rewarding Language Models, Zhaoyang Wang+, ICLR25 Comment#1212を改善した研究OpenReview:https://openreview.net/forum?id=Vf6RDObyEFこの方向性の研究はおもしろい ... #Pocket#NLP#LanguageModel#Attention#AttentionSinks
Issue Date: 2025-04-05 When Attention Sink Emerges in Language Models: An Empirical View, Xiangming Gu+, ICLR25 CommentSink Rateと呼ばれる、全てのheadのFirst Tokenに対するattention scoreのうち（layer l * head h個存在する）、どの程度の割合のスコアが閾値を上回っているかを表す指標を提案#1860の先行研究 ... #Analysis#Pretraining#Pocket#NLP#LanguageModel#Supervised-FineTuning (SFT)
Issue Date: 2025-03-27 Overtrained Language Models Are Harder to Fine-Tune, Jacob Mitchell Springer+, ICLR25 Comment著者によるポスト:https://x.com/jacspringer/status/1904960783341023521?s=46&t=Y6UuIHB0Lv0IpmFAjlc2-Q事前学習のトークン数を増やすとモデルのsensitivityが増し、post-trainingでのパフォーマンスの劣化 ... #Pocket#NLP#LanguageModel#Alignment#DPO#PostTraining#Diversity
Issue Date: 2025-02-01 Diverse Preference Optimization, Jack Lanchantin+, ICLR25 Comment元ポスト:https://x.com/jaseweston/status/1885399530419450257?s=46&t=Y6UuIHB0Lv0IpmFAjlc2-QOpenReview: https://openreview.net/forum?id=pOq9vDIYevDPOと同じ最適化方 ... #Pocket#Attention#AttentionSinks
Issue Date: 2025-04-05 Efficient Streaming Language Models with Attention Sinks, Guangxuan Xiao+, ICLR24 CommentAttention Sinkという用語を提言した研究#1860の先行研究 ... #Pocket#NLP#Dataset#LanguageModel#LLMAgent
Issue Date: 2025-04-02 WebArena: A Realistic Web Environment for Building Autonomous Agents, Shuyan Zhou+, ICLR24 CommentWebにおけるさまざまなrealisticなタスクを評価するためのベンチマーク<img width="855" alt="Image" src="https://github.com/user-attachments/assets/693f0512-d279-416e-acbe-d146a0b74 ... #NLP#LanguageModel#DataGeneration
Issue Date: 2023-04-25 WizardLM: Empowering Large Language Models to Follow Complex Instructions, Xu+, Microsoft_Peking University, ICLR24 Commentinstruction trainingは大きな成功を収めているが、人間がそれらのデータを作成するのはコストがかかる。また、そもそも複雑なinstructionを人間が作成するのは苦労する。そこで、LLMに自動的に作成させる手法を提案している（これはself instructと一緒）。データを生成す ... #Pretraining#Pocket#NLP#LanguageModel
Issue Date: 2024-09-26 UL2: Unifying Language Learning Paradigms, Yi Tay+, N_A, ICLR23 CommentOpenReview:https://openreview.net/forum?id=6ruVLB727MC ... #NeuralNetwork#NLP#LanguageModel#Chain-of-Thought
Issue Date: 2023-04-27 Automatic Chain of Thought Prompting in Large Language Models, Zhang+, Shanghai Jiao Tong University, ICLR23 CommentLLMによるreasoning chainが人間が作成したものよりも優れていることを示しているとのこと #532 よりclusteringベースな手法を利用することにより、誤りを含む例が単一のクラスタにまとめられうことを示し、これにより過剰な誤ったデモンストレーションが軽減されることを示した。手法の ... #Pocket#NLP#LanguageModel#PEFT(Adaptor/LoRA)#PostTraining#Admin'sPick
Issue Date: 2025-05-12 LoRA: Low-Rank Adaptation of Large Language Models, Edward J. Hu+, ICLR22 CommentOpenrReview:https://openreview.net/forum?id=nZeVKeeFYf9LoRAもなんやかんやメモってなかったので追加。事前学習済みのLinear Layerをfreezeして、freezeしたLinear Layerと対応する低ランクの行列A,Bを別途定義し、 ... #Pretraining#Pocket
Issue Date: 2025-01-06 Towards Continual Knowledge Learning of Language Models, Joel Jang+, ICLR22 #NeuralNetwork#MachineLearning#Grokking
Issue Date: 2023-04-25 GROKKING: GENERALIZATION BEYOND OVERFIT- TING ON SMALL ALGORITHMIC DATASETS, Power+, ICLR21 Workshop Comment学習後すぐに学習データをmemorizeして、汎化能力が無くなったと思いきや、10^3ステップ後に突然汎化するという現象（Grokking）を報告  Samplingを提案した研究 ... #DocumentSummarization#Supervised#Pocket#NLP#Abstractive
Issue Date: 2017-12-31 A Deep Reinforced Model for Abstractive Summarization, Paulus+（with Socher）, ICLR18 #NeuralNetwork#Pocket#NLP#MoE(Mixture-of-Experts)
Issue Date: 2025-04-29 Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, Noam Shazeer+, ICLR17 CommentMixture-of-Experts (MoE) Layerを提案した研究 ... #NeuralNetwork#Sentence#Embeddings#NLP#Admin'sPick
Issue Date: 2017-12-28 A structured self-attentive sentence embedding, Li+ （Bengio group）, ICLR17 CommentOpenReview:https://openreview.net/forum?id=BJC_jUqxe ... #RecommenderSystems#SessionBased#SequentialRecommendation#Admin'sPick
Issue Date: 2019-08-02 SESSION-BASED RECOMMENDATIONS WITH RECURRENT NEURAL NETWORKS, Hidasi+, ICLR16 CommentRNNを利用したsequential recommendation (session-based recommendation)の先駆け的論文。日本語解説: https://qiita.com/tatamiya/items/46e278a808a51893deac ... #NeuralNetwork#MachineTranslation#Pocket#NLP#Attention#Admin'sPick
Issue Date: 2025-05-12 Neural Machine Translation by Jointly Learning to Align and Translate, Dzmitry Bahdanau+, ICLR15 Comment(Cross-)Attentionを初めて提案した研究。メモってなかったので今更ながら追加。Attentionはここからはじまった（と認識している） ...
Issue Date: 2025-04-18 AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents, Christopher Rawles+, ICLR25 CommentAndroid環境でのPhone Useのベンチマーク ... #Analysis#MachineLearning#Pocket#NLP#LanguageModel#Alignment#Hallucination#DPO#Repetition
Issue Date: 2025-04-18 Learning Dynamics of LLM Finetuning, Yi Ren+, ICLR25 Comment元ポスト:https://x.com/joshuarenyi/status/1913033476275925414?s=46&t=Y6UuIHB0Lv0IpmFAjlc2-Q解説ポスト:https://x.com/hillbig/status/1917189793588613299?s=46&t=Y ... #Pocket#NLP#RewardHacking
Issue Date: 2025-04-06 CREAM: Consistency Regularized Self-Rewarding Language Models, Zhaoyang Wang+, ICLR25 Comment#1212を改善した研究OpenReview:https://openreview.net/forum?id=Vf6RDObyEFこの方向性の研究はおもしろい ... #Pocket#NLP#LanguageModel#Attention#AttentionSinks
Issue Date: 2025-04-05 When Attention Sink Emerges in Language Models: An Empirical View, Xiangming Gu+, ICLR25 CommentSink Rateと呼ばれる、全てのheadのFirst Tokenに対するattention scoreのうち（layer l * head h個存在する）、どの程度の割合のスコアが閾値を上回っているかを表す指標を提案#1860の先行研究 ... #Analysis#Pretraining#Pocket#NLP#LanguageModel#Supervised-FineTuning (SFT)
Issue Date: 2025-03-27 Overtrained Language Models Are Harder to Fine-Tune, Jacob Mitchell Springer+, ICLR25 Comment著者によるポスト:https://x.com/jacspringer/status/1904960783341023521?s=46&t=Y6UuIHB0Lv0IpmFAjlc2-Q事前学習のトークン数を増やすとモデルのsensitivityが増し、post-trainingでのパフォーマンスの劣化 ... #Pocket#NLP#LanguageModel#Alignment#DPO#PostTraining#Diversity
Issue Date: 2025-02-01 Diverse Preference Optimization, Jack Lanchantin+, ICLR25 Comment元ポスト:https://x.com/jaseweston/status/1885399530419450257?s=46&t=Y6UuIHB0Lv0IpmFAjlc2-QOpenReview: https://openreview.net/forum?id=pOq9vDIYevDPOと同じ最適化方 ... #Pocket#Attention#AttentionSinks
Issue Date: 2025-04-05 Efficient Streaming Language Models with Attention Sinks, Guangxuan Xiao+, ICLR24 CommentAttention Sinkという用語を提言した研究#1860の先行研究 ... #Pocket#NLP#Dataset#LanguageModel#LLMAgent
Issue Date: 2025-04-02 WebArena: A Realistic Web Environment for Building Autonomous Agents, Shuyan Zhou+, ICLR24 CommentWebにおけるさまざまなrealisticなタスクを評価するためのベンチマーク<img width="855" alt="Image" src="https://github.com/user-attachments/assets/693f0512-d279-416e-acbe-d146a0b74 ... #NLP#LanguageModel#DataGeneration
Issue Date: 2023-04-25 WizardLM: Empowering Large Language Models to Follow Complex Instructions, Xu+, Microsoft_Peking University, ICLR24 Commentinstruction trainingは大きな成功を収めているが、人間がそれらのデータを作成するのはコストがかかる。また、そもそも複雑なinstructionを人間が作成するのは苦労する。そこで、LLMに自動的に作成させる手法を提案している（これはself instructと一緒）。データを生成す ... #Pretraining#Pocket#NLP#LanguageModel
Issue Date: 2024-09-26 UL2: Unifying Language Learning Paradigms, Yi Tay+, N_A, ICLR23 CommentOpenReview:https://openreview.net/forum?id=6ruVLB727MC ... #NeuralNetwork#NLP#LanguageModel#Chain-of-Thought
Issue Date: 2023-04-27 Automatic Chain of Thought Prompting in Large Language Models, Zhang+, Shanghai Jiao Tong University, ICLR23 CommentLLMによるreasoning chainが人間が作成したものよりも優れていることを示しているとのこと #532 よりclusteringベースな手法を利用することにより、誤りを含む例が単一のクラスタにまとめられうことを示し、これにより過剰な誤ったデモンストレーションが軽減されることを示した。手法の ... #Pocket#NLP#LanguageModel#PEFT(Adaptor/LoRA)#PostTraining#Admin'sPick
Issue Date: 2025-05-12 LoRA: Low-Rank Adaptation of Large Language Models, Edward J. Hu+, ICLR22 CommentOpenrReview:https://openreview.net/forum?id=nZeVKeeFYf9LoRAもなんやかんやメモってなかったので追加。事前学習済みのLinear Layerをfreezeして、freezeしたLinear Layerと対応する低ランクの行列A,Bを別途定義し、 ... #Pretraining#Pocket
Issue Date: 2025-01-06 Towards Continual Knowledge Learning of Language Models, Joel Jang+, ICLR22 #NeuralNetwork#MachineLearning#Grokking
Issue Date: 2023-04-25 GROKKING: GENERALIZATION BEYOND OVERFIT- TING ON SMALL ALGORITHMIC DATASETS, Power+, ICLR21 Workshop Comment学習後すぐに学習データをmemorizeして、汎化能力が無くなったと思いきや、10^3ステップ後に突然汎化するという現象（Grokking）を報告  Samplingを提案した研究 ... #DocumentSummarization#Supervised#Pocket#NLP#Abstractive
Issue Date: 2017-12-31 A Deep Reinforced Model for Abstractive Summarization, Paulus+（with Socher）, ICLR18 #NeuralNetwork#Pocket#NLP#MoE(Mixture-of-Experts)
Issue Date: 2025-04-29 Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer, Noam Shazeer+, ICLR17 CommentMixture-of-Experts (MoE) Layerを提案した研究 ... #NeuralNetwork#Sentence#Embeddings#NLP#Admin'sPick
Issue Date: 2017-12-28 A structured self-attentive sentence embedding, Li+ （Bengio group）, ICLR17 CommentOpenReview:https://openreview.net/forum?id=BJC_jUqxe ... #RecommenderSystems#SessionBased#SequentialRecommendation#Admin'sPick
Issue Date: 2019-08-02 SESSION-BASED RECOMMENDATIONS WITH RECURRENT NEURAL NETWORKS, Hidasi+, ICLR16 CommentRNNを利用したsequential recommendation (session-based recommendation)の先駆け的論文。日本語解説: https://qiita.com/tatamiya/items/46e278a808a51893deac ... #NeuralNetwork#MachineTranslation#Pocket#NLP#Attention#Admin'sPick
Issue Date: 2025-05-12 Neural Machine Translation by Jointly Learning to Align and Translate, Dzmitry Bahdanau+, ICLR15 Comment(Cross-)Attentionを初めて提案した研究。メモってなかったので今更ながら追加。Attentionはここからはじまった（と認識している） ...