A community-driven registry for Claude, Cursor, Windsurf, Cline & more. Not affiliated with Anthropic.
Are you the author? Sign in to claim
Awesome LLM Papers and repos on very comprehensive topics.
We provide awesome papers and repos on very comprehensive topics as follows.
CoT / VLM / Quantization / Grounding / Text2IMG&VID / Prompt Engineering / Prompt Tuning / Reasoning / Robot / Agent / Planning / Reinforcement-Learning / Feedback / In-Context-Learning / Few-Shot / Zero-Shot / Instruction Tuning / PEFT / RLHF / RAG / Embodied / VQA / Hallucination / Diffusion / Scaling / Context-Window / WorldModel / Memory / Zero-Shot / RoPE / Speech / Perception / Survey / Segmentation / Learge Action Model / Foundation / RoPE / LoRA / PPO / DPO
We strongly recommend checking our Notion table for an interactive experience.
| Category | Title | Links | Date |
|---|---|---|---|
| Zero-shot | Can Foundation Models Perform Zero-Shot Task Speci fication For Robot Manipulation? | ||
| World-model | Leveraging Pre-trained Large Language Models to Co nstruct and Utilize World Models for Model-based Task Planning | ArXiv | 2023/05/07 |
| World-model | Learning and Leveraging World Models in Visual Rep resentation Learning | ||
| World-model | Language Models Meet World Models | ArXiv | |
| World-model | Learning to Model the World with Language | ArXiv | |
| World-model | Diffusion World Model | ArXiv | |
| World-model | Learning to Model the World with Language | ArXiv | |
| VisualPrompt | Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models | ArXiv | |
| VisualPrompt | Making Large Multimodal Models Understand Arbitrar y Visual Prompts | ArXiv | |
| VisualPrompt | What does CLIP know about a red circle? Visual pro mpt engineering for VLMs | ArXiv | |
| VisualPrompt | MOKA: Open-Vocabulary Robotic Manipulation through Mark-Based Visual Prompting | ArXiv | |
| VisualPrompt | SoM : Set-of-Mark PromptingUnleashes Extraordinary Visual Grounding in GPT-4V | ArXiv | |
| VisualPrompt | Set-of-Mark Prompting Unleashes Extraordinary Visu al Grounding in GPT-4V | ArXiv, GitHub | |
| Video | MA-LMM: Memory-Augmented Large Multimodal Model fo r Long-Term Video Understanding | ArXiv | |
| ViFM, Video | InternVideo2: Scaling Video Foundation Models for Multimodal Video Understanding | ArXiv, GitHub | |
| VLM, World-model | Large World Model | ArXiv | |
| VLM, VQA | CogVLM: Visual Expert for Pretrained Language Mode ls | ArXiv | 2023/11/06 |
| VLM, VQA | Chameleon: Plug-and-Play Compositional Reasoning w ith Large Language Models | ArXiv | 2023/04/19 |
| VLM, VQA | DeepSeek-VL: Towards Real-World Vision-Language Un derstanding01 | ||
| VLM | PaLM: Scaling Language Modeling with Pathways | ArXiv | 2022/04/05 |
| VLM | ScreenAI: A Vision-Language Model for UI and Infog raphics Understanding | ||
| VLM | MoE-LLaVA: Mixture of Experts for Large Vision-Lan guage Models | ArXiv, GitHub | |
| VLM | LLaVA-NeXT: Improved reasoning, OCR, and world kno wledge | GitHub | |
| VLM | Mini-Gemini: Mining the Potential of Multi-modalit y Vision Language Models | ArXiv | |
| Text-to-Image, World-model | World Model on Million-Length Video And Language W ith RingAttention | ArXiv | |
| Tex2Img | Be Yourself: Bounded Attention for Multi-Subject T ext-to-Image Generation | ArXiv | |
| Temporal | Explorative Inbetweening of Time and Space | ArXiv | |
| Survey, Video | Video Understanding with Large Language Models: A Survey | ArXiv | |
| Survey, VLM | MM-LLMs: Recent Advances in MultiModal Large Langu age Models | ||
| Survey, Training | Understanding LLMs: A Comprehensive Overview from Training to Inference | ArXiv | |
| Survey, TimeSeries | Large Models for Time Series and Spatio-Temporal D ata: A Survey and Outlook | ||
| Survey | Efficient Large Language Models: A Survey | ArXiv, GitHub | |
| Sora, Text-to-Video | Sora: A Review on Background, Technology, Limitati ons, and Opportunities of Large Vision Models | ||
| Sora, Text-to-Video | Mora: Enabling Generalist Video Generation via A M ulti-Agent Framework | ArXiv | |
| Segmentation | LISA: Reasoning Segmentation via Large Language Mo del | ArXiv | |
| Segmentation | GRES: Generalized Referring Expression Segmentatio n | ||
| Segmentation | Generalized Decoding for Pixel, Image, and Languag e | ArXiv | |
| Segmentation | SEEM: Segment Everything Everywhere All at Once | ArXiv, GitHub | |
| Segmentation | SegGPT: Segmenting Everything In Context | ArXiv | |
| Segmentation | Grounded SAM: Assembling Open-World Models for Div erse Visual Tasks | ArXiv | |
| Scaling | Leave No Context Behind: Efficient Infinite Contex t Transformers with Infini-attention | ArXiv | |
| SLM, Scaling | Textbooks Are All You Need | ArXiv | |
| Robot, Zero-shot | BC-Z: Zero-Shot Task Generalization with Robotic I mitation Learning | ArXiv | |
| Robot, Zero-shot | Universal Manipulation Interface: In-The-Wild Robo t Teaching Without In-The-Wild Robots | ArXiv | |
| Robot, Zero-shot | Mirage: Cross-Embodiment Zero-Shot Policy Transfer with Cross-Painting | ArXiv | |
| Robot, Task-Decompose, Zero-shot | Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents | ArXiv | 2022/01/18 |
| Robot, Task-Decompose | SayPlan: Grounding Large Language Models using 3D Scene Graphs for Scalable Robot Task Planning | ArXiv | 2023/07/12 |
| Robot, TAMP | LLM3:Large Language Model-based Task and Motion Pl anning with Motion Failure Reasoning | ArXiv | |
| Robot, TAMP | Task and Motion Planning with Large Language Model s for Object Rearrangement | ArXiv | |
| Robot, Survey | Toward General-Purpose Robots via Foundation Model s: A Survey and Meta-Analysis | ArXiv | 2023/12/14 |
| Robot, Survey | Language-conditioned Learning for Robotic Manipula tion: A Survey | ArXiv | 2023/12/17 |
| Robot, Survey | Robot Learning in the Era of Foundation Models: A Survey | ArXiv | 2023/11/24 |
| Robot, Survey | Real-World Robot Applications of Foundation Models : A Review | ArXiv | |
| Robot | OK-Robot: What Really Matters in Integrating Open- Knowledge Models for Robotics | ArXiv | |
| Robot | RoCo: Dialectic Multi-Robot Collaboration with Lar ge Language Models | ArXiv | |
| Robot | Interactive Language: Talking to Robots in Real Ti me | ArXiv | |
| Robot | Reflexion: Language Agents with Verbal Reinforceme nt Learning | ArXiv | 2023/03/20 |
| Robot | Generative Expressive Robot Behaviors using Large Language Models | ArXiv | |
| Robot | RoboCat: A Self-Improving Generalist Agent for Rob otic Manipulation | ||
| Robot | Introspective Tips: Large Language Model for In-Co ntext Decision Making | ArXiv | |
| Robot | PIVOT: Iterative Visual Prompting Elicits Actionab le Knowledge for VLMs | ArXiv | |
| Robot | OCI-Robotics: Object-Centric Instruction Augmentat ion for Robotic Manipulation | ArXiv | |
| Robot | DeliGrasp: Inferring Object Mass, Friction, and Co mpliance with LLMs for Adaptive and Minimally Deforming Grasp Policies | ArXiv | |
| Robot | VoxPoser: Composable 3D Value Maps for Robotic Man ipulation with Language Models | ArXiv | |
| Robot | Creative Robot Tool Use with Large Language Models | ArXiv | |
| Robot | AutoTAMP: Autoregressive Task and Motion Planning with LLMs as Translators and Checkers | ArXiv | |
| RoPE | RoFormer: Enhanced Transformer with Rotary Positio n Embedding | ArXiv | |
| Resource | [Resource] Paperswithcode | ArXiv | |
| Resource | [Resource] huggingface | ArXiv | |
| Resource | [Resource] dailyarxiv | ArXiv | |
| Resource | [Resource] Connectedpapers | ArXiv | |
| Resource | [Resource] Semanticscholar | ArXiv | |
| Resource | [Resource] AlphaSignal | ArXiv | |
| Resource | [Resource] arxiv-sanity | ArXiv | |
| Reinforcement-Learning, VIMA | FoMo Rewards: Can we cast foundation models as rew ard functions? | ArXiv | |
| Reinforcement-Learning, Robot | Towards A Unified Agent with Foundation Models | ArXiv | |
| Reinforcement-Learning | Large Language Models Are Semi-Parametric Reinforc ement Learning Agents | ArXiv | |
| Reinforcement-Learning | RLang: A Declarative Language for Describing Parti al World Knowledge to Reinforcement Learning Agents | ArXiv | |
| Reasoning, Zero-shot | Large Language Models are Zero-Shot Reasoners | ArXiv | |
| Reasoning, VLM, VQA | MM-REACT: Prompting ChatGPT for Multimodal Reasoni ng and Action | ArXiv | 2023/03/20 |
| Reasoning, Table | Large Language Models are few(1)-shot Table Reason ers | ArXiv | |
| Reasoning, Symbolic | Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoning | ArXiv | |
| Reasoning, Survey | Reasoning with Language Model Prompting: A Survey | ArXiv | |
| Reasoning, Robot | AlphaBlock: Embodied Finetuning for Vision-Languag e Reasoning in Robot Manipulation | ArXiv | |
| Reasoning, Reward | LET’S REWARD STEP BY STEP: STEP-LEVEL REWARD MODEL AS THE NAVIGATORS FOR REASONING | ArXiv | |
| Reasoning, Reinforcement-Learning | ReFT: Reasoning with Reinforced Fine-Tuning | ||
| Reasoning | Selection-Inference: Exploiting Large Language Mod els for Interpretable Logical Reasoning | ArXiv | |
| Reasoning | ReConcile: Round-Table Conference Improves Reasoni ng via Consensus among Diverse LLMs. | ArXiv | |
| Reasoning | Self-Discover: Large Language Models Self-Compose Reasoning Structures | ArXiv | |
| Reasoning | Chain-of-Thought Reasoning Without Prompting | ArXiv | |
| Reasoning | Contrastive Chain-of-Thought Prompting | ArXiv | |
| Reasoning | Rephrase and Respond(RaR) | ||
| Reasoning | Take a Step Back: Evoking Reasoning via Abstractio n in Large Language Models | ArXiv | |
| Reasoning | STaR: Bootstrapping Reasoning With Reasoning | ArXiv | 2022/05/28 |
| Reasoning | The Impact of Reasoning Step Length on Large Langu age Models | ArXiv | |
| Reasoning | Beyond Natural Language: LLMs Leveraging Alternati ve Formats for Enhanced Reasoning and Communication | ArXiv | |
| Reasoning | Large Language Models as General Pattern Machines | ArXiv | |
| RLHF, Reinforcement-Learning, Survey | A Survey of Reinforcement Learning from Human Feed back | ||
| RLHF | Secrets of RLHF in Large Language Models Part II: Reward Modeling | ArXiv | |
| RAG, Temporal Logics | FreshLLMs: Refreshing Large Language Models with S earch Engine Augmentation | ArXiv | |
| RAG, Survey | Large Language Models for Information Retrieval: A Survey | ||
| RAG, Survey | Retrieval-Augmented Generation for Large Language | ||
| RAG, Survey | Retrieval-Augmented Generation for Large Language Models: A Survey | ArXiv | |
| RAG | Training Language Models with Memory Augmentation | ||
| RAG | Self-RAG: Learning to Retrieve, Generate, and Crit ique through Self-Reflection | ||
| RAG | RAG-Fusion: a New Take on Retrieval-Augmented Gene ration | ArXiv | |
| RAG | RAFT: Adapting Language Model to Domain Specific R AG | ArXiv | |
| RAG | Adaptive-RAG: Learning to Adapt Retrieval-Augmente d Large Language Models through Question Complexity | ArXiv | |
| RAG | RAG vs Fine-tuning: Pipelines, Tradeoffs, and a Ca se Study on Agriculture | ArXiv | |
| RAG | Fine-Tuning or Retrieval? Comparing Knowledge Inje ction in LLMs | ArXiv | |
| Quantization, Scaling | SliceGPT: Compress Large Language Models by Deleti ng Rows and Columns | ArXiv | |
| Prompting, Survey | A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applications | ArXiv | |
| Prompting, Robot, Zero-shot | Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning | ArXiv | |
| Prompting | Contrastive Chain-of-Thought Prompting | ArXiv | |
| PersonalCitation, Robot | Text2Motion: From Natural Language Instructions to Feasible Plans | ArXiv | |
| Perception, Video, Vision | CLIP4Clip: An Empirical Study of CLIP for End to E nd Video Clip Retrieval | ArXiv | |
| Perception, Task-Decompose | DoReMi: Grounding Language Model by Detecting and Recovering from Plan-Execution Misalignment | ArXiv | 2023/07/01 |
| Perception, Robot, Segmentation | Language Segment-Anything | ||
| Perception, Robot | LiDAR-LLM: Exploring the Potential of Large Langua ge Models for 3D LiDAR Understanding | ArXiv | 2023/12/21 |
| Perception, Reasoning, Robot | Reasoning Grasping via Multimodal Large Language M odel | ArXiv | |
| Perception, Reasoning | DetGPT: Detect What You Need via Reasoning | ArXiv | |
| Perception, Reasoning | Lenna: Language Enhanced Reasoning Detection Assis tant | ArXiv | |
| Perception | Simple Open-Vocabulary Object Detection with Visio n Transformers | ArXiv | 2022/05/12 |
| Perception | Grounded Language-Image Pre-training | ArXiv | 2021/12/07 |
| Perception | Grounding DINO: Marrying DINO with Grounded Pre-Tr aining for Open-Set Object Detection | ArXiv | 2023/03/09 |
| Perception | PointCLIP: Point Cloud Understanding by CLIP | ArXiv | 2021/12/04 |
| Perception | DINO: DETR with Improved DeNoising Anchor Boxes fo r End-to-End Object Detection | ArXiv | |
| Perception | Recognize Anything: A Strong Image Tagging Model | ArXiv | |
| Perception | Simple Open-Vocabulary Object Detection with Visio n Transformers | ArXiv | |
| Perception | Sigmoid Loss for Language Image Pre-Training | ArXiv | |
| Package | LlamaIndex | GitHub | |
| Package | LangChain | GitHub | |
| Package | h2oGPT | GitHub | |
| Package | Dify | GitHub | |
| Package | Alpaca-LoRA | GitHub | |
| Package | Promptlayer | GitHub | |
| Package | unsloth | GitHub | |
| Package | Instructor: Structured LLM Outputs | GitHub | |
| PRM | Let's Verify Step by Step | ArXiv | |
| PRM | Let's reward step by step: Step-Level reward model as the Navigators for Reasoning | ArXiv | |
| PPO, RLHF, Reinforcement-Learning | Secrets of RLHF in Large Language Models Part I: P PO | ArXiv | 2024/02/01 |
| Open-source, VLM | OpenFlamingo: An Open-Source Framework for Trainin g Large Autoregressive Vision-Language Models | ArXiv | 2023/08/02 |
| Open-source, SLM | RecurrentGemma: Moving Past Transformers for Effic ient Open Language Models | ArXiv | |
| Open-source, Perception | Grounding DINO: Marrying DINO with Grounded Pre-Tr aining for Open-Set Object Detection | ArXiv | |
| Open-source | Gemma: Introducing new state-of-the-art open model s | ArXiv | |
| Open-source | Mistral 7B | ArXiv | |
| Open-source | Qwen Technical Report | ArXiv | |
| Navigation, Reasoning, Vision | NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models | ArXiv | |
| Natural-Language-as-Polices, Robot | RT-H: Action Hierarchies Using Language | ArXiv | |
| Multimodal, Robot, VLM | Open-World Object Manipulation using Pre-trained V ision-Language Models | ArXiv | 2023/03/02 |
| Multimodal, Robot | MOMA-Force: Visual-Force Imitation for Real-World Mobile Manipulation | ArXiv | 2023/08/07 |
| Multimodal, Robot | Flamingo: a Visual Language Model for Few-Shot Lea rning | ArXiv | 2022/04/29 |
| Multi-Images, VLM | Mantis: Multi-Image Instruction Tuning | ArXiv, GitHub | |
| MoE | Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity | ArXiv | |
| MoE | Sparse MoE as the New Dropout: Scaling Dense and S elf-Slimmable Transformers | ArXiv | |
| Mixtral, MoE | Mixtral of Experts | ArXiv | |
| Memory, Robot | LLM as A Robotic Brain: Unifying Egocentric Memory and Control | ArXiv | 2023/04/19 |
| Memory, Reinforcement-Learning | Semantic HELM: A Human-Readable Memory for Reinfor cement Learning | ||
| Math, Reasoning | DeepSeekMath: Pushing the Limits of Mathematical R easoning in Open Language Models | ArXiv, GitHub | |
| Math, PRM | Math-Shepherd: Verify and Reinforce LLMs Step-by-s tep without Human Annotations | ArXiv | |
| Math | WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instruct | ArXiv | |
| Math | Llemma: An Open Language Model For Mathematics | ArXiv | |
| Low-level-action, Robot | SayTap: Language to Quadrupedal Locomotion | ArXiv | 2023/06/13 |
| Low-level-action, Robot | Prompt a Robot to Walk with Large Language Models | ArXiv | 2023/09/18 |
| LoRA, Scaling | LoRA: Low-Rank Adaptation of Large Language Models | ArXiv | |
| LoRA, Scaling | Vera: A General-Purpose Plausibility Estimation Mo del for Commonsense Statements | ArXiv | |
| LoRA | LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report | ArXiv | |
| Lab | Tencent AI Lab - AppAgent, WebVoyager | ||
| Lab | DeepWisdom - MetaGPT | ||
| Lab | Reworkd AI - AgentGPT | ||
| Lab | OpenBMB - ChatDev, XAgent, AgentVerse | ||
| Lab | XLANG NLP Lab - OpenAgents | ||
| Lab | Rutgers University, AGI Research - OpenAGI | ||
| Lab | Knowledge Engineering Group (KEG) & Data Mining at Tsinghua University - CogVLM | ||
| Lab | OpenGVLab | GitHub | |
| Lab | Imperial College London - Zeroshot trajectory | ||
| Lab | sensetime | ||
| Lab | tsinghua | ||
| Lab | Fudan NLP Group | ||
| Lab | Penn State University | ||
| LLaVA, VLM | TinyLLaVA: A Framework of Small-scale Large Multim odal Models | ArXiv | |
| LLaVA, MoE, VLM | MoE-LLaVA: Mixture of Experts for Large Vision-Lan guage Models | ||
| LLaMA, Lightweight, Open-source | MobiLlama: Towards Accurate and Lightweight Fully Transparent GPT | ||
| LLM, Zero-shot | GPT4Vis: What Can GPT-4 Do for Zero-shot Visual Re cognition? | ArXiv | 2023/11/27 |
| LLM, Temporal Logics | NL2TL: Transforming Natural Languages to Temporal Logics using Large Language Models | ArXiv | 2023/05/12 |
| LLM, Survey | A Survey of Large Language Models | ArXiv | 2023/03/31 |
| LLM, Spacial | Can Large Language Models be Good Path Planners? A Benchmark and Investigation on Spatial-temporal Reasoning | ArXiv | 2023/10/05 |
| LLM, Scaling | BitNet: Scaling 1-bit Transformers for Large Langu age Models | ArXiv | |
| LLM, Robot, Task-Decompose | Do As I Can, Not As I Say: Grounding Language in R obotic Affordances | ArXiv | 2022/04/04 |
| LLM, Robot, Survey | Large Language Models for Robotics: A Survey | ArXiv | |
| LLM, Reasoning, Survey | Towards Reasoning in Large Language Models: A Surv ey | ArXiv | 2022/12/20 |
| LLM, Quantization | The Era of 1-bit LLMs: All Large Language Models a re in 1.58 Bits | ArXiv | |
| LLM, PersonalCitation, Robot, Zero-shot | Language Models as Zero-Shot Trajectory Generators | ArXiv | |
| LLM, PersonalCitation, Robot | Tree-Planner: Efficient Close-loop Task Planning w ith Large Language Models01 | ||
| LLM, Open-source | A self-hosted, offline, ChatGPT-like chatbot, powe red by Llama 2. 100% private, with no data leaving your device. | GitHub | |
| LLM, Open-source | OpenFlamingo: An Open-Source Framework for Trainin g Large Autoregressive Vision-Language Models | ArXiv | 2023/08/02 |
| LLM, Open-source | InstructBLIP: Towards General-purpose Vision-Langu age Models with Instruction Tuning | ArXiv | 2023/05/11 |
| LLM, Open-source | ChatBridge: Bridging Modalities with Large Languag e Model as a Language Catalyst | ArXiv | 2023/05/25 |
| LLM, Memory | MemoryBank: Enhancing Large Language Models with L ong-Term Memory | ArXiv | 2023/05/17 |
| LLM, Leaderboard | LMSYS Chatbot Arena Leaderboard | ||
| LLM | Language Models are Few-Shot Learners | ArXiv | 2020/05/28 |
| Intaractive, OpenGVLab, VLM | InternGPT: Solving Vision-Centric Tasks by Interac ting with ChatGPT Beyond Language | ArXiv | 2023/05/09 |
| Instruction-Turning, Survey | Is Prompt All You Need? No. A Comprehensive and Br oader View of Instruction Learning | ||
| Instruction-Turning, Survey | Vision-Language Instruction Tuning: A Review and A nalysis | ArXiv | |
| Instruction-Turning, Survey | A Closer Look at the Limitations of Instruction Tu ning | ArXiv | |
| Instruction-Turning, Survey | A Survey on Data Selection for LLM Instruction Tun ing | ArXiv | |
| Instruction-Turning, Self | Self-Instruct: Aligning Language Models with Self- Generated Instructions | ArXiv | |
| Instruction-Turning, LLM, Zero-shot | Finetuned Language Models Are Zero-Shot Learners | ArXiv | 2021/09/03 |
| Instruction-Turning, LLM, Survey | Instruction Tuning for Large Language Models: A Su rvey | ||
| Instruction-Turning, LLM, PEFT | Visual Instruction Tuning | ArXiv | 2023/04/17 |
| Instruction-Turning, LLM, PEFT | LLaMA-Adapter: Efficient Fine-tuning of Language M odels with Zero-init Attention | ArXiv | 2023/03/28 |
| Instruction-Turning, LLM | Training language models to follow instructions wi th human feedback | ArXiv | 2022/03/04 |
| Instruction-Turning, LLM | MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models | ArXiv | 2023/04/20 |
| Instruction-Turning, LLM | Self-Instruct: Aligning Language Models with Self- Generated Instructions | ArXiv | 2022/12/20 |
| Instruction-Turning | A Closer Look at the Limitations of Instruction Tu ning | ||
| Instruction-Turning | Exploring Format Consistency for Instruction Tunin g | ||
| Instruction-Turning | Exploring the Benefits of Training Expert Language Models over Instruction Tuning | ArXiv | 2023/02/06 |
| Instruction-Turning | Tuna: Instruction Tuning using Feedback from Large Language Models | ArXiv | 2023/03/06 |
| In-Context-Learning, Vision | What Makes Good Examples for Visual In-Context Lea rning? | ||
| In-Context-Learning, Vision | Visual Prompting via Image Inpainting | ArXiv | |
| In-Context-Learning, Video | Prompting Visual-Language Models for Efficient Vid eo Understanding | ||
| In-Context-Learning, VQA | VisualCOMET: Reasoning about the Dynamic Context o f a Still Image | ArXiv | 2020/04/22 |
| In-Context-Learning, VQA | SINC: Self-Supervised In-Context Learning for Visi on-Language Tasks | ArXiv | 2023/07/15 |
| In-Context-Learning, Survey | A Survey on In-context Learning | ArXiv | |
| In-Context-Learning, Scaling | Structured Prompting: Scaling In-Context Learning to 1,000 Examples | ArXiv | 2020/03/06 |
| In-Context-Learning, Scaling | Rethinking the Role of Scale for In-Context Learni ng: An Interpretability-based Case Study at 66 Billion Scale | ArXiv | 2022/03/06 |
| In-Context-Learning, Reinforcement-Learning | AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agents | ArXiv | |
| In-Context-Learning, Prompt-Tuning | Visual Prompt Tuning | ArXiv | |
| In-Context-Learning, Perception, Vision | Visual In-Context Prompting | ArXiv | |
| In-Context-Learning, Many-Shot, Reasoning | Many-Shot In-Context Learning | ArXiv | |
| In-Context-Learning, Instruction-Turning | In-Context Instruction Learning | ||
| In-Context-Learning | ReAct: Synergizing Reasoning and Acting in Languag e Models | ArXiv | 2023/03/20 |
| In-Context-Learning | Small Models are Valuable Plug-ins for Large Langu age Models | ArXiv | 2023/05/15 |
| In-Context-Learning | Generative Agents: Interactive Simulacra of Human Behavior | ArXiv | 2023/04/07 |
| In-Context-Learning | Beyond the Imitation Game: Quantifying and extrapo lating the capabilities of language models | ArXiv | 2022/06/09 |
| In-Context-Learning | What does CLIP know about a red circle? Visual pro mpt engineering for VLMs | ArXiv | |
| In-Context-Learning | Can large language models explore in-context? | ArXiv | |
| Image, LLaMA, Perception | LLaMA-VID: An Image is Worth 2 Tokens in Large Lan guage Models | ArXiv | |
| Hallucination, Survey | Combating Misinformation in the Age of LLMs: Oppor tunities and Challenges | ArXiv | |
| Gym, PPO, Reinforcement-Learning, Survey | Can Language Agents Approach the Performance of RL ? An Empirical Study On OpenAI Gym | ArXiv | |
| Grounding, Reinforcement-Learning | Grounding Large Language Models in Interactive Env ironments with Online Reinforcement Learning | ArXiv | |
| Grounding, Reasoning | Visually Grounded Reasoning across Languages and C ultures | ArXiv | |
| Grounding | V-IRL: Grounding Virtual Intelligence in Real Life | ||
| Google, Grounding | GLaMM: Pixel Grounding Large Multimodal Model | ArXiv | |
| Generation, Survey | Advances in 3D Generation: A Survey | ||
| Generation, Robot, Zero-shot | Zero-Shot Robotic Manipulation with Pretrained Ima ge-Editing Diffusion Models | ArXiv | |
| Generation, Robot, Zero-shot | Towards Generalizable Zero-Shot Manipulationvia Tr anslating Human Interaction Plans | ||
| GPT4V, Robot, VLM | Closed-Loop Open-Vocabulary Mobile Manipulation wi th GPT-4V | ArXiv | |
| GPT4, LLM | GPT-4 Technical Report | ArXiv | 2023/03/15 |
| GPT4, Instruction-Turning | INSTRUCTION TUNING WITH GPT-4 | ArXiv | |
| GPT4, Gemini, LLM | Gemini vs GPT-4V: A Preliminary Comparison and Com bination of Vision-Language Models Through Qualitative Cases | ArXiv | 2023/12/22 |
| Foundation, Robot, Survey | Foundation Models in Robotics: Applications, Chall enges, and the Future | ArXiv | 2023/12/13 |
| Foundation, LLaMA, Vision | VisionLLaMA: A Unified LLaMA Interface for Vision Tasks | ArXiv | |
| Foundation, LLM, Open-source | Code Llama: Open Foundation Models for Code | ArXiv | |
| Foundation, LLM, Open-source | LLaMA: Open and Efficient Foundation Language Mode ls | ArXiv | 2023/02/27 |
| Feedback, Robot | Correcting Robot Plans with Natural Language Feedb ack | ArXiv | |
| Feedback, Robot | Learning to Learn Faster from Human Feedback with Language Model Predictive Control | ArXiv | |
| Feedback, Robot | REFLECT: Summarizing Robot Experiences for Failure Explanation and Correction | ArXiv | 2023/06/27 |
| Feedback, In-Context-Learning, Robot | InCoRo: In-Context Learning for Robotics Control w ith Feedback Loops | ArXiv | |
| Evaluation, LLM, Survey | A Survey on Evaluation of Large Language Models | ArXiv | |
| Evaluation | simple-evals | GitHub | |
| End2End, Multimodal, Robot | VIMA: General Robot Manipulation with Multimodal P rompts | ArXiv | 2022/10/06 |
| End2End, Multimodal, Robot | PaLM-E: An Embodied Multimodal Language Model | ArXiv | 2023/03/06 |
| End2End, Multimodal, Robot | Physically Grounded Vision-Language Models for Rob otic Manipulation | ArXiv | 2023/09/05 |
| Enbodied | Embodied Question Answering | ArXiv | |
| Embodied, World-model | Language Models Meet World Models: Embodied Experi ences Enhance Language Models | ||
| Embodied, Robot, Task-Decompose | Embodied Task Planning with Large Language Models | ArXiv | 2023/07/04 |
| Embodied, Robot | Large Language Models as Generalizable Policies fo r Embodied Tasks | ArXiv | |
| Embodied, Reasoning, Robot | Natural Language as Policies: Reasoning for Coordi nate-Level Embodied Control with LLMs | ArXiv, GitHub | 2024/03/20 |
| Embodied, LLM, Robot, Survey | The Development of LLMs for Embodied Navigation | ArXiv | 2023/11/01 |
| Driving, Spacial | GPT-Driver: Learning to Drive with GPT | ArXiv | 2023/10/02 |
| Drive, Survey | A Survey on Multimodal Large Language Models for A utonomous Driving | ArXiv | |
| Distilling, Survey | A Survey on Knowledge Distillation of Large Langua ge Models | ||
| Distilling | Distilling Step-by-Step! Outperforming Larger Lang uage Models with Less Training Data and Smaller Model Sizes01 | ArXiv | |
| Diffusion, Text-to-Image | Mastering Text-to-Image Diffusion: Recaptioning, P lanning, and Generating with Multimodal LLMs | ArXiv | |
| Diffusion, Survey | On the Design Fundamentals of Diffusion Models: A Survey | ArXiv | |
| Diffusion, Robot | 3D Diffusion Policy | ArXiv | |
| Diffusion | A latent text-to-image diffusion model | ||
| Demonstration, GPT4, PersonalCitation, Robot, VLM | GPT-4V(ision) for Robotics: Multimodal Task Planni ng from Human Demonstration | ||
| Datatset, LLM, Survey | A Survey on Data Selection for Language Models | ArXiv | |
| Datatset, Instruction-Turning | REVO-LION: EVALUATING AND REFINING VISION LANGUAGE INSTRUCTION TUNING DATASETS | ||
| Datatset, Instruction-Turning | Synthetic Data (Almost) from Scratch: Generalized Instruction Tuning for Language Models | ||
| Datatset | PRM800K: A Process Supervision Dataset | GitHub | |
| Data-generation, Robot | GenSim: Generating Robotic Simulation Tasks via La rge Language Models | ArXiv | 2023/10/02 |
| Data-generation, Robot | RoboGen: Towards Unleashing Infinite Data for Auto mated Robot Learning via Generative Simulation | ArXiv | 2023/11/02 |
| DPO, PPO, RLHF | A Comprehensive Survey of LLM Alignment Techniques : RLHF, RLAIF, PPO, DPO and More | ArXiv | |
| DPO | Is DPO Superior to PPO for LLM Alignment? A Compre hensive Study | ArXiv | |
| Context-Window, Scaling | Infini-gram: Scaling Unbounded n-gram Language Mod els to a Trillion Tokens | ||
| Context-Window, Scaling | LONGNET: Scaling Transformers to 1,000,000,000 Tok ens | ArXiv | 2023/07/01 |
| Context-Window, Reasoning, RoPE, Scaling | Resonance RoPE: Improving Context Length Generaliz ation of Large Language Models | ArXiv | |
| Context-Window, LLM, RoPE, Scaling | LongRoPE: Extending LLM Context Window Beyond 2 Mi llion Tokens | ArXiv | |
| Context-Window, Foundation, Gemini, LLM, Scaling | Gemini 1.5: Unlocking multimodal understanding acr oss millions of tokens of context | ||
| Context-Window, Foundation | Mamba: Linear-Time Sequence Modeling with Selectiv e State Spaces | ArXiv | |
| Context-Window | RoFormer: Enhanced Transformer with Rotary Positio n Embedding | ArXiv | |
| Context-Awere, Context-Window | DynaCon: Dynamic Robot Planner with Contextual Awa reness via LLMs | ArXiv | |
| Computer-Resource, Scaling | FlashAttention: Fast and Memory-Efficient Exact At tention with IO-Awareness | ArXiv | |
| Compress, Scaling | (Long)LLMLingua: Enhancing Large Language Model In ference via Prompt Compression | ArXiv | |
| Compress, Quantization, Survey | A Survey on Model Compression for Large Language M odels | ArXiv | |
| Compress, Prompting | Learning to Compress Prompts with Gist Tokens | ArXiv | |
| Code-as-Policies, VLM, VQA | Visual Programming: Compositional visual reasoning without training | ArXiv | 2022/11/18 |
| Code-as-Policies, Robot | SMART-LLM: Smart Multi-Agent Robot Task Planning u sing Large Language Models | ArXiv | 2023/09/18 |
| Code-as-Policies, Robot | RoboScript: Code Generation for Free-Form Manipula tion Tasks across Real and Simulation | ArXiv | |
| Code-as-Policies, Robot | Creative Robot Tool Use with Large Language Models | ArXiv | |
| Code-as-Policies, Reinforcement-Learning, Reward | Code as Reward: Empowering Reinforcement Learning with VLMs | ArXiv | |
| Code-as-Policies, Reasoning, VLM, VQA | ViperGPT: Visual Inference via Python Execution fo r Reasoning | ArXiv | 2023/03/14 |
| Code-as-Policies, Reasoning | Chain of Code: Reasoning with a Language Model-Aug mented Code Emulator | ArXiv | |
| Code-as-Policies, PersonalCitation, Robot, Zero-shot | Socratic Models: Composing Zero-Shot Multimodal Re asoning with Language | ArXiv | 2022/04/01 |
| Code-as-Policies, PersonalCitation, Robot, State-Manage | Statler: State-Maintaining Language Models for Emb odied Reasoning | ArXiv | 2023/06/30 |
| Code-as-Policies, PersonalCitation, Robot | ProgPrompt: Generating Situated Robot Task Plans u sing Large Language Models | ArXiv | 2022/09/22 |
| Code-as-Policies, PersonalCitation, Robot | RoboCodeX:Multi-modal Code Generation forRobotic B ehavior Synthesis | ArXiv | |
| Code-as-Policies, PersonalCitation, Robot | RoboGPT: an intelligent agent of making embodied l ong-term decisions for daily instruction tasks | ||
| Code-as-Policies, PersonalCitation, Robot | ChatGPT for Robotics: Design Principles and Model Abilities | ||
| Code-as-Policies, Multimodal, OpenGVLab, PersonalCitation, Robot | Instruct2Act: Mapping Multi-modality Instructions to Robotic Actions with Large Language Model | ArXiv | 2023/05/18 |
| Code-as-Policies, Embodied, PersonalCitation, Robot | Code as Policies: Language Model Programs for Embo died Control | ArXiv | 2022/09/16 |
| Code-as-Policies, Embodied, PersonalCitation, Reasoning, Robot, Task-Decompose | Inner Monologue: Embodied Reasoning through Planni ng with Language Models | ArXiv | |
| Code-LLM, Front-End | Design2Code: How Far Are We From Automating Front- End Engineering? | ArXiv | |
| Code-LLM | StarCoder 2 and The Stack v2: The Next Generation | ||
| Chain-of-Thought, Reasoning, Table | Chain-of-table: Evolving tables in the reasoning c hain for table understanding | ArXiv | |
| Chain-of-Thought, Reasoning, Survey | Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters | ArXiv | 2023/12/20 |
| Chain-of-Thought, Reasoning, Survey | A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future | ArXiv | 2023/09/27 |
| Chain-of-Thought, Reasoning | Chain-of-Thought Prompting Elicits Reasoning in La rge Language Models | ArXiv | 2022/01/28 |
| Chain-of-Thought, Reasoning | Tree of Thoughts: Deliberate Problem Solving with Large Language Models | ArXiv | 2023/05/17 |
| Chain-of-Thought, Reasoning | Multimodal Chain-of-Thought Reasoning in Language Models | ArXiv | 2023/02/02 |
| Chain-of-Thought, Reasoning | Verify-and-Edit: A Knowledge-Enhanced Chain-of-Tho ught Framework | ArXiv | 2023/05/05 |
| Chain-of-Thought, Reasoning | Skeleton-of-Thought: Large Language Models Can Do Parallel Decoding | ArXiv | 2023/07/28 |
| Chain-of-Thought, Reasoning | Rethinking with Retrieval: Faithful Large Language Model Inference | ArXiv | 2022/12/31 |
| Chain-of-Thought, Reasoning | Self-Consistency Improves Chain of Thought Reasoni ng in Language Models | ArXiv | 2022/03/21 |
| Chain-of-Thought, Reasoning | Chain-of-Thought Hub: A Continuous Effort to Measu re Large Language Models' Reasoning Performance | ArXiv | 2023/05/26 |
| Chain-of-Thought, Reasoning | Skeleton-of-Thought: Prompting LLMs for Efficient Parallel Generation | ArXiv | |
| Chain-of-Thought, Prompting | Chain-of-Thought Reasoning Without Prompting | ArXiv | |
| Chain-of-Thought, Planning, Reasoning | SelfCheck: Using LLMs to Zero-Shot Check Their Own Step-by-Step Reasoning | ArXiv | 2023/08/01 |
| Chain-of-Thought, In-Context-Learning, Self | Measuring and Narrowing the Compositionality Gap i n Language Models | ArXiv | 2022/10/07 |
| Chain-of-Thought, In-Context-Learning, Self | Self-Polish: Enhance Reasoning in Large Language M odels via Problem Refinement | ArXiv | 2023/05/23 |
| Chain-of-Thought, In-Context-Learning | Chain-of-Table: Evolving Tables in the Reasoning C hain for Table Understanding | ArXiv | |
| Chain-of-Thought, In-Context-Learning | Self-Refine: Iterative Refinement with Self-Feedba ck | ArXiv | 2023/03/30 |
| Chain-of-Thought, In-Context-Learning | Plan-and-Solve Prompting: Improving Zero-Shot Chai n-of-Thought Reasoning by Large Language Models | ArXiv | 2023/05/06 |
| Chain-of-Thought, In-Context-Learning | PAL: Program-aided Language Models | ArXiv | 2022/11/18 |
| Chain-of-Thought, In-Context-Learning | Reasoning with Language Model is Planning with Wor ld Model | ArXiv | 2023/05/24 |
| Chain-of-Thought, In-Context-Learning | Least-to-Most Prompting Enables Complex Reasoning in Large Language Models | ArXiv | 2022/05/21 |
| Chain-of-Thought, In-Context-Learning | Complexity-Based Prompting for Multi-Step Reasonin g | ArXiv | 2022/10/03 |
| Chain-of-Thought, In-Context-Learning | Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations | ArXiv | 2022/05/24 |
| Chain-of-Thought, In-Context-Learning | Algorithm of Thoughts: Enhancing Exploration of Id eas in Large Language Models | ArXiv | 2023/08/20 |
| Chain-of-Thought, GPT4, Reasoning, Robot | Look Before You Leap: Unveiling the Power ofGPT-4V in Robotic Vision-Language Planning | ArXiv | 2023/11/29 |
| Chain-of-Thought, Embodied, Robot | EgoCOT: Embodied Chain-of-Thought Dataset for Visi on Language Pre-training | ArXiv | |
| Chain-of-Thought, Embodied, PersonalCitation, Robot, Task-Decompose | EmbodiedGPT: Vision-Language Pre-Training via Embo died Chain of Thought | ArXiv | 2023/05/24 |
| Chain-of-Thought, Code-as-Policies, PersonalCitation, Robot | Demo2Code: From Summarizing Demonstrations to Synt hesizing Code via Extended Chain-of-Thought | ArXiv | |
| Chain-of-Thought, Code-as-Policies | Chain of Code: Reasoning with a Language Model-Aug mented Code Emulator | ArXiv | |
| Caption, Video | PLLaVA : Parameter-free LLaVA Extension from Image s to Videos for Video Dense Captioning | ArXiv | |
| Caption, VLM, VQA | Caption Anything: Interactive Image Description wi th Diverse Multimodal Controls | ArXiv | 2023/05/04 |
| CRAG, RAG | Corrective Retrieval Augmented Generation | ArXiv | |
| Brain, Instruction-Turning | Instruction-tuning Aligns LLMs to the Human Brain | ArXiv | |
| Brain, Conscious | Could a Large Language Model be Conscious? | ||
| Brain | LLM-BRAIn: AI-driven Fast Generation of Robot Beha viour Tree based on Large Language Model | ArXiv | |
| Brain | A Neuro-Mimetic Realization of the Common Model of Cognition via Hebbian Learning and Free Energy Minimization | ||
| Benchmark, Sora, Text-to-Video | LIDA: A Tool for Automatic Generation of Grammar-A gnostic Visualizations and Infographics using Large Language Models01 | ||
| Benchmark, In-Context-Learning | ARB: Advanced Reasoning Benchmark for Large Langua ge Models | ArXiv | 2023/07/25 |
| Benchmark, In-Context-Learning | PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Change | ArXiv | 2022/06/21 |
| Benchmark, GPT4 | Sparks of Artificial General Intelligence: Early e xperiments with GPT-4 | ||
| Awesome Repo, VLM | awesome-vlm-architectures | GitHub | |
| Awesome Repo, Survey | LLMSurvey | GitHub | |
| Awesome Repo, Robot | Awesome-LLM-Robotics | GitHub | |
| Awesome Repo, Reasoning | Awesome LLM Reasoning | GitHub | |
| Awesome Repo, Reasoning | Awesome-Reasoning-Foundation-Models | GitHub | |
| Awesome Repo, RLHF, Reinforcement-Learning | Awesome RLHF (RL with Human Feedback) | GitHub | |
| Awesome Repo, Perception, VLM | Awesome Vision-Language Navigation | GitHub | |
| Awesome Repo, Package | Awesome LLMOps | GitHub | |
| Awesome Repo, Multimodal | Awesome-Multimodal-LLM | GitHub | |
| Awesome Repo, Multimodal | Awesome-Multimodal-Large-Language-Models | GitHub | |
| Awesome Repo, Math, Science | Awesome Scientific Language Models | GitHub | |
| Awesome Repo, LLM, Vision | LLM-in-Vision | GitHub | |
| Awesome Repo, LLM, VLM | Multimodal & Large Language Models | GitHub | |
| Awesome Repo, LLM, Survey | Awesome-LLM-Survey | GitHub | |
| Awesome Repo, LLM, Robot | Everything-LLMs-And-Robotics | GitHub | |
| Awesome Repo, LLM, Leaderboard | LLM-Leaderboard | GitHub | |
| Awesome Repo, LLM | Awesome-LLM | GitHub | |
| Awesome Repo, Korean | awesome-korean-llm | GitHub | |
| Awesome Repo, Japanese, LLM | 日本語LLMまとめ | GitHub | |
| Awesome Repo, In-Context-Learning | Paper List for In-context Learning | GitHub | |
| Awesome Repo, IROS, Robot | IROS2023PaperList | GitHub | |
| Awesome Repo, Hallucination, Survey | A Survey on Hallucination in Large Language Models : Principles, Taxonomy, Challenges, and Open Questions | ArXiv, GitHub | |
| Awesome Repo, Embodied | Awesome Embodied Vision | GitHub | |
| Awesome Repo, Diffusion | Awesome-Diffusion-Models | GitHub | |
| Awesome Repo, Compress | Awesome LLM Compression | GitHub | |
| Awesome Repo, Chinese | Awesome-Chinese-LLM | GitHub | |
| Awesome Repo, Chain-of-Thought | Chain-of-ThoughtsPapers | GitHub | |
| Automate, Prompting | Large Language Models Are Human-Level Prompt Engin eers | ArXiv | 2022/11/03 |
| Automate, Chain-of-Thought, Reasoning | Automatic Chain of Thought Prompting in Large Lang uage Models | ArXiv | 2022/10/07 |
| Audio2Video, Diffusion, Generation, Video | EMO: Emote Portrait Alive - Generating Expressive Portrait Videos with Audio2Video Diffusion Model under Weak Conditions | ArXiv | |
| Audio | Robust Speech Recognition via Large-Scale Weak Sup ervision | ||
| Apple, VLM | MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training | ArXiv | |
| Apple, VLM | Guiding Instruction-based Image Editing via Multim odal Large Language Models | ArXiv | |
| Apple, Robot | Large Language Models as Generalizable Policies fo r Embodied Tasks | ArXiv | |
| Apple, LLM, Open-source | OpenELM: An Efficient Language Model Family with O pen Training and Inference Framework | ArXiv | |
| Apple, LLM | ReALM: Reference Resolution As Language Modeling | ArXiv | |
| Apple, LLM | LLM in a flash: Efficient Large Language Model Inf erence with Limited Memory | ArXiv | |
| Apple, In-Context-Learning, Perception | SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding | ArXiv | |
| Apple, Code-as-Policies, Robot | Executable Code Actions Elicit Better LLM Agents | ArXiv | |
| Apple | Ferret-v2: An Improved Baseline for Referring and Grounding with Large Language Models | ArXiv | |
| Apple | Ferret-UI: Grounded Mobile UI Understanding with M ultimodal LLMs | ArXiv | |
| Apple | Ferret: Refer and Ground Anything Anywhere at Any Granularity | ArXiv | |
| Anything, LLM, Open-source, Perception, Segmentation | Segment Anything | ArXiv | 2023/04/05 |
| Anything, Depth | Depth Anything: Unleashing the Power of Large-Scal e Unlabeled Data | ArXiv | |
| Anything, Caption, Perception, Segmentation | Segment and Caption Anything | ArXiv | |
| Anything, CLIP, Perception | SAM-CLIP: Merging Vision Foundation Models towards Semantic and Spatial Understanding | ArXiv | |
| Agent-Project, Code-LLM | open-interpreter | GitHub | |
| Agent, Web | WebLINX: Real-World Website Navigation with Multi- Turn Dialogue | ArXiv | |
| Agent, Web | WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models | ArXiv | |
| Agent, Web | OmniACT: A Dataset and Benchmark for Enabling Mult imodal Generalist Autonomous Agents for Desktop and Web | ArXiv | |
| Agent, Web | OS-Copilot: Towards Generalist Computer Agents wit h Self-Improvement | ArXiv | |
| Agent, Video-for-Agent | Video as the New Language for Real-World Decision Making | ||
| Agent, VLM | AssistGPT: A General Multi-modal Assistant that ca n Plan, Execute, Inspect, and Learn | ArXiv | |
| Agent, Tool | Gorilla: Large Language Model Connected with Massi ve APIs | ArXiv | |
| Agent, Tool | ToolLLM: Facilitating Large Language Models to Mas ter 16000+ Real-world APIs | ArXiv | |
| Agent, Survey | A Survey on Large Language Model based Autonomous Agents | ArXiv | 2023/08/22 |
| Agent, Survey | The Rise and Potential of Large Language Model Bas ed Agents: A Survey | ArXiv | 2023/09/14 |
| Agent, Survey | Agent AI: Surveying the Horizons of Multimodal Int eraction | ArXiv | |
| Agent, Survey | Large Multimodal Agents: A Survey | ArXiv | |
| Agent, Soft-Dev | Communicative Agents for Software Development | GitHub | |
| Agent, Soft-Dev | MetaGPT: Meta Programming for A Multi-Agent Collab orative Framework | ArXiv | |
| Agent, Robot, Survey | A Survey on LLM-based Autonomous Agents | GitHub | |
| Agent, Reinforcement-Learning, Reward | Reward Design with Language Models | ArXiv | 2023/02/27 |
| Agent, Reinforcement-Learning, Reward | EAGER: Asking and Answering Questions for Automati c Reward Shaping in Language-guided RL | ArXiv | 2022/06/20 |
| Agent, Reinforcement-Learning, Reward | Text2Reward: Automated Dense Reward Function Gener ation for Reinforcement Learning | ArXiv | 2023/09/20 |
| Agent, Reinforcement-Learning | Eureka: Human-Level Reward Design via Coding Large Language Models | ArXiv | 2023/10/19 |
| Agent, Reinforcement-Learning | Language to Rewards for Robotic Skill Synthesis | ArXiv | 2023/06/14 |
| Agent, Reinforcement-Learning | Language Instructed Reinforcement Learning for Hum an-AI Coordination | ArXiv | 2023/04/13 |
| Agent, Reinforcement-Learning | Guiding Pretraining in Reinforcement Learning with Large Language Models | ArXiv | 2023/02/13 |
| Agent, Reinforcement-Learning | STARLING: SELF-SUPERVISED TRAINING OF TEXTBASED RE INFORCEMENT LEARNING AGENT WITH LARGE LANGUAGE MODELS | ||
| Agent, Reasoning, Zero-shot | Agent Instructs Large Language Models to be Genera l Zero-Shot Reasoners | ArXiv | 2023/10/05 |
| Agent, Reasoning | Pangu-Agent: A Fine-Tunable Generalist Agent with Structured Reasoning | ArXiv | |
| Agent, Reasoning | AGENT INSTRUCTS LARGE LANGUAGE MODELS TO BE GENERA L ZERO-SHOT REASONERS | ArXiv | |
| Agent, Multimodal, Robot | A Generalist Agent | ArXiv | 2022/05/12 |
| Agent, Multi | War and Peace (WarAgent): Large Language Model-bas ed Multi-Agent Simulation of World Wars | ArXiv | |
| Agent, MobileApp | You Only Look at Screens: Multimodal Chain-of-Acti on Agents | ArXiv, GitHub | |
| Agent, Minecraft, Reinforcement-Learning | RLAdapter: Bridging Large Language Models to Reinf orcement Learning in Open Worlds | ||
| Agent, Minecraft | Voyager: An Open-Ended Embodied Agent with Large L anguage Models | ArXiv | 2023/05/25 |
| Agent, Minecraft | Describe, Explain, Plan and Select: Interactive Pl anning with Large Language Models Enables Open-World Multi-Task Agents | ArXiv | 2023/02/03 |
| Agent, Minecraft | LARP: Language-Agent Role Play for Open-World Game s | ArXiv | |
| Agent, Minecraft | Steve-Eye: Equipping LLM-based Embodied Agents wit h Visual Perception in Open Worlds | ArXiv | |
| Agent, Minecraft | S-Agents: Self-organizing Agents in Open-ended Env ironment | ArXiv | |
| Agent, Minecraft | Ghost in the Minecraft: Generally Capable Agents f or Open-World Environments via Large Language Models with Text-based Knowledge and Memory01 | ArXiv | |
| Agent, Memory, RAG, Robot | RAP: Retrieval-Augmented Planning with Contextual Memory for Multimodal LLM Agents | ArXiv | 2024/02/06 |
| Agent, Memory, Minecraft | JARVIS-1: Open-World Multi-task Agents with Memory -Augmented Multimodal Language Models | ArXiv | 2023/11/10 |
| Agent, LLM, Planning | LLM-Planner: Few-Shot Grounded Planning for Embodi ed Agents with Large Language Models | ArXiv | |
| Agent, Instruction-Turning | AgentTuning: Enabling Generalized Agent Abilities For LLMs | ArXiv | |
| Agent, Game | LEARNING EMBODIED VISION-LANGUAGE PRO- GRAMMING FR OM INSTRUCTION, EXPLORATION, AND ENVIRONMENTAL FEEDBACK | ArXiv | |
| Agent, GUI, Web | "What’s important here?": Opportunities and Challe nges of Using LLMs in Retrieving Informatio from Web Interfaces | ArXiv | |
| Agent, GUI, MobileApp | Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception | ||
| Agent, GUI, MobileApp | AppAgent: Multimodal Agents as Smartphone Users | ArXiv | |
| Agent, GUI, MobileApp | You Only Look at Screens: Multimodal Chain-of-Acti on Agents | ||
| Agent, GUI | CogAgent: A Visual Language Model for GUI Agents | ArXiv | |
| Agent, GUI | ScreenAgent: A Computer Control Agent Driven by Vi sual Language Large Model | GitHub | |
| Agent, GUI | SeeClick: Harnessing GUI Grounding for Advanced Vi sual GUI Agents | ArXiv | |
| Agent, GPT4, Web | GPT-4V(ision) is a Generalist Web Agent, if Ground ed | ArXiv | |
| Agent, Feedback, Reinforcement-Learning, Robot | Accelerating Reinforcement Learning of Robotic Man ipulations via Feedback from Large Language Models | ArXiv | 2023/11/04 |
| Agent, Feedback, Reinforcement-Learning | AdaRefiner: Refining Decisions of Language Models with Adaptive Feedback | ArXiv | 2023/09/29 |
| Agent, End2End, Game, Robot | An Interactive Agent Foundation Model | ArXiv | |
| Agent, Embodied, Survey | Application of Pretrained Large Language Models in Embodied Artificial Intelligence | ArXiv | |
| Agent, Embodied, Robot | AutoRT: Embodied Foundation Models for Large Scale Orchestration of Robotic Agents | ArXiv | |
| Agent, Embodied, Robot | OPEx: A Component-Wise Analysis of LLM-Centric Age nts in Embodied Instruction Following | ArXiv | |
| Agent, Embodied | OpenAgents: An Open Platform for Language Agents i n the Wild | ArXiv, GitHub | |
| Agent, Embodied | LLM-Planner: Few-Shot Grounded Planning for Embodi ed Agents with Large Language Models | ArXiv | |
| Agent, Embodied | Embodied Multi-Modal Agent trained by an LLM from a Parallel TextWorld | ArXiv | |
| Agent, Embodied | Octopus: Embodied Vision-Language Programmer from Environmental Feedback | ||
| Agent, Embodied | Embodied Task Planning with Large Language Models | ArXiv | |
| Agent, Diffusion, Speech | NaturalSpeech 3: Zero-Shot Speech Synthesis with F actorized Codec and Diffusion Models | ArXiv | |
| Agent, Code-as-Policies | Executable Code Actions Elicit Better LLM Agents | ArXiv | 2024/01/24 |
| Agent, Code-LLM, Code-as-Policies, Survey | If LLM Is the Wizard, Then Code Is the Wand: A Sur vey on How Code Empowers Large Language Models to Serve as Intelligent Agents | ArXiv | |
| Agent, Code-LLM | TaskWeaver: A Code-First Agent Framework | ||
| Agent, Blog | LLM Powered Autonomous Agents | ArXiv | |
| Agent, Awesome Repo, LLM | Awesome-Embodied-Agent-with-LLMs | GitHub | |
| Agent, Awesome Repo, LLM | CoALA: Awesome Language Agents | ArXiv, GitHub | |
| Agent, Awesome Repo, Embodied, Grounding | XLang Paper Reading | GitHub | |
| Agent, Awesome Repo | Awesome AI Agents | GitHub | |
| Agent, Awesome Repo | Autonomous Agents | GitHub | |
| Agent, Awesome Repo | Awesome-Papers-Autonomous-Agent | GitHub | |
| Agent, Awesome Repo | Awesome Large Multimodal Agents | GitHub | |
| Agent, Awesome Repo | LLM Agents Papers | GitHub | |
| Agent, Awesome Repo | Awesome LLM-Powered Agent | GitHub | |
| Agent | XAgent: An Autonomous Agent for Complex Task Solvi ng | ||
| Agent | LLM-Powered Hierarchical Language Agent for Real-t ime Human-AI Coordination | ArXiv | |
| Agent | AgentVerse: Facilitating Multi-Agent Collaboration and Exploring Emergent Behaviors | ArXiv | |
| Agent | Agents: An Open-source Framework for Autonomous La nguage Agents | ArXiv, GitHub | |
| Agent | AutoAgents: A Framework for Automatic Agent Genera tion | GitHub | |
| Agent | DSPy: Compiling Declarative Language Model Calls i nto Self-Improving Pipelines | ArXiv | |
| Agent | AutoGen: Enabling Next-Gen LLM Applications via Mu lti-Agent Conversation | ArXiv | |
| Agent | CAMEL: Communicative Agents for “Mind” Exploration of Large Language Model Society | ArXiv | |
| Agent | XAgent: An Autonomous Agent for Complex Task Solvi ng | ArXiv | |
| Agent | Generative Agents: Interactive Simulacra of Human Behavior | ArXiv | |
| Agent | LLM+P: Empowering Large Language Models with Optim al Planning Proficiency | ArXiv | 2023/04/22 |
| Agent | AgentSims: An Open-Source Sandbox for Large Langua ge Model Evaluation | ArXiv | 2023/08/08 |
| Agent | Agents: An Open-source Framework for Autonomous La nguage Agents | ArXiv | |
| Agent | MindAgent: Emergent Gaming Interaction | ArXiv | |
| Agent | InfiAgent: A Multi-Tool Agent for AI Operating Sys tems | ||
| Agent | Predictive Minds: LLMs As Atypical Active Inferenc e Agents | ||
| Agent | swarms | GitHub | |
| Agent | ScreenAgent: A Vision Language Model-driven Comput er Control Agent | ArXiv | |
| Agent | AssistGPT: A General Multi-modal Assistant that ca n Plan, Execute, Inspect, and Learn | ArXiv | |
| Agent | PromptAgent: Strategic Planning with Language Mode ls Enables Expert-level Prompt Optimization | ArXiv | |
| Agent | Cognitive Architectures for Language Agents | ArXiv | |
| Agent | AIOS: LLM Agent Operating System | ArXiv | |
| Agent | LLM as OS, Agents as Apps: Envisioning AIOS, Agent s and the AIOS-Agent Ecosystem | ArXiv | |
| Agent | Towards General Computer Control: A Multimodal Age nt for Red Dead Redemption II as a Case Study | ||
| Affordance, Segmentation | ManipVQA: Injecting Robotic Affordance and Physica lly Grounded Information into Multi-Modal Large Language Models | ArXiv | |
| Action-Model, Agent, LAM | LaVague | GitHub | |
| Action-Generation, Generation, Prompting | Prompt a Robot to Walk with Large Language Models | ArXiv | |
| APIs, Agent, Tool | Gorilla: Large Language Model Connected with Massi ve APIs | ArXiv | |
| AGI, Survey | Levels of AGI: Operationalizing Progress on the Pa th to AGI | ArXiv | |
| AGI, Brain | When Brain-inspired AI Meets AGI | ArXiv | |
| AGI, Brain | Divergences between Language Models and Human Brai ns | ArXiv | |
| AGI, Awesome Repo, Survey | Awesome-LLM-Papers-Toward-AGI | GitHub | |
| AGI, Agent | OpenAGI: When LLM Meets Domain Experts | ||
| 3D, Open-source, Perception, Robot | 3D-LLM: Injecting the 3D World into Large Language Models | ArXiv | 2023/07/24 |
| 3D, GPT4, VLM | GPT-4V(ision) is a Human-Aligned Evaluator for Tex t-to-3D Generation | ArXiv | |
| ChatEval: Towards Better LLM-based Evaluators thro ugh Multi-Agent Debate | ArXiv | 2023/08/14 |
A Comprehensive Benchmark to Evaluate LLMs as Agents (ICLR'24)
Pocket Flow: Codebase to Tutorial
干净、强大、属于你的 AI Agent 平台 --AI agents, without the clutter.
An AI-powered custom node for ComfyUI designed to enhance workflow automation and provide intelligent assistance
