Difference between revisions of "AI tricks"
KevinYager (talk | contribs) (→Position Bias) |
KevinYager (talk | contribs) (→Position Bias) |
||
| (5 intermediate revisions by the same user not shown) | |||
| Line 2: | Line 2: | ||
* 2025-03: [https://arxiv.org/abs/2503.04818 Prompting Science Report 1: Prompt Engineering is Complicated and Contingent] | * 2025-03: [https://arxiv.org/abs/2503.04818 Prompting Science Report 1: Prompt Engineering is Complicated and Contingent] | ||
* 2024-06: [https://arxiv.org/abs/2406.06608 The Prompt Report: A Systematic Survey of Prompting Techniques] | * 2024-06: [https://arxiv.org/abs/2406.06608 The Prompt Report: A Systematic Survey of Prompting Techniques] | ||
| + | * 2025-06: [https://arxiv.org/abs/2506.05614 Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Tasks] | ||
| + | * 2025-07: [https://arxiv.org/abs/2507.13334 A Survey of Context Engineering for Large Language Models] | ||
| + | * 2025-08: [https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5375404 Prompting Science Report 3: I'll pay you or I'll kill you -but will you care?] | ||
==In-Context Learning== | ==In-Context Learning== | ||
| Line 8: | Line 11: | ||
==Chain of Thought (CoT)== | ==Chain of Thought (CoT)== | ||
| − | "Let's think step-by-step" | + | * 2022-05: [https://arxiv.org/abs/2205.11916 Large Language Models are Zero-Shot Reasoners] "Let's think step-by-step" |
| − | * [https://arxiv.org/abs/2406.07496 TextGrad: Automatic "Differentiation" via Text] | + | * 2024-06: [https://arxiv.org/abs/2406.07496 TextGrad: Automatic "Differentiation" via Text] |
| + | * 2025-06: [https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5285532 Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting] | ||
==Multi-step== | ==Multi-step== | ||
| Line 22: | Line 26: | ||
* 2024-11: [https://arxiv.org/abs/2411.10541 Does Prompt Formatting Have Any Impact on LLM Performance?] | * 2024-11: [https://arxiv.org/abs/2411.10541 Does Prompt Formatting Have Any Impact on LLM Performance?] | ||
| − | ==Position Bias== | + | ==Brittleness== |
| + | * 2025-03: [https://arxiv.org/abs/2503.01781 Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Models] | ||
| + | |||
| + | ===Position Bias=== | ||
* 2023-07: [https://arxiv.org/abs/2307.03172 Lost in the Middle: How Language Models Use Long Contexts] | * 2023-07: [https://arxiv.org/abs/2307.03172 Lost in the Middle: How Language Models Use Long Contexts] | ||
* 2024-11: [https://arxiv.org/abs/2411.01101 Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problems] | * 2024-11: [https://arxiv.org/abs/2411.01101 Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problems] | ||
* 2025-02: [https://arxiv.org/abs/2502.01951 On the Emergence of Position Bias in Transformers] | * 2025-02: [https://arxiv.org/abs/2502.01951 On the Emergence of Position Bias in Transformers] | ||
| + | * 2025-07: [https://arxiv.org/abs/2507.22887 Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning] | ||
* '''Testing models:''' | * '''Testing models:''' | ||
** [https://github.com/gkamradt/LLMTest_NeedleInAHaystack?utm_source=chatgpt.com Needle-in-a-Haystack tests] | ** [https://github.com/gkamradt/LLMTest_NeedleInAHaystack?utm_source=chatgpt.com Needle-in-a-Haystack tests] | ||
Latest revision as of 11:22, 7 August 2025
Contents
Prompt Engineering
- 2025-03: Prompting Science Report 1: Prompt Engineering is Complicated and Contingent
- 2024-06: The Prompt Report: A Systematic Survey of Prompting Techniques
- 2025-06: Which Prompting Technique Should I Use? An Empirical Investigation of Prompting Techniques for Software Engineering Tasks
- 2025-07: A Survey of Context Engineering for Large Language Models
- 2025-08: Prompting Science Report 3: I'll pay you or I'll kill you -but will you care?
In-Context Learning
- 2020-05: Language Models are Few-Shot Learners
- 2025-03: Learning to Search Effective Example Sequences for In-Context Learning
Chain of Thought (CoT)
- 2022-05: Large Language Models are Zero-Shot Reasoners "Let's think step-by-step"
- 2024-06: TextGrad: Automatic "Differentiation" via Text
- 2025-06: Prompting Science Report 2: The Decreasing Value of Chain of Thought in Prompting
Multi-step
Tool-use, feedback, agentic
Retrieval-Augmented Generation (RAG)
Input/Output Formats
- 2024-08: LLMs Are Biased Towards Output Formats! Systematically Evaluating and Mitigating Output Format Bias of LLMs
- 2024-11: Does Prompt Formatting Have Any Impact on LLM Performance?
Brittleness
Position Bias
- 2023-07: Lost in the Middle: How Language Models Use Long Contexts
- 2024-11: Self-Consistency Falls Short! The Adverse Effects of Positional Bias on Long-Context Problems
- 2025-02: On the Emergence of Position Bias in Transformers
- 2025-07: Where to show Demos in Your Prompt: A Positional Bias of In-Context Learning
- Testing models:
- Needle-in-a-Haystack tests
- 2023-08: LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding
- 2024-02: ∞Bench: Extending Long Context Evaluation Beyond 100K Tokens
- 2024-06: [Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models https://arxiv.org/abs/2406.11230]
- 2024-07: Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystack
- 2025-04: Reasoning on Multiple Needles In A Haystack
- Mitigation:
