Tag: language-models (61 references)
LLM Social Simulations Are a Promising Research Method
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
Exploring the limits of strong membership inference attacks on large language models
Extending "GPTs Are GPTs" to Firms
OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens
Rethinking machine unlearning for large language models
Comprehensive review of machine unlearning in LLMs, aiming to eliminate undesirable data influence (sensitive or illegal information) while maintaining essential knowledge generation. Envisions LLM unlearning as a pivotal element in life-cycle management for developing safe, secure, trustworthy, and resource-efficient generative AI.
The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm
The Rise of AI-Generated Content in Wikipedia
What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions
Large language models reduce public knowledge sharing on online Q&A platforms
GPTs are GPTs: Labor Market Impact Potential of LLMs
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
StarCoder 2 and The Stack v2: The Next Generation
LLM Dataset Inference: Did you train on my dataset?
Scalable Data Ablation Approximations for Language Models through Modular Training and Merging
A Canary in the AI Coal Mine: American Jews May Be Disproportionately Harmed by Intellectual Property Dispossession in Large Language Model Training
Data Flywheels for LLM Applications
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
LLM Unlearning via Loss Adjustment with Only Forget Data
FLAT is a loss adjustment approach which maximizes f-divergence between the available template answer and the forget answer with respect to the forget data. Demonstrates superior unlearning performance compared to existing methods while minimizing impact on retained capabilities, tested on Harry Potter dataset and MUSE Benchmark.
Enhancing Training Data Attribution for Large Language Models with Fitting Error Consideration
Enhances training data attribution methods for large language models including LLaMA2, QWEN2, and Mistral by considering fitting error in the attribution process.
Position Paper: Data-Centric AI in the Age of Large Language Models
Position paper identifying four specific scenarios centered around data for LLMs, covering data-centric benchmarks and data curation, data attribution, knowledge transfer, and inference contextualization.
Alpaca: A Strong, Replicable Instruction-Following Model
Quantifying Memorization Across Neural Language Models
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
A Watermark for Large Language Models
Textbooks Are All You Need II: phi-1.5 technical report
Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
OWASP Top 10 for Large Language Model Applications
DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning
Comprehensive library and empirical study of coreset selection methods for deep learning, finding that random selection remains a strong baseline across many settings.
Training Compute-Optimal Large Language Models
Shows that current LLMs are significantly undertrained. For compute-optimal training, model size and training tokens should scale equally. Introduces Chinchilla (70B params, 1.4T tokens) which outperforms larger models like Gopher (280B) trained on less data.
Training language models to follow instructions with human feedback
Probabilistic Machine Learning: An Introduction
Introducing Whisper
Robust Speech Recognition via Large-Scale Weak Supervision
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
Extracting Training Data from Large Language Models
Measuring Mathematical Problem Solving With the MATH Dataset
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Ethical and Social Risks of Harm from Language Models
Language Models are Few-Shot Learners
Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning
Scaling Laws for Neural Language Models
Establishes power-law scaling relationships between language model performance and model size, dataset size, and compute, spanning seven orders of magnitude.
Deep Double Descent: Where Bigger Models and More Data Hurt
Demonstrates that double descent occurs across model size, training epochs, and dataset size in modern deep networks. Introduces effective model complexity to unify these phenomena and shows regimes where more data hurts.
interpreting GPT: the logit lens
Common voice: A massively-multilingual speech corpus
The Secret Sharer: Measuring Unintended Memorization in Neural Networks
Excavating AI: The Politics of Images in Machine Learning Training Sets
A Survey on Image Data Augmentation for Deep Learning
Comprehensive survey of image data augmentation techniques for deep learning, covering geometric transformations, color space transforms, kernel filters, mixing images, random erasing, and neural style transfer approaches.
Troubling Trends in Machine Learning Scholarship
Active Learning for Convolutional Neural Networks: A Core-Set Approach
Defines active learning as core-set selection, choosing points such that a model trained on the subset is competitive for remaining data. Provides theoretical bounds via k-Center problem.