Tag: ml-methods (80 references)
LLM Social Simulations Are a Promising Research Method
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
Exploring the limits of strong membership inference attacks on large language models
Extending "GPTs Are GPTs" to Firms
OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens
The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm
To code, or not to code? exploring impact of code in pre-training
The Rise of AI-Generated Content in Wikipedia
Poisoning Web-Scale Training Datasets is Practical
What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions
Large language models reduce public knowledge sharing on online Q&A platforms
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
GPTs are GPTs: Labor Market Impact Potential of LLMs
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
StarCoder 2 and The Stack v2: The Next Generation
LLM Dataset Inference: Did you train on my dataset?
Scalable Data Ablation Approximations for Language Models through Modular Training and Merging
A Canary in the AI Coal Mine: American Jews May Be Disproportionately Harmed by Intellectual Property Dispossession in Large Language Model Training
Data {Flywheels} for {LLM} {Applications}
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
Alpaca: A Strong, Replicable Instruction-Following Model
LEACE: Perfect linear concept erasure in closed form
Quantifying Memorization Across Neural Language Models
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
Algorithmic Collective Action in Machine Learning
Provides theoretical framework for algorithmic collective action, showing that small collectives can exert significant control over platform learning algorithms through coordinated data strategies.
A Watermark for Large Language Models
Textbooks Are All You Need II: phi-1.5 technical report
Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
OWASP Top 10 for Large Language Model Applications
TRAK: Attributing Model Behavior at Scale
Introduces TRAK (Tracing with the Randomly-projected After Kernel), a data attribution method that is both effective and computationally tractable for large-scale models by leveraging random projections.
DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning
Comprehensive library and empirical study of coreset selection methods for deep learning, finding that random selection remains a strong baseline across many settings.
Training Data Influence Analysis and Estimation: A Survey
Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learning
Generalizes Data Shapley using Beta weighting functions, providing noise-reduced data valuation that better handles outliers and mislabeled data detection.
Training language models to follow instructions with human feedback
Probabilistic Machine Learning: An introduction
Why Black Box Machine Learning Should Be Avoided for High-Stakes Decisions, in Brief
Beyond neural scaling laws: beating power law scaling via data pruning
Introducing Whisper
Robust Speech Recognition via Large-Scale Weak Supervision
Training Compute-Optimal Large Language Models
Shows that current LLMs are significantly undertrained. For compute-optimal training, model size and training tokens should scale equally. Introduces Chinchilla (70B params, 1.4T tokens) which outperforms larger models like Gopher (280B) trained on less data.
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
Extracting Training Data from Large Language Models
Measuring Mathematical Problem Solving With the MATH Dataset
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Ethical and Social Risks of Harm from Language Models
Language Models are Few-Shot Learners
Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning
Scaling Laws for Neural Language Models
Establishes power-law scaling relationships between language model performance and model size, dataset size, and compute, spanning seven orders of magnitude.
Coresets for Data-efficient Training of Machine Learning Models
Introduces CRAIG (Coresets for Accelerating Incremental Gradient descent), selecting subsets that approximate full gradient for 2-3x training speedups while maintaining performance.
Estimating Training Data Influence by Tracing Gradient Descent
Introduces TracIn, which computes influence of training examples by tracing how test loss changes during training. Uses first-order gradient approximation and saved checkpoints for scalability.
In Pursuit of Interpretable, Fair and Accurate Machine Learning for Criminal Recidivism Prediction
Deep Double Descent: Where Bigger Models and More Data Hurt
Demonstrates that double descent occurs across model size, training epochs, and dataset size in modern deep networks. Introduces effective model complexity to unify these phenomena and shows regimes where more data hurts.
Common voice: A massively-multilingual speech corpus
Reconciling modern machine-learning practice and the classical bias–variance trade-off
The Secret Sharer: Measuring Unintended Memorization in Neural Networks
Excavating AI: The Politics of Images in Machine Learning Training Sets
On the Accuracy of Influence Functions for Measuring Group Effects
Model Cards for Model Reporting
A Survey on Image Data Augmentation for Deep Learning
Comprehensive survey of image data augmentation techniques for deep learning, covering geometric transformations, color space transforms, kernel filters, mixing images, random erasing, and neural style transfer approaches.
CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
Combines cutting and mixing: patches from one image replace regions in another, with labels mixed proportionally. Improves over Cutout by using cut pixels constructively rather than zeroing them out.
The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
Troubling Trends in Machine Learning Scholarship
Active Learning for Convolutional Neural Networks: A Core-Set Approach
Defines active learning as core-set selection, choosing points such that a model trained on the subset is competitive for remaining data. Provides theoretical bounds via k-Center problem.
mixup: Beyond Empirical Risk Minimization
Introduces mixup, a data augmentation technique that trains on convex combinations of input pairs and their labels. Simple, data-independent, and model-agnostic approach that improves generalization and robustness.
Deep learning scaling is predictable, empirically
Understanding Black-box Predictions via Influence Functions
Uses influence functions from robust statistics to trace model predictions back to training data, identifying training points most responsible for a given prediction.
Improved Regularization of Convolutional Neural Networks with Cutout
Introduces Cutout, a regularization technique that randomly masks square regions of input images during training. Inspired by dropout but applied to inputs, encouraging models to learn from partially visible objects.
Poisoning Attacks against Support Vector Machines
Investigates poisoning attacks against SVMs where adversaries inject crafted training data to increase test error. Uses gradient ascent to construct malicious data points.
Curriculum Learning
Introduces curriculum learning: training models on examples of increasing difficulty. Shows this acts as a continuation method for non-convex optimization, improving both convergence speed and final generalization.
Active Learning Literature Survey
Canonical survey of active learning covering uncertainty sampling, query-by-committee, expected error reduction, variance reduction, and density-weighted methods. Establishes foundational taxonomy for the field.