Tag: ml-methods (109 references)
LLM Social Simulations Are a Promising Research Method
DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning
Can We Trust AI Benchmarks? An Interdisciplinary Review of Current Issues in AI Evaluation
Exploring the limits of strong membership inference attacks on large language models
Extending "GPTs Are GPTs" to Firms
OLMoTrace: Tracing Language Model Outputs Back to Trillions of Training Tokens
Welcome to the Era of Experience
The Leaderboard Illusion
The Multilingual Alignment Prism: Aligning Global and Local Preferences to Reduce Harm
To code, or not to code? exploring impact of code in pre-training
The Rise of AI-Generated Content in Wikipedia
Poisoning Web-Scale Training Datasets is Practical
What is Your Data Worth to GPT? LLM-Scale Data Valuation with Influence Functions
Large language models reduce public knowledge sharing on online Q&A platforms
GPTs are GPTs: Labor Market Impact Potential of LLMs
Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training
The Data Provenance Initiative: A Large Scale Audit of Dataset Licensing & Attribution in AI
Large-scale audit of over 1,800 text AI datasets analyzing trends, permissions of use and global representation. Found frequent miscategorization of licences on dataset hosting sites, with licence omission rates of more than 70% and error rates of more than 50%. Released the Data Provenance Explorer tool for practitioners.
StarCoder 2 and The Stack v2: The Next Generation
LLM Dataset Inference: Did you train on my dataset?
Scalable Data Ablation Approximations for Language Models through Modular Training and Merging
A Canary in the AI Coal Mine: American Jews May Be Disproportionately Harmed by Intellectual Property Dispossession in Large Language Model Training
Data Flywheels for LLM Applications
AI models collapse when trained on recursively generated data
Landmark study showing that indiscriminate use of model-generated content in training causes irreversible defects in resulting models, where tails of original content distribution disappear. Model collapse is a degenerative learning process where models forget improbable events over time. Demonstrates this across LLMs, VAEs, and GMMs.
Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Research
Aya Model: An Instruction Finetuned Open-Access Multilingual Language Model
Alpaca: A Strong, Replicable Instruction-Following Model
LEACE: Perfect linear concept erasure in closed form
Quantifying Memorization Across Neural Language Models
Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4
Direct Preference Optimization: Your Language Model is Secretly a Reward Model
Algorithmic Collective Action in Machine Learning
Provides theoretical framework for algorithmic collective action, showing that small collectives can exert significant control over platform learning algorithms through coordinated data strategies.
A Watermark for Large Language Models
Textbooks Are All You Need II: phi-1.5 technical report
Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment
SILO Language Models: Isolating Legal Risk In a Nonparametric Datastore
OWASP Top 10 for Large Language Model Applications
TRAK: Attributing Model Behavior at Scale
Introduces TRAK (Tracing with the Randomly-projected After Kernel), a data attribution method that is both effective and computationally tractable for large-scale models by leveraging random projections.
DeepCore: A Comprehensive Library for Coreset Selection in Deep Learning
Comprehensive library and empirical study of coreset selection methods for deep learning, finding that random selection remains a strong baseline across many settings.
Training Data Influence Analysis and Estimation: A Survey
Training Compute-Optimal Large Language Models
Shows that current LLMs are significantly undertrained. For compute-optimal training, model size and training tokens should scale equally. Introduces Chinchilla (70B params, 1.4T tokens) which outperforms larger models like Gopher (280B) trained on less data.
Beta Shapley: a Unified and Noise-reduced Data Valuation Framework for Machine Learning
Generalizes Data Shapley using Beta weighting functions, providing noise-reduced data valuation that better handles outliers and mislabeled data detection.
Training language models to follow instructions with human feedback
Probabilistic Machine Learning: An Introduction
Why Black Box Machine Learning Should Be Avoided for High-Stakes Decisions, in Brief
Beyond neural scaling laws: beating power law scaling via data pruning
The Stack: A Permissively Licensed Source Code Dataset
Introducing Whisper
Robust Speech Recognition via Large-Scale Weak Supervision
On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?
Machine Unlearning
Introduces SISA (Sharded, Isolated, Sliced, Aggregated) training for efficient exact machine unlearning. Partitions data into shards with separate models, enabling targeted retraining when data must be forgotten.
Extracting Training Data from Large Language Models
Measuring Mathematical Problem Solving With the MATH Dataset
The Pile: An 800GB Dataset of Diverse Text for Language Modeling
Ethical and Social Risks of Harm from Language Models
Language Models are Few-Shot Learners
Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning
Scaling Laws for Neural Language Models
Establishes power-law scaling relationships between language model performance and model size, dataset size, and compute, spanning seven orders of magnitude.
Exploring Research Interest in Stack Overflow -- A Systematic Mapping Study and Quality Evaluation
Coresets for Data-efficient Training of Machine Learning Models
Introduces CRAIG (Coresets for Accelerating Incremental Gradient descent), selecting subsets that approximate full gradient for 2-3x training speedups while maintaining performance.
Deep Double Descent: Where Bigger Models and More Data Hurt
Demonstrates that double descent occurs across model size, training epochs, and dataset size in modern deep networks. Introduces effective model complexity to unify these phenomena and shows regimes where more data hurts.
interpreting GPT: the logit lens
Estimating Training Data Influence by Tracing Gradient Descent
Introduces TracIn, which computes influence of training examples by tracing how test loss changes during training. Uses first-order gradient approximation and saved checkpoints for scalability.
In Pursuit of Interpretable, Fair and Accurate Machine Learning for Criminal Recidivism Prediction
Common voice: A massively-multilingual speech corpus
Reconciling modern machine-learning practice and the classical bias–variance trade-off
The Secret Sharer: Measuring Unintended Memorization in Neural Networks
Excavating AI: The Politics of Images in Machine Learning Training Sets
Data Shapley: Equitable Valuation of Data for Machine Learning
Face Recognition Vendor Test (FRVT) Part 3: Demographic Effects
BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain
First demonstration of backdoor attacks on deep neural networks. Shows that small trigger patterns in training data cause models to misclassify any input containing the trigger (e.g., stop signs with stickers classified as speed limits).
On the Accuracy of Influence Functions for Measuring Group Effects
Model Cards for Model Reporting
A Survey on Image Data Augmentation for Deep Learning
Comprehensive survey of image data augmentation techniques for deep learning, covering geometric transformations, color space transforms, kernel filters, mixing images, random erasing, and neural style transfer approaches.
CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features
Combines cutting and mixing: patches from one image replace regions in another, with labels mixed proportionally. Improves over Cutout by using cut pixels constructively rather than zeroing them out.
The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standards
Troubling Trends in Machine Learning Scholarship
Active Learning for Convolutional Neural Networks: A Core-Set Approach
Defines active learning as core-set selection, choosing points such that a model trained on the subset is competitive for remaining data. Provides theoretical bounds via k-Center problem.
mixup: Beyond Empirical Risk Minimization
Introduces mixup, a data augmentation technique that trains on convex combinations of input pairs and their labels. Simple, data-independent, and model-agnostic approach that improves generalization and robustness.
Improved Regularization of Convolutional Neural Networks with Cutout
Introduces Cutout, a regularization technique that randomly masks square regions of input images during training. Inspired by dropout but applied to inputs, encouraging models to learn from partially visible objects.
Deep learning scaling is predictable, empirically
Understanding Black-box Predictions via Influence Functions
Uses influence functions from robust statistics to trace model predictions back to training data, identifying training points most responsible for a given prediction.
Towards Making Systems Forget with Machine Unlearning
First formal definition of machine unlearning. Proposes converting learning algorithms into summation form to enable efficient data removal without full retraining. Foundational work establishing the unlearning problem.
Poisoning Attacks against Support Vector Machines
Investigates poisoning attacks against SVMs where adversaries inject crafted training data to increase test error. Uses gradient ascent to construct malicious data points.
Curriculum Learning
Introduces curriculum learning: training models on examples of increasing difficulty. Shows this acts as a continuation method for non-convex optimization, improving both convergence speed and final generalization.
Causality: Models, Reasoning, and Inference
Foundational book on causal inference introducing structural causal models, do-calculus, and counterfactual reasoning. Unifies graphical models with potential outcomes framework. Second edition with expanded coverage.
Active Learning Literature Survey
Canonical survey of active learning covering uncertainty sampling, query-by-committee, expected error reduction, variance reduction, and density-weighted methods. Establishes foundational taxonomy for the field.
Causal Inference Using Potential Outcomes: Design, Modeling, Decisions
Comprehensive overview of the potential outcomes framework for causal inference. Covers experimental design, observational studies, propensity scores, and the fundamental problem of causal inference.