SOHAM DE

RECENT TALKS

• State Space Models and the return of RNNs
  Keynote at ICML Workshop on Next Generation of Sequence Modeling Architectures. (2024)
  
  
• High-Accuracy Differentially Private Image Classification: slides
  At Google Brain, University College London, University of Maryland, Security & Privacy in ML Seminar, Oracle Labs and Indian Institute of Technology Kharagpur. (2022)
  
  
• Normalizer-Free Networks: slides
  At Google Brain, University College London and Swiss Federal Institute of Technology Lausanne (EPFL). (2021)

SELECTED PAPERS

• Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Models
  Soham De, Samuel L. Smith, Anushan Fernando, Aleksandar Botev, George Cristian-Muraru, et al
  Preprint
  paper / code / checkpoints / bibtex / tl;dr
  
  @article{de2024griffin,
  title={Griffin: Mixing gated linear recurrences with local attention for efficient language models},
  author={De, Soham and Smith, Samuel L and Fernando, Anushan and Botev, Aleksandar and Cristian-Muraru, George and Gu, Albert and Haroun, Ruba and Berrada, Leonard and Chen, Yutian and Srinivasan, Srivatsan and others},
  journal={arXiv preprint arXiv:2402.19427},
  year={2024}
  }
  
  
  We propose Hawk (a purely linear RNN) and Griffin (hybrid model) language models that are competitive at scale with well-tuned transformers and have fast training and inference.
  
  
  
• Resurrecting Recurrent Neural Networks for Long Sequences
  Antonio Orvieto, Samuel L Smith, Albert Gu, Anushan Fernando, Caglar Gulcehre, Razvan Pascanu, Soham De
  ICML 2023
  paper / bibtex / tl;dr
  
  @article{orvieto2023resurrecting,
  title={Resurrecting recurrent neural networks for long sequences},
  author={Orvieto, Antonio and Smith, Samuel L and Gu, Albert and Fernando, Anushan and Gulcehre, Caglar and Pascanu, Razvan and De, Soham},
  journal={arXiv preprint arXiv:2303.06349},
  year={2023}
  }
  
  
  We show how to design deep RNNs to learn long-range dependencies in sequential data.
  
  
  
• Unlocking High-Accuracy Differentially Private Image Classification through Scale
  Soham De, Leonard Berrada, Jamie Hayes, Samuel L Smith, Borja Balle
  Preprint
  paper / code / bibtex / tl;dr
  
  @article{de2022unlocking,
  title={Unlocking high-accuracy differentially private image classification through scale},
  author={De, Soham and Berrada, Leonard and Hayes, Jamie and Smith, Samuel L and Balle, Borja},
  journal={arXiv preprint arXiv:2204.13650},
  year={2022}
  }
  
  
  Machine learning models that were trained under standard training pipelines can leak the data it was trained on. This presents privacy risks when the training data contains sensitive information. Differential privacy is a technology that can be deployed at training time to mitigate these privacy risks, but they often incur significant reduction in model performance. In this work, we make substantial progress towards unlocking high-accuracy training of image classification models under differential privacy.
  
  
  
• High-Performance Large-Scale Image Recognition Without Normalization
  Andrew Brock, Soham De, Sam Smith, Karen Simonyan
  ICML 2021
  paper / code / bibtex / tl;dr
  
  @article{brock2021high,
  title={High-Performance Large-Scale Image Recognition Without Normalization},
  author={Brock, Andrew and De, Soham and Smith, Samuel L and Simonyan, Karen},
  journal={arXiv preprint arXiv:2102.06171},
  year={2021}
  }
  
  
  BatchNorm is a key component of most vision models, but it has many undesirable properties stemming from its dependence on the batch size and the fact that it introduces interactions between examples. We propose NFNets, a new family of ResNet models, that achieve state-of-the-art performance on ImageNet across a range of training latencies and is up to 8.7 times faster than EfficientNets, without any normalization layers.
  
  
  
• On the Origin of Implicit Regularization in Stochastic Gradient Descent
  Sam Smith, Benoit Dherin, David Barrett, Soham De
  ICLR 2021
  paper / bibtex / tl;dr
  
  @inproceedings{smith2021origin,
  title={On the Origin of Implicit Regularization in Stochastic Gradient Descent},
  author={Smith, Samuel L and Dherin, Benoit and Barrett, David GT and De, Soham},
  booktitle={9th International Conference on Learning Representations, {ICLR}},
  year={2021}
  }
  
  
  We show that, when using small finite learning rates, the path taken by Random Shuffling SGD coincides with the path taken by gradient flow when minimizing a modified loss function. This modified loss function contains an implicit source of regularization which enhances the test accuracies achieved by deep networks on standard benchmarks.
  
  
  
• Batch Normalization Biases Residual Blocks Towards the Identity Function in Deep Networks
  Soham De, Sam Smith
  NeurIPS 2020
  paper / bibtex / tl;dr
  
  @article{de2020batch,
  title={Batch normalization biases residual blocks towards the identity function in deep networks},
  author={De, Soham and Smith, Sam},
  journal={Advances in Neural Information Processing Systems},
  volume={33},
  year={2020}
  }
  
  
  We show that batch normalization biases residual blocks towards the identity function in deep networks early in training. This dramatically increases the largest trainable depth. We can recover this benefit with a simple change to the initialization scheme.
  
  
  
• Training Quantized Nets: A Deeper Understanding
  Hao Li*, Soham De*, Zheng Xu, Christoph Studer, Hanan Samet, Tom Goldstein
  NeurIPS 2017
  paper / poster / bibtex / tl;dr
  
  @inproceedings{li2017training,
  title={Training quantized nets: A deeper understanding},
  author={Li, Hao and De, Soham and Xu, Zheng and Studer, Christoph and Samet, Hanan and Goldstein, Tom},
  booktitle={Advances in Neural Information Processing Systems},
  pages={5811--5821},
  year={2017}
  }
  
  
  Neural net parameters can often be compressed down to just one single bit without a significant loss in network performance, yielding a huge reduction in model size and computational workload. We develop a theory of quantized nets, and explain the performance of algorithms for weight quantization.