Abstract: Rail switches are critical infrastructure components of a railroad network, that must maintain high-levels of reliable operation. Given the vast number and variety of switches that can exist across a rail network, there is an immediate need for robust automated methods of detecting switch degradations and failures without expensive add-on equipment. In this work, we explore two recent machine learning frameworks for classifying various switch degradation indicators: (1) a featureless recurrent neural network called a Long Short-Term Memory (LSTM) architecture, and (2), the Deep Wavelet Scattering Transform (DWST), which produces features that are locally time invariant and stable to time-warping deformations. We describe both methods as they apply to rail switch monitoring and demonstrate their feasibility on a dataset captured under the service conditions by Alstom Corporation. For multiple categories of degradation types, the baseline models consistently achieve near-perfect accuracies and are competitive with the manual analysis conducted by human switch-maintenance experts.