Were RNNs all we needed?