Transformers and large language models (LLMs) have had enormous success in recent years. Yet they remain poorly understood, in particular why and how they work. We are trying to answer such questions using tools such as mathematical analysis and mechanistic interpretability. One area where these models perform poorly is Continual Learning. We are working on alternate solutions, for example based on models such as biological neural networks.