By combining ab initio simulations with state-of-the-art electron microscopy and electron energy loss spectroscopy, we study the mechanism of electron beam damage in nitrogen-doped graphene and carbon nanotubes. Our results show that the incorporation of nitrogen atoms results in noticeable knock-on damage in these structures already at an acceleration voltage of 80 kV, at which essentially no damage is created in pristine structures at corresponding doses. Contrary to an early estimate predicting rapid destruction via sputtering of the nitrogen atoms, in the case of substitutional doping, damage is initiated by displacement of carbon atoms neighboring the nitrogen dopant, leading to the conversion of substitutional dopant sites into pyridinic ones. Although such events are relatively rare at 80 kV, they become significant at higher voltages typically used in electron energy loss spectroscopy studies. Correspondingly, we measured an energy loss spectrum time series at 100 kV that provides direct evidence for such conversions in nitrogen-doped single-walled carbon nanotubes, in excellent agreement with our theoretical prediction. Besides providing an improved understanding of the irradiation stability of these structures, we show that structural changes cannot be neglected in their characterization employing high-energy electrons., T.S. received support from the Finnish Foundation for Technology Promotion. T.S., H.J., and E.I.K. further acknowledge financial support from the CNB-E project of the Aalto MIDE program and from the LiBaCAM project of TEKES. J.K. and A.V.K. acknowledge financial support from Helsinki University Funds and Academy of Finland through several projects as well as CSC Ltd. Finland for computational resources. S.K., U.K., and J.C.M. acknowledge the financial support by the DFG (German Research Foundation) and the Ministry of Science, Research and the Arts (MWK) of Baden-Wuerttemberg in the frame of the SALVE (Sub Angstrom Low-Voltage Electron microscopy) project. V.S. acknowledges an EC Grant No. 266391 related to the project ELECTROGRAPH (FP7/2007-2013)., Peer reviewed