Introduction: Since 2004, the International Olympic Committee (IOC) store all samples collected at summer Olympic Games (OG) for retrospective re-analysis with more advanced analytical techniques to catch doping athletes. Methods: All announced Anti-Doping Rule Violations (ADRVs) from IOC re-tests of the 2004, 2008 and 2012 OG (via IOC, International Federations and Athletics Integrity Unit public data) and other ADRVs confirmed to impact OG results from 1968 to 2012 (via the list of Doping Irregularities on olympedia.org) were collated to investigate how many medals have been impacted by ADRVs, when the ADRV was identified relative to the OG in question and its cause. Results: One hundred and thirty-four medals were impacted by ADRVs but only 26% of these ADRVs were identified at the time of the OG. Most ADRVs impacting medal results (74%) were identified retrospectively, either from events prior to the OG (17%) or via IOC re-tests of samples from 2004, 2008 and 2012 (57%). ADRVs impacting medal results from these re-tests took a mean of 6.8 ± 2.0 years to be announced relative to the end of the OG in which the medal was originally won. Exogenous Anabolic Androgenic Steroid metabolites were present in 90% of all athlete (n = 142) samples from IOC re-tests with dehydrochloromethyltestosterone and stanozolol accounting for 79% of detected substances. Athletics (n = 64) and weightlifting (n = 62) were the most affected sports. Conclusion: This analysis shows the frequency of targeted pre-OG Out-of-Competition testing should increase. We advocate for long-term sample storage to continue and additionally incorporate novel and potentially complementary technologies/sample matrices.