The use of crowdsourced data is growing rapidly, particularly in ornithology. Citizen science greatly contributes to our knowledge, however, little is known about the reliability of data collected in that way. We found, using an online picture quiz, that self-proclaimed expert birders were more likely to misidentify common British bird species as exotic or rare species, compared to people who rated their own expertise more modestly. This finding suggests that records of rare species should always be considered with caution even if the reporters consider themselves to be experts. In general, however, we show that self-rated expertise in bird identification skills is a reliable predictor of correct species identification. Implementing the collection of data on self-rated expertise is easy and low-cost. We therefore suggest it as a useful tool to statistically account for variability in bird identification skills of citizen science participants and to improve the accuracy of identification data collected by citizen science projects.