Modern anesthetic compounds and advanced monitoring tools have revolutionized the field of medicine, allowing for complex surgical procedures to occur safely and effectively. Faster induction times and quicker recovery periods of current anesthetic agents have also helped reduce health care costs significantly. Moreover, extensive research has allowed for a better understanding of anesthetic modes of action, thus facilitating the development of more effective and safer compounds. Notwithstanding the realization that anesthetics are a prerequisite to all surgical procedures, evidence is emerging to support the notion that exposure of the developing brain to certain anesthetics may impact future brain development and function. Whereas the data in support of this postulate from human studies is equivocal, the vast majority of animal research strongly suggests that anesthetics are indeed cytotoxic at multiple brain structure and function levels. In this review, we first highlight various modes of anesthetic action and then debate the evidence of harm from both basic science and clinical studies perspectives. We present evidence from animal and human studies vis-à-vis the possible detrimental effects of anesthetic agents on both the young developing and the elderly aging brain while discussing potential ways to mitigate these effects. We hope that this review will, on the one hand, invoke debate vis-à-vis the evidence of anesthetic harm in young children and the elderly, and on the other hand, incentivize the search for better and less toxic anesthetic compounds.