The integration of AI in healthcare shows promising benefits, such as improved efficiency and patient care. However, challenges exist regarding oversight and regulation. Adaptive algorithms raise concerns about ongoing monitoring and potential bias. Security and privacy risks stem from data storage and inference capabilities. Determining acceptable error rates and liability for algorithmic errors is crucial. President Biden's executive order highlights the need for comprehensive regulation and data privacy legislation. Options for oversight range from government-led regulation to third-party organizations' involvement, with a likely public-private partnership. The FDA's role includes evaluating AI as medical devices, but additional infrastructure is needed for post-market surveillance. Despite excitement about AI's potential, caution is warranted to ensure patient safety and equitable healthcare.