Because Audacity uses Windows audio drivers, the sample rate also needs to match in the Windows Sound control panel (both the recording and playback tabs).
Unfortunately, Microsoft keep changing how to get to the Windows Sound control panel. These instructions are believed to be up to date: Windows: accessing the Windows Sound controls - Audacity Manual