Main issues is high level of ambient noise.
As @RayDunzl explained above, as long as the signal you're measuring is 20 dB above the noise, you will achieve a measurement accuracy of about 0.05 dB (noncoherent 0 dB + -20 dB = 0.05 dB). Even just 10 dB (0.5 dB accuracy) would probably be good enough for room correction.
In addition, the windowing that is normally done on frequency response sweep measurements (including REW measurements, by default) will filter out most of the noise, so the effective noise on the measurement will not be as bad as the broadband noise you measured. This is even more effective on long sweeps.
Bottom line: don't worry about it. REW frequency response measurements are highly resilient to noise. Summoning @JohnPM to confirm.
I suspect some of the answers in this thread are confused because it's really not clear from your post whether you are worried about noise for the purposes of measurement, or for the purposes of listening. From the title of the thread I assumed the former.