I’m pretty sure google will still have your house number associated with your house. You’re just making it harder for people to be sure they’re at the right house when they show up there, like if you fell or choked on your meal and needed an ambulance.