The search indexes (almost) every field in every contact so you don’t need to just use name or address. In your example you could search for “blue door” and get a result. Or if you have a unique identifier like a national ID you could search for that either by typing in the number or soon, using barcode search. Unfortunately the search works particularly poorly for addresses where the number is usually too short to index.
I think we need to take another look at freetext search. The existing solution doesn’t perform well and frequently causes misunderstandings or fails to account for use cases such as yours. For example, we index all fields in the contact, but it’s clear most people just search by the name of the person/place so it would be much more efficient to just search the name, perhaps efficient enough that we could search much shorter keywords, potentially even single words
Tagging @michael here in case he has any thoughts about the overall search experience.