Nothing — at least in our section of the bird world — is all one color. Everything as a mix. I stuck with birds that are more brown than any other color. A lot of the females of a species are brown and striped or speckled. Being less conspicuous helps keep the babies safe.
The brown one that you see with the red House Finch is his mate. Only the males are red. They girls look like some kind of sparrow. Otherwise, there are Carolina Wrens and Chipping Sparrows. There are other sparrows and in the winter, the Goldfinches are not in “breeding colors” and look more brown than bright yellow. It’s interesting in the spring, watching them change color. Pieces of them turn bright yellow and not necessarily contiguous pieces. So you’ll see one with a bright yellow head, but the rest of him is rather tan. Or he’ll have one yellow wing and a few bright tail feathers. The House Finch become less red in the winter, but they don’t become a completely different color. They become rather rusty rather than true red. The wrens and sparrows are permanently brown. Most of the Mourning Doves are some shade of brown, but some are gray or “off-white.”
Not all birds change colors and not all birds are different colors male to female. You just have to get to know your local bird population. After a while, you can spot them out of the corner of your eye, but it takes a while.