On a recent trip out to the seafront, I was suprised at how many others were taking photos (Probably mainly trying out Christmas presents!).
On the whole people totally ignored the camera, but I did notice one person going to have a look at some plaques just after I'd snapped them. I got the impression he hadn't noticed them before...
The only comments I've ever had have been along the lines of pleasant interested chat, though I do some rather odd photography. I wouldn't worry about comments, they are usually a benefit!
In some parts of london you'll probably be in the minority without a camera