Hi Keith,
The BMFA insist that when flying fpv you should have a spotter standing beside you (preferably on a buddy box) and that the model should never be out of visual sight.
Their argument is exactly what you asked, under the goggles it is impossible to determine actual position, height etc and to see if there are other aircraft (full size) in the airspace. To me this makes common sense, but is being fought by the FPV lobby, who have threatened to set up their own organisation.
Like you, I belive e there is a law in the UK which prevents anybody other than the military flying drones. I think it was Liverpool Police who fell foul of this flying a heli with camera's sensors et al, and were told to stop! :shock: