This seems to be required for ptr_elem_ptr with storage buffers. Note that
this does not imply that the pointer can be regarded as physical too.

Some variants of ptr_elem_ptr will need to be forbidden