remote/local multi-user xwindow question

[Alan Young]

I have a central media/family computer that everyone in the house can
access.  I have several computers for individual peoples use; a
windows box for each myself and my mom, a linux box for me and three
chrome os netbooks for my older kids.  I also have two android phones
and an android tablet, but I'm not sure I want to connect to the
family box with those devices, at least in the way I'm asking about.

What I want to be able to do is allow a user to login to or create
their own xwindows session on the family box from any location
(locally or their personal computer, e.g., a chromebook), detach from
the xwindow session and reattach later (à la gnu-screen) from any
location.

I have been able to successfully setup X11vnc to display :0 for
whatever user is currently logged in.  While I've been able to play
some decent practical jokes on my kids with that, it doesn't really do
what I need.

Any pointers?
-- 
Alan Young