...and some of it is far off course - perhaps part of the alignment problem is a) us aligning to value (inc moral) realism, and b) designing/nudging AI that will want to track this too.

...If the 'real' game theory is discoverable (perhaps something akin to value (inc moral) realism), then AI may try to discover and align to this, thinking other advanced civs discover and align to it as well. I think parts of human values approximately tracks some of this value realism, ...

...about the universe, how that factors into offence/defence + defection/cooperation tradeoffs, and ultimately what other mature civilisations align to. I think most mature civs will lean defensive/cooperative - in that I think there are dominant instrumental reasons to do this...

one could think of schelling points in galactic game theory as totems which rational mature civs naturally come to rally around - operating like some kind of cosmic leviathan - which results in alignment pressure upon any rational agent that can reason adequately about it.

thus an ASI thinking really long term may consider what the cosmic commons of mature civs may value+want, and how their superior cognitive capabilities have shaped their values+wants (which I think deeply influences convergence to coordination over wasteful defection and resource burning war)...

Robin Hanson's talk 'Our Big Oops: We Broke Humanity’s Superpower' is on at Feb 28th 13:30 PST at Future Day
www.scifuture.org/events/futur...

.."as that needs natural selection of whole cultures. We now have less variety, weaker selection pressures, and faster changes from context and cultural activism. Our options to fix are neither easy nor attractive." - see www.scifuture.org/our-big-oops...

Hi Ken, hope to see you at Future Day