rexec, spike, skynet

Picture the scene, you're sitting in DoC, tapping away at the keyboard - and suddenly you wonder if anyone you know is near-by, on the machines. Now, you could just get up (remembering to put a screensaver on your machine, lest someone

alias vim emacs
you), but this is DoC. The DoC lab consists of what, 200 machines? It would be far too time consuming to go round each one and look for that person, so you script it.

The original (and I'll admit still the best) script is of course rexec (

~deb02/scripts/rexec
, coupled with its getcomps program (which currently seems chmod-hidden unless I've been outta the loop) - this curtosy of Dan. This (perl) script is fairly portable, usually works, and isn't too slow. It lets you run any program.

Next up came 'spike' (

~ms02/bin/bin/spike
), which is written again in perl, by Matt. You can therefore be assured that it does a lot of funky voodoo, involving forking and making trees of helpers to find your friend. Unfortunately all it can do is find a friend - from a fixed machine lies, which is a bit of a pain when you want to run sicstus on 200 machines (and I've done that before).

Matt also came up with some scripts that used a database and a web-interface where you could register that you are logged on into the db, and other ppl could look you up from that. I don't think it took off.

So the next offering in this chain is SkyNet ( ~toa02/Python/Skynet/skynet ), by moi. I thought it would be a good idea to write it in Python, and nick Matt's idea of making tree's of helpers. Python wasn't a good idea, forking and making trees seems to be a mixed blessing.

Skynet uses an up2date version of ssh that I have sitting in /vol/bitbucket - this means I can get ssh to terminate from trying to connect to windows booted machines quickly (Dan's solution to this was to ping the machine first). It also has timeouts so that if a machine can't load the script, vol/bitbucket or anything else, the script times out that machine (I believe spike does something similar - rexec doesn't tho :().

Unfortunately it does have the minor disadvantage that because its Python, it has some bizarre requirements (oh, and Ctrl+C doesn't quit it nicely either). If it tried to load on a machine without X it complains, and some versions of libc on some machines don't work with the version of python required (that sits in /vol/linux/mandrake).

So anyways, the benchmark. I'm sitting here on my machine, and I want to find out where my mate is. So, lets time them all (lab machines only): (I've picked some random machine with some1 logged in to find. - This isn't a really scientific test btw).)

spike - 3mins 10s

21:48:29 - toa02@vertex13:~
>~ms02/bin/bin/spike bm04
Searching lab machines running sshd for "bm04" (391:19)
+++++++++......+..+.+..+.+..++++sync20
...+..+.+.done
21:51:39 - toa02@vertex13:~ 

rexec - 19s

21:54:01 - toa02@vertex13:~
>~fpr02/scripts/getcomps -ALL | rexec -C -S "who | grep -E bm04" | & grep -v unknown
    bm04     :0           Mar  4 16:23
21:54:20 - toa02@vertex13:~

skynet - 38s

21:58:58 - toa02@vertex13:~
>~fpr02/scripts/getcomps -ALL | ~/Python/Skynet/skynet "who | grep -E bm04"
[ vertex23 ] -> [ sync20 ] bm04     :0           Mar  4 16:23
21:59:36 - toa02@vertex13:~  

So, for the main lab rexec's better. Now lets try the real test - all of the doc dns ;).

rexec - got stuck

22:02:43 - toa02@vertex13:~                                                     
>~toa02/Python/Skynet/allMachines.sh | rexec -C -S "who | grep -E bm04" | & grep
 -v unknown
. junk output
.
.

skynet - 1min 11s

22:07:46 - toa02@vertex13:~
>~toa02/Python/Skynet/allMachines.sh | ~/Python/Skynet/skynet "who | grep -E bm04"
.
. junk output
.
[ howlite ] -> [ sync20 ] bm04     :0           Mar  4 16:23
.
. junk output
.
22:08:57 - toa02@vertex13:~

Oh, I made skynet print out the chain of computers it used, here's the chain of ssh links it formed when I told it to use 2 workers per machine as opposed to 25, across as many of the 1800 machines in the DoC dns it could connect to (which was only 270 :s):

[ charis ]
[ charis ] -> [ joubert ]
[ fornax ]
[ fornax ] -> [ boss ]
[ fornax ] -> [ boss ] -> [ apple ]
[ fornax ] -> [ galileo ]
[ fornax ] -> [ giotto ]
[ jannu ]
[ jannu ] -> [ corvette ]
[ jannu ] -> [ corvette ] -> [ gramsci ]
[ jannu ] -> [ corvette ] -> [ gramsci ] -> [ manta ]
[ jannu ] -> [ corvette ] -> [ marina ]
[ jannu ] -> [ corvette ] -> [ peter ]
[ jannu ] -> [ corvette ] -> [ santiago ]
[ jannu ] -> [ corvette ] -> [ santiago ] -> [ klimt ]
[ jannu ] -> [ corvette ] -> [ santiago ] -> [ zodiac ]
[ jannu ] -> [ corvette ] -> [ tone ]
[ jannu ] -> [ corvette ] -> [ zorro ]
[ jannu ] -> [ imp ]
[ jannu ] -> [ imp ] -> [ luna01 ]
[ jannu ] -> [ imp ] -> [ luna01 ] -> [ ixi15 ]
[ jannu ] -> [ imp ] -> [ luna01 ] -> [ ixi16 ]
[ jannu ] -> [ imp ] -> [ luna02 ]
[ jannu ] -> [ imp ] -> [ luna04 ]
[ jannu ] -> [ imp ] -> [ luna04 ] -> [ ixi01 ]
[ jannu ] -> [ imp ] -> [ luna04 ] -> [ ixi02 ]
[ jannu ] -> [ imp ] -> [ luna04 ] -> [ ixi03 ]
[ jannu ] -> [ imp ] -> [ luna05 ]
[ jannu ] -> [ imp ] -> [ luna05 ] -> [ tungsten ]
[ jannu ] -> [ imp ] -> [ luna05 ] -> [ tungsten ] -> [ elgar ]
[ jannu ] -> [ imp ] -> [ luna07 ]
[ jannu ] -> [ imp ] -> [ luna08 ]
[ jannu ] -> [ imp ] -> [ luna08 ] -> [ lightyear ]
[ jannu ] -> [ imp ] -> [ luna08 ] -> [ lightyear ] -> [ lightyear ]
[ jannu ] -> [ leo ]
[ jannu ] -> [ leo ] -> [ loch03 ]
[ jannu ] -> [ leo ] -> [ loch04 ]
[ jannu ] -> [ leo ] -> [ loch04 ] -> [ thrush ]
[ jannu ] -> [ leo ] -> [ loch05 ]
[ jannu ] -> [ leo ] -> [ loch05 ] -> [ kiwi ]
[ jannu ] -> [ leo ] -> [ loch06 ]
[ jannu ] -> [ leo ] -> [ loch07 ]
[ jannu ] -> [ sepultura ]
[ lyra ]
[ lyra ] -> [ cetus ]
[ malubiting ]
[ malubiting ] -> [ howlite ]
[ malubiting ] -> [ howlite ] -> [ balch ]
[ malubiting ] -> [ howlite ] -> [ balch ] -> [ blueglass ]
[ malubiting ] -> [ howlite ] -> [ balch ] -> [ blueglass ] -> [ long ]
[ malubiting ] -> [ howlite ] -> [ balch ] -> [ blueglass ] -> [ s-bahn ]
[ malubiting ] -> [ howlite ] -> [ balch ] -> [ hopkins ]
[ malubiting ] -> [ howlite ] -> [ fir ]
[ malubiting ] -> [ howlite ] -> [ flusser ]
[ malubiting ] -> [ howlite ] -> [ flusser ] -> [ lexington ]
[ malubiting ] -> [ onyx ]
[ malubiting ] -> [ onyx ] -> [ davinci ]
[ malubiting ] -> [ sard ]
[ malubiting ] -> [ sard ] -> [ amber ]
[ malubiting ] -> [ sard ] -> [ republic ]
[ malubiting ] -> [ zoisite ]
[ malubiting ] -> [ zoisite ] -> [ changtse ]
[ malubiting ] -> [ zoisite ] -> [ corundum ]
[ malubiting ] -> [ zoisite ] -> [ sachow ]
[ savoia ]
[ savoia ] -> [ vertex39 ]
[ savoia ] -> [ vertex40 ]
[ savoia ] -> [ vertex41 ]
[ savoia ] -> [ vertex41 ] -> [ vertex36 ]
[ savoia ] -> [ vertex41 ] -> [ vertex37 ]
[ savoia ] -> [ vertex41 ] -> [ vertex38 ]
[ savoia ] -> [ vertex42 ]
[ savoia ] -> [ vertex42 ] -> [ vertex33 ]
[ savoia ] -> [ vertex42 ] -> [ vertex34 ]
[ savoia ] -> [ vertex42 ] -> [ vertex34 ] -> [ vertex32 ]
[ savoia ] -> [ vertex42 ] -> [ vertex35 ]
[ savoia ] -> [ vertex42 ] -> [ vertex35 ] -> [ vertex31 ]
[ savoia ] -> [ vertex43 ]
[ savoia ] -> [ vertex43 ] -> [ dynamic13 ]
[ savoia ] -> [ vertex43 ] -> [ dynamic13 ] -> [ vertex22 ]
[ savoia ] -> [ vertex43 ] -> [ dynamic13 ] -> [ vertex23 ]
[ savoia ] -> [ vertex43 ] -> [ dynamic13 ] -> [ vertex23 ] -> [ vertex21 ]
[ savoia ] -> [ vertex43 ] -> [ dynamic13 ] -> [ vertex24 ]
[ savoia ] -> [ vertex43 ] -> [ vertex28 ]
[ savoia ] -> [ vertex43 ] -> [ vertex30 ]
[ savoia ] -> [ vertex43 ] -> [ vertex30 ] -> [ vertex25 ]
[ savoia ] -> [ vertex43 ] -> [ vertex30 ] -> [ vertex26 ]
[ savoia ] -> [ vertex43 ] -> [ vertex30 ] -> [ vertex27 ]
[ savoia ] -> [ vertex44 ]
[ savoia ] -> [ vertex44 ] -> [ vertex14 ]
[ savoia ] -> [ vertex44 ] -> [ vertex15 ]
[ savoia ] -> [ vertex44 ] -> [ vertex16 ]
[ savoia ] -> [ vertex44 ] -> [ vertex17 ]
[ savoia ] -> [ vertex44 ] -> [ vertex17 ] -> [ vertex12 ]
[ savoia ] -> [ vertex44 ] -> [ vertex17 ] -> [ vertex13 ]
[ savoia ] -> [ vertex44 ] -> [ vertex18 ]
[ savoia ] -> [ vertex44 ] -> [ vertex18 ] -> [ vertex03 ]
[ savoia ] -> [ vertex44 ] -> [ vertex18 ] -> [ vertex04 ]
[ savoia ] -> [ vertex44 ] -> [ vertex18 ] -> [ vertex04 ] -> [ vertex02 ]
[ savoia ] -> [ vertex44 ] -> [ vertex18 ] -> [ vertex05 ]
[ savoia ] -> [ vertex44 ] -> [ vertex18 ] -> [ vertex05 ] -> [ vertex01 ]
[ savoia ] -> [ vertex44 ] -> [ vertex18 ] -> [ vertex06 ]
[ savoia ] -> [ vertex44 ] -> [ vertex18 ] -> [ vertex06 ] -> [ dynamic01 ]
[ savoia ] -> [ vertex44 ] -> [ vertex19 ]
[ savoia ] -> [ vertex44 ] -> [ vertex19 ] -> [ dynamic11 ]
[ savoia ] -> [ vertex44 ] -> [ vertex19 ] -> [ vertex10 ]
[ savoia ] -> [ vertex44 ] -> [ vertex19 ] -> [ vertex10 ] -> [ vertex07 ]
[ savoia ] -> [ vertex44 ] -> [ vertex19 ] -> [ vertex10 ] -> [ vertex08 ]
[ savoia ] -> [ vertex44 ] -> [ vertex19 ] -> [ vertex11 ]
[ savoia ] -> [ vertex44 ] -> [ vertex19 ] -> [ vertex11 ] -> [ vertex09 ]
[ savoia ] -> [ vertex45 ]
[ savoia ] -> [ vertex45 ] -> [ brouwer ]
[ savoia ] -> [ vertex45 ] -> [ brouwer ] -> [ lecture145 ]
[ savoia ] -> [ vertex45 ] -> [ flora ]
[ savoia ] -> [ vertex45 ] -> [ flora ] -> [ glyph07 ]
[ savoia ] -> [ vertex45 ] -> [ flora ] -> [ glyph07 ] -> [ glyph06 ]
[ savoia ] -> [ vertex45 ] -> [ flora ] -> [ glyph07 ] -> [ glyph06 ] -> [ fuaim ]
[ savoia ] -> [ vertex45 ] -> [ flora ] -> [ glyph07 ] -> [ redcar ]
[ savoia ] -> [ vertex45 ] -> [ flora ] -> [ helium ]
[ savoia ] -> [ vertex45 ] -> [ glyph09 ]
[ savoia ] -> [ vertex45 ] -> [ zeus ]
[ savoia ] -> [ vertex45 ] -> [ zeus ] -> [ glyph08 ]
[ savoia ] -> [ vertex45 ] -> [ zeus ] -> [ glyph08 ] -> [ shasta ]
[ savoia ] -> [ vertex45 ] -> [ zeus ] -> [ lenin ]
[ savoia ] -> [ vertex46 ]
[ savoia ] -> [ vertex46 ] -> [ ascot ]
[ savoia ] -> [ vertex46 ] -> [ ascot ] -> [ glyph15 ]
[ savoia ] -> [ vertex46 ] -> [ ascot ] -> [ glyph16 ]
[ savoia ] -> [ vertex46 ] -> [ ascot ] -> [ glyph18 ]
[ savoia ] -> [ vertex46 ] -> [ ascot ] -> [ glyph18 ] -> [ active20 ]
[ savoia ] -> [ vertex46 ] -> [ ascot ] -> [ glyph18 ] -> [ active20 ] -> [ active10 ]
[ savoia ] -> [ vertex46 ] -> [ ascot ] -> [ glyph18 ] -> [ active20 ] -> [ active10 ] -> [ active04 ]
[ savoia ] -> [ vertex46 ] -> [ ascot ] -> [ glyph18 ] -> [ active20 ] -> [ glyph11 ]
[ savoia ] -> [ vertex46 ] -> [ ascot ] -> [ glyph18 ] -> [ active20 ] -> [ glyph11 ] -> [ active06 ]
[ savoia ] -> [ vertex46 ] -> [ ascot ] -> [ glyph18 ] -> [ glyph12 ]
[ savoia ] -> [ vertex46 ] -> [ ascot ] -> [ glyph18 ] -> [ glyph12 ] -> [ active12 ]
[ savoia ] -> [ vertex46 ] -> [ ascot ] -> [ glyph18 ] -> [ glyph12 ] -> [ active14 ]
[ savoia ] -> [ vertex46 ] -> [ ascot ] -> [ glyph18 ] -> [ glyph12 ] -> [ active14 ] -> [ active11 ]
[ savoia ] -> [ vertex46 ] -> [ glyph30 ]
[ savoia ] -> [ vertex46 ] -> [ glyph30 ] -> [ glyph29 ]
[ savoia ] -> [ vertex46 ] -> [ glyph32 ]
[ savoia ] -> [ vertex46 ] -> [ glyph32 ] -> [ glyph26 ]
[ savoia ] -> [ vertex46 ] -> [ glyph32 ] -> [ glyph28 ]
[ savoia ] -> [ vertex46 ] -> [ holt ]
[ savoia ] -> [ vertex46 ] -> [ holt ] -> [ glyph23 ]
[ savoia ] -> [ vertex46 ] -> [ holt ] -> [ glyph24 ]
[ savoia ] -> [ vertex46 ] -> [ holt ] -> [ glyph25 ]
[ savoia ] -> [ vertex46 ] -> [ holt ] -> [ glyph25 ] -> [ glyph02 ]
[ savoia ] -> [ vertex46 ] -> [ holt ] -> [ glyph25 ] -> [ glyph20 ]
[ savoia ] -> [ vertex46 ] -> [ holt ] -> [ glyph25 ] -> [ glyph20 ] -> [ glyph19 ]
[ savoia ] -> [ vertex47 ]
[ savoia ] -> [ vertex47 ] -> [ dante ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ fusion10 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ fusion11 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ fusion12 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ fusion13 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ fusion13 ] -> [ fusion08 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ fusion13 ] -> [ fusion09 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ quantum01 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ quantum01 ] -> [ fusion05 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ quantum01 ] -> [ fusion06 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ quantum01 ] -> [ fusion06 ] -> [ fusion04 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ quantum01 ] -> [ fusion07 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ quantum01 ] -> [ fusion07 ] -> [ fusion03 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ quantum02 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ quantum02 ] -> [ fusion01 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ quantum02 ] -> [ fusion02 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ quantum02 ] -> [ fusion02 ] -> [ texel40 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ quantum02 ] -> [ fusion02 ] -> [ texel41 ]
[ savoia ] -> [ vertex47 ] -> [ dante ] -> [ quantum02 ] -> [ fusion02 ] -> [ zephyr ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel35 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel35 ] -> [ texel33 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel36 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel36 ] -> [ texel31 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel36 ] -> [ texel32 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel37 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel37 ] -> [ texel28 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel37 ] -> [ texel29 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel37 ] -> [ vantage ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel38 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel38 ] -> [ texel24 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel38 ] -> [ texel24 ] -> [ texel22 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel38 ] -> [ texel25 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel38 ] -> [ texel25 ] -> [ texel21 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel38 ] -> [ texel26 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel38 ] -> [ texel26 ] -> [ texel17 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel38 ] -> [ texel26 ] -> [ texel19 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel39 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel39 ] -> [ texel11 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel39 ] -> [ texel13 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel39 ] -> [ texel15 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel39 ] -> [ texel15 ] -> [ texel08 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel39 ] -> [ texel16 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel39 ] -> [ texel16 ] -> [ texel01 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel39 ] -> [ texel16 ] -> [ texel03 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel39 ] -> [ texel16 ] -> [ texel04 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel39 ] -> [ texel16 ] -> [ texel04 ] -> [ sunbeam ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel39 ] -> [ texel16 ] -> [ texel04 ] -> [ sync03 ]
[ savoia ] -> [ vertex47 ] -> [ glyph01 ] -> [ texel39 ] -> [ texel16 ] -> [ texel04 ] -> [ sync03 ] -> [ sync28 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync20 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync20 ] -> [ sync19 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync21 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync22 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync22 ] -> [ sync18 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync23 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync23 ] -> [ sync16 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync23 ] -> [ sync16 ] -> [ sync13 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ ariane ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ ariane ] -> [ visual05 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ sync01 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ sync01 ] -> [ visual01 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ sync01 ] -> [ visual10 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ sync01 ] -> [ visual11 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ sync01 ] -> [ visual11 ] -> [ rover ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ sync01 ] -> [ visual12 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ sync01 ] -> [ visual12 ] -> [ mini ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ sync01 ] -> [ visual14 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ visual06 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ visual07 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ visual07 ] -> [ visual04 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ visual09 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ visual09 ] -> [ visual03 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ visual09 ] -> [ visual03 ] -> [ snegopad ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ visual09 ] -> [ visual03 ] -> [ visual02 ]
[ savoia ] -> [ vertex47 ] -> [ glyph10 ] -> [ sync27 ] -> [ visual09 ] -> [ visual24 ]
[ savoia ] -> [ vertex47 ] -> [ quantum13 ]
[ savoia ] -> [ vertex47 ] -> [ quantum14 ]
[ savoia ] -> [ vertex47 ] -> [ quantum14 ] -> [ quantum09 ]
[ savoia ] -> [ vertex47 ] -> [ quantum14 ] -> [ quantum10 ]
[ savoia ] -> [ vertex47 ] -> [ quantum15 ]
[ savoia ] -> [ vertex47 ] -> [ quantum15 ] -> [ quantum06 ]
[ savoia ] -> [ vertex47 ] -> [ quantum15 ] -> [ quantum07 ]
[ savoia ] -> [ vertex47 ] -> [ quantum15 ] -> [ quantum07 ] -> [ quantum05 ]
[ savoia ] -> [ vertex47 ] -> [ quantum15 ] -> [ quantum08 ]
[ savoia ] -> [ vertex47 ] -> [ quantum15 ] -> [ quantum08 ] -> [ quantum04 ]
[ taurus ]
[ taurus ] -> [ gypsum ]
[ taurus ] -> [ pearl ]
[ taurus ] -> [ platinum ]
[ taurus ] -> [ willow ]
[ taurus ] -> [ willow ] -> [ farabi ]
[ taurus ] -> [ willow ] -> [ ixi ]
[ yangra ]
[ yangra ] -> [ picard ]
[ yangra ] -> [ sevendust ]
[ yangra ] -> [ sevendust ] -> [ dynamic04 ]
[ yangra ] -> [ sevendust ] -> [ dynamic04 ] -> [ dynamic19 ]
[ yangra ] -> [ sevendust ] -> [ dynamic04 ] -> [ dynamic19 ] -> [ vertex51 ]
[ yangra ] -> [ sevendust ] -> [ dynamic04 ] -> [ dynamic19 ] -> [ vertex52 ]
[ yangra ] -> [ sevendust ] -> [ dynamic04 ] -> [ dynamic19 ] -> [ vertex53 ]
[ yangra ] -> [ sevendust ] -> [ dynamic04 ] -> [ dynamic19 ] -> [ vertex53 ] -> [ dynamic15 ]
[ yangra ] -> [ sevendust ] -> [ dynamic04 ] -> [ vertex58 ]
[ yangra ] -> [ sevendust ] -> [ dynamic04 ] -> [ vertex59 ]
[ yangra ] -> [ sevendust ] -> [ dynamic04 ] -> [ vertex59 ] -> [ vertex57 ]
[ yangra ] -> [ sevendust ] -> [ dynamic04 ] -> [ vertex60 ]
[ yangra ] -> [ sevendust ] -> [ dynamic04 ] -> [ vertex60 ] -> [ vertex54 ]
[ yangra ] -> [ sevendust ] -> [ dynamic04 ] -> [ vertex60 ] -> [ vertex55 ]
[ yangra ] -> [ sevendust ] -> [ dynamic04 ] -> [ vertex60 ] -> [ vertex56 ]
[ yangra ] -> [ sevendust ] -> [ dynamic26 ]
[ yangra ] -> [ sevendust ] -> [ dynamic28 ]
[ yangra ] -> [ sevendust ] -> [ dynamic32 ]
[ yangra ] -> [ sevendust ] -> [ dynamic32 ] -> [ dynamic02 ]
[ yangra ] -> [ sevendust ] -> [ dynamic32 ] -> [ dynamic20 ]
[ yangra ] -> [ sevendust ] -> [ dynamic32 ] -> [ dynamic23 ]
[ yangra ] -> [ theatre343 ]
[ yangra ] -> [ theatre343 ] -> [ maria ]
[ yangra ] -> [ theatre343 ] -> [ spitfire ]