Khmelnitsky test: Mephisto III-S Glasgow
 

Hi chess friends,

Obviously the Glasgow program is a dear one for many, and is especially put under spotlights (with some question marks!) since the Mephisto Phoenix availability.

A new field has been opened for experiments, thanks to the huge acceleration the Mephisto Phoenix can provide.

I own no chess computer running the Glasgow program, but I can run Franz's awesome CB-Emu suite, to get an accurate emulation (yes, it deserves being named a suite rather than a program).

Several brilliant assets of CB-Emu facilitate testing activities: the ability to speed-up the run (within the performance limits of your host PC), the MessChess plugin layer, and the recently added ability to setup positions and play additional moves using referee mode (where available at the chess computer side, of course).

And the Mephisto IIIS Glasgow is available, with three versions: 7.2Mhz to reproduce an original 68000 CPU throttled with wait states, 12Mhz for the fast 68000 without wait states, and 130Mhz to provide a flavor of the Phoenix Revelation II performance.

My 10 years old laptop cannot sustain the 130Mhz version, but can run both original versions with 500% speed. The Info-2 mode of the chess computer displays the thinking time used, so I can set level-9 (analysis) and stop the thinking process on time. Then I can display the score using Info-1.

OK, all the required tools are available to enable running the Khmelnitsky test!

Next step has been to select versions, and associated speed. Of course, both original devices must enter the test, of course the new Mephisto Phoenix as well. I wanted also an intermediate version, significantly faster than the original, but not too much as this would widely extend the time required to run the test. I chose the Phoenix Revelation version (38Mhz equivalent).

For speed scaling, I leveraged the data provided by Egbert:
How does it translate into timing for my test? As some may know I use 3 minutes thinking time per position in the test; and if a move per move evaluation is needed, I use 1 minute per move. It is just a conventional choice of mine, but as I run the KT with several chess computers, I stick to it to enable cross-comparisons under the very same conditions. Don't change the measuring tool, as a golden rule!

I so let the emulator think 3mn per position / 1mn per move for both original IIIS Glasgow devices (7.2 and 12Mhz). The elapsed time to reach it is thus 36s / 12s thanks to the 5 times speed-up.

For the ~38Mz Revelation version, the allowed thinking time raises to 9mn30s per position / 3mn10s per move using the 12Mhz original emulation. That's already a lot of time, but the 500% speed reduces the elapsed to 1mn54s / 38s.

Last but not least, the Mephisto Phoenix (480Mhz equivalent) requires 2h thinking time per position (to mitigate the 12Mhz actual run) and 40mn per move. Despite the emulator speed-up, that is 24mn elapsed per position, and 8mn per move.

At this stage you should be aware of:
1) Ruud has been very successful in speeding up the program, hats off!
2) the KT is going to require a bit of time and efforts, to complete.

I kicked-off the the test a week ago, and I just reached 40%. So, for completeness + data graphs to be produced with a minimal analysis, expect a couple of weeks delay.

So yes, that is a teaser!:D
mit freundlichen Grußen,
Eric

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Thank you Eric for this beautiful report. I am very curious to see how your tuned Mephisto Glasgow will perform in the Khmelnitsky test. I suspect it will be very different in the individual areas. I expect the biggest drops in the endgame and in tactics.

Greetings
Egbert

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Zitat: Last but not least benötigt der Mephisto Phoenix (480-MHz-Äquivalent) 2 Stunden Bedenkzeit pro Position (um den tatsächlichen 12-MHz-Lauf abzuschwächen) und 40 Minuten pro Zug. Trotz der Emulator-Beschleunigung sind 24 Minuten pro Position und 8 Minuten pro Zug vergangen.

Ähm, irgendwie verstehe ich nur Bahnhof, wieso ist der Phoenix langsamer??:raffnix:

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Eigentlich ist nur das PC-emulierte Glasgow-Programm langsam. Die Denkzeit muss verlängert werden, um die gleichen Knoten zu berechnen, wie es der schnelle Mephisto Phoenix in kürzerer Zeit tut.
Schönen Gruß,
Eric

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

65% completed...

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Aha, die Knotenzahl ist die feste, unveränderliche Größe, nach der sich die anderen Parameter richten müssen.
Das ist so einfach wie die Einstein'sche Relativitätstheorie. Dort ist die Lichtgeschwindigkeit die unter allen Umständen unveränderliche Größe. Folglich müssen Massen und Längen sich ändern. Daher hört man in diesem Zusammenhang von extrem schweren Raumschiffen und zusammengestauchten Erdbällen. So schließt sich der Kreis und Physik ist wie Phoenix. Hahaha.

Gruß!

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Ich habe mal nachgemessen:
Mame auf PC erreicht nicht die Performance der Emulation auf dem Phoenix, da auf dem PC Dinge mitemuliert werden, die man nicht braucht.

Der Phoenix kommt auf die 66-fache Geschwindigkeit.

Mit Laptop aus dem Jahr 2019 kommt bei mir auf etwas über 15-fache Geschwindigkeit. An Board ist ein Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz.

Möglicherweise wird hier aber wegen der Bildwiederholrate ausgebremst.

Ohne Videoausgabe komme ich in die Nähe der 40-fachen Beschleunigung:
Folglich müsste ich bei 15-facher Geschwindigkeit etwas mehr als viermal so lange warten bis die gleiche Rechentiefe wie bei der 66-fachen Geschwindigkeit erreicht ist.

Speed 200 sagt übrigens, dass ich die 200-fache Geschwindigkeit haben wollte.

Ergänzung: Mit der 66-fachen Geschwindigkeit beziehe ich mich auf der Aussage der Millennium-Seite, die mit der Urgeschwindigkeit von ca. 7.2MHz vergleicht. Welche Geschwindigkeit Mame/Mess als 100% annimmt (7.2MHz oder 12MHz) weiß ich leider nicht.

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Hallo Markus,

vielen Dank für diesen informativen Beitrag! :)

Gruß
Egbert

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

85% completed, next post will provide results :sabber:
MfG

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Hello Eric,

I'm curious. :)

Best regards
Egbert

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Hi, time for the outcome... I shall split the data across several posts, for an easyer reading.

Let's start with a first graph displaying the skills domains of the slow Mephisto IIIS Glasgow, the one set with CPU waits (7.2 Mhz equivalent):

https://www.schachcomputer.info/foru...1&d=1675451664

The peak strength is strategy, a usually weak domain for oldies, but known to have a large impact (with tactics) on the overall strength. That is a much uncommon feature, revealing deeply thought chess software!

Opening management is strong, then middlegame and endgame loose some strength, a usual trend with our chess computers. Level of play is very weak in standard endgame positions (lack of theoretical knowledge), and the program is rather blind or reluctant to sacrifices. Tactics and calculation ability are limited, whilst threats recognition and defense are fine.

The overall KT score is 1374, to be taken with a grain of salt: the test has not been designed nor standardized for chess computers (nevertheless, some kind of standardization can be applied, I shall do my best in a later post). So, let's compare profiles to a 1374 Elo player:

https://www.schachcomputer.info/foru...1&d=1675452939

Playing level is roughly on par with regards to tactical skills (tactics, recognizing threats, attack, counterattack and defense), and as well about calculations ability, middlegame and endgame. This provides a very human-like distribution of skills, highlighted by the better strategy management. The lack of endgame theory (standard positions) is the most obvious weakness compared to the human player.

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Let's continue with the fast version of the Mephisto IIIS Glasgow: without wait states, thus 12Mhz efficient.

https://www.schachcomputer.info/foru...1&d=1675454153

The profile is a bit more round, with slight enhancements for some weak points of the slow version: tactics, attack, sacrifice. Opening (playing skills once out of book) and strategy, already strong, keep on progressing. No hope for endgame theory, the slight increase in speed cannot make for the lack of knowledge.

The overall KT score is 1441, so let's check with a puny human:

https://www.schachcomputer.info/foru...1&d=1675454801

Still this much unusual domination as far as strategy is concerned! The player should better wait for the endgame. Tactical domains keep on being on par. An interesting opponent for the human player!

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Let's speed up the device, using the Phoenix Revelation (simulated as explained top of the thread); 38Mhz equivalent:

https://www.schachcomputer.info/foru...1&d=1675455331

The overall pattern looks different. The strategy is no more the peak strength, the tactical domains now prevail, undoubtly with regards to far better calculations ability. Leveraging calculations, the Glasgow program can now strongly counterattack, and even commit or recognize sacrifice opportunities. Opening to middlegame phase is very strong.

The overall KT score is 1541, 100 Elo points stronger than the previous 12Mhz.
Compared to a similar strength human:

https://www.schachcomputer.info/foru...1&d=1675456077

No more any advantage in strategy, the human is slightly stronger at that level. Nevertheless, he will have a hard time until late middlegame!

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Hi Tibono,

your Tests are very interesting.

Thank You

best regards

Markus

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Now the long awaited cherry on the cake, the Mephisto Phoenix profile, with an overwhelming 480Mhz speed equivalence:

https://www.schachcomputer.info/foru...1&d=1675456577

Well, the bad news is strategy is now one weak point amongst few others at similar level (like attack and endgame). Not that surprising, more speed is not more knowledge! For the very same reason, standard engame positions do not progress much. On another hand, tactical capabilities reach a significantly higher level. Still not a great attacker, nevertheless now stronger defender and counterattacker. Defense is commonly strong with other chess computers, a quality the Glasgow program did not highlight until this large acceleration.

What about a human opponent? The KT score being 1755, here you are:

https://www.schachcomputer.info/foru...1&d=1675457391

The endgame knowledge of the human player is overwhelming. The Glasgow must rely on its strong opening and middlegame phase, then expect some tactical opportunity to avoid the endgame, at least get some concrete advantage before entering it.

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Of course I cannot leave without a synthesis: all in one graph!
A larger one to better zoom on the skills profiles:

https://www.schachcomputer.info/foru...1&d=1675458166

There is a striking oddity in this graph, but I think it can be explained: not only the strategic skills did not step forward with the acceleration, as previously observed, but the graph displays a regression! The good strategic guidelines embedded in the Glasgow program weight higher with short thinking time; deeper calculations steer the score away because of tactical connsiderations, that is my opinion. Maybe strategic elements are mainly computed at root level?
Another small oddity lies in the calculations, where the fast IIIS performs less then the slow one; but otherwise the calculations scores look to range accordingly to expectations. The fast IIIS must have been on a wrong track with some position, there is of course some alea related to the current score at the time the thinking process is stopped.
Opening, middlegame, tactics, threat recognition and sacrifice progress rather evenly and significantly along with speed.
As already pointed out, defense skills were stuck without any significant progress, until the large acceleration provided by the Mephisto Phoenix. Counterattack is sort of similar, with unleashed power starting with the Revelation. And the Glasgow program kept brakes on attacking skills, but that is not uncommon with our dear chess computers (a bit passive, lacking any attacking plan and awaiting tactical opportunities).

Feel welcome to comment further!
I'll be back very soon with additional data to share, but enough for today.
MfG, Eric

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Hi Eric,

Thank you very much for your detailed analysises.

Like your excellent analysis of the Chessmaster Diamond on your website, they provide valuable information about the strength and weaknesses of the engines.

I've a question regarding the test setting though:

I found a description in German language, how the Khmelnitsky test distinguishes between strategy, tactics, opening, endgame, etc.
(For those, who are interested:
https://www.freechess.info/ueebersic...ing-guide.html)

But where are the human player profiles taken from?

Best regards
Markus

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Hello Eric,

Thank you very much for this very informative test series. Especially the improved defensive capability of the Mephisto Glasgow on the Mephisto Phoenix I could also observe in the context of my competitions! The overall weak endgame performances, even under Mephisto Phoenix also correspond to my expectations and are also present in practice. I also share your assessments, which are based on your test. Really great work Eric! :top:

Greetings
Egbert

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Hi Markus, they are extracted from the data of the very same Khmelnitsky's test. The Elo values come from a data sheet in the book, providing two columns per domain (attack, counterattack,...); one column being the % of your score in the domain, the other the corresponding Elo observed per statistics across a number of players. 99% score is achieved by 2500+ players, 3% by 815 or so players. In between, the distribution vary according to each domain. Elo values for intermediate % values (they are provided per 3% step, so 3%, 6%, 9%... up to 96%, 99%) rely on interpolation.

Well, the data sheet can be red the other way around: I am a 1400 Elo player, which score (%) am I supposed to achieve per domain of chess knowledge? An interpolation can be performed between two Elo values in order to get an accurate percentage within the 3% spread.
Of course, the computer-player comparison cannot usefully rely on an Elo graph (this would just add a circle to the basic Elo-graph, plotting the player's strength). So the comparison graph is based on %, as well displaying usual, average strengths and weaknesses of the human player.
By the way, time ago I used this idea (not mentioned within the book!) to provide a graph displaying how the skills develop along to the Elo, and posted about in the Hiarcs forum. Here is the graph:

https://www.schachcomputer.info/foru...1&d=1675497228

and my comments:

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Now back to my mainstream, the Glasgow program.
This table is quite simple: Speed in Mhz, ratio based on the 7.2 basic speed, log(base 10) of the ratio, and achieved KT Elo from the test. The base of the log function has no influence on the below graph pattern:

https://www.schachcomputer.info/foru...1&d=1675498435

The curve's closeness to a straight line is obvious. This means the Elo growth follows the usual rule (a constant value gain per doubling in speed) even when reaching sky-high acceleration. I was expecting more of a drop in the curve from the Mephisto Phoenix, it is not so, well done from the Glasgow program!
I added a polynomial regression curve to get the displayed formula we will use later on.

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Next step is an easy one: let's find out the Elo gain value per doubling. No need for hard calculations, the worksheet does it for us: just scale the speed according to number of doubling, and add a linear regression line - the slope factor is the value:

https://www.schachcomputer.info/foru...1&d=1675500253

The gain is 61.48 Elo points per doubling (on average, according to the Khmelnitsky test).

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

So far we "only" leveraged the Khmelnitsky test data, in a rather straightforward way. Let's now explore a more hazardous area: try to standardize the data, introducing the knowledge we have thanks to the Wiki-Elo-List.
As the KT results do comply with the rule of thumb (constant Elo gain per doubling), to my understanding we only need a shift in the KT score to better align with the Wiki-Elo-List (meaning: add a constant value).
Do you remember the polynomial formula displayed above? We juste need to substitute a better value to the final constant (and keep the x² and x part of the formula unchanged).
So far, the scope of the test only includes two devices for which the Wiki-Elo-List provides values: the fast MephiIIIS and the Ph.Revelation. But we have a third device with a known Elo within the list: the Ph.Revelation II. Let's use it as well, as the polynomial formula can provide an estimate for its KT Elo:

The "constant-free" formula is -21.3015036253151*LOG10(SMhz/7.2)^2+244.626198127018*LOG10(SMhz/7.2)
(see above formula within the graph)
The delta is the Wiki-Elo minus the constant-free value; from which an average value is calculated.
The standardized KT Elo is of course the same polynomial pattern with the average delta as a new constant: -21.3015036253151*LOG10(SMhz/7.2)^2+244.626198127018*LOG10(SMhz/7.2)+1700.7

This provides this new graph with "better" Elo values (better meaning standardized according to the Wiki-Elo-List data available so far):

https://www.schachcomputer.info/foru...1&d=1675503031

'x' is still a bit of a complicated value (log of the speed ratio), so let's expand the x axis to the SMhz (speed in Mhz) value, and add a logarithmic regression curve to get a less cumbersome formula:

https://www.schachcomputer.info/foru...1&d=1675503317

New formula displayed: 89.7821030247115*LN(SMhz)+1531.25499276664
or, should you prefer the log function: 206.730932042356*LOG10(SMhz)+1531,25499276664

Resulting tab, using either of the above:

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Hello Eric,

very nice, the Elo numbers should now harmonise very well with our existing lists. I am curious, should Mephisto Glasgow on Mephisto Phoenix actually get to approx. 2080 Elo in my competitions, that would be a real surprise for me. My experience has shown that the performance of Mephisto Glasgow is extremely dependent on the opponent. Very nice to see in the modest result against the Revelation II AE CXG Sphinx 40 with approx. 1983 Elo. Against the Novag Super Expert C, however, it was already around 2070 Elo and currently against Mephisto Dallas 68000, 12 MHz it looks like an even better result than 2070 Elo could come out. But of course after 3 games it is not yet possible to make a reliable forecast. I hope that all my results can contribute to calibrating your highly interesting Khmelnitsky test.

Greetings
Egbert

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

As a conclusion to the outcomes of the test, the KT figures (for those who prefer figures to graphs :D):


MfG, Eric.

AW: Khmelnitsky test: Mephisto IIIS Glasgow
 

Good morning Eric,

tournament practice with the Mephisto Glasgow shows that the programme characteristics do indeed correspond to the profile you determined in the "Khmelnitsky test". I am impressed by the significance of this test procedure! If Mephisto Glasgow were improved in terms of endgame theory and endgames in general, it could rise to even higher playing strength regions, especially under Mephisto Phoenix.

Greetings
Egbert