Game opening checker

[Ferdy]

Here is a tool to parse pgn file produced by Shredder GUI and extract games
when there are engine move score that are too high for either white or black. This tool is intended for Frank, but might be useful to others too.

Required files:
1. pgn-extract-17-21.exe
2. your pgn file from Shredder gui output

Sample run.

Code: Select all

GOC - Game Opening Checker v3.1

    This will read pgn produced by Shredder GUI and creates
    new pgn file based on engine evaluation found in game move
    notation on positions just after the opening book moves.
    If white move score >= +N cp in any 3 moves after the opening
    then save that game. Likewise if black move score <= -M cp in
    any 3 moves after the opening then also save that game.

    Required files:
    1. pgn-extract-17-21.exe
    2. <your_game_filename>.pgn
       These games should be from Shredder GUI with move comments.

    New in version 3.1
    1. Add option to set white and black score threshold

enter your game filename? fizbotimeforfeit.pgn
fizbotimeforfeit.pgn is missing ?!
enter again your game filename? fizbo_1.5_timeforfeit.pgn
enter white high score in cp? 70
enter black high score in cp? -40

Converting every game to a single line, please wait ...
Conversion is completed !!
duration (sec): 0.0

processing pgn file, please wait ...
game 11
pgn processing is completed!!
total games read: 11
duration (sec): 0.5
games saved with troubled opening: 3

Done!!
Press enter key to exit

Extracted games are saved in

Code: Select all

troubled_opening-fizbo_1.5_timeforfeit.pgn

This is on overwrite mode.

Python source is available and exe file for windows.
Game parsing uses python-chess module, conversion of
game into 1 line uses pgn-extract-17-21.exe file.
Conversion of python source to exe file uses pyinstaller program.

Links:
https://python-chess.readthedocs.org/en ... in-v0-11-0

http://www.cs.kent.ac.uk/people/staff/djb/pgn-extract/

https://pypi.python.org/pypi/PyInstaller/2.0

Python source is written using python version 2.7.6.
The function

Code: Select all

def get_eval_and_time(self, comment):

can be modified to read other output from different gui. At the moment this is only for shredder output pgn.

Here is the source. It is a mess, no time to refactor it further.

Code: Select all

# opening_line_checker_v3dot1.py
# Developed under python 2.7.6
"""
    This will read pgn produced by Shredder GUI and creates
    new pgn file based on engine evaluation found in game move
    notation on positions just after the opening book moves.
    If white move score >= +N cp in any 3 moves after the opening
    then save that game. Likewise if black move score <= -M cp in
    any 3 moves after the opening then also save that game.

    Required files:
    1. pgn-extract-17-21.exe
    2. <your_game_filename>.pgn
       These games should be from Shredder GUI with move comments.

    New in version 3.1
    1. Add option to set white and black score threshold
"""

from threading import Thread
import chess
import sys
from chess import pgn
import os
import time
import subprocess


# Constants
APP_NAME = "GOC - Game Opening Checker"
APP_VERSION = "3.1"


def app_name():
    """ Shows app name and version """
    print '%s v%s' %(APP_NAME, APP_VERSION)


def delete_file(fn):
    """ Delete fn file if it exists """
    if os.path.isfile(fn):
        os.remove(fn)
    
    
class GameOpeningParser(Thread):
    """ Object to process game in pgn file """
    
    def __init__(self, input_pgn_fn, pgn_extract_fn, w_highscore, b_highscore):
        Thread.__init__(self)
        self.input_pgn_fn = input_pgn_fn
        self.output_pgn_fn = 'troubled_opening-' + self.input_pgn_fn
        self.pgn_extract_fn = pgn_extract_fn
        self.pgn_file_to_process = 'app_temp_file-' + self.input_pgn_fn
        self.w_highscore = w_highscore
        self.b_highscore = b_highscore

        # Delete existing temp pgn file to process
        delete_file(self.pgn_file_to_process)

        # Delete existing pgn output file
        delete_file(self.output_pgn_fn)

    def run(self):
        """ Executes process_pgn() """
        self.process_pgn()

    def save_pgn(self, ga):
        """ Write the game in a pgn file """
        with open(self.output_pgn_fn, 'a') as pgnfo:
            pgnfo.write(str(ga) + '\n\n')

    def pgn_to_single_line(self):
        """ Run pgn-extract-17-21 to transform the pgn to a single line per game """
        print '\nConverting every game to a single line, please wait ...'
        t1 = time.clock()
        debug = False
        arg = self.pgn_extract_fn + ' -w60000 -o' +\
              self.pgn_file_to_process + ' ' + self.input_pgn_fn
        p = subprocess.Popen(arg, stdin = subprocess.PIPE,\
                           stdout = subprocess.PIPE, stderr=subprocess.STDOUT)
        if debug:
            for lines in iter(p.stdout.readline, ''):
                print lines.strip()
        p.communicate()

        # Show messages after completion
        if os.path.isfile(self.pgn_file_to_process):
            print ('Conversion is completed !!')
        else:
            print ('Warning single line conversion has failed ?!')
        print ('duration (sec): %0.1f\n' % (1.0*(time.clock() - t1)))

    def troubled_opening(self, side_, out_ofbookcount_, eval_, time_):
        """ Will determine if the opening position is not good """
        white_moveinrange = (side_ and out_ofbookcount_['W'] >= 1 and\
                             out_ofbookcount_['W'] <= 3)
        white_highval = (side_ and eval_ >= self.w_highscore)
        white_usetime = (time_ > 0)
        
        black_moveinrange = (not side_ and out_ofbookcount_['B'] >= 1 and\
                             out_ofbookcount_['B'] <= 3)
        black_highval = (not side_ and eval_ <= self.b_highscore)
        black_usetime = (time_ > 0)
        
        if (white_moveinrange and white_highval and white_usetime) or\
               (black_moveinrange and black_highval and black_usetime):
            return True
        return False

    def get_eval_and_time(self, comment):
        """ Returns eval in cp and time in sec based from move comments
            found in the game notations
        """
        # [%eval 85,29] [%emt 0:00:25] time
        # [%eval 0,0] [%emt 0:00:00] (h6)
        # [%eval 0,0] [%emt 0: 00:00]
        # [%eval -41, 22] [%emt 0:00: 14] (Kg7)
        
        a = comment.split(',')

        # Get eval
        b = a[0].strip()
        c = b.split(' ')
        d = c[1].strip()
        evalv = int(d)

        # Get time
        b = a[1].strip()
        c = b.split(':')
        d = c[2].strip()  # get the sec
        e = d.split(']')
        f = e[0].strip()
        sec = int(f)

        d = c[1].strip()  # get the minute
        minute = int(d)

        d = c[0].strip()  # get hr
        e = d.split(' ')
        f = e[-1].strip()
        hour = int(f)
        time_sec = hour*60*60 + minute*60 + sec
        
        return evalv, time_sec        

    def process_pgn(self):
        """ parse pgn file using python-chess modules """
        with open(self.pgn_file_to_process, 'r') as pgn_fo:
            game = chess.pgn.read_game(pgn_fo)
            game_cnt = 0  # Count of games read
            num_game_saved = 0

            print 'processing pgn file, please wait ...'
            t1 = time.clock()

            # Loop thru the games
            while (game):            
                game_cnt += 1
                save_game = False
                out_ofbookcount = {'W':0, 'B':0}
                w_out_of_book_cnt = 0
                b_out_of_book_cnt = 0
                print('game %d \r' % (game_cnt)),
                
                # Loop thru the position within a game
                game_node = game
                while len(game_node.variations):
                    curr_node = game_node.variation(0)
                    side = curr_node.board().turn

                    # Get and process comments
                    try:
                        com = curr_node.variations[0].comment

                        # Get eval and time                    
                        ev, tim = self.get_eval_and_time(com)

                        # Update out_ofbook counter                        
                        if side and tim > 0:
                            out_ofbookcount['W'] += 1
                        elif tim > 0 and not side:
                            out_ofbookcount['B'] += 1

                        # Write pgn if criteria is true
                        if self.troubled_opening(side, out_ofbookcount, ev, tim):
                            num_game_saved += 1
                            self.save_pgn(game)
                            save_game = True                 
                    except:
                        # Last position has no comment
                        # print curr_node.board().fen()
                        pass

                    # parse next game if current game is saved
                    if save_game:
                        break

                    # read next pos
                    game_node = curr_node
                
                # Read next game
                game = chess.pgn.read_game(pgn_fo)

        # Delete temp file after use
        delete_file(self.pgn_file_to_process)

        print ('\npgn processing is completed!!')
        print ('total games read: %d' %(game_cnt))
        print ('duration (sec): %0.1f' % (1.0*(time.clock() - t1)))        
        print ('games saved with troubled opening: %d' % (num_game_saved))
        
    
def main(argv):
    """ starts at main """
    
    app_name()
    print(__doc__)

    # Vars
    pgnin_fn = '1.pgn'    
    pgnextract_fn = 'pgn-extract-17-21.exe'

    # User input
    pgnin_fn = raw_input('enter your game filename? ')
    
    # Verify presence of input pgn file
    while True:
        if not os.path.isfile(pgnin_fn):
            print ('%s is missing ?!' % pgnin_fn)
            pgnin_fn = raw_input('enter again your game filename? ')
        else:
            break

    try:
        white_highscore = int(raw_input('enter white high score in cp? '))
    except:
        white_highscore = 70
        
    try:
        black_highscore = int(raw_input('enter black high score in cp? '))
    except:
        black_highscore = -40
        
    # Verify presence of pgn-extract-17-21.exe
    if not os.path.isfile(pgnextract_fn):
        print ('%s is missing ?!' % pgnextract_fn)
        raw_input('Press enter to exit')
        sys.exit(1)
    
    # Declare a class variable ga
    ga = GameOpeningParser(pgnin_fn, pgnextract_fn,\
                           white_highscore, black_highscore)
    
    # Reformat a new pgn file into 1 line per game
    ga.pgn_to_single_line()
    
    # Start
    ga.start()
    
    # Join threads
    ga.join()
    
    print ('\nDone!!')
    raw_input('Press enter key to exit')
    

if __name__ == "__main__":
    main(sys.argv[1:])

Dowload the exe file.
http://www.mediafire.com/download/0w322 ... v3dot1.rar

To Frank,
Could you host the exe file and the source. Regarding the source, just copy it and paste to notepad for example, then save it as

Code: Select all

opening_line_checker_v3dot1.py

Thanks Frank.
If you have other request on this tool, don't hesitate to tell me.

The sample output from sample run.

Code: Select all

[Event "40 Züge in 10 min"]
[Site "fizbo_15-x64, FCP-2"]
[Date "2015.09.07"]
[Round "3.10"]
[White "Protector 1.8.0 x64"]
[Black "Fizbo 1.5 x64"]
[Result "1-0"]
[EventDate "2015.??.??"]
[PlyCount "77"]

1. Nc3 { [%eval 0,0] [%emt 0:00:00] } c5 { [%eval 0,0] [%emt 0:00:00] } 2. Nf3 { [%eval 0,0] [%emt 0:00:00] } Nc6 { [%eval 0,0] [%emt 0:00:00] } 3. d4 { [%eval 0,0] [%emt 0:00:00] } cxd4 { [%eval 0,0] [%emt 0:00:00] } 4. Nxd4 { [%eval 0,0] [%emt 0: 00:00] } Nf6 { [%eval 0,0] [%emt 0:00:00] } 5. g3 { [%eval 0,0] [%emt 0:00:00] } d5 { [%eval 0,0] [%emt 0:00:00] } 6. Bg2 { [%eval 0,0] [%emt 0:00:00] } e5 { [%eval 0, 0] [%emt 0:00:00] } 7. Nb3 { [%eval 0,0] [%emt 0:00:00] } d4 { [%eval 0,0] [%emt 0: 00:00] } 8. Nb1 { [%eval 0,0] [%emt 0:00:00] } Be6 { [%eval 0,0] [%emt 0:00:00] } 9. c3 { [%eval 0,0] [%emt 0:00:00] } Bd5 { [%eval 0,0] [%emt 0:00:00] } 10. O-O { [%eval 0,0] [%emt 0:00:00] } Bxg2 { [%eval 0,0] [%emt 0:00:00] } 11. Kxg2 { [%eval 0,0] [%emt 0:00:00] } Qd5+ { [%eval 0,0] [%emt 0:00:00] } 12. f3 { [%eval 0,0] [%emt 0:00:00] } Rd8 { [%eval 0,0] [%emt 0:00:00] } 13. Bg5 { [%eval -51,21] [%emt 0:00:22] } Be7 { [%eval -67,19] [%emt 0:00:10] } 14. Bxf6 { [%eval -58,22] [%emt 0: 00:22] } Bxf6 { [%eval 0,0] [%emt 0:00:00] } 15. e4 { [%eval -47,21] [%emt 0:00:27] } dxe3 { [%eval -81,21] [%emt 0:00:27] } 16. Qxd5 { [%eval -42,23] [%emt 0:00:14] } Rxd5 { [%eval 0,0] [%emt 0:00:00] } 17. Na3 { [%eval -32,24] [%emt 0:00:29] } a5 { [%eval -61,21] [%emt 0:00:19] (O-O) } 18. Rfd1 { [%eval 0,21] [%emt 0:00:15] (Sc4) } Rxd1 { [%eval -114,21] [%emt 0:00:12] } 19. Rxd1 { [%eval 0,23] [%emt 0:00: 26] } a4 { [%eval -97,22] [%emt 0:00:13] (O-O) } 20. Nc5 { [%eval 17,21] [%emt 0: 00:16] } Be7 { [%eval -56,21] [%emt 0:00:14] } 21. Rd5 { [%eval 18,22] [%emt 0:00: 18] } O-O { [%eval -37,21] [%emt 0:00:30] (Lxc5) } 22. Nc4 { [%eval 7,23] [%emt 0: 00:51] } Rd8 { [%eval -32,21] [%emt 0:00:14] } 23. Rxd8+ { [%eval 35,23] [%emt 0: 00:13] } Nxd8 { [%eval 0,0] [%emt 0:00:00] } 24. Nd3 { [%eval 50,24] [%emt 0:00:16] } f6 { [%eval -33,20] [%emt 0:00:15] } 25. Nb6 { [%eval 55,23] [%emt 0:00:12] (Sxe3) } Ne6 { [%eval -58,19] [%emt 0:00:33] (a3) } 26. Nxa4 { [%eval 90,24] [%emt 0:00:17] (Kf1) } f5 { [%eval -46,20] [%emt 0:00:28] (Kf7) } 27. Nxe5 { [%eval 134, 23] [%emt 0:00:21] (Sb6) } f4 { [%eval -29,20] [%emt 0:00:30] (Ld6) } 28. Nb6 { [%eval 124,24] [%emt 0:00:15] (b4) } g5 { [%eval -28,21] [%emt 0:00:26] (Ld6) } 29. Nc8 { [%eval 143,22] [%emt 0:00:17] (Kf1) } Bd8 { [%eval -21,20] [%emt 0:00: 24] (Lf8) } 30. Nd6 { [%eval 145,21] [%emt 0:00:12] (Sd3) } Bc7 { [%eval 0,20] [%emt 0:00:15] (Sc5) } 31. Nec4 { [%eval 151,21] [%emt 0:00:16] } h5 { [%eval 0,22] [%emt 0:00:24] (Sc5) } 32. Nxb7 { [%eval 241,21] [%emt 0:00:15] (Kf1) } e2 { [%eval 4,22] [%emt 0:00:23] (Kf8) } 33. Kf2 { [%eval 260,22] [%emt 0:00:22] } fxg3+ { [%eval 128,24] [%emt 0:01:43] } 34. hxg3 { [%eval 281,24] [%emt 0:01:33] } Bxg3+ { [%eval 128,24] [%emt 0:00:37] } 35. Kxe2 { [%eval 271,22] [%emt 0:00:09] } h4 { [%eval 149,23] [%emt 0:00:52] } 36. Kf1 { [%eval 286,21] [%emt 0:00:09] } Bb8 { [%eval 176,22] [%emt 0:00:29] (Kf8) } 37. a4 { [%eval 303,21] [%emt 0:00:20] (Sbd6) } Nf4 { [%eval 149,19] [%emt 0:00:05] (Kf8) } 38. Nbd6 { [%eval 333,21] [%emt 0:00:27] (Kg1) } Ba7 { [%eval 195,18] [%emt 0:00:07] } 39. Ne5 { [%eval 346, 19] [%emt 0:00:04] time } 1-0

[Event "40 Züge in 10 min"]
[Site "fizbo_15-x64, FCP-2"]
[Date "2015.09.08"]
[Round "5.34"]
[White "Zappa Mexico II x64"]
[Black "Fizbo 1.5 x64"]
[Result "1-0"]
[EventDate "2015.??.??"]
[PlyCount "79"]

1. f4 { [%eval 0,0] [%emt 0:00:00] } g6 { [%eval 0,0] [%emt 0:00:00] } 2. e4 { [%eval 0,0] [%emt 0:00:00] } Bg7 { [%eval 0,0] [%emt 0:00:00] } 3. Nf3 { [%eval 0, 0] [%emt 0:00:00] } c5 { [%eval 0,0] [%emt 0:00:00] } 4. g3 { [%eval 0,0] [%emt 0: 00:00] } Nc6 { [%eval 0,0] [%emt 0:00:00] } 5. Bg2 { [%eval 0,0] [%emt 0:00:00] } d6 { [%eval 0,0] [%emt 0:00:00] } 6. O-O { [%eval 0,0] [%emt 0:00:00] } Nf6 { [%eval 0, 0] [%emt 0:00:00] } 7. d3 { [%eval 0,0] [%emt 0:00:00] } O-O { [%eval 0,0] [%emt 0: 00:00] } 8. h3 { [%eval 0,0] [%emt 0:00:00] } Rb8 { [%eval 0,0] [%emt 0:00:00] } 9. Qe1 { [%eval 0,0] [%emt 0:00:00] } b5 { [%eval 0,0] [%emt 0:00:00] } 10. Qf2 { [%eval 0,0] [%emt 0:00:00] } Nd7 { [%eval 0,0] [%emt 0:00:00] } 11. Nc3 { [%eval 0, 0] [%emt 0:00:00] } e6 { [%eval 0,0] [%emt 0:00:00] } 12. a4 { [%eval 0,0] [%emt 0: 00:00] } b4 { [%eval 0,0] [%emt 0:00:00] } 13. Nb5 { [%eval -30,16] [%emt 0:01:15] } Qe7 { [%eval -80,20] [%emt 0:00:10] } 14. e5 { [%eval -33,16] [%emt 0:00:43] } dxe5 { [%eval -68,21] [%emt 0:00:13] } 15. Nxe5 { [%eval -27,16] [%emt 0:00:33] } Ncxe5 { [%eval -67,20] [%emt 0:00:10] } 16. Nxa7 { [%eval -23,16] [%emt 0:00:37] } b3 { [%eval -82,20] [%emt 0:00:08] (Lb7) } 17. fxe5 { [%eval -91,15] [%emt 0:01:53] (Le3) } Nxe5 { [%eval -86,21] [%emt 0:00:11] } 18. Nb5 { [%eval -93,15] [%emt 0:00: 34] } bxc2 { [%eval -123,21] [%emt 0:00:22] } 19. Be3 { [%eval -71,15] [%emt 0:00: 18] } c1=Q { [%eval -148,22] [%emt 0:00:18] (Dd7) } 20. Raxc1 { [%eval -88,14] [%emt 0:00:17] } Nxd3 { [%eval -131,22] [%emt 0:00:11] } 21. Bxc5 { [%eval -88,14] [%emt 0:00:00] } Qd8 { [%eval -148,22] [%emt 0:00:22] } 22. Qe2 { [%eval -99,14] [%emt 0:00:31] (Dc2) } Nxc1 { [%eval -193,22] [%emt 0:00:25] } 23. Rxc1 { [%eval -99,13] [%emt 0:00:00] } Qg5 { [%eval -198,23] [%emt 0:00:20] } 24. Rc2 { [%eval -99,12] [%emt 0:00:00] } Ba6 { [%eval -213,22] [%emt 0:00:13] (Td8) } 25. Bxf8 { [%eval -103,14] [%emt 0:00:13] } Bxf8 { [%eval 0,0] [%emt 0:00:00] (Kxf8) } 26. Kh2 { [%eval -102,15] [%emt 0:00:10] } Bd6 { [%eval -204,22] [%emt 0:00:15] (Lxb5) } 27. Qd3 { [%eval -118,14] [%emt 0:00:16] } Be5 { [%eval -290,22] [%emt 0:00:34] (Td8) } 28. Re2 { [%eval -133,14] [%emt 0:00:17] (Tc5) } Bxb5 { [%eval -349,22] [%emt 0:00:14] } 29. axb5 { [%eval -133,13] [%emt 0:00:00] } Rxb5 { [%eval -359,22] [%emt 0:00:10] } 30. b3 { [%eval -142,14] [%emt 0:00:18] (De3) } h5 { [%eval -394, 23] [%emt 0:00:16] } 31. Re4 { [%eval -143,15] [%emt 0:00:13] (De3) } h4 { [%eval -480,24] [%emt 0:00:21] } 32. Rxh4 { [%eval -145,16] [%emt 0:00:13] } Qxh4 { [%eval 0,0] [%emt 0:00:00] } 33. Qxb5 { [%eval -145,16] [%emt 0:00:00] } Qxg3+ { [%eval -580,25] [%emt 0:00:14] } 34. Kg1 { [%eval -145,15] [%emt 0:00:00] } Bd4+ { [%eval -611,28] [%emt 0:00:35] (De1+) } 35. Kh1 { [%eval -143,9] [%emt 0:00:00] } Qe3 { [%eval -632,28] [%emt 0:01:01] } 36. Qf1 { [%eval -151,17] [%emt 0:00:29] } Kg7 { [%eval -698,28] [%emt 0:00:45] (Dxb3) } 37. b4 { [%eval -159,16] [%emt 0:00: 08] } f5 { [%eval -807,26] [%emt 0:02:06] } 38. Bc6 { [%eval -172,16] [%emt 0:00: 08] } e5 { [%eval -807,23] [%emt 0:00:03] } 39. b5 { [%eval -192,15] [%emt 0:00:13] (Kh2) } e4 { [%eval -943,19] [%emt 0:00:12] (f4) } 40. b6 { [%eval -210,14] [%emt 0:00:03] (h4) time } 1-0

[Event "40 Züge in 10 min"]
[Site "fizbo_15-x64, FCP-2"]
[Date "2015.09.08"]
[Round "8.10"]
[White "Fizbo 1.5 x64"]
[Black "Protector 1.8.0 x64"]
[Result "0-1"]
[EventDate "2015.??.??"]
[PlyCount "76"]

1. d4 { [%eval 0,0] [%emt 0:00:00] } Nf6 { [%eval 0,0] [%emt 0:00:00] } 2. c4 { [%eval 0,0] [%emt 0:00:00] } c5 { [%eval 0,0] [%emt 0:00:00] } 3. d5 { [%eval 0,0] [%emt 0:00:00] } e6 { [%eval 0,0] [%emt 0:00:00] } 4. Nc3 { [%eval 0,0] [%emt 0:00: 00] } d6 { [%eval 0,0] [%emt 0:00:00] } 5. Nf3 { [%eval 0,0] [%emt 0:00:00] } exd5 { [%eval 0,0] [%emt 0:00:00] } 6. cxd5 { [%eval 0,0] [%emt 0:00:00] } g6 { [%eval 0, 0] [%emt 0:00:00] } 7. h3 { [%eval 0,0] [%emt 0:00:00] } Bg7 { [%eval 0,0] [%emt 0: 00:00] } 8. e4 { [%eval 0,0] [%emt 0:00:00] } O-O { [%eval 0,0] [%emt 0:00:00] } 9. Bd3 { [%eval 0,0] [%emt 0:00:00] } a6 { [%eval 0,0] [%emt 0:00:00] } 10. O-O { [%eval 0,0] [%emt 0:00:00] } Re8 { [%eval 0,0] [%emt 0:00:00] } 11. a4 { [%eval 0, 0] [%emt 0:00:00] } Nbd7 { [%eval 0,0] [%emt 0:00:00] } 12. Bf4 { [%eval 0,0] [%emt 0:00:00] } Qc7 { [%eval 0,0] [%emt 0:00:00] } 13. Qd2 { [%eval 69,18] [%emt 0:00:30] } c4 { [%eval 73,18] [%emt 0:00:24] } 14. Bc2 { [%eval 74,19] [%emt 0:00: 10] } Rb8 { [%eval 60,20] [%emt 0:00:40] (b6) } 15. Rfe1 { [%eval 86,19] [%emt 0: 00:14] (Tfc1) } b6 { [%eval 66,19] [%emt 0:00:15] (Sh5) } 16. Rac1 { [%eval 82,19] [%emt 0:00:18] (Tab1) } Bb7 { [%eval 64,20] [%emt 0:00:41] } 17. Bh6 { [%eval 78, 20] [%emt 0:00:23] (De2) } Nc5 { [%eval 45,19] [%emt 0:00:17] (Dc5) } 18. Bxg7 { [%eval 79,19] [%emt 0:00:19] } Kxg7 { [%eval 49,20] [%emt 0:00:42] } 19. Nd4 { [%eval 83,19] [%emt 0:00:15] (Te3) } Qd8 { [%eval 44,19] [%emt 0:00:14] (Sfd7) } 20. Nc6 { [%eval 98,21] [%emt 0:00:36] (Tcd1) } Bxc6 { [%eval 54,21] [%emt 0:00: 14] } 21. dxc6 { [%eval 0,0] [%emt 0:00:00] } Rc8 { [%eval 60,21] [%emt 0:00:16] } 22. Rcd1 { [%eval 84,21] [%emt 0:00:23] (Dd4) } Rxc6 { [%eval 33,22] [%emt 0:00: 16] } 23. e5 { [%eval 103,22] [%emt 0:00:11] } Nfd7 { [%eval 13,21] [%emt 0:00:23] } 24. Qd5 { [%eval 97,22] [%emt 0:00:13] (exd6) } Qc8 { [%eval 3,22] [%emt 0:00:11] (Tc8) } 25. exd6 { [%eval 52,23] [%emt 0:00:20] } Rxe1+ { [%eval 24,24] [%emt 0:00: 17] } 26. Rxe1 { [%eval 0,0] [%emt 0:00:00] } Nf6 { [%eval 11,24] [%emt 0:00:13] } 27. Qe5 { [%eval 33,22] [%emt 0:00:17] (Dd4) } Qe6 { [%eval 0,23] [%emt 0:00:17] } 28. Qxe6 { [%eval 30,22] [%emt 0:00:09] } Nxe6 { [%eval 0,25] [%emt 0:00:14] } 29. Rd1 { [%eval 11,22] [%emt 0:00:32] } Kf8 { [%eval 0,23] [%emt 0:00:13] (Tc8) } 30. g3 { [%eval 63,19] [%emt 0:00:34] (f4) } Ke8 { [%eval -34,23] [%emt 0:00:17] } 31. Be4 { [%eval 0,20] [%emt 0:00:15] (Se4) } Nxe4 { [%eval -37,24] [%emt 0:00:20] } 32. Nxe4 { [%eval 0,0] [%emt 0:00:00] } Kd8 { [%eval -40,24] [%emt 0:00:27] } 33. Nf6 { [%eval -8,23] [%emt 0:01:51] (d7) } h5 { [%eval -54,22] [%emt 0:00:17] (Sc5) } 34. Ng8 { [%eval 0,24] [%emt 0:00:18] } b5 { [%eval -51,22] [%emt 0:00:19] (Kd7) } 35. axb5 { [%eval -43,23] [%emt 0:00:49] } axb5 { [%eval -70,23] [%emt 0:00:19] } 36. Nf6 { [%eval -42,22] [%emt 0:00:27] (Sh6) } b4 { [%eval -111,23] [%emt 0:00: 19] (Sc5) } 37. Ra1 { [%eval -26,22] [%emt 0:00:05] } c3 { [%eval -111,24] [%emt 0: 00:18] (Kc8) } 38. bxc3 { [%eval -149,24] [%emt 0:00:38] } b3 { [%eval -111,24] [%emt 0:00:22] time } 0-1

From first game,

13. Bg5 { [%eval -51,21] [%emt 0:00:22] } Be7 { [%eval -67,19] [%emt 0:00:10] }

[Frank Quisinsky]

Hello Ferdinand,

great ...
Looks like good but I can't checked it until Sunday in the evening (I am not at home). I will do that Sunday in the evening / Monday in the morning.

Most interesting in additional ...
To check now all the games I added in my rating list. Very easy I can see which games I overview (I checked all by hand in the past) in the last two years. This games can be replayed. And of course I can deactivated all the lines in my opening book if your tool find out critical games.

You do so many things for me ... thinking on all the time you need for it. We should organice a meeting!

Have many thanks ...
AGAIN

PS:
I had forgotten ...

Example:
If I have 1.500 newe games ... (after my short holiday in Berlin)
Tool should be saved all critical games in one database and all other games in an other database. So I can added directly "Nothing to do games" in my main database and can checked by hand the critical games only!

Best
Frank

[Ferdy]

PS:
I had forgotten ...

Example:
If I have 1.500 newe games ... (after my short holiday in Berlin)
Tool should be saved all critical games in one database and all other games in an other database. So I can added directly "Nothing to do games" in my main database and can checked by hand the critical games only!

Right, say you have src.pgn as source file, the tool will save suspect opening
in troubled-opening_src.pgn, the src.pgn is still there. So I need to save the games in a separate file that are not in troubled-opening_src.pgn. No problem, I can do this using pgn-extract, I will save this in good-opening_src.pgn.
So overall you can have 3 files,
src.pgn
troubled-opening_src.pgn
good-opening_src.pgn

I will be releasing an update, with some optimization and this new feature to save good-opening games.

[Frank Quisinsky]

Hi Ferdinand,

back from my Berlin trip.
Have to work (professional work) today and will looking after work in the late evening.

Later ...

Best
Frank

[Ferdy]

Hi Ferdinand,

back from my Berlin trip.
Have to work (professional work) today and will looking after work in the late evening.

Later ...

Best
Frank

I am re-designing everything. This tool will use the power of pgn-extract and python-chess. PGN and EPD/FEN processing, game analysis, epd analysis, test suite creator, etc.

Code: Select all

Universal PGN and EPD Tool v1

    Processes games and epd files based from customized commands implemented. When
    using high-score command, the source pgn file must be from Shredder GUI.

    :: Required files ::
    1. pgn-extract-17-21.exe
    2. your_game.pgn file

    :: Example command usage ::
    1. Extract games with draw result and with user specified moves
       short-draw my_games.pgn
    2. Extract games with time forfeit, using pgn output from Shredder GUI
       time-forfeit tourney2.pgn

enter command: help

:: HELP ::

A. command format:
command <source pgn file>
example:
short-draw mygames.pgn

B. supported commands:
high-score   >> by Frank
short-draw   >> by Frank
long-game    >> by Frank
time-forfeit >> by Frank
unique       >> remove duplicates
clean        >> remove comments, vars and NAGs

enter command: clean WorldCup2015Round1.pgn
Cleaning games, please wait ...
Cleaning games is completed!!
Total games saved: 128
Games are saved in clean-WorldCup2015Round1.pgn
Duration (sec): 0.125

enter command: 

I am testing now your commands and will upload the files later.

[Ferdy]

Sample run. You may need to type help for supported commands.

Code: Select all

Universal PGN and EPD Tool v1

    Processes games and epd files based from customized commands implemented.

    :: Required files ::
    1. pgn-extract-17-21.exe
    2. your_game.pgn file

    :: Example command usage ::
    1. Extract games with draw result and with user specified moves
       short-draw my_games.pgn
    2. Extract games with time forfeit, using pgn output from Shredder GUI
       time-forfeit tourney2.pgn
    3. Type help to display available commands

    :: Thanks to the following ::
    1. pgn-extract
       http://www.cs.kent.ac.uk/people/staff/djb/pgn-extract/
    2. python-chess
       https://github.com/niklasf/python-chess/tree/v0.11.1
    3. python 2.7.6
       https://www.python.org/download/releases/2.7.6/

enter command: help

Code: Select all

:: HELP ::

A. command format:
command <source pgn file>
example:
short-draw mygames.pgn

B. supported commands:
high-score   >> by Frank
short-draw   >> by Frank
long-game    >> by Frank
time-forfeit >> by Frank
unique       >> remove duplicates
clean        >> remove comments, vars and NAGs
credit       >> to show acknowledgment

Extract games with high score in opening from 1.pgn

Code: Select all

enter command: high-score 1.pgn
enter white high score in cp? +70
enter black high score in cp? -40

Code: Select all

high score, white: +70, black: -40

Converting every game to a single line, please wait ...
Conversion is completed !!
Total games: 12
Duration (sec): 0.028

Extracting games with high score in the opening, please wait ...
games 12
Done extracting games with high score!!
Total games read: 12
Total games saved: 3
Games are saved in high-score-1.pgn
Duration (sec): 0.958

Extracting games opposite to criteria, please wait ...
Extraction is completed !!
Games are saved in not-high-score-1.pgn
Total games: 9
Duration (sec): 0.064

Normalizing pgn file, please wait ...
Done!! normalizing pgn file

enter command:

Download the exe file.
http://www.mediafire.com/download/1t854 ... ool_v1.rar

Download the python source.
http://www.mediafire.com/view/xw6j4u6nj ... tool_v1.py

You may post your features to be added in this tool.

[Frank Quisinsky]

Hi Ferdiand,

I try it out ...

Please download the chaos Fizbo 1.5 *.pgn file, Shredder GUI produced. Such a file I got during the round robin. Included are around 60 lost on time games and a lot of games I have to checked the opening lines.

Short draw games ... no problems, it works.

With time-forfeit I got error messages (error reading move comments), also if I try to start "high-score 1.pgn" for an example. Tool stopped very fast.

Again, can be see easy with the Fizbo 1.5 file I am using for my first look!
Please download:
http://www.amateurschach.de/download/test-ferdinand.zip

That are the Fizbo 1.5 games I have after many problems during the time I am not home, time forfeits, crashes and so one ... nothing to do with your tool but with this Fizbo file you can test it out perfectly and you can see what I mean.

Best
Frank

[Ferdy]

Hi Ferdiand,

I try it out ...

Please download the chaos Fizbo 1.5 *.pgn file, Shredder GUI produced. Such a file I got during the round robin. Included are around 60 lost on time games and a lot of games I have to checked the opening lines.

Short draw games ... no problems, it works.

With time-forfeit I got error messages (error reading move comments), also if I try to start "high-score 1.pgn" for an example. Tool stopped very fast.

Again, can be see easy with the Fizbo 1.5 file I am using for my first look!
Please download:
http://www.amateurschach.de/download/test-ferdinand.zip

That are the Fizbo 1.5 games I have after many problems during the time I am not home, time forfeits, crashes and so one ... nothing to do with your tool but with this Fizbo file you can test it out perfectly and you can see what I mean.

Best
Frank

I have downloaded the fizbo file. There are comments here that are different. Like this,

Code: Select all

14. Bc2 { -0.24/18 19s }

I expected something like this,

Code: Select all

6. Ng5 { [%eval 0,0] [%emt 0:00:00] }

Are these game comments coming from Shredder gui? I need to know the format of move comments from all of your pgn files. If only this two, then I will revise the code to parse those type of comments.

[Ferdy]

Version 2, it is now able to read a different type of comment. Tested with sample Fizbo games.

Download exe file.
http://www.mediafire.com/download/piur1 ... ool_v2.rar

Download python source.
http://www.mediafire.com/view/68o2ss1cu ... tool_v2.py

[Frank Quisinsky]

Hi Ferdinand,

it works ...
Really great ... all what I need for checking new games, Shredder GUI saved.

My steps with new games in the past:

1. Loading the *.pgn database from Shredder in ChessBase GUI.
2. Looking for fast draw games, time-forfeit games, games with 300 moves (bad games).
3. Looking for games with bad openings.
4. Set "." in Shredder *.sto file for replaying the games during the test-run.
5. Cancelt with "F" bad book lines in my *.bkt opeining book.

6. After a test-run I am looking for games same ECO code produced, double games undo move 8 or higher. Such games are replayed after a test-run too.

Point 6 I forget.

Example:
Deuterium got after the Fizbo 1.5 test-run 50 new games.
Game 24 and Game 48 is ECO Code B11. Double game undo move 8 ... I replayed one of the games (latest game, game 48).

Game 4 and game 9 = B38 ... double game undo move 7 ... I do nothing.

After this ...
I added all the Fizbo games in my main databases, test-run ended and added the new opening book version on my site.

Best
Frank

PS:
Interesting is now to use the tool for checking the complete 115.000 database. So I can find out games with bad opening lines I overview. And for sure to use your tool during a test-run.