BayesElo recompiled and faster

[Brunetti]

Hi,
since a significant amount of time is spent reading PGN files, and for those managing a list, the PGNs being read are almost always the same with only minor additions, might it be a good idea to save all imported PGNs in a specific format (compressed or not) for quicker imports in subsequent sessions?

Alex

[mar]

Hi,
since a significant amount of time is spent reading PGN files, and for those managing a list, the PGNs being read are almost always the same with only minor additions, might it be a good idea to save all imported PGNs in a specific format (compressed or not) for quicker imports in subsequent sessions?

Alex

hi Alex,

probably, but there might be a better way - if you mean a pure rating list, then you don't really care about any moves/engine output in comments,
so it might be useful to filter and save a minimal pgn with results only and load that into bayeselo instead
(CCRL already does that for computing the rating history by day)

[mar]

The blitz update is already fairly reasonable - less than an hour and a half for me and Gabor, but the 4015 one is where a reduction will be really useful. I can't recall how long that took last time I ran it, maybe close to 3 hours.

ok, that doesn't sound too bad - I think Graham was talking about 9 hours iirc?

[Modern Times]

ok, that doesn't sound too bad - I think Graham was talking about 9 hours iirc?

I think he runs it on a SATA HDD. My times are NVME SSD. But it never ever took that long for me, even on slow hardware of the past. Just as well we turned off the pure lists, that adds a lot of time on top of that.

[Graham Banks]

ok, that doesn't sound too bad - I think Graham was talking about 9 hours iirc?

I think he runs it on a SATA HDD. My times are NVME SSD. But it never ever took that long for me, even on slow hardware of the past. Just as well we turned off the pure lists, that adds a lot of time on top of that.

The first time one runs the update, it takes longer, I suspect.

[Brunetti]

probably, but there might be a better way - if you mean a pure rating list, then you don't really care about any moves/engine output in comments,
so it might be useful to filter and save a minimal pgn with results only and load that into bayeselo instead

I already do this, but the loading time could be significantly improved, especially if you have numerous PGNs, and merging them all together has other drawbacks.

Alex

[Modern Times]

The first time one runs the update, it takes longer, I suspect.

No it doesn't, unless you're including the time it takes on a clean installation checking all the historic pgns. Once they are checked they don't get checked again unless there is a trigger.

Anyway - I just ran 4015, it took 3 hrs 45 mins, a bit longer than I remember. I'm now running again with Martin's changes and we will see what time saving there is.

[mar]

I already do this, but the loading time could be significantly improved, especially if you have numerous PGNs, and merging them all together has other drawbacks.

Alex

ok then - another thing that comes to mind is parallelization - I can imagine either a special command to read multiple pgns at once
or simply make readpgn read async and then sync and merge on other commands
I'm a bit worried that this might slow down readpgn a bit along with increased memory usage - now I think definitely a separate command
maybe "readpgnasync" - so you would use say 20 readpgnasync in a row, then it'd sync on the next command and the rest would work as usual

as for a custom binary format - I don't like that idea much. but of course the download comes with source code so anything is possible

[Roland Chastain]

Hi Martin.

Thank you for sharing. The program works very well.

I noticed a very small thing. It happened that I made a wrong manipulation and loaded an empty PGN file.

Here is the output of the program:

Code: Select all

0 game(s) loaded, 0 game(s) with unknown result ignored.
terminate called after throwing an instance of 'std::bad_array_new_length'
  what():  std::bad_array_new_length

Maybe you could add a check in case someone loads by mistake an empty file, so that the program exit more gracefully?

Regards.

Roland

[mar]

hi Roland,

this thread is half a year old, I've since moved on to other things (I don't even know where I have the modified source code anymore)

the whole point was to speed up bayeselo for CCRL updates, fixing old bugs was never the goal, sorry.

I think this minor annoyance isn't a serious issue, but thanks for reporting anyway!