swami wrote:
I hope STS can be used for the clone detection test as it also comes with the partial credit moves which just maximizes the probing and the choices of engines will be more easily and comprehensively assessed.
Next version will be released probably by the end of this month, and we will have 1000 positions.
I thought the point of STS was that there was an objectively best move (as well as a possibly 2nd or 3rd best for partial credit). If this is the case, then the better the programs are the more they would look like each other in terms of STS results. If anything, the positions you have rejected are more likely to be good test candidates, because assumably you rejected them as not having a clear best move. Or better yet, the positions you did not even consider using, because it is completely unclear what the best move might be.
-Sam
Yes, you have raised an interesting point. I've usually sent about 160-200 positions to Dann, of which 100 gets selected because they have the best moves as well as partial credit ones. What about the rejected ones? They are rejected because there's no best objective solution.
So, yes, rejected positions might be the good test candidates because it assesses engine's input in positions where there's no clear best move. Perhaps Dann had saved a list of all the rejected positions for each one of the test?
swami wrote:
I hope STS can be used for the clone detection test as it also comes with the partial credit moves which just maximizes the probing and the choices of engines will be more easily and comprehensively assessed.
Next version will be released probably by the end of this month, and we will have 1000 positions.
I thought the point of STS was that there was an objectively best move (as well as a possibly 2nd or 3rd best for partial credit). If this is the case, then the better the programs are the more they would look like each other in terms of STS results. If anything, the positions you have rejected are more likely to be good test candidates, because assumably you rejected them as not having a clear best move. Or better yet, the positions you did not even consider using, because it is completely unclear what the best move might be.
-Sam
Yes, you have raised an interesting point. I've usually sent about 160-200 positions to Dann, of which 100 gets selected because they have the best moves as well as partial credit ones. What about the rejected ones? They are rejected because there's no best objective solution.
So, yes, rejected positions might be the good test candidates because it assesses engine's input in positions where there's no clear best move. Perhaps Dann had saved a list of all the rejected positions for each one of the test?
I have decided that the entire idea is a very bad one.
Are Rybka and Naum clones of each other?
How about Booot and ProDeo?
Chessmaster and Delfi?
This is an incredibly thorough analysis of engines that "think alike" and yet what it shows me is that there is no correlation to geneology to be assumed from this.
swami wrote:
I hope STS can be used for the clone detection test as it also comes with the partial credit moves which just maximizes the probing and the choices of engines will be more easily and comprehensively assessed.
Next version will be released probably by the end of this month, and we will have 1000 positions.
I thought the point of STS was that there was an objectively best move (as well as a possibly 2nd or 3rd best for partial credit). If this is the case, then the better the programs are the more they would look like each other in terms of STS results. If anything, the positions you have rejected are more likely to be good test candidates, because assumably you rejected them as not having a clear best move. Or better yet, the positions you did not even consider using, because it is completely unclear what the best move might be.
-Sam
Yes, you have raised an interesting point. I've usually sent about 160-200 positions to Dann, of which 100 gets selected because they have the best moves as well as partial credit ones. What about the rejected ones? They are rejected because there's no best objective solution.
So, yes, rejected positions might be the good test candidates because it assesses engine's input in positions where there's no clear best move. Perhaps Dann had saved a list of all the rejected positions for each one of the test?
I have decided that the entire idea is a very bad one.
Are Rybka and Naum clones of each other?
How about Booot and ProDeo?
Chessmaster and Delfi?
This is an incredibly thorough analysis of engines that "think alike" and yet what it shows me is that there is no correlation to geneology to be assumed from this.
I think that the number of moves is clearly too small and they are from games so they are not independent.
It is possible that a program was involved in some tactical games with many forced moves.
Taking fixed positions is clearly better then playing games and calculating ponder hit.
swami wrote:
I hope STS can be used for the clone detection test as it also comes with the partial credit moves which just maximizes the probing and the choices of engines will be more easily and comprehensively assessed.
Next version will be released probably by the end of this month, and we will have 1000 positions.
I thought the point of STS was that there was an objectively best move (as well as a possibly 2nd or 3rd best for partial credit). If this is the case, then the better the programs are the more they would look like each other in terms of STS results. If anything, the positions you have rejected are more likely to be good test candidates, because assumably you rejected them as not having a clear best move. Or better yet, the positions you did not even consider using, because it is completely unclear what the best move might be.
-Sam
Yes, you have raised an interesting point. I've usually sent about 160-200 positions to Dann, of which 100 gets selected because they have the best moves as well as partial credit ones. What about the rejected ones? They are rejected because there's no best objective solution.
So, yes, rejected positions might be the good test candidates because it assesses engine's input in positions where there's no clear best move. Perhaps Dann had saved a list of all the rejected positions for each one of the test?
I have decided that the entire idea is a very bad one.
Are Rybka and Naum clones of each other?
How about Booot and ProDeo?
Chessmaster and Delfi?
This is an incredibly thorough analysis of engines that "think alike" and yet what it shows me is that there is no correlation to geneology to be assumed from this.
Hi Dann,
we currently have problems with the ponder hit stats, new GUIs/versions have changed the PGN output and this has broken some ponder hit results.
It is a lot of work to fix and Kirill has a big list of enhancements and changes he wants to make.
swami wrote:
I hope STS can be used for the clone detection test as it also comes with the partial credit moves which just maximizes the probing and the choices of engines will be more easily and comprehensively assessed.
Next version will be released probably by the end of this month, and we will have 1000 positions.
I thought the point of STS was that there was an objectively best move (as well as a possibly 2nd or 3rd best for partial credit). If this is the case, then the better the programs are the more they would look like each other in terms of STS results. If anything, the positions you have rejected are more likely to be good test candidates, because assumably you rejected them as not having a clear best move. Or better yet, the positions you did not even consider using, because it is completely unclear what the best move might be.
-Sam
Yes, you have raised an interesting point. I've usually sent about 160-200 positions to Dann, of which 100 gets selected because they have the best moves as well as partial credit ones. What about the rejected ones? They are rejected because there's no best objective solution.
So, yes, rejected positions might be the good test candidates because it assesses engine's input in positions where there's no clear best move. Perhaps Dann had saved a list of all the rejected positions for each one of the test?
I have decided that the entire idea is a very bad one.
Are Rybka and Naum clones of each other?
How about Booot and ProDeo?
Chessmaster and Delfi?
This is an incredibly thorough analysis of engines that "think alike" and yet what it shows me is that there is no correlation to geneology to be assumed from this.
There is difference with this ponder hit statistics: The positions are not the same for each engine. In other words, the comparison between engine A vs B is done with positions played in only their games. B vs C have another set of positions. This introduces a huge amount of noise that makes the whole thing impossible to handle.
Again, the previous technique does not establish genealogy, but tries to address statistical similarity in move selection with a clustering technique. Yes, it is a bad idea to wildly extrapolate conclusions beyond what it represent. It just add an objective measure (at least try) to the perception of some people when they say "This engine is plays similar to this one".
I would like to add that I really like the idea of a tool to highlight similar move selection. While IMO not proof one way or the other it will make a useful tool for possible clone detection. It will also highlight engines with very different move selection, ideal for selecting analysis partners.
A public tool that also allows you to substitute your own positions and experiment with different time controls would be FANTASTIC.
swami wrote:
I hope STS can be used for the clone detection test as it also comes with the partial credit moves which just maximizes the probing and the choices of engines will be more easily and comprehensively assessed.
Next version will be released probably by the end of this month, and we will have 1000 positions.
I thought the point of STS was that there was an objectively best move (as well as a possibly 2nd or 3rd best for partial credit). If this is the case, then the better the programs are the more they would look like each other in terms of STS results. If anything, the positions you have rejected are more likely to be good test candidates, because assumably you rejected them as not having a clear best move. Or better yet, the positions you did not even consider using, because it is completely unclear what the best move might be.
-Sam
Yes, you have raised an interesting point. I've usually sent about 160-200 positions to Dann, of which 100 gets selected because they have the best moves as well as partial credit ones. What about the rejected ones? They are rejected because there's no best objective solution.
So, yes, rejected positions might be the good test candidates because it assesses engine's input in positions where there's no clear best move. Perhaps Dann had saved a list of all the rejected positions for each one of the test?
I have decided that the entire idea is a very bad one.
Are Rybka and Naum clones of each other?
How about Booot and ProDeo?
Chessmaster and Delfi?
This is an incredibly thorough analysis of engines that "think alike" and yet what it shows me is that there is no correlation to geneology to be assumed from this.
I think that the number of moves is clearly too small and they are from games so they are not independent.
It is possible that a program was involved in some tactical games with many forced moves.
Taking fixed positions is clearly better then playing games and calculating ponder hit.
Uri
The number of moves is from 306 to 2716. It will include all phases of the game, since the moves are counted from games played between the opponents. If we ignore all of those pairs for which the number of moves counted is less than 1000, I don't think it changes anything.
swami wrote:
I hope STS can be used for the clone detection test as it also comes with the partial credit moves which just maximizes the probing and the choices of engines will be more easily and comprehensively assessed.
Next version will be released probably by the end of this month, and we will have 1000 positions.
I thought the point of STS was that there was an objectively best move (as well as a possibly 2nd or 3rd best for partial credit). If this is the case, then the better the programs are the more they would look like each other in terms of STS results. If anything, the positions you have rejected are more likely to be good test candidates, because assumably you rejected them as not having a clear best move. Or better yet, the positions you did not even consider using, because it is completely unclear what the best move might be.
-Sam
Yes, you have raised an interesting point. I've usually sent about 160-200 positions to Dann, of which 100 gets selected because they have the best moves as well as partial credit ones. What about the rejected ones? They are rejected because there's no best objective solution.
So, yes, rejected positions might be the good test candidates because it assesses engine's input in positions where there's no clear best move. Perhaps Dann had saved a list of all the rejected positions for each one of the test?
I have decided that the entire idea is a very bad one.
Are Rybka and Naum clones of each other?
How about Booot and ProDeo?
Chessmaster and Delfi?
This is an incredibly thorough analysis of engines that "think alike" and yet what it shows me is that there is no correlation to geneology to be assumed from this.
There is difference with this ponder hit statistics: The positions are not the same for each engine. In other words, the comparison between engine A vs B is done with positions played in only their games. B vs C have another set of positions. This introduces a huge amount of noise that makes the whole thing impossible to handle.
Again, the previous technique does not establish genealogy, but tries to address statistical similarity in move selection with a clustering technique. Yes, it is a bad idea to wildly extrapolate conclusions beyond what it represent. It just add an objective measure (at least try) to the perception of some people when they say "This engine is plays similar to this one".
Miguel
This I do agree with. I think that if we do not try to read more into it than we should, then it may be a good idea and not a bad one. But if we are using this idea alone to assume some sort of guilt, then I think it is the worst idea imaginable.
Shaun wrote:I would like to add that I really like the idea of a tool to highlight similar move detection. While IMO not proof one way or the other it will make a useful tool both for possible clone detection. It will also highlight engines with very different move selection, ideal for selecting analysis partners.
I think the latter is the most useful side of the technique. If you want to pick partners, choose some that are in complete different branches.
A public tool that also allows you to substitute your own positions and experiment with different time controls would be FANTASTIC.
swami wrote:
I hope STS can be used for the clone detection test as it also comes with the partial credit moves which just maximizes the probing and the choices of engines will be more easily and comprehensively assessed.
Next version will be released probably by the end of this month, and we will have 1000 positions.
I thought the point of STS was that there was an objectively best move (as well as a possibly 2nd or 3rd best for partial credit). If this is the case, then the better the programs are the more they would look like each other in terms of STS results. If anything, the positions you have rejected are more likely to be good test candidates, because assumably you rejected them as not having a clear best move. Or better yet, the positions you did not even consider using, because it is completely unclear what the best move might be.
-Sam
Yes, you have raised an interesting point. I've usually sent about 160-200 positions to Dann, of which 100 gets selected because they have the best moves as well as partial credit ones. What about the rejected ones? They are rejected because there's no best objective solution.
So, yes, rejected positions might be the good test candidates because it assesses engine's input in positions where there's no clear best move. Perhaps Dann had saved a list of all the rejected positions for each one of the test?
I have decided that the entire idea is a very bad one.
Are Rybka and Naum clones of each other?
How about Booot and ProDeo?
Chessmaster and Delfi?
This is an incredibly thorough analysis of engines that "think alike" and yet what it shows me is that there is no correlation to geneology to be assumed from this.
I think that the number of moves is clearly too small and they are from games so they are not independent.
It is possible that a program was involved in some tactical games with many forced moves.
Taking fixed positions is clearly better then playing games and calculating ponder hit.
Uri
The number of moves is from 306 to 2716. It will include all phases of the game, since the moves are counted from games played between the opponents. If we ignore all of those pairs for which the number of moves counted is less than 1000, I don't think it changes anything.
