Ideas from other engines ok?

Discussion of anything and everything relating to chess playing software and machines.

Moderators: bob, hgm, Harvey Williamson

Forum rules
This textbox is used to restore diagrams posted with the [d] tag before the upgrade.
BubbaTough
Posts: 1154
Joined: Fri Jun 23, 2006 3:18 am

Re: Ideas from other engines ok?

Post by BubbaTough » Sat Jul 09, 2011 5:22 pm

Just to back Bob's primary point a little, the author of Naum once said he auto-tuned his evaluation to match Rybka's moves. I do not know if this is true, but it is completely plausible, and one could imagine his program choosing mostly the same moves, while the underlying code was different and original. The main point is its possible to have 2 different pieces of code, both original, choosing mostly the same moves.

-Sam

bob wrote: Do you not think that someone could write a _detailed_ specification for an evaluation function, describing what to evaluate, how to evaluate it, the order things should be tested, and the partial scores to be used, and then give those specs to two different people, and get two pieces of code that will look similar in structure, but won't be identical? Those "detailed specs" are the crux here, and I would say that the two resulting evaluations are "the same". So it is a complex problem, with reasonable answers after considering all angles.

Adam Hair
Posts: 3201
Joined: Wed May 06, 2009 8:31 pm
Location: Fuquay-Varina, North Carolina

Re: Ideas from other engines ok?

Post by Adam Hair » Sat Jul 09, 2011 7:12 pm

BubbaTough wrote:Just to back Bob's primary point a little, the author of Naum once said he auto-tuned his evaluation to match Rybka's moves. I do not know if this is true, but it is completely plausible, and one could imagine his program choosing mostly the same moves, while the underlying code was different and original. The main point is its possible to have 2 different pieces of code, both original, choosing mostly the same moves.

-Sam
Most likely Alexander Naumov did auto-tune Naum to act more like Rybka. But, Strelka 2.0 beta (with sources) had been around for 6-7 months before Naum 3.0 appeared. It is plausible to think that those sources helped to make Naum more like Rybka 1.0, and then subsequent tuning helped to make it more like Rybka 2.3.2a.

I am hoping to find a situation where two engines play similar moves but have different code. I'm not sure that Naum and Rybka qualify.

Adam

Addendum: It is also plausible that Strelka 2.0 beta was adjusted to play more like Rybka 2.3.2a.

bob
Posts: 20642
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: Ideas from other engines ok?

Post by bob » Sat Jul 09, 2011 8:15 pm

Uri Blass wrote:
bob wrote:
rbarreira wrote:
bob wrote:
lkaufman wrote:
bob wrote:
h1a8 wrote:Let's assume that someone want's to write an engine using the ideas of other engines while absolutely copying no code (not even in structure). Let's assume they can also achieve certain tricks as another engine but using a different means to achieve it.

Is it

1. Legally and WCCC ok
2. Legally ok but not WCCC ok
3. Neither Legally or WCCC ok ?
Legal everywhere...
I'm surprised at your answer Bob. If the new engine plays pretty much the same moves with similar evals (perhaps scaled) as the old one, regardless of how different the code is I think it would not be allowed in WCCC. Don't you agree?
No I don't. I think it would lead to a protest, an investigation, but if code was not copied then I don't think there is any way there would be a problem. Notice that taking a C program and converting it to assembly, or to Ada, doesn't qualify, as that is copying, of course.
In that case, the fact that Rybka used the same eval features as Fruit does not count as evidence of plagiarism?
Not just "features". There is much more to the comparison than that. Eval the same concepts, in the same way, in the same order, with the same partial bonuses and penalties. With the only difference being bitboard vs mailbox does count as evidence. But just because we both evaluate passed pawns does not support such a claim until the procedure itself is examined.
This is simply wrong.
The difference between fruit and Rybka1 beta is more than bitboard vs mailbox.

It is a fact that rybka1 beta has material imbalance tables that fruit does not have and it is a fact that fruit has endgame knowledge that Rybka does not have.

Even if Rybka copied part of fruit's evaluation to bitboard then it is still misleading to claim that rybka and fruit
"Eval the same concepts, in the same way, in the same order, with the same partial bonuses and penalties" and it is misleading to claim that
"the only difference being bitboard vs mailbox"
All you have to do is read first, THEN post. Nobody has said that Rybka 1.0 beta has the _identical_ evaluation from fruit. What we have said is that most (not all, but most) of fruit's eval is present in rybka 1.0 beta, and that that stuff is identical with the exception of the bitboard to mailbox conversion. NOBODY has said that rybka 1.0 beta does not have _additional_ evaluation terms that Fruit doesn't have. And in fact, the opposite has already been stated many times. Ditto for the search. There are additions to the fruit search in Rybka 1.0 beta. But that doesn't mean that most of the fruit search is not present.

I see nothing misleading about my statement. We are comparing _only_ the parts of the fruit eval that remain in Rybka 1.0 beta. And there, my statement is crystal clear and dead accurate.

lkaufman
Posts: 3760
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Ideas from other engines ok?

Post by lkaufman » Sat Jul 09, 2011 10:50 pm

bob wrote:
rbarreira wrote:
bob wrote:
rbarreira wrote:
bob wrote:
lkaufman wrote:
bob wrote:
h1a8 wrote:Let's assume that someone want's to write an engine using the ideas of other engines while absolutely copying no code (not even in structure). Let's assume they can also achieve certain tricks as another engine but using a different means to achieve it.

Is it

1. Legally and WCCC ok
2. Legally ok but not WCCC ok
3. Neither Legally or WCCC ok ?
Legal everywhere...
I'm surprised at your answer Bob. If the new engine plays pretty much the same moves with similar evals (perhaps scaled) as the old one, regardless of how different the code is I think it would not be allowed in WCCC. Don't you agree?
No I don't. I think it would lead to a protest, an investigation, but if code was not copied then I don't think there is any way there would be a problem. Notice that taking a C program and converting it to assembly, or to Ada, doesn't qualify, as that is copying, of course.
In that case, the fact that Rybka used the same eval features as Fruit does not count as evidence of plagiarism?
Not just "features". There is much more to the comparison than that. Eval the same concepts, in the same way, in the same order, with the same partial bonuses and penalties. With the only difference being bitboard vs mailbox does count as evidence. But just because we both evaluate passed pawns does not support such a claim until the procedure itself is examined.
For an engine to play exactly the same moves with the same evals as another one (as Larry hypothesized), it would need to use the same concepts with the same partial bonuses and penalties. You said that would be OK as long as no code is copied.
You go from "pretty much the same" to "exactly the same". That changes everything. Two strong evaluations _must_ be fairly correlated in their final score, wouldn't you think? Otherwise one is wrong and therefore can't be "strong" compared to the other.

But don't change the words, then ask why I think that is true. Most strong programs will agree on the move in a high percentage of the cases. But they won't agree _exactly_ and that wasn't originally posited.
Even identical engines won't always play the same move in timed play, or even in fixed depth play if MP is involved. Of course two strong engines will play the same move more often than not. Let's say that two unrelated strong engines (for example SF and Rybka) play the same move in timed play 70% of the time (I don't know the real figure). If another engine comes along and plays the same move as one of these programs 97% of the time, I would call it a clone or at least a derivative. If another engine plays the same move maybe 80% of the time, but this percentage increases to 97% if we reduce the allowed time, then I would consider it an improved derivative. There has to be some percentage short of 100% that would convince you that one program is derived from another, especially with regard to the eval. Some day there may be agreement on the proper eval for a chess program, but we are far from that point now. Just compare evals of SF or Crafty with Houdini for example.

bob
Posts: 20642
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: Ideas from other engines ok?

Post by bob » Sun Jul 10, 2011 1:57 am

lkaufman wrote:
bob wrote:
rbarreira wrote:
bob wrote:
rbarreira wrote:
bob wrote:
lkaufman wrote:
bob wrote:
h1a8 wrote:Let's assume that someone want's to write an engine using the ideas of other engines while absolutely copying no code (not even in structure). Let's assume they can also achieve certain tricks as another engine but using a different means to achieve it.

Is it

1. Legally and WCCC ok
2. Legally ok but not WCCC ok
3. Neither Legally or WCCC ok ?
Legal everywhere...
I'm surprised at your answer Bob. If the new engine plays pretty much the same moves with similar evals (perhaps scaled) as the old one, regardless of how different the code is I think it would not be allowed in WCCC. Don't you agree?
No I don't. I think it would lead to a protest, an investigation, but if code was not copied then I don't think there is any way there would be a problem. Notice that taking a C program and converting it to assembly, or to Ada, doesn't qualify, as that is copying, of course.
In that case, the fact that Rybka used the same eval features as Fruit does not count as evidence of plagiarism?
Not just "features". There is much more to the comparison than that. Eval the same concepts, in the same way, in the same order, with the same partial bonuses and penalties. With the only difference being bitboard vs mailbox does count as evidence. But just because we both evaluate passed pawns does not support such a claim until the procedure itself is examined.
For an engine to play exactly the same moves with the same evals as another one (as Larry hypothesized), it would need to use the same concepts with the same partial bonuses and penalties. You said that would be OK as long as no code is copied.
You go from "pretty much the same" to "exactly the same". That changes everything. Two strong evaluations _must_ be fairly correlated in their final score, wouldn't you think? Otherwise one is wrong and therefore can't be "strong" compared to the other.

But don't change the words, then ask why I think that is true. Most strong programs will agree on the move in a high percentage of the cases. But they won't agree _exactly_ and that wasn't originally posited.
Even identical engines won't always play the same move in timed play, or even in fixed depth play if MP is involved. Of course two strong engines will play the same move more often than not. Let's say that two unrelated strong engines (for example SF and Rybka) play the same move in timed play 70% of the time (I don't know the real figure). If another engine comes along and plays the same move as one of these programs 97% of the time, I would call it a clone or at least a derivative. If another engine plays the same move maybe 80% of the time, but this percentage increases to 97% if we reduce the allowed time, then I would consider it an improved derivative. There has to be some percentage short of 100% that would convince you that one program is derived from another, especially with regard to the eval. Some day there may be agreement on the proper eval for a chess program, but we are far from that point now. Just compare evals of SF or Crafty with Houdini for example.
Sure, but let's first define the parameters.

1. Just play the same move n% of the time. N has to be quite high to be fairly certain the two are related.

2. Show the same PV n% of the time. N can be lower, because there are more opportunities for differencies to show up.

3. Show the same PV with the same score N % of the time. Now N can be even lower, because we are getting too much similarity when you see the same score and PV, particularly at the same depth (although in the case of Rybka, this can be problematic if the program falsely reports depth (or even scores)).

But just the same move is, to me, not really convincing unless the percentage is quite high. But add in the PV or score or both, and it drops right on down...

lkaufman
Posts: 3760
Joined: Sun Jan 10, 2010 5:15 am
Location: Maryland USA
Contact:

Re: Ideas from other engines ok?

Post by lkaufman » Sun Jul 10, 2011 2:57 am

bob wrote:
lkaufman wrote: Even identical engines won't always play the same move in timed play, or even in fixed depth play if MP is involved. Of course two strong engines will play the same move more often than not. Let's say that two unrelated strong engines (for example SF and Rybka) play the same move in timed play 70% of the time (I don't know the real figure). If another engine comes along and plays the same move as one of these programs 97% of the time, I would call it a clone or at least a derivative. If another engine plays the same move maybe 80% of the time, but this percentage increases to 97% if we reduce the allowed time, then I would consider it an improved derivative. There has to be some percentage short of 100% that would convince you that one program is derived from another, especially with regard to the eval. Some day there may be agreement on the proper eval for a chess program, but we are far from that point now. Just compare evals of SF or Crafty with Houdini for example.
Sure, but let's first define the parameters.

1. Just play the same move n% of the time. N has to be quite high to be fairly certain the two are related.

2. Show the same PV n% of the time. N can be lower, because there are more opportunities for differencies to show up.

3. Show the same PV with the same score N % of the time. Now N can be even lower, because we are getting too much similarity when you see the same score and PV, particularly at the same depth (although in the case of Rybka, this can be problematic if the program falsely reports depth (or even scores)).

But just the same move is, to me, not really convincing unless the percentage is quite high. But add in the PV or score or both, and it drops right on down...

Then I would say we are in pretty good agreement. But note that score comparisons are tricky, because not all engines define 1.00 as the average value of an extra pawn, and obviously just multiplying or dividing the score by a constant is a trivial change. Furthermore some programs modify the score in a non-linear way; both Rybka 2.xx and Houdini do this (Houdini for the stated reason of having a given score mean a given win percentage). So I would put a lot more weight on the PV than on the reported score. But even this is tricky, as even a 1 centipawn randomizer will very often produce quite different PVs. Best is to undo any "fudging" of the reported score, if you are able to do this, and then compare the scores to the program it is suspected of cloning. In the case of Houdini, someone even made a version that undid this score transformation, making its similarity to Ivanhoe much more obvious.
For me it is obvious when one program more or less clones the eval of another. I analyze all the major openings with any strong new program in the IDeA tree of Aquarium. If two programs share a very similar eval (I mean much closer than could be attributed to independent work), the two trees will look extremely similar. So basically I can recognize a clone of the eval I did for Rybka 3 quite easily. Probably I could have done the same with Fruit and Rybka 1 had I been interested in them.

bob
Posts: 20642
Joined: Mon Feb 27, 2006 6:30 pm
Location: Birmingham, AL

Re: Ideas from other engines ok?

Post by bob » Sun Jul 10, 2011 4:51 pm

lkaufman wrote:
bob wrote:
lkaufman wrote: Even identical engines won't always play the same move in timed play, or even in fixed depth play if MP is involved. Of course two strong engines will play the same move more often than not. Let's say that two unrelated strong engines (for example SF and Rybka) play the same move in timed play 70% of the time (I don't know the real figure). If another engine comes along and plays the same move as one of these programs 97% of the time, I would call it a clone or at least a derivative. If another engine plays the same move maybe 80% of the time, but this percentage increases to 97% if we reduce the allowed time, then I would consider it an improved derivative. There has to be some percentage short of 100% that would convince you that one program is derived from another, especially with regard to the eval. Some day there may be agreement on the proper eval for a chess program, but we are far from that point now. Just compare evals of SF or Crafty with Houdini for example.
Sure, but let's first define the parameters.

1. Just play the same move n% of the time. N has to be quite high to be fairly certain the two are related.

2. Show the same PV n% of the time. N can be lower, because there are more opportunities for differencies to show up.

3. Show the same PV with the same score N % of the time. Now N can be even lower, because we are getting too much similarity when you see the same score and PV, particularly at the same depth (although in the case of Rybka, this can be problematic if the program falsely reports depth (or even scores)).

But just the same move is, to me, not really convincing unless the percentage is quite high. But add in the PV or score or both, and it drops right on down...

Then I would say we are in pretty good agreement. But note that score comparisons are tricky, because not all engines define 1.00 as the average value of an extra pawn, and obviously just multiplying or dividing the score by a constant is a trivial change. Furthermore some programs modify the score in a non-linear way; both Rybka 2.xx and Houdini do this (Houdini for the stated reason of having a given score mean a given win percentage). So I would put a lot more weight on the PV than on the reported score. But even this is tricky, as even a 1 centipawn randomizer will very often produce quite different PVs. Best is to undo any "fudging" of the reported score, if you are able to do this, and then compare the scores to the program it is suspected of cloning. In the case of Houdini, someone even made a version that undid this score transformation, making its similarity to Ivanhoe much more obvious.
For me it is obvious when one program more or less clones the eval of another. I analyze all the major openings with any strong new program in the IDeA tree of Aquarium. If two programs share a very similar eval (I mean much closer than could be attributed to independent work), the two trees will look extremely similar. So basically I can recognize a clone of the eval I did for Rybka 3 quite easily. Probably I could have done the same with Fruit and Rybka 1 had I been interested in them.
I've previously mentioned that problem, already. I have seen 100 to 256 (I think Deep Blue used 256 for pawns, perhaps another contemporary program does/did as well). The IDEA comparison seems reasonable to me, as an automated way of detecting this problem, although it still needs a human's "eye" to see the similarities.

Here's a couple of examples of "sharing ideas but not code."

Back in the middle 90's, Bruce mentioned to me that he was playing around with the idea that after you had gotten to the part of the move ordering where you have tried a few moves based on the history heuristic, that we all stop using the history heuristic after some N moves. He commented that he was playing around with the idea than when he turned the history counter code off, he also was trying to reduce the depth, or outright prune at the last few plies, and that it made his program very fast. But it was also dangerous, for obvious reasons.

I experimented with that idea quite a bit, but at the time, our depths were in the 8-10-12 range on a pentium Pro, and it was beyond risky, it hurt. But he didn't suggest code, just an idea. I did some implementations and tested. And changed things and tested again, until I decided that this was perhaps an idea for future study as it wasn't paying off. Nobody would have said that our two approaches were similar at all, but that the underlying idea, that for what we now call "late moves in the move list," one might expend less effort on them, is a reasonable idea, yet our implementations shared nothing but a vague idea and not code.

Another example, singular extensions. Hsu/Campbell wrote a paper that described a hellishly complex implementation of this. Bruce and I discussed it over quite a while and we came up with an idea of "what about searching all moves, at the front of search, with a reduced depth and a lowered window, to see if just one will fail high? And if it does, flag it as singular, and then do the normal search. If you have a flagged singular move in that normal search, extend it. Bruce seemed to produce some code that helped, although he did not test thoroughly enough to quantify any real elo gain. But it did seem stronger. I tried a couple of approaches, and none passed my testing as better. We compared code, and we had done things terribly differently, yet with the same basic idea. Yet it worked for him (apparently) and did not for me. Same idea, drastically different implementation and code, and different results.

That's why I say "copying ideas" is fine. But by "ideas" I do not mean "look at someone's code, and write your own code that does things exactly as they do them, in the same order, in the same way, with the same bonuses and penalties, etc. That's copying more than an idea, that's copying an implementation.

Ferdy
Posts: 4113
Joined: Sun Aug 10, 2008 1:15 pm
Location: Philippines

Re: Ideas from other engines ok?

Post by Ferdy » Mon Jul 11, 2011 4:24 am

Don wrote:
Ferdy wrote:
Don wrote:
h1a8 wrote:Let's assume that someone want's to write an engine using the ideas of other engines while absolutely copying no code (not even in structure). Let's assume they can also achieve certain tricks as another engine but using a different means to achieve it.

Is it

1. Legally and WCCC ok
2. Legally ok but not WCCC ok
3. Neither Legally or WCCC ok ?
None of this is a problem. There is no patent on these ideas. In fact I don't think there is anything wrong with reverse engineering a program for ideas - it's not like ideas for chess programs are "owned" by anyone. The concept that if I use an idea first nobody else can is just ridiculous.
Talking about ideas.
(1) Looking for ideas thru reverse engineering and use it to gain money is wrong?
(2) Looking for ideas thru reverse engineering and share those ideas to the community for progress is fine?
(3) Looking for ideas thru reverse engineering and use it but not use it to gain money is alright?
(4) Examining the open sources and use ideas from there to gain money is not ok?
(5) Examining the open sources and use ideas from there to improve his program but not to gain money is ok?
I personally think the money issue is not relevant. The answer to your question 1 is the same as the answer to question 2 in my opinion.

By including the phrase "for money" you can pre-condition the response. There is a while science around how to ask questions (and which order to ask them) which can greatly influence the answer people will give, often by leading them down a certain path implied by the questions. In critical thinking you have to sort through the nonsense. In a lot of cases information is added that is irrelevant to the answer, but can influence the response.

Here is a silly example of the same question being posed differently to influence the response:

1. Do you think illegal cloning for profit by unscrupulous people who cannot write a program on their own should be permitted?

2. Do you think that getting good ideas from the computer chess community should be consider a bad thing?

Each wording is designed to illicit a different response and implicitly emphasize different aspects of the issue, in one case to make the practice look as bad as possible and in the other case to make it look like a noble thing.
Let me answer your questions.
1. Do you think illegal cloning for profit by unscrupulous people who cannot write a program on their own should be permitted?
No. This is illegal cloning and using it for profit does not make it honorable either.
2. Do you think that getting good ideas from the computer chess community should be consider a bad thing?
It depends on how you get the ideas and whether you take it for profit.
a) If you get the ideas thru discussions then fine.
b) If you get the ideas thru open sources and you will not use it for profit then fine.
c) If you get the ideas thru open sources and use it for profit and did not acknowledge the author/source of the idea then that is not good.
d) If you get the ideas by reverse eng'g the program and that program is not patented and use it not for profit then it is fine.

Using/improving ideas from others and sharing back derived ideas will contribute to progress. Using ideas from others and hide and use it for profit is not good. I admire a person who uses ideas from others and acknowledge the person/source of the ideas, share his ideas back to the community for progress, he may use it for profit does not matter to me.

Post Reply