Everything I touch gets simplified...

[maksimKorzh]

Hi guys

Quite surprisingly my work on integrating a custom array based move generator (to play Chienese chess Xiangqi) into Stockfish
goes well - currently I already have a pseudo legal move generator generating moves, encoding them into integers and populating
move list. I've disabled bitboards, changed the internal board array's size (11*14), altered functions to parse FEN and print board.
(I'm working within files position.cpp, position.h, movegen.cpp, search.cpp, types.h, uci.cpp)
https://github.com/maksimKorzh/xiangqi-stockfish

To test above you can build it via:

Code: Select all

make build make build ARCH=x86-64 comp=gcc

And then try commands:
d // SF specific to print board, works within UCI loop => would print xiangqi board
position startpos moves dddd // would invoke move generator or move validation (doesn't validate yet) and prints the generated moves
Next step is to alter sf's pos.do_move() and pos.undo.move() and run perft test.

And now regarding the title of this post.
Probably it happens due to my dumb nature (see avatar). By fact I'm just disabling the most sf's native functionality and replacing it
with the patterns I've worked out within my own engines. I can already see that what I wanted initially to achieve - combine
a custom movegen generating moves for xiangqi with powerful sf's search is possible, however in order to do it I'll need to strip
lot's of stuff from search as well, just a few tech examples to show what I mean:
1. SF scores moves for move ordering using A specific structure RootMoves which is getting populated by c++'s emplace_back() method
2. Obviously the search itself is invoked within the main thread context (assuming 1 thread by default)

What I'm gonna do is to score moves during move generation and get rid of threads because it's extremely hard to debug when
the position instance within the thread might not always be the same as the one initialized globally - I first encountered that when
tried to run perft depth 1 - it generated moves but definitely for somewhat another position that I've set up but if do the same directly
it works perfectly well.

So it's really hard to say whether what I came up with eventually would still be possible to call stockfish or not...
Things that would remain from sf:
1. C++ (it's the biggest pain for me for C++ is just too complicated for me with tons of seemingly unnecessary features complicating the development)
2. Sources spitted into files (you know I love one source file engines regardless of programming language)
3. Position class and it's interface (would get significantly reduced later I hope)
4. Search routine

Now A natural question is arising why not to just right an angine from scratch and reuse sf's search?
well, I wanted to go that way, but in that case I wouldn't learn about how sf works itself and I need to say
that having a pain with C++ and world class project implementation is a really nice source of inspiration.
The more I understand how the original code was working (before I've burnt the bridges) the more amazed I get.
So probably the main reason of doing this is is to learn new things from top engine.
However it's a "learning by breaking" approach - just like when I was a kid I could break a radio to see "what is inside"...
Obviously radio stopped working forever after that and I was hated by my dad for that...
This time I'm turning palace into a barn but you just imagine how much fun that brings)

[mar]

And now regarding the title of this post.
Probably it happens due to my dumb nature (see avatar). By fact I'm just disabling the most sf's native functionality and replacing it
with the patterns I've worked out within my own engines. I can already see that what I wanted initially to achieve - combine
a custom movegen generating moves for xiangqi with powerful sf's search is possible, however in order to do it I'll need to strip
lot's of stuff from search as well, just a few tech examples to show what I mean:
1. SF scores moves for move ordering using A specific structure RootMoves which is getting populated by c++'s emplace_back() method
2. Obviously the search itself is invoked within the main thread context (assuming 1 thread by default)

What I'm gonna do is to score moves during move generation and get rid of threads because it's extremely hard to debug when
the position instance within the thread might not always be the same as the one initialized globally - I first encountered that when
tried to run perft depth 1 - it generated moves but definitely for somewhat another position that I've set up but if do the same directly
it works perfectly well.

umm...
I think SF (and many other engines) simply do command processing in the main thread. search gets its own thread. this is what I do as well, why?
because polling commands from inside search seems ugly - using a separate thread you can simply use a blocking read in the main thread, nothing fancy.

there's a reason why root moves deserve special handling, you're going to score root moves in move generator? good luck with that

also - your problems debugging search seem like you were trying to debug a multithreaded search? but if you set number of search threads to 1 you already avoid these problems - no need to remove smp functionality, if that's what you mean.

so: butchering is not simplification!

I'm disappointed because the title of this thread seems like a clickbait

EDIT: if you genuinely believe that everything you touch turns into pure gold, maybe you should offer your services to Linus,
the kernel is 10+M lines of code, I'm sure it would benefit greatly from your simplifications

[maksimKorzh]

And now regarding the title of this post.
Probably it happens due to my dumb nature (see avatar). By fact I'm just disabling the most sf's native functionality and replacing it
with the patterns I've worked out within my own engines. I can already see that what I wanted initially to achieve - combine
a custom movegen generating moves for xiangqi with powerful sf's search is possible, however in order to do it I'll need to strip
lot's of stuff from search as well, just a few tech examples to show what I mean:
1. SF scores moves for move ordering using A specific structure RootMoves which is getting populated by c++'s emplace_back() method
2. Obviously the search itself is invoked within the main thread context (assuming 1 thread by default)

What I'm gonna do is to score moves during move generation and get rid of threads because it's extremely hard to debug when
the position instance within the thread might not always be the same as the one initialized globally - I first encountered that when
tried to run perft depth 1 - it generated moves but definitely for somewhat another position that I've set up but if do the same directly
it works perfectly well.

umm...
I think SF (and many other engines) simply do command processing in the main thread. search gets its own thread. this is what I do as well, why?
because polling commands from inside search seems ugly - using a separate thread you can simply use a blocking read in the main thread, nothing fancy.

there's a reason why root moves deserve special handling, you're going to score root moves in move generator? good luck with that

also - your problems debugging search seem like you were trying to debug a multithreaded search? but if you set number of search threads to 1 you already avoid these problems - no need to remove smp functionality, if that's what you mean.

so: butchering is not simplification!

I'm disappointed because the title of this thread seems like a clickbait

Hi Martin, didn't mean to disappoint you, I just didn't know english word "butchering" in English before you've used it)
Yes, butchering Stockfish - that's the correct name for this thread, I'll try to rename it!

re: polling commands from inside search seems ugly
- I know, but initially I wanted to use SF's search to search moves for xiangqi.
I just want to focus on this first and not getting bothered by the complicated stuff I do not fully understand and
which is not the priority at the moment. By disabling threads I can now write and debug code. I couldn't do so before.

re: you're going to score root moves in move generator? good luck with that
- I did this many time before - it's a simple and straightforward way of doing things which always WORKS.

re: seem like you were trying to debug a multithreaded search?
- No, default number of threads in sf I've forked is set to 1, however the fact that position instance gets initialized twice -
Ones globally and then within the current thread context. Well at least this is how Fabian Fichter, author of Fairy Stockfish
has explained to me:

The initialization via FEN indeed is a bit tricky, since first the position of the UCI loop gets updated, and then an FEN is generated again and fed to the position objects of the threads. Therefore, if set() and fen() are not consistent, the search might get a different FEN than you put in.

see details here: https://github.com/maksimKorzh/xiangqi- ... h/issues/1

re: so: butchering is not simplification!
- I should call the thread Butchering Stockfish - this sounds way more cool) (Already renamed title, not sure if it works)

re: I'm disappointed because the title of this thread seems like a clickbait
- Martin, I'm sorry ,I didn't mean to dissapoint anyone. It's just a way to share the experience of learning something you.
I now understand that it offends you for some reason but I didn't ever meant that. And this is not a clickbait.
The way engine would work eventually would be way more simple compared to how it works now.
You may blame me for still calling it stockfish - I would accept that.
However I think it's better to clearly announce own work as a derivative rather than being hated later for
cloning sf. I still can't call it "original work" because the main power of the engine would come from sf's search,
so probably a :"butchered stockfish derivative" is a good name. Need to add it to readme)))

[maksimKorzh]

To moderators:
Could you please rename this thread to "Butchering Stockfish" - This is suggested by Martin Sedlak and this really represents the process being described much better. I ask this to avoid further confusions.

DISCLAIMER:
I couldn't change the initial post, so here's an edit:
If you're an experienced C++ dev getting hurt when principles forming the basic philosophy of the language are violated to the ground
you better simply don't read this. I'm not about offending anyone's development principles and best practices.
I do what I do because I can do it only this way.
If sharing this is inappropriate please delete this thread to avoid fighting in comments.
I don't mean to attack anyone and I'm sorry if the way I am (and write code) offends anyone.

[mar]

re: you're going to score root moves in move generator? good luck with that
- I did this many time before - it's a simple and straightforward way of doing things which always WORKS.

it will work, but is it optimal? unless you have godlike move ordering, you can do better in terms of ordering root moves.
that is - by ordering likely new PV candidates first (e.g. if you run out of time). also you won't be quite able to handle multipv this way

[mar]

I don't mean to attack anyone and I'm sorry if the way I am (and write code) offends anyone.

nah, what bothers me (I'm not offended for sure) are misleading thread titles
if you write "Everything I touch gets simplified" - I read as "I shit gold" in this context, but maybe that's just me.

of course, many people here do that (e.g. "human readable" Pyrrhic)

I'm simply allergic to bullshit, that's all.

[maksimKorzh]

re: you're going to score root moves in move generator? good luck with that
- I did this many time before - it's a simple and straightforward way of doing things which always WORKS.

it will work, but is it optimal? unless you have godlike move ordering, you can do better in terms of ordering root moves.
that is - by ordering likely new PV candidates first (e.g. if you run out of time). also you won't be quite able to handle multipv this way

Good question.
No, it's not optimal, it only gives a basic MVV_LVA, killer and history moves scoring.
However this is the approach where I can quickly get the engine actually searching.

I'm butchering stockfish not because I want to annoy someone.
I do it because more sophisticated ways of development available for most guys here
are simply not available for me due to my dumbness.
I just say if I learned to code how normal people do I would never ever write anything in my life, I swear.
But with this approach I'm currently working on xiangqi.com project, my engine Wukong https://github.com/maksimKorzh/wukong-xiangqi
Has already been integrated with xiangqi.com and now available for playing: https://play.xiangqi.com/game/computer
It's good enough to play versus humans who don't know how to play.
However later we'll want to make some server side processing with the engine (e.g. for cheating detection)
and a c++ engine would be the best option because of speed. (Wukong can run in UCI mode as well via node JS, but it's JS, it's too slow)
So here's one of the reasons why I'm doing this.

Well obviously Fairy Stockfish and Chameleon (sf7 derivative to play xiangqi) can be used, BUT
their move generators are slow as hell. For instance my Wukong, written is JS has slightly faster perft then Chameleon
and only slightly slower than Fairy Stockfish. I asked Fabian Fichter, author of Fairy Stockfish about that and... he wasn't surprised -
generalization of movegen for tons of variant resulted in removeing code optimizing movegen for chess.

Here's a video proving my words of perft results comparison:


Obviously you can check it on your own side)

I strongly believe that every developer, regardless of his "path" should be constantly improving.
For me butchering stockfish is the chance to touch "how things should be done".
The fact that I'm butchering everything now doesn't mean that I don't do things later the right way.
I'm actually going to!

You just try to understand that the reason for butchering is having too many distracting factors:
1. C++ (I only coded in C, C++ is a pain for me, calling a function from different file was a great issue for me couple of days ago, but not now already)
2. Threads (I understand what are threads, why are they used and how cool it is to use them, however working with sync code is easier)
3. Stockfish's architecture (Say year ago I was opening sf's sources and couldn't understand a single line of code. Now I can alter sources and still compile without errors and warnings, meanwhile achieving desired functionallity)

So by starting this thread I simply want to share my happiness of being able to work with a monster I couldn't even come close to not very long time ago.
Hope now it's clear.

[maksimKorzh]

I don't mean to attack anyone and I'm sorry if the way I am (and write code) offends anyone.

nah, what bothers me (I'm not offended for sure) are misleading thread titles
if you write "Everything I touch gets simplified" - I read as "I shit gold" in this context, but maybe that's just me.

of course, many people here do that (e.g. "human readable" Pyrrhic)

I'm simply allergic to bullshit, that's all.

re: "I shit gold"
- this is true, I actually do. But not to produce a crap but because this is the only available way of learning new stuff for me.
I wanted to call it like so, but I thought word "shit" might get moderated. Anyway the way how you read it is exactly what I meant, I swear!
Thank you for yet another brilliant title for my next youtube videos)

re: of course, many people here do that (e.g. "human readable" Pyrrhic)
- this thread is addressed to them mainly I believe

re: I'm simply allergic to bullshit, that's all.
- That's the reason for my apologies. This is internet and the main thing is to stay tolerant, I tried to but you're right - misleading title is the issue. I've
already asked moderators to rename it to "Butchering Stockfish" . I'm not kidding or trollig, I'm 100% serious. I spent lots of time when titling
my youtube videos exactly for this reason. You just helped me to be more precise while giving a name to what I'm doing.

[mar]

re: "I shit gold"
- this is true, I actually do. But not to produce a crap but because this is the only available way of learning new stuff for me.
I wanted to call it like so, but I thought word "shit" might get moderated. Anyway the way how you read it is exactly what I meant, I swear!
Thank you for yet another brilliant title for my next youtube videos)

haha, nice! you made my day

[maksimKorzh]

re: "I shit gold"
- this is true, I actually do. But not to produce a crap but because this is the only available way of learning new stuff for me.
I wanted to call it like so, but I thought word "shit" might get moderated. Anyway the way how you read it is exactly what I meant, I swear!
Thank you for yet another brilliant title for my next youtube videos)

haha, nice! you made my day

you made my as well