Jjosh
is creating LeelensteinSelect a membership level
Leelenstein Fan
Limited (26 remaining)
$3
per month
Pledge/Show support for Leelenstein. Get early access to these networks
Top Fan!
Limited (0 remaining)
$5
per month
Show extra support. Possible exclusive posts, and access to network checkpoints at some LR cycle points. (When LR drops to zero). Preview of future performance that is possibly (but unlikely) stronger than final.
Einstein
Limited (0 remaining)
$10
per month
Includes all previous rewards! New discord roll. Limited to One!
About
Leela developer/tester/enthusiast. Gauging interest on current projects. Working on supervised learning now with "Leelenstein", combined with some other lc0 branch things.
Leelenstein started out mostly CCRL games, with some 40b and T30 games. Currently uses CCRL games + more T30 games + SGDR + GGT + SE, but started with the same weights as the previous to retain some information from those other games.
(There have been several different SL runs with some nice nets picked from each, and some slightly different datasets.)
Trade Penalty is also recommended (and used at CCCC in current tournament).
SGDR is new method for learning rate scheduling.
https://arxiv.org/abs/1608.03983
GGT is new SL optimizer.
https://arxiv.org/pdf/1806.02958.pdf
Gradient Noise (adding to improve start and generalization)
https://arxiv.org/abs/1511.06807
Initialization
https://medium.com/@prateekvishnu/xavier-and-he-normal-he-et-al-initialization-8e3d7a087528
Tested significantly stronger than Gull 3 on 30CPU with a 2080 TI, and beat the Sufi 32930 in a tactics test, but lost head to head.
EDIT: 8.0 added batch renormalization https://arxiv.org/abs/1702.03275
9.0 added a new optimizer
10 added label smoothing for policy
11.0 added some t40 data which didn't seem to help, but 11.1 changed some smoothing parameters and gained a fair bit
12.0 started using Leelenstein to fix the training data. Mostly value head tweaks. Changed policy head.
13.0 added some t40 games, changed value head, starting to do more policy data changes.
Has gone from being weaker than 32930 H2H to stronger than 41800.
Leelenstein started out mostly CCRL games, with some 40b and T30 games. Currently uses CCRL games + more T30 games + SGDR + GGT + SE, but started with the same weights as the previous to retain some information from those other games.
(There have been several different SL runs with some nice nets picked from each, and some slightly different datasets.)
Trade Penalty is also recommended (and used at CCCC in current tournament).
SGDR is new method for learning rate scheduling.
https://arxiv.org/abs/1608.03983
GGT is new SL optimizer.
https://arxiv.org/pdf/1806.02958.pdf
Gradient Noise (adding to improve start and generalization)
https://arxiv.org/abs/1511.06807
Initialization
https://medium.com/@prateekvishnu/xavier-and-he-normal-he-et-al-initialization-8e3d7a087528
Tested significantly stronger than Gull 3 on 30CPU with a 2080 TI, and beat the Sufi 32930 in a tactics test, but lost head to head.
EDIT: 8.0 added batch renormalization https://arxiv.org/abs/1702.03275
9.0 added a new optimizer
10 added label smoothing for policy
11.0 added some t40 data which didn't seem to help, but 11.1 changed some smoothing parameters and gained a fair bit
12.0 started using Leelenstein to fix the training data. Mostly value head tweaks. Changed policy head.
13.0 added some t40 games, changed value head, starting to do more policy data changes.
Has gone from being weaker than 32930 H2H to stronger than 41800.
Become a patron to
41
Be part of the community
Connect via private message
Recent posts by Jjosh
How it works
Get started in 2 minutes
Choose a membership
Sign up
Add a payment method
Get benefits