@eyedeekay
&zzz
+R4SAS
+RN
+RN_
+Stormycloud
+T3s|4
+dr|z3d
+eche|off
+hagen
+hk
+lance074
+mareki2p
+orignal
+postman
+qend-irc2p
+snex
Arch
BubbRubb1
Danny
DeltaOreo
FreefallHeavens
HowardPlayzOfAdmin
Irc2PGuest22497
Irc2PGuest2488320
Irc2PGuest81774
Irc2PGuest89334
Onn4l7h
Onn4|7h
SigSegv
Sisyphus
Sleepy
SlippyJoe
T3s|4_
Teeed
Yotsu
acetone_
ardu
b3t4f4c3__
cumlord
dr4wd3_
eyedeekay_bnc
f00b4r_
idontpee
nnm
not_bob_afk
ohThuku1
poriori_
profetik1
r00tobo_BNC
rapidash
rascal
shiver_
solidx66
thetia
u5657
uop23ip
w8rabbit
weko
wew
wodencafe2
x74a6
eyedeekay
No problem, I noticed because the dev builds were not coming through, if the javadoc fails to build in CI then the i2pupdate.zip I publish to github doesn't get built
eyedeekay
Was on a plane most of today working tonight, git-repair finally seems to have finished which tentatively seems to have solved the runaway git processes but the RAM is still reporting significant over-allocation so I'll have to see
eyedeekay
Will spend the next few hours watching
eyedeekay
I think the 500 may have been a coincidence, the only evidence of anything so far is more OOM's
eyedeekay
I'm also about to send my first successful I2NP message to the live net.
zzz
eyedeekay, my branch half-vanished, I repushed, and now back to the 500 again. Re-branched to a new name, same thing
eyedeekay
OK that's a clue then. Hang on to your branches for the rest of the day, I'm going to take down the clearnet side for several hours while I try to reproduce the issue. Expect longer i2pgit.org outages for 24 hours while I dig around in it.
eyedeekay
Maybe reducing the log noise will help me find what's actually wrong
eyedeekay
Crash happened this morning during another git cat-file --batch-check call, but the stacktrace shows bleve was where it crashed. I'm going to try backporting the outstanding PR that patches out bleve since apparently it is not obeying my instructions to not use the indexer and see if that fixes it
zzz
ok thanks maybe try to create your own PR and see what happens
zzz
the URL from yesterday returns 500 still
eyedeekay
That's what I'm going to do
zzz
and this one redirects to that one: git.idk.i2p/zzz/i2p.i2p/pulls/new/tunnel-limiter
zzz
is it 500 for you?
eyedeekay
Thanks, dunno yet I'm checking it now
zzz
I also did a new branch this morning which is tunnel-limiter2 it does the same thing
eyedeekay
Yup that's a 500, which is probably going to be helpful
zzz
super
eyedeekay
A non-crashing 500 too, that's interesting
eyedeekay
Maybe not the OOM here
zzz
yeah different from all previous problems
eyedeekay
Yeah. The thing I changed yesterday was to turn on a new gitea feature that disables external access to expensive pages which may have reduced the ram explosion issues but this looks different, maybe easier to spot
zzz
note that I did a different PR #519 on sunday and did not hit this issue
eyedeekay
Yeah that seems odd also, possibly related to the settings change I don't know how though, the binary on the server is the same so it's not some gitea code change
eyedeekay
AHA. New error message found.
eyedeekay
Going to chase this for a while
eyedeekay
OK some recent checkins are missing but I'll re-sync them, branch information is repaired in gitea which may have been the root of the 500, still hang on to the branches until tomorrow while I dig around some more, still expect extended downtime today, will make sure it's all good to accept new PR's by tomorrow
eyedeekay
I *may* have also fixed the OOM by doing this but I can't be sure yet
zzz
should I retry the PR or wait until you tell me?
eyedeekay
Not yet, I'm going to finish reading a few more forum threads but this is looking promising, this is the longest it's stayed up for 3 weeks without crash-and-restart
eyedeekay
Probably about an hour before I'm reasonably sure it's good
zzz
ok
eyedeekay
Change took effect without a restart which is giving me some weird feedback, gitea is still requesting 700GB of RAM from the server but it's not crashing anymore, which is progress, if a little odd
eyedeekay
OK go for it
eyedeekay
I'm fairly certain I just took #520 with the test PR I made so I think your broken PR is fully gone
eyedeekay
There's another issue with the other WIP PQ precalc thread PR but I think I'll have to resolve that with a manual merge
zzz
success, thanks eyedeekay
eyedeekay
No problem, glad I got it before the flyfi cut out