y2kboy23
dr|z3d After a few hours, I had no tunnels due to thinking it had a "high load". The router only had about 16 jobs queued up.
dr|z3d
high job lab, y2kboy23, or high system load?
dr|z3d
*lag
dr|z3d
And when you say no tunnels, no transit tunnels or?
dr|z3d
if (SystemVersion.getCPULoad() > 95 && SystemVersion.getCPULoadAvg() > 90) {
dr|z3d
setTunnelStatus(_x("Rejecting all tunnel requests" + ":<br>" + _x("High system load")));
y2kboy23
It said high load
dr|z3d
That will reject requests if your cpu is at 95% current and 90% avg for a minute.
dr|z3d
should be a transient issue.
dr|z3d
so when you say no tunnels, you already had plenty of transit tunnels and it was rejecting, or? and for how long?
y2kboy23
Not sure about participating but exploratory, snark, IRC shared all were at zero
y2kboy23
I had 8 active peers.
dr|z3d
ok. and what's the rough spec on that system as seen by java? cores/ram?
y2kboy23
In the event log, I see a message about a symmetric nat. I don't have that normally.
dr|z3d
that'll be a recent upstream commit, probably. let me find it.
y2kboy23
And the build I'm on right now does not have those commits from zzz
dr|z3d
I thought you were on I2P+ master? In any event, symmetric NAT and full cone NAT has been seeing a lot of work lately.
y2kboy23
I jumped back some commits just to get it up and running.
dr|z3d
ok
y2kboy23
I checkedout at this commit gitlab.com/i2pplus/I2P.Plus/-/commit/9a8c6ad1c3ef6e61ca0c77f728ca1b3882599512
y2kboy23
And things are stable
dr|z3d
that's still got the stackoverflow bug, but as you've noticed, it's kinda rare.
y2kboy23
Sorry. My IRC client gives up too quickly
dr|z3d
you configured your ping timeout in your client?
dr|z3d
ok, suspiciously low traffic on one router and lots of build failures, testing patch...
dr|z3d
sounds somewhat like the issue you had.
y2kboy23
my client doesn't allow me to change ping timeout without editing the code. I may just drop ZNC in front of it and call it a day.
dr|z3d
ok, latest patch looks ok, router has bounced back to normal operation, will push the latest commit to master, y2kboy23
dr|z3d
Blinded message
dr|z3d
ok, merged.
y2kboy23
dr|z3d I'll pull and build.
dr|z3d
what's confusing things a little is the huge variation in bandwidth requirements from transit tunnels, and the wildly oscillating transit request numbers. really hard to get a handle on how the network's performing and your local router.
dr|z3d
I see spikes of up to 6MB/s or more for a few minutes, and then, *boom* it's gone.
y2kboy23
Yea, it seems like spikes of traffic every now and then
dr|z3d
here it's every now. not much letup, just a massive saw wave of on/off.
y2kboy23
dr|z3d I'm on the latest version. I'll let you know if I get back into that low connection mode
dr|z3d
thanks, y2kboy23
dr|z3d
main changes are the router search concurrency to bring down job lag and cpu load earlier, lower known peer thresholds to start scaling concurrency back.
T3s|4
dr|z3d: is the build server down? I've been stuck at Contacting i2p+ update server for more than 30 mins - even with hard refreshes
dr|z3d
anyways, whereas earlier I was seeing poor perf on one router, lower transit tunnels than expected, low b/w usage, things have picked up to normal levels with the latest build. so it doesn't look too bad from here at least.
dr|z3d
update server's normally always up. don't know what's the issue with that, seeing that here also from remote routers.
dr|z3d
lots of weirdness lately that's a bit hard to dissect. could be some protocol bugs, could be my bugs, could be a network attack.
T3s|4
dr|z3d: I now see my build success rate is down to 7%. I'll check if a restart helps...
dr|z3d
how long you been running the latest build, T3s|4?
dr|z3d
or rather, how latest is latest?
T3s|4
about an hour - and this version: 3b56e6fb (Build date: 2023-01-22 05:10:08 UTC)
dr|z3d
let's see where you're at.
dr|z3d
oh, wait. that's pretty old in the scheme of things.
dr|z3d
if you can't communicate with skank.i2p, pull the latest update from: gitlab.com/i2pplus/I2P.Plus/-/jobs/artifacts/master/raw/i2pupdate.zip?job=Java8
T3s|4
yeah, I know, but my fiber cable failed yesterday, and was replaced today
dr|z3d
as I was saying to y2kboy23, some weird shit happening lately.
T3s|4
^noted
dr|z3d
ah, ok, T3s|4.. just we're testing a bunch of fixes that if they hit your router, hit hard, but if they don't, then not an issue. until it is :)
dr|z3d
do check /logs if router's performing below par.
dr|z3d
build success is currently on the low side, anyways. weko on i2pd was seeing 15% earlier, I'm seeing 26% on a recently updated router.
T3s|4
ok dr|z3d - grabbed it from gitlab - restarting
y2kboy23
dr|z3d My guts says it's an attack or an attempt to disrupt. The whole dread drama probably brought more eyes to i2p.
dr|z3d
build success at 47% on one router currently.
dr|z3d
so definitely super variable right now.
y2kboy23
dr|z3d Happened again. Have two different exceptions. Cake.i2p links incoming once I upload.
dr|z3d
gah, ok, y2kboy23, thanks.
T3s|4
dr|z3d: now running 09332f6d (Build date: 2023-01-24 04:07:34 UTC) - and my router is showing an available update from 05:16 yesterday!
dr|z3d
T3s|4: that'll be the last time your router successfully checked for an update, and won't take into account any manually installed updates outside of the console's updater.
dr|z3d
thanks, y2kboy23, checking now...
dr|z3d
T3s|4: safe to ignore, in any event.
dr|z3d
if you did hit the update button and a newer version was available on skank, you'd pull that instead.
T3s|4
yep, ofc
dr|z3d
y2kboy23: ok, thanks for those logs, I think I've identified the problematic lines, new master commit about to land.
dr|z3d
done.
y2kboy23
dr|z3d Got it up and running. Too hard to tell if things are better. Nothing in the logs yet so that's a positive.
dr|z3d
y2kboy23: ok, thanks. I nailed a couple of possible issues.
y2kboy23
Thanks for all of your help!
dr|z3d
thank you for all your reports and your patience :)
y2kboy23
Uhh. 1200 dht peers on snark. Is that right?
y2kboy23
I haven't seen it that high before.
dr|z3d
y2kboy23: yeah, that's fine. peers will expire after a while.
dr|z3d
if you restart a few times in a short period, peers will accumulate before they have time to expire.
dr|z3d
if the error logs reappear, I've got more refinements that I will commit once I've tested them locally, y2kboy23
T3s|4
dr|z3d: been running 6e0b65f5 (Build date: 2023-01-24 05:51:28 UTC) for several hours; build success is back up to ~33%, which is about average on my end
T3s|4
dr|z3d: what's the `...date Runner` from 2023/01/24 07:44:06.413 ERROR […date Runner] …oleUpdateManager: Failed net.i2p.router.update.UnsignedUpdateRunner ROUTER_UNSIGNED HTTP skank.i2p/dev/i2pupdate.zip for ROUTER_UNSIGNED:
T3s|4
just the UnsignedUpdateRunner mentioned later?
dr|z3d
that's just the updater failing to connect to skank by the looks of things, T3s|4
dr|z3d
just building a new update that fixes some snark issues, and there's no update at /dev/ until I upload.
dr|z3d
so could be that, too.
dr|z3d
ok, /dev/ update uploaded.
dr|z3d
now to build I2PSnark standalone for notbobski
dr|z3d
ok, i2psnark standalone available from skank.i2p/dev/i2psnark-standalone.zip or eekmit7xiyu3vjgovu756xj3rzbhycwzgx4gmnyqxau64yzckiluoxad.onion/dev/i2psnark-standalone.zip if skank's playing hard to get.
dr|z3d
oh, I just realized. I might have borked the updater when I tried to remove the non-skank autoupdates. gah.
dr|z3d
actually, no, I don't think I did. but checking I can update from any of the routers I'm overseeing here.
dr|z3d
T3s|4: there's also the graceful_update script in i2p/ that can pull from various places.
dr|z3d
ok, confirmed, updates should be working again, though currently access is pretty spotty.
XANA
dr|z3d, you up?
dr|z3d
what's up, XANA?
XANA
so after update my router sometimes reach like 4 Mbps up/down
dr|z3d
spikes, or sustained?
XANA
usually it stays like apro 1 MBps
XANA
1-2 with sometimes spike going 3-4
dr|z3d
yeah, welcome to the network. lots of abnormal traffic loads lately.
XANA
Looking at /graphs
XANA
though more speed = better network?
dr|z3d
if you're on a 2.0.0-1+ build, you should now have a graph option on stats under Tunnels named TunnelBuildSuccessAvg .. try enabling that one.
XANA
I'm on 2.1.0-1+
dr|z3d
you've got a Build Success readout in the sidebar congestion section?
dr|z3d
you can also graph jvm cpu load via /configstats -> Router / CpuLoad
XANA
where?
XANA
under tunnels there is nothing like this
XANA
I have cpu load bandwitchand number of peers
dr|z3d
No Build Success value under Tunnels in sidebar, not congestion, sorry?
XANA
nope
XANA
only service utility transit total concurenncy and ShareRatio
dr|z3d
ok, you're a few builds behind.
XANA
trying to update but getting Contacting I2P+ update server…
dr|z3d
updates are behaving a bit oddly, so you may not have been informed of anything newer.
XANA
I mean I downloaded some update but not done restart
XANA
let me do it now
dr|z3d
best to make sure you've got the latest.
dr|z3d
wget gitlab.com/i2pplus/I2P.Plus/-/jobs/artifacts/master/raw/i2p.tar.bz2?job=Java8 and then rename to i2pupdate.zip
dr|z3d
rm ~/.i2p/i2pupdate.zip
dr|z3d
and then cp that new update to ~/i2p/
XANA
dr|z3d, nothing changed
dr|z3d
give it 10m, XANA.
XANA
im at 2.1.0-1+
dr|z3d
needs 10m for the average to be calculated.
dr|z3d
assuming you got the latest build.
XANA
I downloaded from skunk s website though still get info that update is ready to download
XANA
Jan 23 2023 5:15 AM
XANA
though I see Build Success
XANA
dr|z3d, so I guess I will need to live with this pop up now :D
dr|z3d
until you need update to something more recent. though you can collapse that section, as you can most sidebar sections...
XANA
ah got it
XANA
what is normal value for Build Success?
dr|z3d
obviously you hadn't discovered the collapsibles before.. :)
dr|z3d
30% and above is fine.
dr|z3d
20% or lower isn't so great. value is for the last 10m of builds, so will vary every 10m.
XANA
btw what is pack200?
dr|z3d
it's a compressor for java files.
XANA
cause i2p renamed that update to bad update :D
dr|z3d
used to be in Java until they removed it from Java14. I2P+ has the code that Java had before it was removed. allows for compressed i2pupdate.zip files @ ~50% size.
XANA
getting like 50% sucess
dr|z3d
that's good. it does vary considerably.
XANA
why so many pages get The connection was reset
XANA
is this page still up? echelon.i2p
dr|z3d
no
XANA
ah so dead links
dr|z3d
will tell you.
dr|z3d
or rather: notbob.i2p/cgi-bin/jump.cgi?q=echelon.i2p
XANA
I'm thinking If I could setup my seedbox to cross seed clearnet to i2p
XANA
hi albat
albat
hi XANA :)
XANA
I have own seedbox for cleanet
albat
cool
y2kboy23
dr|z3d That last commit seemed to fix the issue. I did notice my number of active peers being lower than normal. I usually have around 400 but it's sitting around 100.
dr|z3d
great, y2kboy23. thanks.
dr|z3d
got to the root cause. another couple of related bugs squashed, y2kboy238.
dr|z3d
just about to push to master.
dr|z3d
turns out the bug wasn't firing unless your router was low bandwidth or unreachable (firewalled). wasn't excluding our own routerinfo in a couple of places before determining a router was uninteresting.
dr|z3d
pushed.
dr|z3d
anyone running I2P+, whether on the dev or release branch for updates, should update now.
dr|z3d
console should report a new update regardless if you hit check for updates on /configupdate
dr|z3d
thanks again, y2kboy23, for helping triage the bug.