@eyedeekay
&kytv
&zzz
+R4SAS
+RN
+RN_
+dr|z3d
+orignal
+postman
+wodencafe
Arch
DeltaOreo
FreeRider
FreefallHeavens
Irc2PGuest19353
Irc2PGuest22478
Irc2PGuest48042
Irc2PGuest64530
Irc2PGuest77854
Nausicaa
Onn4l7h
Onn4|7h
Over1
Sisyphus
Sleepy
Soni
T3s|4_
Teeed
aargh3
acetone_
anon4
b3t4f4c3
bak83_
boonst
cumlord
dr4wd3
eyedeekay_bnc
hagen_
khb
not_bob_afk
plap
poriori
profetikla
r3med1tz
rapidash
shiver_1
solidx66
tr
u5657
uop23ip
w8rabbit
weko_
x74a6
zzz
worked for me, as reported
dr|z3d
yeah, minor in the scheme of things. restart of router will probably fix.
UDP
Do you guys know if the wrapper.java.maxmemory flag in wrapper.config scales automatically with the bandwidth set upon first setup? I've had to modify it manually on a couple machines since I have a rather fast internet connection, and it would just keep crashing after 4-5 minutes.
obscuratus
UDP: Yep, you have to set that manually in the wrapper.config
UDP
Seems like it runs out of heap memory with the default value, so just wondering.
UDP
ah gotcha, that's good to know since that doesn't seem to be covered on the website's basic install steps, and I was trying to figure it out :p
UDP
thanks
UDP
got 3-4 nodes running so far, one's done a few terabytes this week
UDP
gonna go get some food, but gonna lurk on here a bit :)
Opicaak
It appears like the network is back to normal, for now.
dr|z3d
like I said to zzz earlier, Opicaak, just because the view from where you are looks ok, it doesn't mean that's the same everywhere. attack isn't hitting all routers at once.
eche|off
hm, memory footprint did went up, some routers did died of 256 MB java heap memory as of OOM, to bad
zzz
you'll be at the meeting today right eche|off ?
zzz
first report of OOMs, thanks
eche|off
with 2.0.1 it was.
zzz
2.1.0?
eche|off
I try to be, need to check if I can get into with laptop. still system broken :_/
eche|off
2.1.0, latest ubunti
eche|off
(sorry, currently no dev build until system is rebuild...)
zzz
what else? fast router? slow? floodfill? any services on it?
eche|off
1 service on it, mde-fast router, defaultiung to 500kb/sec, else default, reachable via IPv4 and IPv6
eche|off
80% share
zzz
ok. did it restart ok by itself and come back up?
eche|off
by default ~2k active peers
eche|off
it did restart itself ~10 times and went of later on
eche|off
restart after 3-5 min of being active
zzz
ok
eche|off
mostly kademlia datastorejobs I see in the out of heap memory errors
zzz
good to know, expected it but did not have any reports yet
zzz
will have a package of fixes later in the week
eche|off
ok
zzz
workaround 1, increase max mem in wrapper.config of course
zzz
workaround 2, when stopped, just delete all but a thousand or so of your RIs
eche|off
just 9k
eche|off
in netDB
eche|off
ah, 19k in peerProfile
eche|off
so, default alue is bad in these situations
eche|off
interstingly the bigger oney do have low numbers now, but guess they are rated low on most peers
eche|off
which shows the net does work
zzz
not_bob, did you resolve the snark recheck-every-startup issue from my forum and if so how? symlinks? plus bug?
dr|z3d
pfft. plus bug my aspidistra.
dr|z3d
that would be in the same category as the double leaseset I2P+ jeopardy bug.
dr|z3d
aka canon bug.
not_bob
zzz: Sometimes it does it now, and other times it does not. I have no idea why.
not_bob
zzz: If I do a bad shutdown it does a re-checke very time.
not_bob
It has nothing to do with symlinks.
dr|z3d
running it with i2pd on the backend, no, not_bob?
not_bob
Yes
dr|z3d
could be session data's not being given enough time to write to disk before shutdown..
not_bob
i2pd backend.
not_bob
My fleet is mostly i2pd at this point due to low memory.
zzz
ok, thx for update
zzz
just asking
not_bob
No worries. The re-checking at start isn't that big of a deal. It just takes a while.
not_bob
I have just shy of 2 TB or torrents loaded on one router.
dr|z3d
my money's on session data being corrupted on shutdown.
zzz
probably a shutdown thing, yeah. not corruption, just not getting it all done
zzz
before the router gives up waiting
dr|z3d
right, same thing really, if the data's not all written in time, then on startup it's, well, corrupted :)
dr|z3d
rotating the save points could mitigate that. so you maintain 2, current and last.
zzz
not_bob, try stopping i2psnark at the same time you click graceful shutdown, should fix it
dr|z3d
if current's not good, rollback to last.
not_bob
Next time I restart that server I'll give it more time before I tell it to restart.
not_bob
zzz: I'll try that and report back later.
zzz
it's not corrupt drz, just didn't get the new one written, so the files are newer than the timestamp in the old one
dr|z3d
why would files be newer if they're already downloaded and are seeding, assuming we're talking about the content here, and not the metadata?
zzz
right, only if incomplete
dr|z3d
so, not_bob's probably is a TB of completed torrents all wanting to do the integrity check dance on startup.
dr|z3d
*problem
dr|z3d
that's why I'm suggesting the config/metadata is being corrupted on shutdown.
not_bob
Yeah, almost all of my torrents are done and just there to upload to others.
zzz
it's also very hard to corrupt a config file because of the way we do it:
zzz
save to x.tmp then rename to x
eyedeekay
Hi everybody, sorry I'm late, got disconnected right before the meeting
eyedeekay
1. Hi
eyedeekay
2. 2.1.0 Status Report
eyedeekay
3. 2.2.0 Development Status
eyedeekay
4. Congestion Throttling
eyedeekay
5. Hypothetical Traffic Management ( Flood of Tor Users)
eyedeekay
6. Stickers for translators
eyedeekay
zzz where do you want to do your report about the DOS? 2 or own topic?
zzz
let's call it 3b)
zzz
or 2b)
zzz
your choice
eyedeekay
OK 2b then
zzz
hi
eyedeekay
1. Hi who all is here today besides me and zzz?
not_bob
Here
echelonMAC
here
obscuratus
Hi
echelonMAC
on replaement system.
eyedeekay
Great turnout, thanks everybody
zzz
irc is laggier than usual so please allow a little extra time for responses
eyedeekay
2. 2.1.0 Status Report
eyedeekay
Thanks zzz I will keep that in mind
eyedeekay
Where to even start... my focus has been on Android UX issues in particular fixing up things I was doing wrong when initializing the app
eyedeekay
Android will get a point release on that account
eyedeekay
The topic I'm sure is on everybody's mind right now is the network being under attack which has shifted everybody's attention
eyedeekay
And that is topic 2b
eyedeekay
Shoot I mixed up the topics, scratch that, that was supposed to go to 3. sorry, long week, lot going on
eyedeekay
zzz would you do me a favor to take 2? I think I am ill-equipped to present on everything which is going on
zzz
sure
zzz
but before I do
zzz
do you and echelonMAC have a 2.1.0 mac easy-install bundle progress report?
eyedeekay
Oh yes I can do that
echelonMAC
in short: eyedeekaym did redo the signing script, I can buiodl and sign, but still wrong signing for apple
eyedeekay
So, we've been closing in on issues, and right now echelonMAC and have different hypothesis on what's going on which we're both pursuing
eyedeekay
My hypothesis is that this all stems from a stale workaround for a bug in Java 14
eyedeekay
Where jpackage is supposed to sign the dylib files that are packed into a disk image(dmg) but which are in fact left unsigned
eyedeekay
zab worked around this by creating the .app directory and signing the stuff inside it then using jpackage only for the final packaging step
eyedeekay
so when we upgraded Java's we started signing everything twice, and we no longer needed to do that signing step manually
eyedeekay
echelonMAC correct me if I'm wrong but you think you have the wrong sort of keys?
echelonMAC
maybe, maybe not, unsure about that
echelonMAC
at least the logs showing this error
zzz
my question was higher level, what's the ETA, is 2.1.0 still a target or has that been abandoned and you're hoping for 2.2.0, or what?
echelonMAC
2.1.0 is stioll the target, but currently no ETA
echelonMAC
I can buidl nearly instant, but digging deeper is currently out of time...
eyedeekay
I still want to cut a release of 2.1.0 as soon as we know we can pass notarization, IMO once we can pass it once we should be able to do it predictably
zzz
eyedeekay more or less committed to early april 2.2.0 in his blog post, you're not endorsing that ETA echelonMAC ?
echelonMAC
I am completely helpless and cant predict a ETA, as I do not full yunderstand signing and building yet.
echelonMAC
more or less, I do follow scripts blindly, if idk can fix the signage, the notarization is a 5 min job.
echelonMAC
IF the sign does work as appple expect it
zzz
ok then I recommend an edit of the blog post eyedeekay, let's not promise things we can't deliver on
eyedeekay
Will do
zzz
ok thanks you ready for my part of 2) ?
eyedeekay
Yes please
zzz
great
zzz
last meeting was one week after the release, now we're 4 weeks out
zzz
my hope was that expl. build success would climb steadily
zzz
from the low-20% to low-30%, and then "pop" out of congestion, back to low 40s
zzz
only the first part happened
zzz
and then we swung back and forth between low 20s and low 30s
zzz
so, we have some theories, see 2b)
zzz
but I'm happy with the performance of 2.1.0 otherwise
zzz
not too many bug reports
zzz
I'll give an overview of what we are fixing in 2b) and 3)
zzz
about 50% of the network has updated to 2.1.0 or the i2pd equivalent
zzz
everybody please update if you haven't
zzz
that's all I have for 2a), but I'll pause for a minute for questions / comments before starting 2b)
zzz
ok, 2b) current network conditions
zzz
over the weekend started an unambiguous attack
zzz
lots of floodfill routers
zzz
for the most part, the network overall, and java routers, are handling it ok
zzz
I do have one report of routers crashing with OOM (out of memory)
zzz
I understand that i2pd routers are really strugging with very low tunnel build success rates
not_bob
My fleet is up to date.
zzz
the attack is starting / stopping / changing several times a day
zzz
so we're only about 60 hours in to understanding it and discussing countermeasures
zzz
remember the tunnel build problem (now much more likely to be classified as a different attack) started Dec. 19 and took us several weeks to understand and address
zzz
so it's early days
not_bob
I have one I2P+ router and it's done well to weather this. But, my i2pd routers not so much. I've seen as low as 3% tunnel build success. I'm currently sitting around 10% on those routers.
zzz
but this is a straightforward sybil / DHT attack, new to us but familiar to anybody doing peer-to-peer / DHT applications
zzz
far too early to say if it will require a release ahead of schedule (for java i2p) or if so when
zzz
EOT, I'll wait a couple minutes for discussion / comments / questions
eyedeekay
Should people who wind up OOM increase the RAM available to their router?
zzz
yeah, that's a straightforward mitigation
zzz
stop your router, edit wrapper.config, restart
zzz
I expect I'll have mitigations in dev builds in a few days
eyedeekay
As I mistakenly stated before, my focus has been on Android UX issues in particular fixing up things I was doing wrong when initializing the app, pretty basic stuff all things considered but all of which was due for improvement
eyedeekay
or outright fixing
eyedeekay
Examples being fixes for subscriptions and custom ports when building tunnels which got in several user's way after a large increase in users with 2.10
dr|z3d
as an aside, and to shine some light on the size of the attack, we've got a user reporting ~30K banned routers. so it's non-trivial :)
dr|z3d
(that'll be shiver, who's here)
mark22k
I got 56005 banned peers.
moristo
Is this the work of a nation state--the banned routers or any other noticable patten?
eyedeekay
Holy moley. I have 11027 and I thought that was a lot
moristo
Spectrum internet was off yesetrday in FL and Italy the day before.
moristo
*yesterday.
zzz
let's get back to 3) please and table further attack discussion until after the meeting
echelonMAC
Banned Peers (57053)
moristo
oh, is there a meeting in progress? My bad.
zzz
eyedeekay, you still with us?
eyedeekay
yeah I'm here
zzz
you have more on 3) or is it my turn?
eyedeekay
started a long one:
eyedeekay
i2p.firefox(Easy-Install for Windows is also getting worked on, more of the work is being done by removal there, updates are getting simplified now that the old admin-style installs are all migrated to user-style installs
eyedeekay
portable USB install support is on the horizon for 2.2.0
eyedeekay
With updates
eyedeekay
Other issues and user-complaints I'm addressing or deciding how to address are on that issue tracker
eyedeekay
EOT for me
zzz
ok you saw the NPE issue in i2p.i2p right?
eyedeekay
Yes I did, hot on the trail
zzz
ok holler if you need help ofc
zzz
3) for me:
zzz
I finished the peer selection refactor I've been working on since september, finally
zzz
I think I fixed the erroneous symmetric nat indications that were in 2.1.0, but need somebody with that problem to test and report
zzz
got a cool new i2psnark search box
zzz
almost done with "congestion caps" (proposal 162)
echelonMAC
:-)
zzz
and some more tweaks to refine our handling of tunnel build congestion
zzz
late last week, I thought I was pretty much done with 2.2.0 and could relax for two months until the release
zzz
so, unfortunately, now we have a lot more to do
zzz
that's the way it goes sometimes
not_bob
Thank you for that, a major quality of life improvement.
zzz
EOT, I'll wait a minute for discussion, then back to you eyedeekay
zzz
haha not_bob you're welcome
eyedeekay
Last call for 3?
eyedeekay
4. Congestion Throttling - I think this was a tunnel_king topic, is tunnel_king here?
zzz
back to you eyedeekay
eyedeekay
4. Congestion Throttling, 5. Hypothetical Traffic Management ( Flood of Tor Users) - both introduced by tunnel_king, last call for tunnel_king
eyedeekay
OK last one, 6. Stickers for translators
eyedeekay
Specifically rules for people receiving stickers outside of the Americas
zzz
this was my topic, only because unresolved since last meeting
zzz
echelonMAC you have an answer?
echelonMAC
not en detail, but who wnats should receive a bunch of stickers if they sent their address
echelonMAC
aka sned a announcement in transifex and send out after receive of address
echelonMAC
but currently no new stickers arrivced here
eyedeekay
Tracking says the 10th
zzz
I'm awaiting echelonMAC to post the instructions on my forum, then I will copy/paste to transifex
echelonMAC
ok
zzz
that's where we've been for a month
zzz
thanks
zzz
EOT on 6) for me, back to you eyedeekay
eyedeekay
Anything else for 6 or for the meeting?
zzz
one closing word for the meeting: zzz.i2p the best place for news and weather, I'll post there when I know more than I do now
zzz
thanks
eyedeekay
Thanks very much for that zzz, and thanks everybody for coming to the meeting
zzz
thanks eyedeekay
zzz
got thru it without disconnects
zzz
anybody with a registered nick that wants voice, speak up before I +m
eyedeekay
See you around IRC and same time next month
justmessin
hey
justmessin
unsure if I can speak in this channel, can anyone hear me? I was going to ask for the logs of this channel/details of the attack. I just recently started using I2P a few weeks ago but have a background in cybersecurity
obscuratus
justmessin: Check major.i2p
not_bob
justmessin: Logs will be posted on geti2p.net/en/meetings
eyedeekay
justmessin you could look at the #ls2 channel logs on major.acetone.i2p
zzz
last call for voice justmessin mark22k moristo or anybody else
justmessin
hey zzz
zzz
does hey mean you want voice?
justmessin
could I get voice please
justmessin
obscuratus not_bob eyedeekay thanks all I will check. would any devs find router stats valuable?
justmessin
i just started recently with i2p this past month but i have everything open, max tunnels quite high etc
zzz
welcome to i2p
zzz
no we don't need stats from anybody
justmessin
thank you, gonna get back to something but then i'll take a look through these logs to see if theres anything i can suggest
justmessin
is this channel okay for questions about routers? i only just recently started hosting one. started with default java i2p default 1G, now using 3G. is there any problem with giving it like 32G?
not_bob
justmessin: The defaults tend to work well for most people. As for more equals better? At some point it's not. And I think the max you can allocate to a jvm is 4 gigs.
justmessin
not_bob thanks. just have spare cpu and bw so want to help the network
not_bob
Thank you for that.
RN
Thanks to the devs for the continuing efforts to unplug the toaster!