Xeha
hows ipv6 support on mobile networks there?
zzz
thanks Xeha
zzz
didn't realize we had an expert here!
zzz
of course you're right, somewhere along the way I redefined what SNAT meant. I'll correct my terminology
zzz
after several weeks I think I have a grip on how to fix symmetric nat detection
zzz
and I've updated the post on my forum about it
zzz
I don't think java i2p has ever had working symmetric nat detection
zzz
due to fundamental flaws in jrandom's design
orignal
about recent network activity
orignal
i2p_in has an idea
orignal
it's coincident with bitcoin release with i2p support
orignal
if many users turned it on and started syching over i2p?
zzz
Bitcoin has had support for I2P since version 22, released September 2021 - zzz.i2p/topics/3078
zzz
However, version 24 was released recently, with support for transient destinations, and I've seen some publicity about that
dr|z3d
I'm just wondering if bitcoin syncing would cause huge spikes or a more distributed network load.
zzz
we know where most of the network activity is coming from orignal
zzz
it's not a mystery
orignal
and where?
zzz
two routers, as discussed here several weeks ago
dr|z3d
yeah, given that we've identified 2 routers that apparently are generating most of the traffic, I'm still hedging on retroshare crap.
orignal
the question is not about number of tunnels
zzz
one that stays on the same IP, and one that is on 150+ IPs
orignal
the question is badwidth
orignal
what data goes through?
zzz
don't know. probably just more tunnel builds
dr|z3d
as soon as I blocked the hashes for the offending routers, traffic went down from 40MB/s to ~6MB/s
dr|z3d
in an idea world they'd be kicked right off the network.
dr|z3d
*ideal
zzz
orignal, it may get even worse for i2pd after the java i2p release, because we aren't banning them properly now. Once we ban them properly, they will only talk to i2pd
orignal
i2pd can handle such traffic easily
orignal
no, tunnel build is just 1K message
orignal
should be high loaded traffic
dr|z3d
it's not about handling the traffic, that's not the issue. the issue is the collateral damage on the entire network.
orignal
again. what's trier target?
zzz
?
dr|z3d
you lost us, orignal..
orignal
their
orignal
where does this traffic go to?
orignal
we see bunch of tunnels
orignal
and a lot of traffic through this tuunel
orignal
but where does it go to?
dr|z3d
_if_ it's retroshare, you've got a shit ton of file transfers happening between clients.
orignal
then why we didn't see it before?
orignal
retroshare has been used for at least 5 years
dr|z3d
apparently it's been a theme since someone rocked up to your forum and wondered why his router was crashing after 18K tunnels and 20GB swap space.
dr|z3d
as for why not before, maybe they introduced a bug before switching over to SAM.
zzz
orignal, we don't know. Could be attack, could be application bug. Doesn't matter to me.
orignal
I can beleive that a bug can cause excessive number of tunnels
orignal
but traffic
orignal
if it's a attack all this hsit must go to a sinle destination
zzz
it _could_ be all tunnel builds. He's almost doubling the number of tunnels in the entire network, and that doesn't count rejections
zzz
if java is rejecting, then the only good tunnels are through i2pd, so those are the only tunnels to use for building more tunnels
orignal
a tunnel build is sigdle 1K message
zzz
yes but he's sending thousands and thousands of build requests
dr|z3d
what I'm observing is a huge hike in part tunnels without the expected hike in b/w usage.
orignal
also why I see 10K routers in netdb now?
orignal
if it's just one router
dr|z3d
that's likely a parallel issue.
dr|z3d
and to my mind, the router count spikes look more like an attack than anything else, but we can only speculate.
zzz
orignal, re: 10k routers, we don't know, might be a separate attack, just started a week ago: stats.i2p/pix/total/routers3_1m.png
dr|z3d
what we need is a global (i2p/i2pd) function to removeLimbs() of said offenders. :)
orignal
I doubt that's offenders
orignal
I think somebody bundled BTC and I2P
zzz
the number of leasesets hasn't significantly changed: stats.i2p/pix/total/leasesets-1mo.png
zzz
so the attack is building thousands of tunnels for one destination, not creating thousands of destinations
zzz
unless they are unpublished
orignal
I gouess they are client
zzz
maybe
zzz
right now I'm thinking a good time to do a Java release would be around Jan. 9. Do you and R4SAS have any plans yet?
orignal
fine for me. let's wait for R4SAS
orignal
see, my theory
orignal
an idiot might has created million tunnels
orignal
to have different address to shit on kislitsa
orignal
it's pretty much possible
orignal
but it doesn't explian traffic
orignal
maybe it's a ddos attacking a signle destination
orignal
then it would be worth to identify it by IBGWs
orignal
and the leaseset
orignal
*then
zzz
could be
zzz
it's really tough to know, because there's at least 4 things happening at once:
zzz
1) the switch to SSU2, and LOTS of SSU2 bugs
zzz
2) a crazy router building thousands of tunnels from one IP
zzz
3) a crazy router building thousands of tunnels from hundreds of IPs
zzz
4) big increase in router infos, unknown cause, only started a week ago
orignal
anb bitcoin release 24
zzz
yeah. Of course, bitcoin-core is probably used mostly as a library in "downstream" projects, so there's probably some lag between the core release and when the features get pulled in to the downstream wallets / miners / etc
zzz
maybe some project just added an i2p option to the UI
zzz
gah. I think it is bitcoin 24
zzz
^^^ orignal please look
zzz
a new sam session + transient dest. per connection
zzz
or not?
zzz
per-connection is what the release notes say, but I didn't believe it: github.com/bitcoin/bitcoin/blob/master/doc/release-notes/release-notes-24.0.1.md
zzz
I completely missed it when reviewing their changes
zzz
dr|z3d, would you please make me a graph of part. tunnels on one of your big routers, starting after the 15th when you banned the two troublemakers?
zzz
I'll want to attach it to a bitcoin bug report if it looks like it correlates
dr|z3d
try this, zzz: skank.i2p/partTunnels-12days.png
dr|z3d
limit is 12K.
zzz
thanks. that correlates pretty closely with my new routers graph
dr|z3d
so basically bitcoin is shitting all over the network? is that the conclusion?
zzz
awaiting confirmation from jonatack on twitter or orignal, yes
zzz
for the part. tunnel / RI burst starting the 19th
zzz
but only if bitcoin is configured for transient, which is the new feature in 24
dr|z3d
ok, well it sounds like they need to back that feature out.
zzz
next release scheduled for May 15 ((
dr|z3d
I also noticed on the i2p bitcoin github page they're recommending a 20 transit tunnel limit when used with i2pd. as such, their contribution to the network seems entirely negative. parasitic.
dr|z3d
piggy back on an established technnology and contribute nothing back to the network. whoever's in charge needs a stern talking to.
dr|z3d
"For example, to limit total I2P traffic to 256KB/s and share 50% of this limit for a maximum of 20 transit tunnels"...
orignal
who is jonatack?
zzz
one of the main bitcoin guys
orignal
release 24 where the clearly sstate about i2p support
dr|z3d
presumably chief muppet.
orignal
hence somebody might have made bundle
orignal
another interesting thing
orignal
there is no transit tunnels blast on ygg-only routers
zzz
orignal, would you please confirm it's a new sam session every connection? github.com/bitcoin/bitcoin/blob/master/src/net.cpp#L499
orignal
why proxy?
zzz
I think they just consider SAM a proxy
orignal
if (m_i2p_sam_session) {
orignal
else
orignal
I don't think so
zzz
m_i2p_sam_session is if configured for a permenent destination
dr|z3d
no, I think they consider I2P the proxy, no?
orignal
yes
orignal
I'm trying to understand the code
orignal
CNodeOptions{ .i2p_sam_session = std::move(i2p_transient_session) });
orignal
so theu create a session then insert it into some strucure
orignal
it seems they create a new session for each peer
zzz
here's my draft bug report:
zzz
The per-connection transient address feature in 24.0.1 is putting a large load on the I2P network.
zzz
I2P isn't really designed to work that way, and some limit on the number of addresses (aka destinations or tunnels) built by the bitcoin client is necessary.
zzz
A high number of addresses will also result in excessive CPU and bandwidth usage by the router to build the tunnels for each.
zzz
Additionally, other routers will reject excessive tunnel building, which will
zzz
increase your resource usage even more as your router retries.
zzz
Depending on your security goals, and the max number of i2p connections you support, there's a few options:
zzz
- Use a single transient address, created at startup
zzz
- Hard limit the number of i2p connections if configured for transient
zzz
- Group connections into a smaller pool of addresses
zzz
- Rate limit new connections or creation of new addresses
zzz
The i2pd router does not limit the number of destinations, I don't think.
zzz
The Java router limits to 100 by default and will reject addresses over that limit.
zzz
If you do choose to continue using multiple addresses,
zzz
I recommend a reasonable maximum number of addresses of around 10 or so,
zzz
together with a rate limit on how frequently you create new ones.
zzz
We ask that you consider including this change in your next point release.
zzz
In the meantime, please update the i2p doc to discourage the transient option.
zzz
Also in that doc, at the bottom, please encourage people to
zzz
share as much bandwidth and transittunnels as they can,
zzz
so that bitcoin users contribute more to the i2p network than they consume.
zzz
Files:
zzz
Thank you.
zzz
Attached are two graphs:
zzz
- Number of tunnels on a sample high-speed i2p router, Dec. 15-26
zzz
- Number of new routers as seen from another router, Nov. 27 - Dec. 26
zzz
any suggestions?
orignal
yes
orignal
basically it mostly causes troubles for his users
orignal
so, i2p_in is real genious
zzz
I wish I had realized what they were doing when I reviewed their code
orignal
howeever
orignal
it answer the question about number of tunnels
orignal
but nadwidth
zzz
this is only cause #4 which started on the 19th. There's still causes #2 and #3
orignal
I think BTC causes a lot of traffic
dr|z3d
one minor suggestion, zzz, re bandwidth contribution.
dr|z3d
perhaps add that the more bandwidth shared, the better for anonymity, since bitcoin over i2p is all about anonymity.
dr|z3d
low bandwidth share == easier to profile bitcoin usage.
zzz
I can't get the add attachment to work on github
dr|z3d
re dest limits, iirc, orignal implemented a default limit of 500 a while back?
orignal
no
orignal
I'm going to do it
orignal
configurable
orignal
like 1000 by default
dr|z3d
ok, my bad, I thought you implemented a 500 limit when we were talking about the retroshare guy a few months ago.
orignal
I was going to
orignal
as usual
zzz
do you think 10 max is a good recommendation to them?
orignal
16
zzz
ok
dr|z3d
I think they've got it into their heads that they don't want any correlation between peers and dests.
orignal
but
orignal
they would also flood local peers.dat
zzz
that's why I said 'depending on your security goals'
orignal
with transient addreses
zzz
I don't know how many simultaneous connections they max out at
zzz
but the worst is they have to create the tunnels before they try to connect, who knows if they blacklist on failure or keep trying
orignal
furthermore
orignal
they must do it asychnosly
orignal
because SESSION CREATE take a lot of time
zzz
yup, adding that
zzz
I'm giving up on the attachments
dr|z3d
upload the files somewhere and add links to them instead?
dr|z3d
geti2p.net for example.
dr|z3d
or that one on skank can be viewed via skank.hides.me/..
zzz
full link?
R4SAS
skank.hides.me died for me
R4SAS
ERR_CONNECTION_CLOSED
zzz
<Vort> zzz: your link in Bitcoin report skank.hides.me/partTunnels-12days.png is not accessible from clearnet
zzz
<
zzz
dr|z3d ^^^
orignal
if you wish I can put it on gostco.in
orignal
it has almost 100% uptime
R4SAS
I also can put it on i2pd.xyz
zzz
ok, get it from skank.i2p/partTunnels-12days.png
zzz
and give me the link
orignal
but gostco.in will be an irony ))
orignal
sec
R4SAS
yup)))
orignal
skank.i2p also seems down ))
zzz
get the IPs of all the bitcoin devs...
R4SAS
in my case: i2pd.xyz/partTunnels-12days.png
zzz
ok I'll use that one, thanks
zzz
you have it R4SAS ?
R4SAS
yes
R4SAS
))))
zzz
thanks, editing ticket now
dr|z3d
i2phides.me/ my bad.
dr|z3d
ok, we got it fixed. good good.
zzz
everybody click the like button on the ticket or whatever the github thing is
orignal
works for me
dr|z3d
not seeing a like/thumbs up button.
R4SAS
for authorized only )
dr|z3d
sure, this much I know :)
orignal
where is it?
orignal
I can't find
dr|z3d
I'm logged in. should see a button.
zzz
guess there isn't one
orignal
I don't see
dr|z3d
it's not the smilie face at the top left is it?
orignal
found
zzz
I guess subscribe is the closest
dr|z3d
yeah, it's the smilie face which brings up a menu.
dr|z3d
not exactly intuitive, but hey, we got there.
zzz
and probably pointless :)
dr|z3d
probably, but that's on you :)
zzz
not all my ideas are good ones...
dr|z3d
at least it indicates support for the bug. whether they listen is a different matter.
dr|z3d
I think you made a good point there re Tor vs I2P.
dr|z3d
the way they're treating dests suggests they haven't grasped the difference between Tor and I2P.
zzz
they readily fixed my previous bug report, but this one is harder
dr|z3d
Tor .onions, cheap. I2P dests, expensive.
zzz
actually my previous bug report helped this situation alot. Previously their transient dests were ElG
zzz
or at least sha1
zzz
can't remember
dr|z3d
should have left then on sha1 and then blacklisted sha1 :)
zzz
I disabled my hop throttle logging a while back but I'm going to turn it back on. It should help protect us
orignal
SAM uses DSA by default
zzz
yeah that was my first issue, telling them to specify the sigtype for transient dests
dr|z3d
trying to remember where we set the scaling for part tunnels.. might be worth visiting that.
dr|z3d
RouterThrottleImpl.java
dr|z3d
getTunnelGrowthFactor() and getTunnelTestTimeGrowthFactor()
zzz
oh wow it's kicking in hard
zzz
this is only half an hour
zzz
343 lb6qOr8pGDwB-nAycRn0W1YZCz5zjelRzeMTNIuUatM=]:
zzz
37 ckPfSv4AbYiMOyzwBe3MgOfUqCPqW4AyfqJp00T~VRI=]:
zzz
16 maANST0mGXJ7rAuejD5Ny-XivMtPYx1wr8NNJbMjqtU=]:
zzz
9 FHKmY--NiehNePEqZ3JM1VgZBkX9E6oFtH2IbBBrI3A=]:
zzz
2 n3B8-z7fKaJy5ihi2Ct46mfOzLUkp-RA0VMKbtK8sos=]:
zzz
2 cLKxGmb2eFvz8wz4VjTgbpeX5J5fi329j~bYJ2GHD2w=]:
zzz
1 JDmACpBIJdQQHnoNr95~0Re9v-d5IkFulzljwZTRSe8=]:
zzz
1 8~TwlvzUK-R2Jn8Sc5VAppO8UU4P0oyyxWDknniVgCA=]:
zzz
this could be as bad as when Vuze almost blew us up, because it's out of our hands
orignal
in 2015
orignal
I remeber when I always saw code 30 for all 3 tunnel participants
zzz
would you consider rate-limiting how fast you send out build requests, total for the whole router?
orignal
own?
zzz
eys
zzz
yes
zzz
looking to see what we do...
orignal
but it wouldn't work
orignal
I send build reqquest at different time for each destination
orignal
they are never sent all together
zzz
yeah but we have a global limiter
orignal
limiting number of local destinatiions yes
orignal
should do
zzz
we have a max of 13 outstanding build requests
zzz
as in, we have sent it out, but no reply or timeout yet
orignal
good point
orignal
will do the same
orignal
I have pending list
zzz
and I thought we also had a rate limit, but I can't find it
zzz
a lot of this was for ElG, so we didn't blow up our own CPU
zzz
doesn't matter so much 20 years later and with X25519
zzz
like this:
dr|z3d
BuildHandler or BuildExecutor probably, zzz.
zzz
// If builds take more than 75 ms, start throttling
zzz
int throttle = (int) (75 * MAX_CONCURRENT_BUILDS / avg);
zzz
if (throttle < allowed) {
zzz
X25519 DH isn't going to take 75 ms
orignal
ofc
R4SAS
re: next release: I really don't know at this moment. first we need to survive until next year :D
zzz
nope, I don't see any separate rate limit, just the max of 13, which together with a first-stage build timeout of 5 seconds, would be a rate limit of 2.6 per second if they all timed out
zzz
you could build faster if you got at least some rejections
zzz
ok R4SAS
zzz
so that's still 1560 builds in 10 minutes even if they all get dropped before they get back to us
R4SAS
anyway release can be started without me, I'll take care of it if I'm available.
orignal
when you are available?
R4SAS
I don't know at the moment. plans are changing too fast now
zzz
I added more to the ticket:
zzz
Other ideas to limit resource usage if you want to stick with one connection per transient address:
zzz
- Limit to one tunnel each way with inbound.quantity=1 outbound.quantity=1
zzz
- Reuse an address if the connection failed
zzz
- Reuse an address after the connection closes
zzz
Last two are subject to your security goals.
zzz
orignal, the main design goal, of course, is to avoid congestion collapse. If more build failures leads to even more build attempts, with no limits, the whole thing blows up
zzz
got confirmation from jonatack on twitter, one dest. per connection