[Kea-users] Raw socket input buffer

classic Classic list List threaded Threaded
5 messages Options
kea
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Kea-users] Raw socket input buffer

kea
Hi,

I have a kea4 server, version 1.1.0, I added hooks that connect
externally (the important information : that hook is "slow", it does TCP
connect)
That part works well

However, when the server takes too many queries (I don't know how "too
many" is), a strange behavior appears: the server keep and answer
outdated requests (some kind of input buffer, I guess), which clients
timed-out in the mean time, leading to another requests etc -> broadcast
storm and global failure

I will try to explain that using the timeline below:
t: client1 make request1-1
    client2 make request2-1
    client3 make request3-1
t + 1: server process request1-1
t + 2: server keep processing request1-1
t + 3: client1 time out request1-1 and make request1-2
       client2 time out request2-1 and make request2-2
       client3 time out request3-1 and make request3-2
t + 4: server send answer1-1, and start processing request2-1 (which is
already timed out by the client2)
       answer1-1 reach client1 and is dropped

t + n: client keep sending new requests, timing out old one
       server keep processing obsolete requests


I tried to reduce /proc/sys/net/core/rmem_max to 4096, this does not fix
the issue (probably because 4096 is still a lot of dhcp-packets :p)

Any tip, trick, configuration or whatever on that issue ?

Best regards,
_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Kea-users] [kea-dev] Raw socket input buffer

Francis Dupont
[hidden email] writes:

> I have a kea4 server, version 1.1.0, I added hooks that connect
> externally (the important information : that hook is "slow", it does TCP
> connect)
> That part works well
>
> However, when the server takes too many queries (I don't know how "too
> many" is), a strange behavior appears: the server keep and answer
> outdated requests (some kind of input buffer, I guess), which clients
> timed-out in the mean time, leading to another requests etc -> broadcast
> storm and global failure

=> as your tentative fix suggested this is a kernel issue: pending
queries are simply queued waiting Kea to read them. IMHO the best is
to add some code in the slow hook to flush them.

> I tried to reduce /proc/sys/net/core/rmem_max to 4096, this does not fix
> the issue (probably because 4096 is still a lot of dhcp-packets :p)
>
> Any tip, trick, configuration or whatever on that issue ?

=> slow processing is incompatible with the DHCP protocol so basically
there is no final fix, only tricks to make things less broken.

Regards

Francis Dupont <[hidden email]>
_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Kea-users] [kea-dev] Raw socket input buffer

Tomek Mrugalski
W dniu 31.07.2017 o 12:36, Francis Dupont pisze:

> [hidden email] writes:
>> I have a kea4 server, version 1.1.0, I added hooks that connect
>> externally (the important information : that hook is "slow", it does TCP
>> connect)
>> That part works well
>>
>> However, when the server takes too many queries (I don't know how "too
>> many" is), a strange behavior appears: the server keep and answer
>> outdated requests (some kind of input buffer, I guess), which clients
>> timed-out in the mean time, leading to another requests etc -> broadcast
>> storm and global failure
>
> => as your tentative fix suggested this is a kernel issue: pending
> queries are simply queued waiting Kea to read them. IMHO the best is
> to add some code in the slow hook to flush them.
>
>> I tried to reduce /proc/sys/net/core/rmem_max to 4096, this does not fix
>> the issue (probably because 4096 is still a lot of dhcp-packets :p)
According to man 7 socket, the minimum value you can set there is 256.
This is a doubled value (see the man page for explanation), which in
principle could allow you to buffer up to 128 bytes. This is definitely
in range of a single DHCP packet. Obviously, nobody recommends setting
it to such a low value. But if you want to drop packets while previous
packet is being processed, this is one thing to possibly try.

>> Any tip, trick, configuration or whatever on that issue ?
>
> => slow processing is incompatible with the DHCP protocol so basically
> there is no final fix, only tricks to make things less broken.
Yes. One day we will have to extend the hooks mechanism to make it
asynchronous. This is still a bit far away, because this most likely
means Kea will need to be multi-threaded or at least multi-process. Both
are non-trivial.

But before we fully embrace that, we could look whether there's a way to
get the timestamp from kernel regarding when the packet was received. If
there's a way to do that, we could reject packets that spent too much
time in a buffer and are now stale. This shouldn't be super complex to
implement, but sadly our schedule for upcoming 1.3 is pretty tight.

But as Francis said, this would be another trick. The ultimate solution
would be to implement async hooks.

Tomek
_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Kea-users] [kea-dev] Raw socket input buffer

Francis Dupont
Tomek Mrugalski writes:
> But as Francis said, this would be another trick. The ultimate solution
> would be to implement async hooks.

=> I am not sure they (async hooks) can save you: if the response depends
on a slow processing the client will timeout anyway.
Note if the problem is a long hook startup you should initiate it
when the hook is loaded (unfortunately today a hook is loaded more than
once?)

Regards

Francis Dupont <[hidden email]>
_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
kea
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [Kea-users] [kea-dev] Raw socket input buffer

kea
There is a mistake in my first mail : the hook is "slow", but fast
enought to handle a request

So when it start flooding, the first request will be processed in time,
then another, slowly sliding into obsoletes requests

Anyway thank you both for the insight, I have now a couple of lead to
work on

On 31/07/2017 17:18, Francis Dupont wrote:

> Tomek Mrugalski writes:
>> But as Francis said, this would be another trick. The ultimate solution
>> would be to implement async hooks.
>
> => I am not sure they (async hooks) can save you: if the response depends
> on a slow processing the client will timeout anyway.
> Note if the problem is a long hook startup you should initiate it
> when the hook is loaded (unfortunately today a hook is loaded more than
> once?)
>
> Regards
>
> Francis Dupont <[hidden email]>
> _______________________________________________
> Kea-users mailing list
> [hidden email]
> https://lists.isc.org/mailman/listinfo/kea-users
>

_______________________________________________
Kea-users mailing list
[hidden email]
https://lists.isc.org/mailman/listinfo/kea-users
Loading...