Re: the best way to match these domains.

On 7/10/2012 12:08 PM, Robert Klemme wrote:
> On Sat, Jul 7, 2012 at 5:32 AM, Eliezer Croitoru <[e-mail]> wrote:
>
>> well this is the reason i am trying to:
>> 1. make it more modular by using methods that can be changed easily
>> 2. thing about efficiency.
>> 3. consult with others.
>>
>> for now there is one guy that requested me for that ACL of deny\allow per
>> ldap group policy.
>> so my main goals now are:
>> 1. fix bugs to make it bug free( i have some that i know of and might have
>> others that i dont ).
>> 2. add a more accurate url match filtering then just host\domain.
>> 3. add user\ip db integration for future filtering\acl capabilities.
>> 4. improve the filtering based on categories\level.
>> 5. add a form that will allow a user to report a false-positive to the
>> admin.
>> 6. add a "user custom allowed\denied domains\urls list".
>> 7. create a category option for the "custom allowed\denied domains\urls" so
>> a user\admin can add to a user specific allowed categories.
>> for the above option i must really think more before implementing the
>> filtering acls as levels or categories etc..
>> 8. content auditor module
>> ( i had in mind to add an option of "content inspector\inspecting\auditing".
>> what i mean is to add a feature that will log requests urls\domains\pages on
>> a db so some human inspection on the content later can be done.
>> so in environment like small isp\office that want to build his own
>> blacklists\categories based on users browsing experience\habits the "content
>> auditor" will get the list from the the DB somehow. )
>> 9. live urls\domains access statistics on a DB for admins.
>> (squid has logs but not live statistics)
>>
>> i had just one simple goal and it became more then just that and i'm happy
>> for that.
>>
>> any ideas on the subjects?
>
> There's probably so much that can be said to all of them but I am
> lacking time.  My first impression was to proceed like this
>
> 1. write down all the _business_ requirements in a structured way (for
> example, it seems to me that you want something like global and local
> lists although I don't see that explicitly mentioned)
> 2. make a designing session to come up with an architecture
> 3. find out how you get from what you have to the new design / architecture
>
> For example: for me it is not clear why you need a RDBMS in there if
> you do not plan for large lists and plan to use its features.
>
> Kind regards
>
> robert
>
Thanks Robert,

"make a designing session to come up with an architecture" ? with who? 
i'm by self on it.

it is not much of a "business" for me.
the main reason i wrote this server was one project specifically to help 
two cache proxy coordinate some data on urls.

as i was building it i have seen that it's pretty simple to add these 
features.

if i had some sponsor for this project i will be more then happy.

about the RDBMS or a DB is more for persistent storage and interface 
integration later.
and also the lists i have are now 1 Million rows long only for one category.
i intend to make it efficient and to use some DB to add new sites into 
the to be checked\categorized list.

i was working with a big filtering system before but it was designed by 
a "big" company that offers a complete solution for http\ssl\https\p2p 
filtering and shaping etc on one machine.

i need someone to work on it with.

as i was testing my server it seems like 4 workers can work under a load 
of more then concurrent 25000 requests and about 4000 requests per 
second on a dumb intel atom with 2GB ram.

Thanks,
Eliezer


-- 
Eliezer Croitoru
https://www1.ngtech.co.il
IT consulting for Nonprofit organizations
eliezer <at> ngtech.co.il