From: Eliezer Croitoru Date: 2012-07-11T11:37:51+09:00 Subject: Re: the best way to match these domains. On 7/10/2012 12:08 PM, Robert Klemme wrote: > On Sat, Jul 7, 2012 at 5:32 AM, Eliezer Croitoru wrote: > >> well this is the reason i am trying to: >> 1. make it more modular by using methods that can be changed easily >> 2. thing about efficiency. >> 3. consult with others. >> >> for now there is one guy that requested me for that ACL of deny\allow per >> ldap group policy. >> so my main goals now are: >> 1. fix bugs to make it bug free( i have some that i know of and might have >> others that i dont ). >> 2. add a more accurate url match filtering then just host\domain. >> 3. add user\ip db integration for future filtering\acl capabilities. >> 4. improve the filtering based on categories\level. >> 5. add a form that will allow a user to report a false-positive to the >> admin. >> 6. add a "user custom allowed\denied domains\urls list". >> 7. create a category option for the "custom allowed\denied domains\urls" so >> a user\admin can add to a user specific allowed categories. >> for the above option i must really think more before implementing the >> filtering acls as levels or categories etc.. >> 8. content auditor module >> ( i had in mind to add an option of "content inspector\inspecting\auditing". >> what i mean is to add a feature that will log requests urls\domains\pages on >> a db so some human inspection on the content later can be done. >> so in environment like small isp\office that want to build his own >> blacklists\categories based on users browsing experience\habits the "content >> auditor" will get the list from the the DB somehow. ) >> 9. live urls\domains access statistics on a DB for admins. >> (squid has logs but not live statistics) >> >> i had just one simple goal and it became more then just that and i'm happy >> for that. >> >> any ideas on the subjects? > > There's probably so much that can be said to all of them but I am > lacking time. My first impression was to proceed like this > > 1. write down all the _business_ requirements in a structured way (for > example, it seems to me that you want something like global and local > lists although I don't see that explicitly mentioned) > 2. make a designing session to come up with an architecture > 3. find out how you get from what you have to the new design / architecture > > For example: for me it is not clear why you need a RDBMS in there if > you do not plan for large lists and plan to use its features. > > Kind regards > > robert > Thanks Robert, "make a designing session to come up with an architecture" ? with who? i'm by self on it. it is not much of a "business" for me. the main reason i wrote this server was one project specifically to help two cache proxy coordinate some data on urls. as i was building it i have seen that it's pretty simple to add these features. if i had some sponsor for this project i will be more then happy. about the RDBMS or a DB is more for persistent storage and interface integration later. and also the lists i have are now 1 Million rows long only for one category. i intend to make it efficient and to use some DB to add new sites into the to be checked\categorized list. i was working with a big filtering system before but it was designed by a "big" company that offers a complete solution for http\ssl\https\p2p filtering and shaping etc on one machine. i need someone to work on it with. as i was testing my server it seems like 4 workers can work under a load of more then concurrent 25000 requests and about 4000 requests per second on a dumb intel atom with 2GB ram. Thanks, Eliezer -- Eliezer Croitoru https://www1.ngtech.co.il IT consulting for Nonprofit organizations eliezer ngtech.co.il