Welcome! Log In Create A New Profile

Advanced

Features I've added to my own client that may be of interest

Posted by btx 
btx
Features I've added to my own client that may be of interest
August 30, 2010 11:59PM
I recently wrote an NMDC bot in Python to cover a few features that aren't really beneficial to the typical DC user (ncurses text mode display being a big one).

One thing I added in were plugins that affected your file list - In addition to your regular share directories, you'd have "virtual share directories" - for example, one I made creates a share name of "Alphabetical File Listing", whose subdirs correspond to the first letter of the files. Since files are requested by root TTH hash, it didn't require any change to existing download routines, just the file list generation function(s). Another plugin groups based on the media type, etc. Since my file share is driven by a SQL database (SQLite3 with the fts3 functionality to handle finding parts of filenames), functions like alphabetization are especially easy, but it's certainly something one could work around if the share was memory-based.

Another feature I added that may not be too Java-friendly is support for the Linux inotify interface, which lets me stop having to scan for new files periodically - it just needs to scan for new files at startup time, and of course when adding a new share.

I also got it working with OSX's file notification service as well - which is journaled, letting the bot detect ffiles that were added since you shut down the bot, but the requirements for this functionality were ridiculous, so I took this functionality out (Apple only seems willing to let you communicate with the file notification service with their API, using their objective C interface.

I also have the often requested feature of giving out slots based on IP address, which the other client authors are very opposed to since it's always possible for separate users to be NAT'd to the same IP address - which is of course valid, but without that protection, any user in multiple hubs with the bot could consume multiple slots.

I don't know how often people actually want these type of features, but if any sound interesting, I maybe could be of some assistance when adding them.

btx
Re: Features I've added to my own client that may be of interest
August 31, 2010 06:07PM
1. Alphabetical listing..
seems bad for most as you would loose the structural information which is mostly more important...

Though is this just your own view of your and others filelists?

2. inotify and alike features exist also on windows... though its really not Java friendly ... I would have to impelment this once for each os and have to deal with ugly C code... so no... maybe Java 7 will support this out of the box... but thats way in the future..


3. I am more for the userbased approach here....I don't see much benefit of IP based slots over user based. Can you give me an example where its beneficial?


about adding features:
Jucy is open source its written in Java and plugin based... so you can create your own plugins!
If you want I can either add them to the update site so they are found when people are looking for extensions.
Another possibility is if you create some plugins I could also add a update site you maintain to jucy or rather some option to add new update sites.
Till now it was just nobody was interested in developing a plugin for Jucy ...

------------------------------------------------------------------------
Freedom is a new religion, the religion of our time.
- Heinrich Heine
btx
Re: Features I've added to my own client that may be of interest
September 02, 2010 12:50AM
You are obviously right about hte alphabetical listing, I was just using that as an example - the idea is alternate organization. The files I usually share are generally single archives, making the alphabetical approach useful for me.

I wasn't sure if there was any library that handled async file events that you could use - I just wanted to mention it just in case it was something that you hadn't thought of.

As for the slots ... first of all, this only becomes an issue when your client is connected to over 1 hub on NMDC protocol. Maybe there's a mechanism for doing this better with ADC, I don't know. Here is the specific scenario I'm talking about:

- Let's say both of us are connected to the same two hubs, both of which use the NMDC protocol.
- My client wants to download segments of a file you happen to be sharing.

When my client searches for this file, it will request the search on each hub, and your cilent would return the same result on each hub.

Whtat ends up happening now is that my client would treat these results like it would treat 2 results from different clients - and there's really no way around that - if the search was passive, you don't even know where the results are coming from, and of course you could be using different nicks on each hub, making it pointless to try to match that way. And so, my client would take up 2 of your slots. This ends up benefitting no one, since the download wouldn't go any faster, so the slot is effectively wasted.

The bot I wrote only gives out 1 slot per IP, which gives a good deal of protection against this problem. Additional requests from the same IP are given the slots full msg.

Obviously, this isn't perfect or impossible to work around, and it's possible to unfairly deny a user a slot... I just don't know what alternative there is, aside from not doing anything at all (which is the approach other client authors have taken).

As for the open src side of things ... I'll have to look into what can be done with Jucy Plugins... I'd be very interested in implementing feature #3, since this is a widespread problem in my community among those of us with dedicated shares. A basic implemention would need a) to store the remote IP address for any download slot you've allocated, and b) When a user requests a file, the last thing to do before actually allocating a slot is check to make sure the downloader's IP isn't already using a slot. If it is, you just treat it like there's no slot available. Personally, I treat mini slots and regular slots differently (so you can download a file and a file list at once), but it's not necessary.

I hope I made more sense in this post smiling smiley

btx
Re: Features I've added to my own client that may be of interest
September 02, 2010 06:32PM
If the search is passive you know exactly where it came from.
For an active search... you still know it because even nmdc puts in the hubaddress there which zou can use to guess the hub..

---
My stance on this topic has always been that if you share 2 hubs with one user then you may get 2 Slots of this user,
so you get twice the speed! 2 Slots on one IP really means twice the speed, if and only if the line is maxed out.

I would assume that maxed out is the default case, if you are not maxed out you probably have a fast line or small share, then this is not really important for you anyway!

Another problem with the one Slot per IP is that NAT have become popular and will become even more popular in future. This penaltiyes university users that have lots of bandwidth but sometimes only a single IP for their dorms.

------------------------------------------------------------------------
Freedom is a new religion, the religion of our time.
- Heinrich Heine
btx
Re: Features I've added to my own client that may be of interest
September 04, 2010 01:37PM
NMDC puts the hubs address in searches? Do you mean in the Pk field of the the Client <-> Client connection?

I am not concerned about the share speed, but the slots - they keep other people from the share. My bot is in 4 hubs, and there are people in all 4 hubs who used to be able to unintentionally monopolize the slots. I only gave you the example with 2 hubs as a simple example, but the real problem is often a lot worse.

THere's no perfect solution to this problem, since of course like you said, 2 separate connections (or more) COULD have the same public IP address. My community's users rarely even have a shared IP, although we've seen a couple from India - never duplicate (yet) however. We DO however, have many many users who are on multiple hubs. It seems like it'd be a reasonable option - especially if it's off by default.

One of DC's greatest features is its lack of a required central infrastructure, allowing communities to have many of their own hubs. I don't think it's unusual to have the multi-user issue I'm talking about.

Anyway, if you see what I'm saying, but just don't agree with the logic, then i'll drop it. I already have a solution for my purpose, and just felt it was a valuable feature. As always, I appreciate your feedback very much.


btx

PS: As for your statement of NAT being more popular in the future ... certainly that's true... until IPv6 takes over smiling smiley (So not for a while smiling smiley
Re: Features I've added to my own client that may be of interest
September 04, 2010 03:55PM
NMDC puts an address string of the hub address in every search result.

usually it looked like this:
$SR <source> <result> <free_slots>/<total_slots><0x05><hub_name> (<hub_ip:listening_port>winking smiley
Hubname has been replaced with the TTH of the file in the last years. But the IP address in the brackets stays.


With one person monopolizing all your slots... you should have this person's queue of your share worked of quite fast.
As you really share 4 hubs with this person ... and the person its lucky.. I just don't see why the Person shouldn't get this 4 slots.



IPv6 will require a lot of NAT-ing to work well and emerging of IPv6 won't bring IPv4 down ... these will coexist for decades.
Also sadly a lot of people see NAT as a convenient security measure, even with IPv6 NATs will stay.

------------------------------------------------------------------------
Freedom is a new religion, the religion of our time.
- Heinrich Heine
btx
Re: Features I've added to my own client that may be of interest
September 09, 2010 02:30PM
QS:

I can't answer whether they SHOULD get 4 slots, since that's an opinion, but I can tell you the point of view that my community has... since slot holders generally can keep their slot as long as they have things to request, having 4 slots could mean that someone with a large queue will monopolize the xfers as long as they have content to download. When the download speed isn't that fast, that can mean 1 user is taking up 4 slots for a long period of time - but not intentionally.

It's not uncommon to have that kind of overlap between users and hubs. DC lets people run their own hubs so easily that there seem to often be many hubs, each of which is happy to have as many good users as it can.

I promise you these aren't hypothetical situations, although if you feel it's best to give out any # of slots to a single user... well it's your client, and I can't argue with how you, the author, wants it to work smiling smiley

And yeah, I was mostly joking about IPV6 hehe smiling smiley That'll be one painful switchover, when and if it ever happens.

btx
Re: Features I've added to my own client that may be of interest
September 09, 2010 06:45PM
The switchover is happening now ... mostly silent...
its not a question of when ... switching over has begun years ago!

------------------------------------------------------------------------
Freedom is a new religion, the religion of our time.
- Heinrich Heine
Sorry, only registered users may post in this forum.

Click here to login