ROM Shepherd Forum

All your ROM collecting needs fulfilled! => General Chat => Topic started by: bigfattony on July 29, 2016 - 15:00:05

Title: FTP Server
Post by: bigfattony on July 29, 2016 - 15:00:05
Important edit: I've started getting SMART errors on one of my drives. As I made the big mistake of using BTRFS RAID 5, this means that my array is very likely to be compromised when I rebuild, or at least put it in an unprotected state.

I'm going to take it offline until I can get everything backed up.

Sorry for the inconvenience and I hope I helped at least somebody out with this service.



Hey guys,

Over the past month or so, I have been setting up my NAS, rebuilding my ROM sets onto it, and trying to set it up for FTP access for people to grab what they need from my sets.

Most of my sets are rebuilt. I need to do a bit of housekeeping, but I think that I am mostly ready to start giving people access.

What I need is some advice.

I have vsftp installed on my server, and I think I have it set up to jail people to specific directories (I have personal stuff on here that isn't for everyone ;) ). Any tips on making sure it works well would be great. Second, I have set FTP users to read only. I want to create a directory to allow people to upload, should they wish. I would like to set this up within FTP, rather than adjusting file permissions.

The current config should jail you to the roms directory. I have a 18mbps upload speed. Think about this when you're downloading many isos, for example. This is my own server. I reserve the right to reboot it, turn it off, or anything like that without notice. I don't care if you want to download all my gamecube ISOs, for example, but keep in mind that if 18 people are doing this, you will get an average 1mbps speed, and it will take a long time. If I reboot in that time, it will make your life more difficult. You might want to consider finding a torrent instead.

As I say I don't care how much you want to download, but if I suspect that anyone is trying to poke around where they shouldn't, I also reserve the right to revoke access.

Anyway, for now, if you want access, PM me for details.

I have probably made a mess of the above post, but never mind.

EDIT:

I currently have the following sets hosted:

TOSEC - All - Complete as of 2016-11-11

No Intro:

Custom - a bit scattered. I have some sets. Not other. Working on them soon.
Discontinued - Complete
Main - I believe I have the current best miss with sets from 2017-01-02.

Amiga - 527
GBA Ecards - 3
New 3ds (DLC) - 1 - I know the best miss for this is 0, but they changed the sets. I have the old style.
Spectrum - 3

Redump

Bios - complete
Main - Again, dats from 2016-09-01. Things are a bit scattered. Small sets are at best fix, I think. Big sets are in much TLC. PSX is complete, or at least, best miss. Gamecube is likely not too far off
Title: Re: FTP Server
Post by: Thallyrion on July 29, 2016 - 16:59:32
Which sets do you have on your FTP? TOSEC? Redump? Something else?
Title: Re: FTP Server
Post by: bigfattony on July 29, 2016 - 18:19:01
Currently:

All available Good (afaik)
All non-good
All Redump bios
some redump main (I'd avoid CD based ones at the moment as things like PSX are a mess and include very incomplete 'sets'. i.e. tracks 2+, but not track 1)
All Tosec main
All tosec pix
I'm in the middle of rebuilding my TOSEC iso
An older set of Gruby's scummvm stuff
Most available no intro main (though I haven't really updated in a month). Might be missing a  set though I forget which
All of no intro discontinued
Hardly any from custom
Mame has been built from 0.170. I will update it when I get chance, but as you can guess, it is incomplete
Trurip is in the rebuld queue
Most of the company files (amiga games)
KGwhdload set
All of Maybe intro

I can get some better stats soon, if you like?
Title: Re: FTP Server
Post by: Andrea71 on July 29, 2016 - 18:22:58
oh wow... great!
Title: Re: FTP Server
Post by: niallquinn on July 29, 2016 - 20:17:25
Be good to have access to decent servers again, like the old days, Zxyo, Edc etc.

:)

NQ.
Title: Re: FTP Server
Post by: bigfattony on July 29, 2016 - 20:32:53
Fingers crossed :)

though one person is having issues, saying that they are interupted and being asked if they want to overwrite the file.

If anyone else is having the issue, let me know. Or, if not, also let me know.

Also, if someone has a solution? I think that will come when we know if this is affecting everyone
Title: Re: FTP Server
Post by: bigfattony on July 29, 2016 - 21:04:41
I ballsed up the user. If you were having trouble connecting, try again. Should hopefully work
Title: Re: FTP Server
Post by: Connie on July 30, 2016 - 18:01:20
Thanks.

When you get a little more 'organised', you might want to consider an 'Upload' or 'New' folder where people can upload new stuff or you can leave a message for fill requests. :)
Title: Re: FTP Server
Post by: bigfattony on July 30, 2016 - 19:38:19
Yeah, that was the idea. I hopefully won't be too long in getting it running well.
Title: Re: FTP Server
Post by: bigfattony on August 06, 2016 - 19:17:39
Erm, so it appears that I may have a problem. In my infinite wisdom to embrace new technology, I decided to use btrfs raid 5 on my server. Then this happens. http://phoronix.com/scan.php?page=news_item&px=Btrfs-RAID-56-Is-Bad

So, I have a potential problem.

What am I going to do about it? Nothing for the time being. I should be able to convert it to RAID10 and rebalance on the fly, but:

/dev/sda         22T   16T  2.8T  85% /mnt/data

Yes, I don't have the space to do so.

I have Crashplan running on that machine. I intend to let it finish its initial backup, and then there won't be as many issues. I will try to keep my sets up to date (currently, most are pre-July, but I intend to change that.)

I had an issue with clrmamapro that I will try to resolve soon, and then I will get back on rebuilding. I still have 3x3TB drives to process, so I might have to prioritise what is available.

I am open to suggestions, by the way. If I'm not making sense at the moment, then it is because I am mostly thinking out loud, so yeah, suggest away.

Thanks for reading.
Title: Re: FTP Server
Post by: sbranzo on August 18, 2016 - 10:30:32
Hi Bigfattony,

yes, the btrfs was an unlucky choice but no worries! It's not like you are sitting on a time bomb!

Yes, your plan B seems good, and i recovered a lot of lost causes with crasplan. What are your uploading speed? Crashplan is known to be slow.

My advice is to NOT upload to your server anymore (to lessen the risk of a parity corruption) and wait for the backup.

If i may be of more help please ask via PM. I may or may not have several servers scattered in the world and a bit of experience in filesystem AND sharing. Another source of information could be www.reddit.com/r/datahoarder or the linked irc channel #datahoarder on freenode.
Title: Re: FTP Server
Post by: bigfattony on September 07, 2016 - 20:38:27
Cheers sbranzo.

/r/datahoarder is actually where I read about btrfs raid5 issue!

Anyway, I am changing from crashplan to amazon cloud drive, as it is cheaper and people have been reporting faster speeds.

I am also in the process of bringing all my roms up to date as of 01/09/2016.

I might need to streamline a bit - move some less popular stuff off. But, I have an awful lot of rudump to add so, it will be worth it :)
Title: Re: FTP Server
Post by: sbranzo on September 13, 2016 - 22:42:50
[You are not allowed to view links] Register or Login
Cheers sbranzo.

/r/datahoarder is actually where I read about btrfs raid5 issue!

Anyway, I am changing from crashplan to amazon cloud drive, as it is cheaper and people have been reporting faster speeds.

I am also in the process of bringing all my roms up to date as of 01/09/2016.

I might need to streamline a bit - move some less popular stuff off. But, I have an awful lot of rudump to add so, it will be worth it :)

Acd seems a good choice. I don't think it's essential but it could be better to encrypt your uploaded linux isos <wink wink>
Encfs could be the right solution.
Title: Re: FTP Server
Post by: locvez on September 15, 2016 - 18:48:08
Thanks for offering to share, bigfattony :) I have sent a pm
Title: Re: FTP Server
Post by: niallquinn on September 15, 2016 - 20:13:43
Just tried it, couldn't connect, has the url or ip address changed?

Thanks,

NQ.
Title: Re: FTP Server
Post by: bigfattony on September 20, 2016 - 10:27:13
Oops!

I got a new router and set up the wrong port. I forwarded 23 instead of 21 because I'm some sort of idiot.

Should be up again now.
Title: Re: FTP Server
Post by: bigfattony on December 13, 2016 - 12:53:20
Updated first post with a bit of information about what is available.
Title: Re: FTP Server
Post by: RooDBwoY on December 30, 2016 - 10:45:49
This is really decent of you. I did a similar thing many years ago with my Sour DnB complete collection and it got absolutely thrashed for a few years.

One thing I would do is change it to an SFTP daemon (SSH File Transfer) as it's loads more secure and will ensure the data is not transmitted in plain text. So by default you'll be on port 22 (which I would also change to stop port scanners easily finding the site). Most decent file transfer programs support FTP/FTPS/SFTP these days so won't be any real hardship for the users.
Title: Re: FTP Server
Post by: bigfattony on January 03, 2017 - 20:31:46
Thanks for the advice.

I had considered sftp, but if I was going to go down that route, I'd prefer to use public key authentication over passwords. I don't really want keyless ssh access to my server. FTPS could work though...

I'm not fussed about the details for the server getting out, per se. I just don't want them to become available via a search engine. I could set up fail2ban to ban brute force attacks, which could be an idea.

Anyway, I'll have a think. My main priority at the moment is to get everything backed up (I made a post previously about being on btrfs RAID 5). Whe that is done, I'll probably nuke and start over with a less problematic raid/raid-like setup. Still undecided.
Title: Re: FTP Server
Post by: RooDBwoY on January 05, 2017 - 17:52:25
Yeah, I see your point on allowing SSH without keys but I'm sure there will be a way to disallow terminal/shell access (which I guess is your concern?) via the daemon. I have an SFTP site that some of our clients use at work and it only allows file transfer (locked to their own root folder), with terminal/shell access and everything else completely disabled.

Haha, I have enough to deal with different types of RAID at work so understand the frustration at anything outside the norm!
Title: Re: FTP Server
Post by: bigfattony on January 06, 2017 - 11:45:08
Normally I'd have just gone for a regular RAID setup, but I thought I'd embrace the new tech with btrfs. Oops!
Title: Re: FTP Server
Post by: bigfattony on June 14, 2017 - 08:52:39
I've made an important edit to the first post, but I'll bump to make sure people notice:

I've started getting SMART errors on one of my drives. As I made the big mistake of using BTRFS RAID 5, this means that my array is very likely to be compromised when I rebuild, or at least put it in an unprotected state.

I'm going to take it offline until I can get everything backed up.

Sorry for the inconvenience and I hope I helped at least somebody out with this service.