Sunday, July 6, 2008

Image upload issues

So Dreamhost recently locked out creation of new files on the account that hosts editthis.info. It turns out that that account had 250,000+ files, and huge file systems have been a problem for them regarding stability. Almost all of the files were in the image upload directory.

The problem is mediawiki creates a large number of directories, thumbnails, and archive files for every image uploaded. Many of the directories it creates are actually empty, and every directory in linux is really just a file.

I went into the image storage location and ran the command:

nohup find -depth -type d -empty -exec rmdir {} \;&

Which deleted all empty directories, which brought me back under quota, but it won't be long before file count gets to a point where real files will have to be deleted... so I am blocking image uploads for right now.

The solution I am looking at is possibly making image uploads a premium service only, and just charging a small cost for this. So many users create wikis on EDI, upload a few images, and then stop using the site. This will make it so those users don't have an impact on the usage, and the more heavier users most likely won't mind paying a few bucks to host their images.

As always, if you ever want you can host images elsewhere, and then include them in your pages. You simply put the url to the image in your page, and it automatically shows up there.

19 comments:

Gavin said...

I would recommend deleting wikis that haven't be updated for 12 months. These abandoned sites are part of the dotsam (i.e. detritus on the Internet) that serve no real purpose by being retained.

As a serious user of editthis.info who uses it to create websites which supply important, timely information to other users, I would gladly pay a bit of cash to guarantee the availability of the site. Availability is far more important to me than the ability to upload images.

If editthis.info starts charging users, I believe most users will presume that the fee guarantees a certain uptime standard.

Rob Kohr said...

Much of the downtime issues has been relating to maintaining stability of a system that is free. Free creates the problem that growth is rapid, random, and generally creates lots of "dotsam." Those abandoned and even worse semi-abandoned sites are a pain to maintain and weed through.

This is unofficial, and also uncertain, but as I am coding the new version of editthis.info to work with mediawiki, I am considering abandoning the free service aspect of the site for new wikis and wikis that wish to be upgraded. Old wikis will remain free in the current mediawiki155 version, but no more free wikis will be able to be created on that codebase. New wikis will be for a small cost on the new mediawiki version, and an old wiki can be upgraded accordingly.

Maintaining a system with thousands of wikis was fun at first, but now has become too much of a burden. As a paid service I can reduce the number of wikis I support, and focus on providing upgrades and better features instead of trying to keep an overladen ship afloat on a shoe-string budget.

Wael said...

well..i would gladly pay a few bucks....but what if people live on the other side of the earth..that would be a problem to send the money....thus u will have much more less wikis then u had expected with this idea... and another thing....a month or so ago the wiki requester would not let me make a wiki in the name of ''DSB Sucks'' but only excpeted things like ''My Wiki'' and would not even allow a wiki in the name of ''School''...can u please explain why????

Sam said...

I think you're doing a great job and I assume most people don't give you enough credit. I'm a serious user of editthis.info, but I'm only part of the SCP community. I think that the admin who originally created the site has abbandoned it.

No matter how little you decide to charge people for hosting images, the SCP community won't be able to contribute because we have no contact with the original admin.

Do you have any suggestions on what to do? I believe that sooner, rather than later we are going to reach our size limit.

Rob Kohr said...

@wael: I don't really see less users uploading images as a problem. The reverse is the problem.

Charging for wiki hosting may be inevitable if I want to improve quality of service and also dedicate time to development

@sam: The admin will not have to be the person who pays for this. I will have it so anyone that uses a wiki can do so.

Wael said...

EVEN WORSE!
http://z11.invisionfree.com/Code_Wiki_Forum/index.php?showtopic=32&st=0
u can stop reading at:

''Yeah, and I'm planning to make a Crash Bandicoot wiki in the near future. What a bummer. ''

Scyphozoa said...

Isn't it possible to switch to a new image hosting service that has better storage capasity.

Tyty said...

I agree with gavin, and deleting older, non used wikis. And about the whole charging people to upload, it's stupid. Wikis are supposed to be free. And if you want to charge, you should allow users to upload a certain amount, and then once they pass a threshold, then you can charge.

Wael said...

i agree with Scyphozoa, just change the image hosting site.

Kremzeek! said...

Maybe not the best place for this, but Vandalpedia has dissapeared. Being the Head Admin of there, I'd like to know WHAT THE HECK IS GOING ON?!?!

Sorry, I'm just a bit of a nervous wreck right now. hehe....

Kremzeek! said...

Oh and if you're looking for an image hosting site, majhost.com is free and has unlimited filesize. It's too good to be true, but it's true.

Wael said...

that can work ^_^

Wael said...

ummm rob???CODE WIKI HAS JUST BEEN ERASED COMPLETELY!!!!JUST LIKE KREM ZEEK IS SAYING?!!?!?HOW DID THIS HAPPEN....PEOPLE PUT THEIR EFFFOROTS I THAT WIKI!!!! HOW DID THIS HAPPEN!?!?!!??!?!

sennin said...

Honestly, I don't care how this problem is resolved... so long as it's resolved. I'm willing to pay a few dollars a month for proper image hosting, after creating an account and realizing just how much of a pain it is without the feature.

Please update us on what's going on.

Thanks.

Frank said...

It doesn't work because it doesn't free any space. Also, move it to a different site hosting service.

Frank said...

I would consider to remove the old wikis and UPGRADE the wiki software to version 1.14.

Frank said...

Visit this link for the Top 10 web hosting services. These I would recommend instead of Dreamhost 'cause some are cheaper, but better than DreamHost. Some such as Immotion even allows you to have your own dedicated server for a reasonable price.

Zapan said...

So a month later, a decision has been made ?
My wiki needs to include logos and maps that should easily be updatable by users. File hosting and user access rights management are the only reasons I choosed editthis.info.

If there is not change in one month, I will have to stop using this wiki and switch to an other system.

Cleft of said...

Lack of image upload has crippled development of the CoD wiki pretty bad.

If you intend to charge for this, can I just go ahead and send you the money and have you turn it back on for our wiki in particular?