This repository contains the code for a platform that manages a test suite bot, sample upload and more. This platform allows for a unified place to report errors, submit samples, view existing samples and more. It was originally developed during GSoC 2015 and rewritten during GSoC 2016.
You can find a running version here: CCExtractor Submission Platform
While CCExtractor is an awesome tool and it works flawlessly most of the time, bugs occur occasionally (as with all existing software). These are usually reported through a variety of channels (private email, mailing list, GitHub, and so on...).
The aim of this project is to build a platform, which is accessible to everyone (after signing up), that provides a single place to upload, view samples and associated test results.
- Nginx (Other possible when modifying the sample download section)
- Python 2.7 (Flask and other dependencies)
- MySQL
- Pure-FTPD with mysql
TBA.
To serve files without any PHP overhead, the X-Accel-Redirect feature of Nginx is used. To enable it, a special section (as seen below) needs to be added to the nginx configuration file:
location /protected/ {
internal;
alias /path/to/storage/of/samples/; # Trailing slash is important!
}
More info on this directive is available at the Nginx wiki.
Other web servers can be configured too (see this excellent SO answer), but will require a small modification in the relevant section of the SampleInfoController that handles the download.
There are a couple of places where you need to take care to set a big enough size (depending on your wishes) when you want to set/increase the upload limit for HTTP uploads.
If the upload is too large, Nginx will throw a 413 Request entity too large.
This can be solved by adding
# Increase Nginx upload limit
client_max_body_size 1G;
And setting it to an appropriate limit.
To allow upload of big files, FTP can be used. Since the goal is to keep the uploaded files of the users anonymous for other users, every user should get it's own FTP account.
Since system accounts pose a possible security threat, virtual accounts using MySQL can be used instead (and it's easier to manage too).
sudo apt-get install pure-ftpd-mysql
If requested, answer the following questions as follows:
Run pure-ftpd from inetd or as a standalone server? <-- standalone
Do you want pure-ftpwho to be installed setuid root? <-- No
All MySQL users will be mapped to this user. Pick a group and user id that is still free
sudo groupadd -g 2015 ftpgroup
sudo useradd -u 2015 -s /bin/false -d /bin/null -c "pureftpd user" -g ftpgroup ftpuser
Edit the /etc/pure-ftpd/db/mysql.conf file (in case of Debian/Ubuntu) so it
matches the next configuration:
MYSQLSocket /var/run/mysqld/mysqld.sock
# user from the DATABASE_USERNAME in the configuration, or a separate one
MYSQLUser user
# password from the DATABASE_PASSWORD in the configuration, or a separate one
MYSQLPassword ftpdpass
# The database name configured in the DATABASE_SOURCE_NAME dsn string in the configuration
MYSQLDatabase pureftpd
# For now we use plaintext. While this is terribly insecure in case of a database leakage, it's not really an issue,
# given the fact that the passwords for the FTP accounts will be randomly generated and hence do not contain sensitive
# user info (we need to show the password on the site after all).
MYSQLCrypt plaintext
# Queries
MYSQLGetPW SELECT Password FROM ftpd WHERE User="\L" AND status="1" AND (ipaccess = "*" OR ipaccess LIKE "\R")
MYSQLGetUID SELECT Uid FROM ftpd WHERE User="\L" AND status="1" AND (ipaccess = "*" OR ipaccess LIKE "\R")
MYSQLGetGID SELECT Gid FROM ftpd WHERE User="\L" AND status="1" AND (ipaccess = "*" OR ipaccess LIKE "\R")
MYSQLGetDir SELECT Dir FROM ftpd WHERE User="\L" AND status="1" AND (ipaccess = "*" OR ipaccess LIKE "\R")
MySQLGetQTAFS SELECT QuotaFiles FROM ftpd WHERE User="\L" AND status="1" AND (ipaccess = "*" OR ipaccess LIKE "\R")
# Override queries for UID & GID
MYSQLDefaultUID 2015 # Set the UID of the ftpuser here
MYSQLDefaultGID 2015 # Set the GID of the ftpgroup here
Create a file /etc/pure-ftpd/conf/ChrootEveryone with the following contents:
yes
And do the same for /etc/pure-ftpd/conf/CreateHomeDir and
/etc/pure-ftpd/conf/CallUploadScript
Then modify the /etc/default/pure-ftpd-common, and configure the next values:
UPLOADSCRIPT=/path/to/cron/upload.sh
UPLOADUID=1234 # User that owns the upload.sh script
UPLOADGID=1234 # Group that owns the upload.sh script
If necessary, you can also set an appropriate value in the Umask file
(/etc/pure-ftpd/conf/Umask).
After this you can restart Pure-FTPD with
sudo /etc/init.d/pure-ftpd-mysql restart
Note: if you don't see a line saying:
Restarting ftp upload handler: pure-uploadscript.
You need to start the pure-uploadscript. This can be done as follows (where 1000 is replaced with the gid & uid specified above):
sudo pure-uploadscript -u 1000 -g 1000 -B -r /home/path/to/src/cron/upload.sh
You can also verify this by running ps aux | grep pure-uploadscript.
If it still doesn't work, rebooting the server might help.
If you want to help this project forward, or have a solution for some of the issues or bugs, don't hesitate to help! You can fork the project, create a branch for the issue/problem/... and afterwards create a pull request for it.
It will be reviewed as soon as possible.
Security is taken seriously, but even though many precautions have been taken, bugs always can occur. If you discover any security related issues, please send an email to [email protected] (GPG key 0xF8643F5B, fingerprint 53FF DE55 6DFC 27C3 C688 1A49 3AFD C9BF F864 3F5B) instead of using the issue tracker, in order to prevent abuse while it's being patched.