Discussion: Split future repos into files + writeup repos?

[abpolym]

Often, people who want to add a writeup (link or local) to the repo download the repo to do so (though they can add writeups using the browser without any additional workload).
However, downloading the repo gradually becomes a tedious process, since the repo gets bigger and bigger during the year.

So my suggestion is to split future repos into two repos: one repo containing files/resources only and one repo containing writeups only. Format could be write-ups-2016 and resources-2016 .

What do you think?

[dhanvi]

IMHO I don't think that a good Idea and even it becomes a tedious task for us 😨 to maintain the repos and it is defiantly becomes a tedious for the users to switch between the two repos even though the url's are given for where to look for the writeups at the end of the tasks all consider the people who doesn't use the beautiful github interface and do it form the terminal ? it becomes very difficult for them to see the links without another terminal 😄

I don't think 1Gb is a big issue for someone who is strongly willing to contribute, these days we are hardly seeing the any pull requests any ways 😛

I think the main issue which we can tackle for next year is creating new more beginner-friendly CONTRIBUTING.md or
may be give the exact details of what we are doing using https://readthedocs.org/ (along with basic introduction to git)

also make more writeup tools 👍

[cm-recruit]

I think using a single branch per ctf might be a good way to go about it. For 2016, you can make an empty branch now then the process is:

1. git clone -b empty_ctf --single-branch [url] (or git checkout empty_ctf if you are the maintainer of the master)
2. git checkout -b [ctf_name]

or if the branch already exists for that ctf:

1. git clone -b [ctf_name] --single-branch [url]

Then once this branch exists the maintainer of the master just needs to:

1. git checkout master
2. git merge [ctf_name]

A single branch per ctf allows people to work on the repository without having to clone the entire master branch as it gets bigger. Github also allows users to download a zip of specific branches from the website which is very nice if people want to download a particular ctf and play with those challenges.

I was messing around with the idea here: https://github.com/cm-recruit/write-ups-2016/tree/master

[abpolym]

@cm-recruit Nice idea with using different branches and just cloning one specific branch. However, there are ~60 CTFs in 2015 and I guess more than that in 2016 - so that means at least 60 branches. Keeping track, merging with the master and telling and making people to download specific branches complicates things and adds at least 15-30m maintaining time for each CTF.

[cm-recruit]

Why do you think it would add 15-30 mins? Ideally, you would only accept pull requests on the ctf specific branches and as a maintainer, merge the new commits from that branch into the master. I don't think it should take more effort than you currently invest. It does mean that people interested in helping should do work per ctf on the individual branches. If they have the space, they can still clone the entire repository and just checkout whichever ctf they want to work on.

I have 2015 split as well: https://github.com/cm-recruit/ctf-pwn-2015

I am going to end removing all the non-pwn challenges on my fork this weekend, but before then I can submit a pull request if you want the branches.

[abpolym]

@cm-recruit Maybe we should try your suggestion for the 2016 repo, not sure.

[cm-recruit]

@YASME-Tim
The way to do it is start it from the beginning with the first ctf of the year. It involves a little more work after you have already started just putting things into the master.

[abpolym]

@cm-recruit I think it requires to add as many branches as possible CTFs occurring in 2016 before the first CTF starts, right? Anyway I think we got another , effortless solution that doesn't disturb the current workflow in a higher scale and makes it possible for other CTF participants to get only the files they need:

• 2 repos:
  • write-ups-<year> - contains ONLY descriptions of challenges and files produced by a write-up (e.g. exploit scripts etc)
  • <resourcesname>-<year> - contains the files needed to solve the task, as referenced in a description
• If you want to add a writeup link or repo-local writeup, just clone write-up-<year>. Add your (small) files needed for the write-up
• If you want to clone only files from a specific directory, just follow these instructions (TODO is a quick actual test if this works)
• A script has to be written that just changes links in the descriptions from write-ups-<year>/taskdir/file to <resourcesname>-<year>/taskdir/file -> simple as a sed or a small python script that has to be called on every new CTF skeleton

Advantages:

• No 80 or so branches
• Less complexity and just basic steps required for someone to push a pull-request
• Less disturbance in the current workflow

Disadvantages:

• Files and Descriptions are split

[cm-recruit]

@YASME-Tim
You would not have to make a branch until the ctf has happened. You just need to have one branch that is your base (just your top-level readmes), and branch off of that one for the most recent ctf that you want to add.

Wow, that svn trick is awesome. I tested it with " svn checkout https://github.com/ctfs/write-ups-2015/trunk/32c3-ctf-2015/pwn/teufel-200" and it worked no problem.

I like the new plan. Good luck, and thanks for maintaining these repos.

[abpolym]

@cm-recruit Ah ok :). Awesome to know, thanks @dhanvi for coming up with the stackoverflow link / suggestion. Guess I can close this issue now! Thanks for suggesting some solutions :)

[dhanvi]

I am still against this Idea of splitting the repo into two parts

I have just made a fresh clone of the 2015 repo and I found that it just took me 500 MiB of data which is not so big to cause a big trouble for the people to clone.

All it creates complexity for our simple repo let's ask @mathiasbynens for his opinion too!

[dhanvi]

Also remember we splitted the main write-ups repo to parts based on the year as the repo size is increasing ctfs/write-ups-2014#292

for me it doesn't make sense to split the repo again just to make the repo super light weight and I don't think there will be more contributors because of this

[abpolym]

$ time git clone https://github.com/ctfs/write-ups-2015.git
Cloning into 'write-ups-2015'...
remote: Counting objects: 19718, done.
remote: Compressing objects: 100% (236/236), done.
remote: Total 19718 (delta 104), reused 0 (delta 0), pack-reused 19471
Receiving objects: 100% (19718/19718), 501.02 MiB | 443.00 KiB/s, done.
Resolving deltas: 100% (7309/7309), done.
Checking connectivity... done.
Checking out files: 100% (3776/3776), done.

real    19m39.882s
user    0m24.581s
sys     0m27.991s

Takes me ~20m. I think people who just want to add a writeup link or writeup don't want to wait that long. Even if they have 3x the download speed of my (considerably) slow speed.

A faster cloning should mean more contributors not being placed on hold by the repo download.

[dhanvi]

I don't think 20m or half an hour is a huge amount of time to clone and I don't think we will have more contributors if we reduce the size to 50 MiB / 5 MiB

I can say for sure that the repo size is not the reason for letting down new contributors

[mathiasbynens]

I think people who just want to add a writeup link or writeup don't want to wait that long. Even if they have 3x the download speed of my (considerably) slow speed.

Those people don’t have to perform a full clone though — they can just use the GitHub web UI to submit a simple PR in a few seconds.

I seriously doubt the repo size is a bottleneck for new contributors.

[abpolym]

Ok, so the result is to not change anything and keep doing what we were doing. Don't split the repo, don't use multiple branches. If you don't want to use the web UI to submit pull requests, you have to clone the whole repo.

[dhanvi]

I second that repo size is a issue for new contributors

so let's keep the repo unchanged 👍