current community

your communities

more stack exchange communities

Stack Exchange
tour help
Take the 2-minute tour ×
Programmers Stack Exchange is a question and answer site for professional programmers interested in conceptual questions about software development. It's 100% free, no registration required.
up vote 9 down vote favorite
1

I'm writing some test code for a feature which processes PDF files. The basic idea behind the tests is that I point them towards some PDFs I've selected specially, they process them and I check that the output is what I expect.

My question is: where should I be storing these large-ish PDFs? Should I check them into version control along with the code? Or put them somewhere else? Obviously, the test code is useless without the PDFs (or even with different PDFs) but still, putting them into our repository feels wrong.

share|improve this question
1  
possible duplicate of Should unit tests be stored in the repository? –  Michael Kjörling 8 hours ago
1  
@MichaelKjörling: Tests != Test Data –  Robert Harvey 8 hours ago

3 Answers 3

up vote 27 down vote

Your version control system should contain everything it needs to build, compile, test, and package an application for distribution (e.g. MSI, RPM). I would also argue build configurations and other scripts should also be in version control.

I should be able to check out a project and have a complete compile, build, and test environment.

There are two approaches to checking in test data. First, you can check in the test data itself (PDFs in this case). Second, you can check in source data that can be used to generate test data (if applicable). This could be a SQL script loaded into a blank database containing test data, or maybe a text-based file that can be compiled into a PDF or other file.

Others may disagree with checking everything into version control, but I have found in my professional experience it is critical to ensuring a complete environment is able to be rebuilt from scratch.

share|improve this answer
5  
Yes. Absolutely yes. It's 2014, there is no justification whatsoever for using revision control that doesn't handle binary files seamlessly. –  Kilian Foth 11 hours ago
    
100% agree. VCS is the system of record for your development effort. –  Chris McCall 10 hours ago
    
I agree, but you definitely want to avoid the situation where you are checking in junk items as well. For example, if the test data includes an "output" folder that contains all the pdf files generated by the tests, then you will want to not include that into repository. But I do agree the tests themselves should be part of the repo as well as any packages needed to run it. –  Kenneth Garza 9 hours ago
    
@KennethGarza It isn't hard, really. As a rule of thumb, any original content (source code, test source code, test data, media, [real] documentation, third party libraries, build scripts, tooling scripts, conversion scripts, etc.) should be included, while all data that can be generated in reasonable time from the original data should not be. Besides, given those are the test outputs, they probably only make sense after running the tests yourself, otherwise you are not testing your program, you are testing the VCS software's ability to preserve the integrity of your files :) –  Thomas 8 hours ago
    
@KennethGarza to expand on Thomas' point, it should be fairly trivial to instruct your VCS to ignore generated files. For example, with SVN, I will check in e.g. an output folder with svn:ignore set to *. That way it is not possible to check in intermediate or generated files without purposefully jumping through hoops, and the project will not show up as dirty after building. –  Snowman 8 hours ago
up vote 8 down vote

If the tests are useless without the setup files that you have prepared, then it makes sense to include the files in your VCS along with the test code.

While the files used in the test aren't code, you can view them as a dependency that the code relies upon. So there is merit in keeping everything together.

As a counterpoint, some VCSs don't handle large binary files well, and others have strong opposition to including any sort of binary file in a VCS. If either of those cases apply to you, then storing the test files in a well known location that is easily accessed would also make sense.

I would also consider putting a comment in the test code that says "relies upon foo.pdf in order to run all tests."

share|improve this answer
    
I don't see anything wrong with having the tests check for the test data, if not found then trying to get it (eg. from a URL) and failing if neither worked. Relying on the network is a bad idea because it makes tests slower and more fragile; but trying is less fragile than not, and automatically getting (and caching locally) the right data is quicker than manually reading docs/comments, getting it and putting it in place. –  Warbo 5 hours ago
up vote 3 down vote

If it's static data, then yes put it in version control. Those files won't really change once they're checked in; they'll either get removed if that functionality's no longer needed, or new test files will be added alongside. Either way, you don't need to worry about poor binary diffs taking up space.

If you're generating test data, eg. randomly, then you should automatically save it when a test fails, but discard it otherwise. Any data saved this way should be turned into regular regression tests, so that those edge-cases are definitely tested in the future rather than relying on the luck of the draw.

share|improve this answer
    
If you're generating test data randomly, then you should really go out and buy a book about writing reproducible automated tests. –  David Wallace 6 hours ago
    
@DavidWallace I dunno; QuickCheck's my current favourite test framework, and that has a load of free resources online (eg. Real World Haskell) –  Warbo 6 hours ago
    
OK, my point is - random test data leads to a world of hurt. Never do it. –  David Wallace 6 hours ago
3  
@DavidWallace So you're saying entire fields like fuzz testing, property checking and statistical software testing are not only wrong, but harmful? –  Warbo 5 hours ago
1  
@DavidWallace random != unreproducible. –  congusbongus 2 hours ago

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.