Handling argument flags

Josay is correct and you should accept his answer. I'd like to mention a few more things.

1. Please use braces everywhere, the code you're using is known to lead to difficult reading and subtle bugs. Consider:

   if (condition)
       a();
       b();
   

   b() will be executed unconditionally.

2. switch wouldn't have helped with your code since you can only switch on integers. It doesn't make the code much shorter unless you have a lot of cases to handle, and the main benefit isn't shorter code anyway: it's better readability.

3. stricmp is Windows-only, the POSIX equivalent is strcasecmp, but I don't think Windows supports it. If you want to be cross-platform, go for Boost and iequals.
4. You're using C++, so you may want to take advantage of it. With your current code it mostly means using string instead of char* and std::cout instead of printf.

I suggest using a table of :

for each entry in table do:
    if entry.text == parameter
        execute entry.function_pointer
    end-if
end-for

Example table:

"WAD", Process_WAD_argument
"VMF", Process_VMF_argument

The nice feature about tables in this scenario is that you can add parameters without changing the parsing code: add another line to the table.

Here's a version of ifError() with reduced nesting level by bailing out with an early return:

void ifError(argFlags& argFlags, char* errorMessage)
{
    argFlags.err = true;
    if (argFlags.sil)
        return;
    if (!argFlags.err)
    {
        printf("Converts DOOM WAD into VMF and/or BSP file(s).\n\n");
        printf("DOOM2BSP [-silent] [-WAD [path][name]] [-VMF [path]] [-BSP [path]]\n\n");
        printf("  -WAD     Location of WAD.\n");
        printf("  -VMF     Create VMF file(s) in path.\n");
        printf("  -BSP     Create BSP file(s) in path.\n");
        printf("  -silent  Disable text output.\n\n");
    }
    printf(errorMessage);
}

In checkDir() you can merge the two if statements. Make sure to keep the same order, because the second condition needs to operate on st which is initialized in the first condition:

bool checkDir(char* path)
{
    struct stat st;
    if (!stat(path, &st) && st.st_mode & S_IFDIR != 0)
            return true;
    return false;
}

There's not much to simplify in handleArgs(). Note that using switch is not possible because in C++ strings cannot be used with switch (neither C-strings, nor std::string.)

I think you should separate the logic parsing the arguments provided from the logic using the argument parsed. At the moment, you are trying to do everything in one place and I'm not quite sure this is correct (for instance, if a filename is also an option name, this leads to an unexpected behavior).

I'd suggest something like (I'm not even sure it would compile but that just to give you an idea) :

void handleArgs(argFlags& argFlags, files& files, int& argc, char**& argv)
{
    int waitingForArg = 0; // To know if we are expecting an other argument to be provided : 0:no, 1:wad, 2:vmf, 3:bsp
    for (int i = 1; i < argc; i++)
    {
        char* arg = argv[i];
        switch (waitingForArg)
        {
            case 0:
                if (!stricmp(arg, "-silent"))
                    argFlags.sil = true;
                else if (!stricmp(arg, "-wad")
                    waitingForArg = 1;
                else if (!stricmp(arg, "-vmf")
                    waitingForArg = 2;
                else if (!stricmp(arg, "-bsp")
                    waitingForArg = 3;
                break;
            case 1:
                argFlags.wadFile = arg; waitingForArg = 0;
                break;
            case 2:
                argFlags.vmfFile = arg; waitingForArg = 0;
                break;
            case 3:
                argFlags.bspFile = arg; waitingForArg = 0;
                break;
        }
    }

    if (waitingForArg)
    {
        ifError(argFlags, "Error: file name must be provided with option -wad, -vmf or -bsp");
        return;
    }

    // Logic involving wadFile, wmfFile, bspFile. Checking if they are set and/or if they correspond to existing files.
}

This is just a first step to make things easier to follow (on top of being more correct and more efficient).

Then to remove code duplication, you might want to do something like using an array to store the different possible options (["-wad","-vmf","-bsp"]). In order to do so, you'd replace the "else if (!stricmp(arg, "OPTIONS"))" with a loops on the different options. Then you might also be happy to store the flags as an array. I think this would make more sense if you had more flags to handle.

I also notice that you have tagged your question as C++ but your code seems to be using some C-style (printf and stuff). This might be something you should fix as well.