Tell me more ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems.. It's 100% free, no registration required.
up vote 2 down vote favorite

I have a huge file (1/2 million lines) of json.

I need to delete a set of entries that contain a certain string.

{
    "bla1": {
        "Part1": "Plop1",
        "Part2": "Plop2",
        "Part3": "BadFling1<stuff>",
        "part4": "Plop4",
    },
    "bla2": {
        "Part1": "Plop1",
        "Part2": "Plop2",
        "Part3": "<stuff>",
        "part4": "Plop4",
    },
    // etc for many more entries
}

All the entries have "BadFling1" as the prefix to the "Part3" entry.

I was wondering how the best way to automate removing all entries that include "BadFling1". So for example the result of removing the bad entry from above is:

{
    "bla2": {
        "Part1": "Plop1",
        "Part2": "Plop2",
        "Part3": "<stuff>",
        "part4": "Plop4",
    },
    // etc for many more entries
}

My first attempt was working but not fast enough (as it was slightly manual).

/BadFling1
qan3k5ddq
:map z n@a

Now hold down the 'z' key.

My vim foo is not strong enough, so I am unsure how to automate the processes better in vim. Any help appreciated.

Alternative approaches in bash (or other command line tools also welcome).

share|improve this question
Why don't you install Node.js (nodejs.org) and, for instance, underscore package (npm install underscore) with an appropriate select command (has command-line interface, see jsonselect.org for syntax)? It is better to use a chisel than a hammer when a chisel is available. – Deer Hunter Dec 18 '12 at 9:07
1  
@DeerHunter Don't chisels usually require hammers to provide force? :P – jw013 Dec 18 '12 at 17:27

4 Answers

up vote 3 down vote accepted

Try this in vim:

:g/BadFling/normal [{V]}d

The :global command runs a command on all lines that match the pattern (I used BadFling as an example - adjust it if necessary). The command being run in this case is the :normal command, which runs normal mode commands. The purpose of this is to use the power of the [{ and ]} vim movement commands which move between brace pairs. The Vd is combo is used to do a linewise delete. This is not as robust as a JSON parser but works assuming that each "blah1" part is contained within its own set of lines and so a linewise delete won't accidentally delete anything that belongs to another block. For example, the linewise delete approach would not work if you had something like

    ... end of block you want to keep
}, "blah1" : {
    block you want removed
}, "blah2" : {
    start of block you want to keep ...
}

Also, the [{ only uses the immediate parent block so if you have further levels of nesting it also won't work.

share|improve this answer
1  
There is also a text object that could be handy for this (e.g. to avoid clobbering the previous visual selection): a{/a}/aB “a {}-delimited block”. With it, the normal-mode command could be da{dd (delete the containing {} block, then the remaining "key": , line); this has the same effect as your visual mode commands, but avoids using visual mode itself. – Chris Johnsen Dec 18 '12 at 3:39
@ChrisJohnsen Yes that would work. The reason I didn't use it is because it has the risk of deleting an extra line when there is a bare {} block, if such a block existed (probably not). – jw013 Dec 18 '12 at 5:16
up vote 1 down vote

Here's a solution in awk:

awk '/".*":\ {/             { open=line; skip_block=0 }
     /"Part3":\ "BadFling1/ { skip_block=1 }
     /},/                   { if (skip_block) { line=open; next } }
     { lines[line++]=$0 }
     END { for (i=0;i<=line;i++) { print lines[i] } }' yourfile > clean

This is not very well tested, but it should get you started. It will work even if the blocks are of variable length and does not care where in the block the disqualifying line is positioned.

Explanation:

line 1: if line matches the start of a block, note the position in the array, mark the block as good so far

line 2: if line matches the disqualifying line, and mark the block

line 3: match the end of a block. if the block is marked, reset the position in the array to where the block started, and skip to the next line

line 4: add the current line to array and increment line counter

line 5: when done reading file, print the array, containing only the "good" blocks

You can implement the same thing in bash, but awk will be a lot faster, and in my opinion this is the sort of things awk is built for, without having to pull out a "heavier" language.

share|improve this answer
up vote 2 down vote

You can do this with grep and diff, if your version of diff is recent enough:

ire@localhost$ grep -B 3 -A 2 BadFling1 huge.json | diff --changed-group-format="%>" --unchanged-group-format="" - huge.json 
{
    "bla2": {
        "Part1": "Plop1",
        "Part2": "Plop2",
        "Part3": "<stuff>",
        "part4": "Plop4",
    },
    // etc for many more entries
}

The grep pulls out the bad records, by extracting the lines surrounding your match. The diff removes them from the original. As mentioned in the comments, this solution requires that the block sizes be consistent, and the matching line to be in the same place within each block (as in your example).

If that were not the case (varying record size, or unreliable positioning of record elements), I would take that as the cue to write a quick parsing script. You can easily and safely remove these records with only a few lines of Python, which has a built-in JSON parser.

share|improve this answer
This is a clever and novel approach. You should mention that it requires the line offset of the pattern line to be consistent within its "block" though. – jw013 Dec 17 '12 at 21:19
@jw013 - Thanks. I've expanded the explanation to make my assumptions clearer. – ire_and_curses Dec 17 '12 at 21:30
up vote 0 down vote

using vim:

:%s/BadFling1//g

will search for all occurrences of "BadFling1" and replace it with "".

share|improve this answer
4  
He wants the whole of the surrounding record removed, not just the line. – ire_and_curses Dec 17 '12 at 20:53

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.