current community

your communities

more stack exchange communities

Stack Exchange
tour help
Take the 2-minute tour ×
Unix & Linux Stack Exchange is a question and answer site for users of Linux, FreeBSD and other Un*x-like operating systems. It's 100% free, no registration required.
up vote 3 down vote favorite

Need some pointing in right direction on script to fetch and regex or sed.

Site24x7 provides a URL with a CSV list of their source IP's used for monitoring. (they also provide other formats, CSV seems the least messed up as their structure leaves a lot to be desired. https://www.site24x7.com/multi-location-web-site-monitoring.html )

Like so:

Country,City,IP Address External
Australia,Sydney,"101.0.67.53"
Australia,Melbourne,"125.214.65.59"
Belgium,Brussels,"87.238.165.164"
Brazil,São Paulo,"200.170.83.170"
Brazil,Rio de Janeiro,"201.20.20.237"
Canada,Toronto,"208.69.56.166,
208.69.56.171,
208.69.56.172 "
Canada,Montreal,"199.204.45.153,
199.204.45.154,
199.204.45.155,
199.204.45.156"

I need to save it as an Allow include file in apache. Like so:

Allow from \
72.5.230.111 \
72.5.230.65 \
72.5.230.84
share|improve this question

2 Answers 2

up vote 1 down vote accepted 
#!/bin/bash

webpage="https://www.site24x7.com/multi-location-web-site-monitoring.html"
csv=$(curl -s "$webpage" | grep 'title="CSV"' | \
    sed 's/^.*href="\(http[^ ]*\)".*$/\1/')

echo -e "Allow from \\"
curl -s "$csv" | egrep -o '[0-9.]{7,15}' | paste -s | sed 's/\t/ \\\n/g'
share|improve this answer
    
Looks Jedi. I'll test in the AM. Thanks, more than expected. –  8None1 Feb 10 '12 at 5:15
up vote 1 down vote

Quick and dirty and not very robust sed(1) one-liner:

echo 'Country,City,IP Address External
Australia,Sydney,"101.0.67.53"
Australia,Melbourne,"125.214.65.59"
Belgium,Brussels,"87.238.165.164"
Brazil,São Paulo,"200.170.83.170"
Brazil,Rio de Janeiro,"201.20.20.237"
Canada,Toronto,"208.69.56.166,
208.69.56.171,
208.69.56.172 "
Canada,Montreal,"199.204.45.153,
199.204.45.154,
199.204.45.155,
199.204.45.156"' | 
    sed -e 1d -e 's/"$//' -e 's/^[^"]*"/Allow from /' -e 's/,$/ \\/'

Output:

Allow from 101.0.67.53
Allow from 125.214.65.59
Allow from 87.238.165.164
Allow from 200.170.83.170
Allow from 201.20.20.237
Allow from 208.69.56.166 \
208.69.56.171 \
208.69.56.172 
Allow from 199.204.45.153 \
199.204.45.154 \
199.204.45.155 \
199.204.45.156

Assumptions:

  • Every IP address group is surrounded with the double quote (") character
  • The " character cannot occur anywhere else except to delimit IP groups.
  • A comma (,) on the end of a line means multiple IP addresses are continued on the next line.
share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.