While loop with result from function - BASH

Your first problem is:

while validmask
do ...

Nothing ever happens there - you call your function without any arguments. And so it returns false and the loop stops.

The next problem is you want to run your function until the return is valid. To do that, you need to use until. I managed to overlook this before.

You need to drop the first if statement and just do:

until validmask "$mask"
do    mask=$(get_new_value)
done

The until loop is the boolean negation of the while. It will run until the command it runs returns true.

It can also be written:

while ! validmask "$mask"
do    mask=$(get_new_mask)
done

You might do the assignment / test at once:

unset mask
until validmask "${mask:=$(get_new_value)}"
do    mask=
done

Another problem is that your validmask function fails to fully validate for a lot of edge cases - especially those which include [*?. I think you should just be using case and nevermind all of the ancillary splitting and variable declarations.

Just rule out bad values:

validmask()
    case    "${1##*[!0-9.]*}"       in
    (.*|*.|*..*|*.*.*.*.*|*[!.][!.][!.][!.]*) ! :;;
    (*[3-9][!.][!.]*|*2[6-9][!.]*|*25[6-9]*)  ! :;;
    (*.*.*.*)       ;;              (*)       ! :;;
    esac

A little demo:

for mask in              \
         0.0.0.0         \
         0.0.0.          \
         0.0.0.1233      \
         0.0.0.233       \
         0.0..233        \
         0.0.2.233       \
         0.5555.2.233    \
         0.55.2.233      \
         .55.2.233       \
         1.55.2.233      \
         255.255.255.255 \
         255.255.256.255
do    validmask "$mask"
      printf "%-16.16s: %.$?0s%.$((!$?*4))s\n%.d" \
             "$mask" bad good  "0$(($?*8))"
      printf "printf's return:\t $?\n\n"
done  2>/dev/null

0.0.0.0         : good
printf's return:     0

0.0.0.          : bad
printf's return:     1

0.0.0.1233      : bad
printf's return:     1

0.0.0.233       : good
printf's return:     0

0.0..233        : bad
printf's return:     1

0.0.2.233       : good
printf's return:     0

0.5555.2.233    : bad
printf's return:     1

0.55.2.233      : good
printf's return:     0

.55.2.233       : bad
printf's return:     1

1.55.2.233      : good
printf's return:     0

255.255.255.255 : good
printf's return:     0

255.255.256.255 : bad
printf's return:     1

Just move your if statement to your while loop.

while true
do
   if ! validmask $mask; then
        mask=$(whiptail --title "xx" --inputbox --nocancel "Bad entry" 3>&1 1>&2 2>&3)
   else
       break
   fi
done