Releases: SpiderLabs/ModSecurity

Note: additional information on the release and some of the key changes will be published separately in short order.

New features and security impacting issues

• Adjust parser activation rules in modsecurity.conf-recommended
  [Issue #2796 - @terjanq, @martinhsv]
• Multipart parsing fixes and new MULTIPART_PART_HEADERS collection
  [Issue #2795 - @terjanq, @martinhsv]

Bug fixes

• Prevent LMDB related segfault
  [Issue #2755, #2761 - @dvershinin]
• Fix msc_transaction_cleanup function comment typo
  [Issue #2788 - @lookat23]
• Fix: MULTIPART_INVALID_PART connected to wrong internal variable
  [Issue #2785 - @martinhsv]
• Restore Unique_id to include random portion after timestamp
  [Issue #2752, #2758 - @datkps11, @martinhsv]

Note: additional information on the release and some of the key changes will be published separately in short order.

New features and security impacting issues

• Adjust parser activation rules in modsecurity.conf-recommended
  [Issue #2799 - @terjanq, @martinhsv]
• Multipart parsing fixes and new MULTIPART_PART_HEADERS collection
  [Issue #2797 - @terjanq, @martinhsv]

Bug fixes

• Limit rsub null termination to where necessary
  [Issue #2794 - @marcstern, @martinhsv]
• IIS: Update dependencies for next planned release
  [@martinhsv]
• XML parser cleanup: NULL duplicate pointer
  [Issue #2760 - @martinhsv]
• Properly cleanup XML parser contexts upon completion
  [Issue #2239 - @argenet]
• Fix memory leak in streams
  [Issue #2208 - @marcstern, @vloup, @JamesColeman-LW]
• Fix: negative usec on log line when data type long is 32b
  [Issue #2753 - @ABrauer-CPT, @martinhsv]
• mlogc log-line parsing fails due to enhanced timestamp
  [Issue #2682 - @bozhinov, @ABrauer-CPT, @martinhsv]
• Allow no-key, single-value JSON body
  [Issue #2735 - @marcstern, @martinhsv]
• Set SecStatusEngine Off in modsecurity.conf-recommended
  [Issue #2717 - @un99known99, @martinhsv]
• Fix memory leak that occurs on JSON parsing error
  [Issue #2236 @argenet, @vloup, @martinhsv]
• Multipart names/filenames may include single quote if double-quote enclosed
  [Issue #2352 @martinhsv]
• Add SecRequestBodyJsonDepthLimit to modsecurity.conf-recommended
  [Issue #2647 @theMiddleBlue, @airween, @877509395 ,@martinhsv]

New features

• Support PCRE2
  [Issue #2668 - @martinhsv]
• Support SecRequestBodyNoFilesLimit
  [Issue #2670 - @airween, @martinhsv]
• Add ctl:auditEngine action support
  [Issue #2606 - @alekravch, @martinhsv]

Bug fixes

• Move PCRE2 match block from member variable
  [@martinhsv]
• Add SecArgumentsLimit, 200007 to modsecurity.conf-recommended
  [Issue #2738 - @jleproust, @martinhsv]
• Fix memory leak when concurrent log includes REMOTE_USER
  [Issue #2727 - @liudongmiao]
• Fix LMDB initialization issues
  [Issue #2688 - @ziollek @martinhsv]
• Fix initcol error message wording
  [Issue #2732 - @877509395, @martinhsv]
• Tolerate other parameters after boundary in multipart C-T
  [Issue #1900 - @martinhsv]
• Add DebugLog message for bad pattern in rx operator
  [Issue #2723 - @martinhsv]
• Fix misuses of LMDB API
  [Issue #2601, #2602 - @hyc]
• Fix duplication typo in code comment
  [Issue #2677 - @gleydsonsoares]
• Fix multiMatch msg, etc, population in audit log
  [Issue #2573 - @Sachin-M-Desai , @martinhsv]
• Fix some name handling for ARGS_*NAMES: regex SecRuleUpdateTargetById, etc.
  [Issue #2627, #2648 - @lontchianicet , @victorserbu2709 , @martinhsv]
• Adjust confusing variable name in setRequestBody method
  [Issue #2635 - @Mesar-Ali , @martinhsv]
• Multipart names/filenames may include single quote if double-quote enclosed
  [Issue #2352 - @martinhsv]
• Add SecRequestBodyJsonDepthLimit to modsecurity.conf-recommended
  [Issue #2647 - @theMiddleBlue , @airween , @877509395 , @martinhsv]

Security issue

• Support configurable limit on depth of JSON parsing (possible DoS issue)
  [@theMiddleBlue, @airween, @dune73, @martinhsv]

Notes

• For Windows, as we are not aware of anyone using the 32-bit installer, only the 64-bit installer is now included
• Users of ModSecurity that cannot update immediately may wish to consult issue #2647, or the related blog post, for mitigation suggestions.

Security issue

• Support configurable limit on depth of JSON parsing (possible DoS issue)
  [@theMiddleBlue, @martinhsv]

New features

• Having ARGS_NAMES, variables proxied
  [@zimmerle, @martinhsv, @KaNikita]
• Use explicit path for cross-compile environments.
  [Issue #2485 - @dtoubelis]
• Fix: FILES variable does not use multipart part name for key
  [Issue #2377 - @martinhsv]
• Regression: Mark the test as failed in case of segfault.
  [@zimmerle]
• GeoIP: switch to GEOIP_MEMORY_CACHE from GEOIP_INDEX_CACHE
  [Issues #2378, #2186 - @defanator]
• Add support to test framework for audit log content verification
  and add regression tests for issues #2000, #2196
• Support configurable limit on number of arguments processed
  [Issue #2234 - @jleproust, @martinhsv]
• Multipart Content-Dispostion should allow field: filename*=
  [@martinhsv]
• Adds support to lua 5.4
  [@zimmerle]
• Add support for new operator rxGlobal
  [@martinhsv]

Bug fixes

• Replaces put with setenv in SetEnv action
  [Issue #2469 - @martinhsv, @WGH-, @zimmerle]
• Regex key selection should not be case-sensitive
  [Issue #2296, #2107, #2297 - @michaelgranzow-avi, @victorhora,
  @airween, @martinhsv, @zimmerle]
• Fix: Only delete Multipart tmp files after rules have run
  [Issue #2427 - @martinhsv]
• Fixed MatchedVar on chained rules
  [Issue #2423, #2435, #2436 - @michaelgranzow-avi]
• Fix maxminddb link on FreeBSD
  [Issue #2131 - @granalberto, @zimmerle]
• Fix IP address logging in Section A
  [Issue #2300 - @inaratech, @zavazingo, @martinhsv]
• rx: exit after full match (remove /g emulation); ensure capture
  groups occuring after unused groups still populate TX vars
  [Issue #2336 - @martinhsv]
• Correct CHANGES file entry for #2234
• Fix rule-update-target for non-regex
  [Issue #2251 - @martinhsv]
• Fix configure script when packaging for Buildroot
  [Issue #2235 - @frankvanbever]
• modsecurity.pc.in: add Libs.private
  [Issue #1918, #2253 - @ffontaine, @Dridi, @victorhora]

Security Impacting Issues

• Handle URI received with uri-fragment
  [@martinhsv]

Enhancements

• Add microsec timestamp resolution to the formatted log timestamp
  [Issue #2095 - @rainerjung]
• Added missing Geo Countries
  [Issue #2123, #2124 - @emphazer]

Bug fixes

• Store temporaries in the request pool for regexes compiled per-request.
  [Issue #890, #2049 - @lightsey]
• Fix other usage of the global pool for request temporaries in re_operators.c
  [Issue #890, #2049 - @lightsey]
• Adds a sanity check before use ctl:ruleRemoveTargetById and ctl:ruleRemoveTargetByMsg.
  [Issue #2033 - @studersi]
• Fix the order of error_msg validation
  [Issue #2128 - @marcstern, @zimmerle]
• When the input filter finishes, check whether we returned data
  [Issue #2091, #2092 - @rainerjung]
• fix: care non-null terminated chunk data
  [Issue #2097 - @orisano]
• Fix for apr_global_mutex_create() crashes with mod_security
  [Issue #1957 - @blappm]
• Fix inet addr handling on 64 bit big endian systems
  [Issue #1980 - @zimmerle, @airween]

Notes

• Windows installer no longer includes OWASP CRS.

New features

• SecRuleUpdateTargetById now supports regular expressions
  [Issue #1872 - @zimmerle, @anush-cr, @victorhora, @j0k2r]
• Adds a new operator verifySVNR that checks for Austrian social
  security numbers.
  [Issue #2063 - @Rufus125]
• Allow 0 length JSON requests.
  [Issue #1822 - @allanbomsft, @zimmerle, @victorhora, @marcstern]
• Adds support to multiple ranges in ctl:ruleRemoveById
  [Issue #1956 - @theseion, @victorhora, @zimmerle]

Bug fixes

• Fix: audit log data omitted when nolog,auditlog
  [@martinhsv]
• Adds missing check for runtime ctl:ruleRemoveByTag
  [Issue #2102, #2099 - @airween]
• Fix: ModSecurity 3.x inspectFile operator does not pass FILES_TMPNAMES parameter to lua engine
  [Issue #2204, #2205 - @kadirerdogan]
• XML: Remove error messages from stderr
  [Issue #2010 - @JaiHarpalani, @zimmerle]
• Filter comment or blank line for pmFromFile operator
  [Issue #1645 - @LeeShan87, @victorhora, @tdoubley]
• Additional adjustment to Cookie header parsing
  [@martinhsv]
• Restore chained rule part H logging to be more like 2.9 behavior
  [Issue #2196 - @martinhsv]
• Small fixes in log messages to help debugging the file upload
  [Issue #2130 - @airween]
• Fix Cookie header parsing issues
  [Issue #2201 - @airween, @martinhsv]
• Fix rules with nolog are logging to part H
  [Issue #2196 - @martinhsv]
• Fix argument key-value pair parsing cases
  [Issue #1904 - @martinhsv]
• Fix: audit log part for response body for JSON format to be E
  [Issue #2066 - @martinhsv, @zimmerle]
• Make sure m_rulesMessages is filled after successful match
  [Issue #2000, #2048 - @victorhora, @defanator]
• Fix @pm lookup for possible matches on offset zero.
  [@zimmerle, @afoxdavidi, @martinhsv, @marshal09]
• Regex lookup on the key name instead of COLLECTION:key
  [@rdiperri-yottaa, @danbiagini-work, @mmelo-yottaa, @zimmerle]
• Missing throw in Operator::instantiate
  [Issue #2106 - @marduone]
• Making block action execution dependent on the SecEngine status
  [Issue #2113, #2111 - @theMiddleBlue, @airween]
• Making block action execution dependent of the SecEngine status
  [Issue #1960 - @theMiddleBlue, @zimmerle, @airween, @victorhora]
• Having body limits to respect the rule engine state
  [@zimmerle]
• Fix variables output in debug logs
  [Issue #2057 - @jleproust]
• Correct typo validade in log output
  [Issue #2059 - @nerrehmit]
• fix/minor: Error encoding hexa decimal.
  [Issue #2068 - @tech-ozon-io]
• Limit more log variables to 200 characters.
  [Issue #2073 - @jleproust]
• parser: fix parsed file names
  [@zimmerle]
• Allow empty anchored variable
  [Issue #2024 - @airween]
• Fixed FILES_NAMES collection after the end of multipart parsing
  [Issue #2016 - @airween]
• Fixed validateByteRange parsing method
  [Issue #2017 - @airween]
• Removes a memory leak on the JSON parser
  [@zimmerle]
• Enables LMDB on the regression tests.
  [Issue #2011, #2008 - @WGH-, @mdunc]
• Fix: Extra whitespace in some configuration directives causing error
  [Issue #2006 - @porjo, @zimmerle]
• Refactoring on Regex and SMatch classes.
  [@WGH-]
• Fixed buffer overflow in Utils::Md5::hexdigest()
  [Issue #2002 - @defanator]
• Implemented merge() method for ConfigInt, ConfigDouble, ConfigString
  [Issue #1990 - @defanator]
• Adds initially support the drop action.
  [@zimmerle]
• Complete merging of particular rule properties
  [Issue #1978 - @defanator]
• Replaces AC_CHECK_FILE with 'test -f'
  [Issue #1984 - @chuckwolber]
• Fix inet addr handling on 64 bit big-endian systems
  [Issue #1980 - @airween]
• Fix tests on FreeBSD
  [Issue #1973 - @defanator]
• Changes ENV test case to read the default MODSECURTIY env var
  [Issue #1969 - @zimmerle, @airween, @inittab]
• Regression: Sets MODSECURITY env var during the tests execution
  [Issue #1969 - @zimmerle, @airween, @inittab]
• Fix setenv action to strdup key=variable
  [@zimmerle]
• Fix "make dist" target to include default configuration
  [Issue #1966 - @defanator]
• Replaced log locking using mutex with fcntl lock
  [Issue #1949, #1927 - @Cloaked9000]
• Correct the usage of modsecurity::Phases::NUMBER_OF_PHASES
  [Issue #1959 - @weliu]
• Rule variable interpolation broken
  [Issue #1961 - @soonum, @zimmerle]
• Make the boundary check less strict as per RFC2046
  [Issue #1943 - @victorhora, @allanbomsft]
• Fix buffer size for utf8toUnicode transformation
  [Issue #1208 - @katef, @victorhora]

Security issue

• Cookie parser problems
  [@airween, @theMiddleBlue, @martinhsv]

Bug fixes

• Fix buffer size for utf8toUnicode transformation
  [Issue #1208 - @katef, @victorhora]
• Fix sanitizing JSON request bodies in native audit log format
  [p0pr0ck5, @victorhora]
• Fix NetBSD build by renaming the hmac function to avoid conflicts
  [Issue #1241 - @victorhora, @joerg, @sevan]
• IIS: Windows build, fix duplicate YAJL dir in script
  [Issue #1612 - @allanbomsft, @victorhora]
• Fix mpm-itk / mod_ruid2 compatibility
  [Issue #712 - @ju5t , @derhansen, @meatlayer, @victorhora]
• potential off by one in parse_arguments
  [Issue #1799 - @tinselcity, @zimmerle]
• Fix utf-8 character encoding conversion
  [Issue #1794 - @tinselcity, @zimmerle]
• Fix ip tree lookup on netmask content
  [Issue #1793 - @tinselcity, @zimmerle]
• build: fix when multiple lines for curl version
  [Issue #1771 - @Artistan]
• Fixes SecConnWriteStateLimit
  [Issue #1545 - @nicjansma]
• Adds missing headers
  [Issue #1454 - @devnexen]

Improvements

• Allow 0 length JSON requests.
  [Issue #1822 - @allanbomsft, @zimmerle, @victorhora, @marcstern]
• Include unanmed JSON values in unnamed ARGS
  [Issue #1577, #1576 - @marcstern, @victorhora, @zimmerle]
• IIS: Update Wix installer to bundle a supported CRS version (3.0)
  [@victorhora, @zimmerle]
• IIS: Update dependencies for Windows build
  [Issue #1848 - @victorhora, @hsluoyz]
• IIS: Set SecStreamInBodyInspection by default on IIS builds (#1299)
  [Issue #1299 - @victorhora]
• IIS: Update modsecurity.conf
  [Issue #788 - @victorhora, @brianclark]
• Add sanity check for a couple malloc() and make code more resilient
  [Issue #979 - @dogbert2, @victorhora, @zimmerle]
• IIS: Remove body prebuffering due to no locking in modsecProcessRequest
  [Issue #1917 - @allanbomsft, @victorhora]
• Code cosmetics: checks if actionset is not null before use it
  [Issue #1556 - @marcstern, @zimmerle, @victorhora]
• Only generate SecHashKey when SecHashEngine is On
  [Issue #1671 - @dmuey, @monkburger, @zimmerle]
• Docs: Reformat README to Markdown and update dependencies
  [Issue #1857 - @hsluoyz, @victorhora]
• IIS: no lock on ProcessRequest. No reload of config.
  [Issue #1826 - @allanbomsft]
• IIS: buffer request body before taking lock
  [Issue #1651 - @allanbomsft]
• good practices: Initialize variables before use it
  [Issue #1889 - Marc Stern]
• Let body parsers observe SecRequestBodyNoFilesLimit
  [Issue #1613 - @allanbomsft]
• IIS: set overrideModeDefault to Allow so that individual websites can
  add <ModSecurity ...> to their web.config file
  [Issue #1781 - @default-kramer]
• modsecurity.conf-recommended: Fix spelling
  [Issue #1721 - @padraigdoran]
• Fix arabic charset in unicode_mapping file
  [Issue #1619 - @alaa-ahmed-a]
• Optionally preallocates memory when SecStreamInBodyInspection is on
  [Issue #1366 - @allanbomsft, @zimmerle]
• Fixed typo in build_yajl.bat
  [Issue #1366 - @allanbomsft]
• Added "empy chunk" check
  [Issue #1347, #1446 - @gravagli, @bostrt, @zimmerle]
• Add capture action to @detectXSS operator
  [Issue #1488, #1482 - @victorhora]
• Fix for wildcard operator when loading conf files on Nginx / IIS
  [Issue #1486, #1285 - @victorhora and @thierry-f-78]
• Set of fixies to make windows build workable with the buildbots
  [Commit 94fe3 - @zimmerle]
• Uses LOG_NO_STOPWATCH instead of DLOG_NO_STOPWATCH
  [Issue #1510 - @marcstern]

New features

• Adds new transaction constructor that accepts the transaction id
  as parameter.
  [Issue #1627 - @defanator, @zimmerle]
• Adds support to UpdateActionById.
  [Issue #1800 - @zimmerle, @victorhora, @NisariAIT]
• Adds support to setenv action.
  [Issue #1044 - @zimmerle]
• Adds support for ctl:requestBodyProcessor=URLENCODED
  [Issue #1797 - @victorhora]
• Implement support for Lua 5.1
  [Issue #1809 - @p0pr0ck5, @victorhora]

Bug fixes

• Fix double macros bug
  [Issue #1943 - @supplient, @zimmerle]
• Override the default status code if not suitable to redirect action
  [Issue #1850 - @zimmerle, @victorhora]
• parser: Fix the support for CRLF configuration files
  [Issue #1945 - @zimmerle, @defanator, @kjakub]
• m_lineNumber in Rule not mapping with the correct line number in file
  [Issue #1844 - @zimmerle, @victorhora, @xizeng]
• Fix the SecUnicodeMapFile and SecUnicodeCodePage
  [0x3094d - @zimmerle, @victorhora]
• Fix crash in msc_rules_add_file() when using disruptive action in chain
  [Issue #1849 - @victorhora, @zimmerle, @rperper]
• Fix memory leak in AuditLog::init()
  [Issue #1897 - @weliu]
• Fix RulesProperties::appendRules()
  [Issue #1901 - @steven-j-wojcik]
• Fix RULE lookup in chained rules
  [0x3077c - @zimmerle]
• Add correct C function prototypes for msc_init and msc_create_rule_set
  [Issue #1922 - @steven-j-wojcik]
• Fix: function m.setvar in Lua scripts and add testcases
  [Issue #1859 - @nowaits, @victorhora]
• Fix SecResponseBodyAccess and ctl:requestBodyAccess directives
  [Issue #1531 - @victorhora, @defanator]
• parser: Fix simple quote setvar in the end of the line
  [Issue #1831 - @zimmerle, @csanders-git]
• Fix pc file
  [Issue #1847 - @gquintard]
• Fix utf-8 character encoding conversion
  [Issue #1794 - @tinselcity, @zimmerle]
• Fixed LMDB collection errors
  [Issue #1787 - @airween, @zimmerle]
• Fix ip tree lookup on netmask content
  [Issue #1793 - @tinselcity, @zimmerle]
• Fix race condition in UniqueId::uniqueId()
  [Issue #1786 - @weliu]
• Fix memory leak in error message for msc_rules_merge C APIs
  [Issue #1765 - @weliu]
• Build System: Fix when multiple lines for curl version.
  [Issue #1771 - @Artistan]
• Fix LDFLAGS for unit tests.
  [Issue #1758 - @smlx]
• Fix STATUS var parsing and accept STATUS_LINE var for v2 backward comp.
  [Issue #1738 - @victorhora]
• Fix broken @detectxss operator regression test case
  [Issue #1739 - @p0pr0ck5]
• Fix memory leak in modsecurity::utils::expandEnv()
  [Issue #1750 - @defanator] - Fix variable FILES_TMPNAMES
  [Issue #1646, #1610 - @victorhora, @zimmerle, @defanator]
• Fix memory leak in Collections
  [Issue #1729, #1730 - @defanator]

Improvements

• Organizes the server logs
  [0xb7c36 and 0x5ac20 - @zimmerle, @steven-j-wojcik]
• Using shared_ptr instead of unique_ptr on rules exceptions
  [Issue #1697 - @zimmerle, @brianp9906, @victorhora, @LeSwiss, @defanator]
• Changes debuglogs schema to avoid unecessary str allocation
  [0xb2840 - @zimmerle]
• Changes the timing to save the rule message
  [0xca270 - @zimmerle]
• @ipMatch "Could not add entry" on slash/32 notation in 2.9.0
  [Issue #849 - @zimmerle, @dune73]
• Using values after transformation at MATCHED_VARS
  [0x14316 - @zimmerle]
• Allow LuaJIT 2.1 to be used
  [Issue #1909 - @victorhora, @mdunc]
• Match m_id JSON log with RuleMessage and v2 format
  [Issue #1185 - @victorhora]
• Adds request IDs and URIs to the debug log
  [Issue #1627 - @defanator, @zimmerle]
• Treating variables exception on load-time instead of run time.
  [0x028e0 and 0x275a1 - @zimmerle]
• Fix OpenBSD build
  [Issue #1841 - @victorhora, @zimmerle, @juanfra684]
• Fix parser to support GeoLookup with MaxMind
  [Issue #1884, #1895 - @victorhora, @everping]
• modsec_rules_check: uses the gnu .la' instead of .a' file
  [Issue #1853 - @ste7677, @victorhora, @zimmerle]
• good practices: Initialize variables before use it
  [Issue #1889 - Marc Stern]
• Add LUA compatibility for CentOS and try to use LuaJIT first if available
  [Issue #1622 - @victorhora, @dmitryzykov]
• Allow LuaJIT to be used
  [Issue #1809 - @victorhora, @p0pr0ck5]
• Variable names must match fully, not partially. Match should be case insensitive.
  [Issue #1818, #1820, #1810, #1808 - @michaelgranzow-avi, @victorhora, @theMiddleBlue, @airween, @zimmerle, @LeeShan87]
• Improves the performance while loading the rules
  [Issue #1735 - @zimmerle, @p0pr0ck5, @victorhora]
• Allow empty strings to be evaluated by regex::searchAll
  [Issue #1799, #1785 - @victorhora, @XuanHuyDuong, @zimmerle]
• Adds basic pkg-config info
  [Issue #1790 - @gquintard, @zimmerle]
• Fixed false positive MULTIPART_UNMATCHED_BOUNDARY errors
  [Issue #1747, #1924 - @airween, @victorhora, @defanator, @zimmerle]
• Changes the behavior of the default sec actions
  [Issue #1629 - @mirkodziadzka-avi, @zimmerle, @victorhora]
• Refactoring on {global,ip,resources,session,tx,user} collections
  [Issue #1754, #1778 - @LeeShan87, @zimmerle, @victorhora, @wwd5613, @sobigboy]
• Return false in SharedFiles::open() when an error happens
  [Issue #1783 - @weliu]
• Use rvalue reference in ModSecurity::serverLog
  [Issue #1769 - @weliu]
• Checks if response body inspection is enabled before process it
  [Issue #1643 - @zoltan-fedor, @dennus, @defanator, @zimmerle]
• Code Cleanup.
  [Issue #1757, #1755, #1756, #1761 - @p0pr0ck5]
• Fix setvar parsing of quoted data
  [Issue #1733, #1759, #1775 - @victorhora, @JaiHarpalani, @defanator]
• Adds time stamp back to the audit logs
  [Issue #1762 - @Pjack, @zimmerle]
• Disables skip counter if debug log is disabled
  [@zimmerle]
• Cosmetics: Represents amount of skipped rules without decimal
  [Issue #1737 - @p0pr0ck5]
• Add missing escapeSeqDecode, urlEncode and trimLeft/Right tfns to parser
  [Issue #1752 - @victorhora]
• Initialize m_dtd member in ValidateDTD class as NULL
  [Issue #1751 - @airween]
• Fix utils::string::ssplit() to handle delimiter in the end of string
  [Issue #1743, #1744 - @defanator]