all client autobahn test should pass ! last failing one was ...

+- zlib/deflate has a bug with windowsbits == 8, so we silently upgrade it to 9/ (fix autobahn test 13.X which uses 8 for the windows size)

CMakeLists.txt

@@ -69,6 +69,7 @@ set( IXWEBSOCKET_HEADERS
     ixwebsocket/IXSocketFactory.h
     ixwebsocket/IXSocketServer.h
     ixwebsocket/IXUrlParser.h
+    ixwebsocket/IXUtf8Validator.h
     ixwebsocket/IXUserAgent.h
     ixwebsocket/IXWebSocket.h
     ixwebsocket/IXWebSocketCloseConstants.h

DOCKER_VERSION

@@ -1 +1 @@
-5.1.4
+6.0.0

README.md

@@ -10,4 +10,4 @@ Interested ? Go read the [docs](https://bsergean.github.io/IXWebSocket/site/) !
 
 IXWebSocket is actively being developed, check out the [changelog](CHANGELOG.md) to know what's cooking. If you are looking for a real time messaging service (the chat-like 'server' your websocket code will talk to) with many features such as history, backed by Redis, look at [cobra](https://github.com/machinezone/cobra).
 
-IXWebSocket is not yet autobahn compliant, but we are working on changing this. See the current compliance [test results](https://bsergean.github.io/IXWebSocket/autobahn/index.html).
+IXWebSocket is not yet autobahn compliant, but we are working on changing this. See the current compliance [test results](https://bsergean.github.io/IXWebSocket/autobahn/index.html). The only tests that are still failing are the Websocket Compression ones (see section 12 and 13).

docs/CHANGELOG.md

@@ -1,6 +1,40 @@
 # Changelog
 All notable changes to this project will be documented in this file.
 
+## [6.0.0] - 2019-09-04
+
+- all client autobahn test should pass !
+- zlib/deflate has a bug with windowsbits == 8, so we silently upgrade it to 9/ (fix autobahn test 13.X which uses 8 for the windows size)
+
+## [5.2.0] - 2019-09-04
+
+- Fragmentation: for sent messages which are compressed, the continuation fragments should not have the rsv1 bit set (fix all autobahn tests for zlib compression 12.X)
+- Websocket Server / do a case insensitive string search when looking for an Upgrade header whose value is websocket. (some client use WebSocket with some upper-case characters)
+
+## [5.1.9] - 2019-09-03
+
+- ws autobahn / report progress with spdlog::info to get timing info
+- ws autobahn / use condition variables for stopping test case + add more logging on errors
+
+## [5.1.8] - 2019-09-03
+
+- Per message deflate/compression: handle fragmented messages (fix autobahn test: 12.1.X and probably others)
+
+## [5.1.7] - 2019-09-03
+
+- Receiving invalid UTF-8 TEXT message should fail and close the connection (fix remaining autobahn test: 6.X UTF-8 Handling)
+
+## [5.1.6] - 2019-09-03
+
+- Sending invalid UTF-8 TEXT message should fail and close the connection (fix remaining autobahn test: 6.X UTF-8 Handling)
+- Fix failing unittest which was sending binary data in text mode with WebSocket::send to call properly call WebSocket::sendBinary instead.
+- Validate that the reason is proper utf-8. (fix autobahn test 7.5.1)
+- Validate close codes. Autobahn 7.9.*
+
+## [5.1.5] - 2019-09-03
+
+Framentation: data and continuation blocks received out of order (fix autobahn test: 5.9 through 5.20 Fragmentation)
+
 ## [5.1.4] - 2019-09-03
 
 Sending invalid UTF-8 TEXT message should fail and close the connection (fix **tons** of autobahn test: 6.X UTF-8 Handling)
@@ -19,17 +53,17 @@ Close connections when reserved bits are used (fix autobahn test: 3.X Reserved B
 
 ## [5.1.0] - 2019-08-31
 
-ws autobahn / Add code to test websocket client compliance with the autobahn test-suite
-add utf-8 validation code, not hooked up properly yet
-Ping received with a payload too large (> 125 bytes) trigger a connection closure
-cobra / add tracking about published messages
-cobra / publish returns a message id, that can be used when 
-cobra / new message type in the message received handler when publish/ok is received (can be used to implement an ack system).
+- ws autobahn / Add code to test websocket client compliance with the autobahn test-suite
+- add utf-8 validation code, not hooked up properly yet
+- Ping received with a payload too large (> 125 bytes) trigger a connection closure
+- cobra / add tracking about published messages
+- cobra / publish returns a message id, that can be used when 
+- cobra / new message type in the message received handler when publish/ok is received (can be used to implement an ack system).
 
 ## [5.0.9] - 2019-08-30
 
-User-Agent header is set when not specified.
-New option to cap the max wait between reconnection attempts. Still default to 10s. (setMaxWaitBetweenReconnectionRetries).
+- User-Agent header is set when not specified.
+- New option to cap the max wait between reconnection attempts. Still default to 10s. (setMaxWaitBetweenReconnectionRetries).
 
 ```
 ws connect --max_wait 5000 ws://example.com # will only wait 5 seconds max between reconnection attempts

docs/design.md

@@ -32,7 +32,6 @@ The regression test is running after each commit on travis.
 
 * On Windows TLS is not setup yet to validate certificates.
 * There is no convenient way to embed a ca cert.
-* No utf-8 validation is made when sending TEXT message with sendText()
 * Automatic reconnection works at the TCP socket level, and will detect remote end disconnects. However, if the device/computer network become unreachable (by turning off wifi), it is quite hard to reliably and timely detect it at the socket level using `recv` and `send` error codes. [Here](https://stackoverflow.com/questions/14782143/linux-socket-how-to-detect-disconnected-network-in-a-client-program) is a good discussion on the subject. This behavior is consistent with other runtimes such as node.js. One way to detect a disconnected device with low level C code is to do a name resolution with DNS but this can be expensive. Mobile devices have good and reliable API to do that.
 * The server code is using select to detect incoming data, and creates one OS thread per connection. This is not as scalable as strategies using epoll or kqueue.
 

ixwebsocket/IXUtf8Validator.h

@@ -0,0 +1,167 @@
+/*
+ * The following code is adapted from code originally written by Bjoern
+ * Hoehrmann <bjoern@hoehrmann.de>. See
+ * http://bjoern.hoehrmann.de/utf-8/decoder/dfa/ for details.
+ *
+ * The original license:
+ *
+ * Copyright (c) 2008-2009 Bjoern Hoehrmann <bjoern@hoehrmann.de>
+ *
+ * Permission is hereby granted, free of charge, to any person obtaining a copy
+ * of this software and associated documentation files (the "Software"), to deal
+ * in the Software without restriction, including without limitation the rights
+ * to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+ * copies of the Software, and to permit persons to whom the Software is
+ * furnished to do so, subject to the following conditions:
+ *
+ * The above copyright notice and this permission notice shall be included in
+ * all copies or substantial portions of the Software.
+ *
+ * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+ * IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+ * FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+ * AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+ * LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+ * OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+ * SOFTWARE.
+*/
+
+/*
+ *  IXUtf8Validator.h
+ *  Author: Benjamin Sergeant
+ *  Copyright (c) 2019 Machine Zone, Inc. All rights reserved.
+ *
+ *  From websocketpp. Tiny modifications made for code style, function names etc...
+ */
+
+#pragma once
+
+#include <cstdint>
+#include <string>
+
+namespace ix
+{
+    /// State that represents a valid utf8 input sequence
+    static unsigned int const utf8_accept = 0;
+    /// State that represents an invalid utf8 input sequence
+    static unsigned int const utf8_reject = 1;
+
+    /// Lookup table for the UTF8 decode state machine
+    static uint8_t const utf8d[] = {
+      0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 00..1f
+      0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 20..3f
+      0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 40..5f
+      0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, // 60..7f
+      1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9,9, // 80..9f
+      7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7,7, // a0..bf
+      8,8,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2,2, // c0..df
+      0xa,0x3,0x3,0x3,0x3,0x3,0x3,0x3,0x3,0x3,0x3,0x3,0x3,0x4,0x3,0x3, // e0..ef
+      0xb,0x6,0x6,0x6,0x5,0x8,0x8,0x8,0x8,0x8,0x8,0x8,0x8,0x8,0x8,0x8, // f0..ff
+      0x0,0x1,0x2,0x3,0x5,0x8,0x7,0x1,0x1,0x1,0x4,0x6,0x1,0x1,0x1,0x1, // s0..s0
+      1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,0,1,1,1,1,1,0,1,0,1,1,1,1,1,1, // s1..s2
+      1,2,1,1,1,1,1,2,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,2,1,1,1,1,1,1,1,1, // s3..s4
+      1,2,1,1,1,1,1,1,1,2,1,1,1,1,1,1,1,1,1,1,1,1,1,3,1,3,1,1,1,1,1,1, // s5..s6
+      1,3,1,1,1,1,1,3,1,3,1,1,1,1,1,1,1,3,1,1,1,1,1,1,1,1,1,1,1,1,1,1, // s7..s8
+    };
+
+    /// Decode the next byte of a UTF8 sequence
+    /**
+     * @param [out] state The decoder state to advance
+     * @param [out] codep The codepoint to fill in
+     * @param [in] byte The byte to input
+     * @return The ending state of the decode operation
+     */
+    inline uint32_t decodeNextByte(uint32_t * state, uint32_t * codep, uint8_t byte)
+    {
+      uint32_t type = utf8d[byte];
+
+      *codep = (*state != utf8_accept) ?
+        (byte & 0x3fu) | (*codep << 6) :
+        (0xff >> type) & (byte);
+
+      *state = utf8d[256 + *state*16 + type];
+      return *state;
+    }
+
+    /// Provides streaming UTF8 validation functionality
+    class Utf8Validator
+    {
+    public:
+        /// Construct and initialize the validator
+        Utf8Validator() : m_state(utf8_accept),m_codepoint(0) {}
+
+        /// Advance the state of the validator with the next input byte
+        /**
+         * @param byte The byte to advance the validation state with
+         * @return Whether or not the byte resulted in a validation error.
+         */
+        bool consume(uint8_t byte)
+        {
+            if (decodeNextByte(&m_state,&m_codepoint,byte) == utf8_reject)
+            {
+                return false;
+            }
+            return true;
+        }
+
+        /// Advance Validator state with input from an iterator pair
+        /**
+         * @param begin Input iterator to the start of the input range
+         * @param end Input iterator to the end of the input range
+         * @return Whether or not decoding the bytes resulted in a validation error.
+         */
+        template <typename iterator_type>
+        bool decode(iterator_type begin, iterator_type end)
+        {
+            for (iterator_type it = begin; it != end; ++it)
+            {
+                unsigned int result = decodeNextByte(
+                    &m_state,
+                    &m_codepoint,
+                    static_cast<uint8_t>(*it)
+                );
+
+                if (result == utf8_reject)
+                {
+                    return false;
+                }
+            }
+            return true;
+        }
+
+        /// Return whether the input sequence ended on a valid utf8 codepoint
+        /**
+         * @return Whether or not the input sequence ended on a valid codepoint.
+         */
+        bool complete()
+        {
+            return m_state == utf8_accept;
+        }
+
+        /// Reset the Validator to decode another message
+        void reset()
+        {
+            m_state = utf8_accept;
+            m_codepoint = 0;
+        }
+    private:
+        uint32_t    m_state;
+        uint32_t    m_codepoint;
+    };
+
+    /// Validate a UTF8 string
+    /**
+     * convenience function that creates a Validator, validates a complete string
+     * and returns the result.
+     */
+    inline bool validateUtf8(std::string const & s)
+    {
+        Utf8Validator v;
+        if (!v.decode(s.begin(),s.end()))
+        {
+            return false;
+        }
+        return v.complete();
+    }
+
+} // namespace ix

ixwebsocket/IXWebSocket.cpp

@@ -8,66 +8,12 @@
 #include "IXSetThreadName.h"
 #include "IXWebSocketHandshake.h"
 #include "IXExponentialBackoff.h"
+#include "IXUtf8Validator.h"
 
 #include <cmath>
 #include <cassert>
+#include <iostream>
 
-namespace
-{
-    //
-    // Stolen from here http://www.zedwood.com/article/cpp-is-valid-utf8-string-function
-    // There doesn't seem to be anything in the C++ library so far to do that.
-    // The closest thing is code for converting from utf-8 to utf-16 or utf-32 but 
-    // that isn't working well for some broken input strings.
-    //
-    bool isValidUtf8(const std::string& str)
-    {
-        size_t i = 0;
-        size_t ix = str.length();
-        int c, n, j;
-        
-        for (; i < ix; i++)
-        {
-            c = (unsigned char) str[i];
-            //if (c==0x09 || c==0x0a || c==0x0d || (0x20 <= c && c <= 0x7e) ) n = 0; // is_printable_ascii
-            if (0x00 <= c && c <= 0x7f)
-            {
-                n = 0; // 0bbbbbbb
-            }
-            else if ((c & 0xE0) == 0xC0)
-            {
-                n = 1; // 110bbbbb
-            }
-            else if ( c==0xed && i<(ix-1) && ((unsigned char)str[i+1] & 0xa0)==0xa0)
-            {
-                return false; //U+d800 to U+dfff
-            }
-            else if ((c & 0xF0) == 0xE0)
-            {
-                n = 2; // 1110bbbb
-            }
-            else if ((c & 0xF8) == 0xF0)
-            {
-                n = 3; // 11110bbb
-            }
-            //else if (($c & 0xFC) == 0xF8) n=4; // 111110bb //byte 5, unnecessary in 4 byte UTF-8
-            //else if (($c & 0xFE) == 0xFC) n=5; // 1111110b //byte 6, unnecessary in 4 byte UTF-8
-            else
-            {
-                return false;
-            }
-
-            for (j=0; j<n && i<ix; j++)
-            {   // n bytes matching 10bbbbbb follow ?
-                if ((++i == ix) || (( (unsigned char)str[i] & 0xC0) != 0x80))
-                {
-                    return false;
-                }
-            }
-        }
-        return true;
-    }
-}
 
 namespace ix
 {
@@ -459,10 +405,10 @@ namespace ix
     WebSocketSendInfo WebSocket::sendText(const std::string& text,
                                           const OnProgressCallback& onProgressCallback)
     {
-        if (!isValidUtf8(text))
+        if (!validateUtf8(text))
         {
-            close(WebSocketCloseConstants::kNormalClosureCode,
-                  WebSocketCloseConstants::kInvalidUtf8);
+            close(WebSocketCloseConstants::kInvalidFramePayloadData,
+                  WebSocketCloseConstants::kInvalidFramePayloadDataMessage);
             return false;
         }
         return sendMessage(text, SendMessageKind::Text, onProgressCallback);

ixwebsocket/IXWebSocketCloseConstants.cpp

@@ -11,6 +11,7 @@ namespace ix
     const uint16_t WebSocketCloseConstants::kNormalClosureCode(1000);
     const uint16_t WebSocketCloseConstants::kInternalErrorCode(1011);
     const uint16_t WebSocketCloseConstants::kAbnormalCloseCode(1006);
+    const uint16_t WebSocketCloseConstants::kInvalidFramePayloadData(1007);
     const uint16_t WebSocketCloseConstants::kProtocolErrorCode(1002);
     const uint16_t WebSocketCloseConstants::kNoStatusCodeErrorCode(1005);
 
@@ -23,5 +24,8 @@ namespace ix
     const std::string WebSocketCloseConstants::kProtocolErrorReservedBitUsed("Reserved bit used");
     const std::string WebSocketCloseConstants::kProtocolErrorPingPayloadOversized("Ping reason control frame with payload length > 125 octets");
     const std::string WebSocketCloseConstants::kProtocolErrorCodeControlMessageFragmented("Control message fragmented");
-    const std::string WebSocketCloseConstants::kInvalidUtf8("Invalid UTF-8");
+    const std::string WebSocketCloseConstants::kProtocolErrorCodeDataOpcodeOutOfSequence("Fragmentation: data message out of sequence");
+    const std::string WebSocketCloseConstants::kProtocolErrorCodeContinuationOpCodeOutOfSequence("Fragmentation: continuation opcode out of sequence");
+    const std::string WebSocketCloseConstants::kInvalidFramePayloadDataMessage("Invalid frame payload data");
+    const std::string WebSocketCloseConstants::kInvalidCloseCodeMessage("Invalid close code");
 }

ixwebsocket/IXWebSocketCloseConstants.h

@@ -18,6 +18,7 @@ namespace ix
         static const uint16_t kAbnormalCloseCode;
         static const uint16_t kProtocolErrorCode;
         static const uint16_t kNoStatusCodeErrorCode;
+        static const uint16_t kInvalidFramePayloadData;
 
         static const std::string kNormalClosureMessage;
         static const std::string kInternalErrorMessage;
@@ -28,6 +29,9 @@ namespace ix
         static const std::string kProtocolErrorReservedBitUsed;
         static const std::string kProtocolErrorPingPayloadOversized;
         static const std::string kProtocolErrorCodeControlMessageFragmented;
-        static const std::string kInvalidUtf8;
+        static const std::string kProtocolErrorCodeDataOpcodeOutOfSequence;
+        static const std::string kProtocolErrorCodeContinuationOpCodeOutOfSequence;
+        static const std::string kInvalidFramePayloadDataMessage;
+        static const std::string kInvalidCloseCodeMessage;
     };
 } // namespace ix

ixwebsocket/IXWebSocketHandshake.cpp

@@ -295,7 +295,7 @@ namespace ix
             return sendErrorResponse(400, "Missing Sec-WebSocket-Key value");
         }
 
-        if (headers["upgrade"] != "websocket")
+        if (!insensitiveStringCompare(headers["upgrade"], "WebSocket"))
         {
             return sendErrorResponse(400, "Invalid or missing Upgrade header");
         }
@@ -326,6 +326,7 @@ namespace ix
         ss << "Sec-WebSocket-Accept: " << std::string(output) << "\r\n";
         ss << "Upgrade: websocket\r\n";
         ss << "Connection: Upgrade\r\n";
+        ss << "Server: " << userAgent() << "\r\n";
 
         // Parse the client headers. Does it support deflate ?
         std::string header = headers["sec-websocket-extensions"];

ixwebsocket/IXWebSocketPerMessageDeflateOptions.cpp

@@ -33,6 +33,8 @@ namespace ix
         _serverNoContextTakeover = serverNoContextTakeover;
         _clientMaxWindowBits = clientMaxWindowBits;
         _serverMaxWindowBits = serverMaxWindowBits;
+
+        sanitizeClientMaxWindowBits();
     }
 
     //
@@ -107,10 +109,22 @@ namespace ix
                 _clientMaxWindowBits =
                     std::min(maxClientMaxWindowBits,
                         std::max(x, minClientMaxWindowBits));
+
+                sanitizeClientMaxWindowBits();
             }
         }
     }
 
+    void WebSocketPerMessageDeflateOptions::sanitizeClientMaxWindowBits()
+    {
+        // zlib/deflate has a bug with windowsbits == 8, so we silently upgrade it to 9
+        // See https://bugs.chromium.org/p/chromium/issues/detail?id=691074
+        if (_clientMaxWindowBits == 8)
+        {
+            _clientMaxWindowBits = 9;
+        }
+    }
+
     std::string WebSocketPerMessageDeflateOptions::generateHeader()
     {
         std::stringstream ss;

ixwebsocket/IXWebSocketPerMessageDeflateOptions.h

@@ -41,5 +41,7 @@ namespace ix
         bool _serverNoContextTakeover;
         int _clientMaxWindowBits;
         int _serverMaxWindowBits;
+
+        void sanitizeClientMaxWindowBits();
     };
 } // namespace ix

ixwebsocket/IXWebSocketTransport.cpp

@@ -37,6 +37,7 @@
 #include "IXWebSocketHttpHeaders.h"
 #include "IXUrlParser.h"
 #include "IXSocketFactory.h"
+#include "IXUtf8Validator.h"
 
 #include <string.h>
 #include <stdlib.h>
@@ -76,6 +77,7 @@ namespace ix
 
     WebSocketTransport::WebSocketTransport() :
         _useMask(true),
+        _compressedMessage(false),
         _readyState(ReadyState::CLOSED),
         _closeCode(WebSocketCloseConstants::kInternalErrorCode),
         _closeReason(WebSocketCloseConstants::kInternalErrorMessage),
@@ -551,19 +553,22 @@ namespace ix
                 || ws.opcode == wsheader_type::PONG
                 || ws.opcode == wsheader_type::CLOSE
             )){
-                // Cntrol messages should not be fragmented
+                // Control messages should not be fragmented
                 close(WebSocketCloseConstants::kProtocolErrorCode,
                       WebSocketCloseConstants::kProtocolErrorCodeControlMessageFragmented);
                 return;
             }
 
+            unmaskReceiveBuffer(ws);
+            std::string frameData(_rxbuf.begin()+ws.header_size,
+                                  _rxbuf.begin()+ws.header_size+(size_t) ws.N);
+
             // We got a whole message, now do something with it:
             if (
                    ws.opcode == wsheader_type::TEXT_FRAME
                 || ws.opcode == wsheader_type::BINARY_FRAME
                 || ws.opcode == wsheader_type::CONTINUATION
             ) {
-                unmaskReceiveBuffer(ws);
 
                 if (ws.opcode != wsheader_type::CONTINUATION)
                 {
@@ -571,6 +576,21 @@ namespace ix
                         (ws.opcode == wsheader_type::TEXT_FRAME)
                         ? MessageKind::MSG_TEXT
                         : MessageKind::MSG_BINARY;
+
+                    _compressedMessage = _enablePerMessageDeflate && ws.rsv1;
+
+                    // Continuation message needs to follow a non-fin TEXT or BINARY message
+                    if (!_chunks.empty())
+                    {
+                        close(WebSocketCloseConstants::kProtocolErrorCode,
+                              WebSocketCloseConstants::kProtocolErrorCodeDataOpcodeOutOfSequence);
+                    }
+                }
+                else if (_chunks.empty())
+                {
+                    // Continuation message need to follow a non-fin TEXT or BINARY message
+                    close(WebSocketCloseConstants::kProtocolErrorCode,
+                          WebSocketCloseConstants::kProtocolErrorCodeContinuationOpCodeOutOfSequence);
                 }
 
                 //
@@ -579,10 +599,11 @@ namespace ix
                 if (ws.fin && _chunks.empty())
                 {
                     emitMessage(_fragmentedMessageKind,
-                                std::string(_rxbuf.begin()+ws.header_size,
-                                            _rxbuf.begin()+ws.header_size+(size_t) ws.N),
-                                ws,
+                                frameData,
+                                _compressedMessage,
                                 onMessageCallback);
+
+                    _compressedMessage = false;
                 }
                 else
                 {
@@ -593,30 +614,26 @@ namespace ix
                     // the internal buffer which is slow and can let the internal OS
                     // receive buffer fill out.
                     //
-                    _chunks.emplace_back(
-                        std::vector<uint8_t>(_rxbuf.begin()+ws.header_size,
-                                             _rxbuf.begin()+ws.header_size+(size_t)ws.N));
+                    _chunks.emplace_back(frameData);
+
                     if (ws.fin)
                     {
                         emitMessage(_fragmentedMessageKind, getMergedChunks(),
-                                    ws, onMessageCallback);
+                                    _compressedMessage, onMessageCallback);
+
                         _chunks.clear();
+                        _compressedMessage = false;
                     }
                     else
                     {
-                        emitMessage(MessageKind::FRAGMENT, std::string(), ws, onMessageCallback);
+                        emitMessage(MessageKind::FRAGMENT, std::string(), false, onMessageCallback);
                     }
                 }
             }
             else if (ws.opcode == wsheader_type::PING)
             {
-                unmaskReceiveBuffer(ws);
-
-                std::string pingData(_rxbuf.begin()+ws.header_size,
-                                     _rxbuf.begin()+ws.header_size + (size_t) ws.N);
-
                 // too large
-                if (pingData.size() > 125)
+                if (frameData.size() > 125)
                 {
                     // Unexpected frame type
                     close(WebSocketCloseConstants::kProtocolErrorCode,
@@ -628,29 +645,23 @@ namespace ix
                 {
                     // Reply back right away
                     bool compress = false;
-                    sendData(wsheader_type::PONG, pingData, compress);
+                    sendData(wsheader_type::PONG, frameData, compress);
                 }
 
-                emitMessage(MessageKind::PING, pingData, ws, onMessageCallback);
+                emitMessage(MessageKind::PING, frameData, false, onMessageCallback);
             }
             else if (ws.opcode == wsheader_type::PONG)
             {
-                unmaskReceiveBuffer(ws);
-                std::string pongData(_rxbuf.begin()+ws.header_size,
-                                     _rxbuf.begin()+ws.header_size + (size_t) ws.N);
-
                 std::lock_guard<std::mutex> lck(_lastReceivePongTimePointMutex);
                 _lastReceivePongTimePoint = std::chrono::steady_clock::now();
 
-                emitMessage(MessageKind::PONG, pongData, ws, onMessageCallback);
+                emitMessage(MessageKind::PONG, frameData, false, onMessageCallback);
             }
             else if (ws.opcode == wsheader_type::CLOSE)
             {
                 std::string reason;
                 uint16_t code = 0;
 
-                unmaskReceiveBuffer(ws);
-
                 if (ws.N >= 2)
                 {
                     // Extract the close code first, available as the first 2 bytes
@@ -660,8 +671,28 @@ namespace ix
                     // Get the reason.
                     if (ws.N > 2)
                     {
-                        reason.assign(_rxbuf.begin()+ws.header_size + 2,
-                                      _rxbuf.begin()+ws.header_size + (size_t) ws.N);
+                        reason = frameData.substr(2, frameData.size());
+                    }
+
+                    // Validate that the reason is proper utf-8. Autobahn 7.5.1
+                    if (!validateUtf8(reason))
+                    {
+                        code = WebSocketCloseConstants::kInvalidFramePayloadData;
+                        reason = WebSocketCloseConstants::kInvalidFramePayloadDataMessage;
+                    }
+
+                    // Validate close codes. Autobahn 7.9.*
+                    // 1014, 1015 are debattable. The firefox MSDN has a description for them
+                    if (code < 1000 || code == 1004 || code == 1006 ||
+                        (code > 1013 && code < 3000))
+                    {
+                        // build up an error message containing the bad error code
+                        std::stringstream ss;
+                        ss << WebSocketCloseConstants::kInvalidCloseCodeMessage
+                           << ": " << code;
+                        reason = ss.str();
+
+                        code = WebSocketCloseConstants::kProtocolErrorCode;
                     }
                 }
                 else
@@ -747,8 +778,7 @@ namespace ix
 
         for (auto&& chunk : _chunks)
         {
-            std::string str(chunk.begin(), chunk.end());
-            msg += str;
+            msg += chunk;
         }
 
         return msg;
@@ -756,21 +786,38 @@ namespace ix
 
     void WebSocketTransport::emitMessage(MessageKind messageKind,
                                          const std::string& message,
-                                         const wsheader_type& ws,
+                                         bool compressedMessage,
                                          const OnMessageCallback& onMessageCallback)
     {
         size_t wireSize = message.size();
 
         // When the RSV1 bit is 1 it means the message is compressed
-        if (_enablePerMessageDeflate && ws.rsv1 && messageKind != MessageKind::FRAGMENT)
+        if (compressedMessage && messageKind != MessageKind::FRAGMENT)
         {
             std::string decompressedMessage;
             bool success = _perMessageDeflate.decompress(message, decompressedMessage);
-            onMessageCallback(decompressedMessage, wireSize, !success, messageKind);
+
+            if (messageKind == MessageKind::MSG_TEXT && !validateUtf8(decompressedMessage))
+            {
+                close(WebSocketCloseConstants::kInvalidFramePayloadData,
+                      WebSocketCloseConstants::kInvalidFramePayloadDataMessage);
+            }
+            else
+            {
+                onMessageCallback(decompressedMessage, wireSize, !success, messageKind);
+            }
         }
         else
         {
-            onMessageCallback(message, wireSize, false, messageKind);
+            if (messageKind == MessageKind::MSG_TEXT && !validateUtf8(message))
+            {
+                close(WebSocketCloseConstants::kInvalidFramePayloadData,
+                      WebSocketCloseConstants::kInvalidFramePayloadDataMessage);
+            }
+            else
+            {
+                onMessageCallback(message, wireSize, false, messageKind);
+            }
         }
     }
 
@@ -819,6 +866,8 @@ namespace ix
             message_end = compressedMessage.end();
         }
 
+        _txbuf.reserve(wireSize);
+
         // Common case for most message. No fragmentation required.
         if (wireSize < kChunkSize)
         {
@@ -906,8 +955,9 @@ namespace ix
             header[0] |= 0x80;
         }
 
-        // This bit indicate that the frame is compressed
-        if (compress)
+        // The rsv1 bit indicate that the frame is compressed
+        // continuation opcodes should not set it. Autobahn 12.2.10 and others 12.X
+        if (compress && type != wsheader_type::CONTINUATION)
         {
             header[0] |= 0x40;
         }

ixwebsocket/IXWebSocketTransport.h

@@ -149,13 +149,16 @@ namespace ix
         // messages (tested messages up to 700M) and we cannot put them in a single
         // buffer that is resized, as this operation can be slow when a buffer has its
         // size increased 2 fold, while appending to a list has a fixed cost.
-        std::list<std::vector<uint8_t>> _chunks;
+        std::list<std::string> _chunks;
 
         // Record the message kind (will be TEXT or BINARY) for a fragmented
         // message, present in the first chunk, since the final chunk will be a
         // CONTINUATION opcode and doesn't tell the full message kind
         MessageKind _fragmentedMessageKind;
 
+        // Ditto for whether a message is compressed
+        bool _compressedMessage;
+
         // Fragments are 32K long
         static constexpr size_t kChunkSize = 1 << 15;
 
@@ -244,7 +247,7 @@ namespace ix
 
         void emitMessage(MessageKind messageKind,
                          const std::string& message,
-                         const wsheader_type& ws,
+                         bool compressedMessage,
                          const OnMessageCallback& onMessageCallback);
 
         bool isSendBufferEmpty() const;

ixwebsocket/IXWebSocketVersion.h

@@ -6,4 +6,4 @@
 
 #pragma once
 
-#define IX_WEBSOCKET_VERSION "5.1.4"
+#define IX_WEBSOCKET_VERSION "6.0.0"

makefile

@@ -63,6 +63,9 @@ test:
 ws_test: ws
 	(cd ws ; env DEBUG=1 PATH=../ws/build:$$PATH bash test_ws.sh)
 
+autobahn_report:
+	cp -rvf ~/sandbox/reports/clients/* ../bsergean.github.io/IXWebSocket/autobahn/
+
 # For the fork that is configured with appveyor
 rebase_upstream:
 	git fetch upstream

test/cmd_websocket_chat.cpp

@@ -206,7 +206,7 @@ namespace
 
     void WebSocketChat::sendMessage(const std::string& text)
     {
-        _webSocket.send(encodeMessage(text));
+        _webSocket.sendBinary(encodeMessage(text));
     }
 
     bool startServer(ix::WebSocketServer& server)
@@ -239,7 +239,7 @@ namespace
                             {
                                 if (client != webSocket)
                                 {
-                                    client->send(msg->str);
+                                    client->sendBinary(msg->str);
                                 }
                             }
                         }

ws/ws_autobahn.cpp

@@ -42,6 +42,23 @@
 #include <ixwebsocket/IXWebSocket.h>
 #include <ixwebsocket/IXSocket.h>
 
+#include <spdlog/spdlog.h>
+
+
+namespace
+{
+    std::string truncate(const std::string& str, size_t n)
+    {
+        if (str.size() < n)
+        {
+            return str;
+        }
+        else
+        {
+            return str.substr(0, n) + "...";
+        }
+    }
+}
 
 namespace ix
 {
@@ -117,7 +134,7 @@ namespace ix
                     ss << "Received " << msg->wireSize << " bytes" << std::endl;
 
                     ss << "autobahn: received message: "
-                       << msg->str
+                       << truncate(msg->str, 40)
                        << std::endl;
 
                     _webSocket.send(msg->str, msg->binary);
@@ -161,7 +178,7 @@ namespace ix
         _webSocket.stop();
     }
 
-    void generateReport(const std::string& url)
+    bool generateReport(const std::string& url)
     {
         ix::WebSocket webSocket;
         std::string reportUrl(url);
@@ -169,14 +186,16 @@ namespace ix
         webSocket.setUrl(reportUrl);
         webSocket.disableAutomaticReconnection();
 
-        std::atomic<bool> done(false);
+        std::atomic<bool> success(true);
+        std::condition_variable condition;
+
         webSocket.setOnMessageCallback(
-            [&done](const ix::WebSocketMessagePtr& msg)
+            [&condition, &success](const ix::WebSocketMessagePtr& msg)
             {
                 if (msg->type == ix::WebSocketMessageType::Close)
                 {
                     std::cerr << "Report generated" << std::endl;
-                    done = true;
+                    condition.notify_one();
                 }
                 else if (msg->type == ix::WebSocketMessageType::Error)
                 {
@@ -186,18 +205,24 @@ namespace ix
                     ss << "Wait time(ms): "    << msg->errorInfo.wait_time   << std::endl;
                     ss << "HTTP Status: "      << msg->errorInfo.http_status << std::endl;
                     std::cerr << ss.str() << std::endl;
+
+                    success = false;
                 }
             }
         );
-        webSocket.start();
 
-        while (!done)
+        webSocket.start();
+        std::mutex mutex;
+        std::unique_lock<std::mutex> lock(mutex);
+        condition.wait(lock);
+        webSocket.stop();
+
+        if (!success)
         {
-            std::chrono::duration<double, std::milli> duration(10);
-            std::this_thread::sleep_for(duration);
+            spdlog::error("Cannot generate report at url {}", reportUrl);
         }
 
-        webSocket.stop();
+        return success;
     }
 
     int getTestCaseCount(const std::string& url)
@@ -208,15 +233,15 @@ namespace ix
         webSocket.setUrl(caseCountUrl);
         webSocket.disableAutomaticReconnection();
 
-        int count = 0;
+        int count = -1;
+        std::condition_variable condition;
 
-        std::atomic<bool> done(false);
         webSocket.setOnMessageCallback(
-            [&done, &count](const ix::WebSocketMessagePtr& msg)
+            [&condition, &count](const ix::WebSocketMessagePtr& msg)
             {
                 if (msg->type == ix::WebSocketMessageType::Close)
                 {
-                    done = true;
+                    condition.notify_one();
                 }
                 else if (msg->type == ix::WebSocketMessageType::Error)
                 {
@@ -226,6 +251,8 @@ namespace ix
                     ss << "Wait time(ms): "    << msg->errorInfo.wait_time   << std::endl;
                     ss << "HTTP Status: "      << msg->errorInfo.http_status << std::endl;
                     std::cerr << ss.str() << std::endl;
+
+                    condition.notify_one();
                 }
                 else if (msg->type == ix::WebSocketMessageType::Message)
                 {
@@ -236,16 +263,18 @@ namespace ix
                 }
             }
         );
+
         webSocket.start();
-
-        while (!done)
-        {
-            std::chrono::duration<double, std::milli> duration(10);
-            std::this_thread::sleep_for(duration);
-        }
-
+        std::mutex mutex;
+        std::unique_lock<std::mutex> lock(mutex);
+        condition.wait(lock);
         webSocket.stop();
 
+        if (count == -1)
+        {
+            spdlog::error("Cannot retrieve test case count at url {}", caseCountUrl);
+        }
+
         return count;
     }
 
@@ -254,14 +283,20 @@ namespace ix
     //
     int ws_autobahn_main(const std::string& url, bool quiet)
     {
-        int N = getTestCaseCount(url);
-        std::cerr << "Test cases count: " << N << std::endl;
+        int testCasesCount = getTestCaseCount(url);
+        std::cerr << "Test cases count: " << testCasesCount << std::endl;
 
-        N++;
-
-        for (int i = 1 ; i < N; ++i)
+        if (testCasesCount == -1)
         {
-            std::cerr << "Execute test case " << i << std::endl;
+            spdlog::error("Cannot retrieve test case count at url {}", url);
+            return 1;
+        }
+
+        testCasesCount++;
+
+        for (int i = 1 ; i < testCasesCount; ++i)
+        {
+            spdlog::info("Execute test case {}", i);
 
             int caseNumber = i;
 
@@ -277,9 +312,7 @@ namespace ix
             testCase.run();
         }
 
-        generateReport(url);
-
-        return 0;
+        return generateReport(url) ? 0 : 1;
     }
 }