hush
/
hush3
9
0
Fork 11
Code Issues 143 Pull Requests 2 Projects Releases 16 Wiki Activity
Hush Full Node software. We were censored from Github, this is where all development happens now. https://hush.is
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
4650 Commits
30 Branches
82 Tags
217 MiB
Tree: cd1fc2434c
Branches Tags
${ item.name }
Create tag ${ searchTerm }
Create branch ${ searchTerm }
from 'cd1fc2434c'
${ noResults }
hush3/src/test/serialize_tests.cpp

106 lines
3.0 KiB
Raw Normal View History

Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
12 years ago
#include <boost/test/unit_test.hpp>
#include <string>
#include <vector>
#include "serialize.h"
using namespace std;
BOOST_AUTO_TEST_SUITE(serialize_tests)
BOOST_AUTO_TEST_CASE(varints)
{
// encode
CDataStream ss(SER_DISK, 0);
CDataStream::size_type size = 0;
for (int i = 0; i < 100000; i++) {
ss << VARINT(i);
size += ::GetSerializeSize(VARINT(i), 0, 0);
BOOST_CHECK(size == ss.size());
}
for (uint64 i = 0; i < 100000000000ULL; i += 999999937) {
ss << VARINT(i);
size += ::GetSerializeSize(VARINT(i), 0, 0);
BOOST_CHECK(size == ss.size());
}
// decode
for (int i = 0; i < 100000; i++) {
Fix signed/unsigned comparison warnings
11 years ago
int j = -1;
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
12 years ago
ss >> VARINT(j);
BOOST_CHECK_MESSAGE(i == j, "decoded:" << j << " expected:" << i);
}
for (uint64 i = 0; i < 100000000000ULL; i += 999999937) {
Fix signed/unsigned comparison warnings
11 years ago
uint64 j = -1;
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
12 years ago
ss >> VARINT(j);
BOOST_CHECK_MESSAGE(i == j, "decoded:" << j << " expected:" << i);
}
Reject non-canonically-encoded sizes The length of vectors, maps, sets, etc are serialized using Write/ReadCompactSize -- which, unfortunately, do not use a unique encoding. So deserializing and then re-serializing a transaction (for example) can give you different bits than you started with. That doesn't cause any problems that we are aware of, but it is exactly the type of subtle mismatch that can lead to exploits. With this pull, reading a non-canonical CompactSize throws an exception, which means nodes will ignore 'tx' or 'block' or other messages that are not properly encoded. Please check my logic... but this change is safe with respect to causing a network split. Old clients that receive non-canonically-encoded transactions or blocks deserialize them into CTransaction/CBlock structures in memory, and then re-serialize them before relaying them to peers. And please check my logic with respect to causing a blockchain split: there are no CompactSize fields in the block header, so the block hash is always canonical. The merkle root in the block header is computed on a vector<CTransaction>, so any non-canonical encoding of the transactions in 'tx' or 'block' messages is erased as they are read into memory by old clients, and does not affect the block hash. And, as noted above, old clients re-serialize (with canonical encoding) 'tx' and 'block' messages before relaying to peers.
11 years ago
}
BOOST_AUTO_TEST_CASE(compactsize)
{
CDataStream ss(SER_DISK, 0);
vector<char>::size_type i, j;
for (i = 1; i <= MAX_SIZE; i *= 2)
{
WriteCompactSize(ss, i-1);
WriteCompactSize(ss, i);
}
for (i = 1; i <= MAX_SIZE; i *= 2)
{
j = ReadCompactSize(ss);
BOOST_CHECK_MESSAGE((i-1) == j, "decoded:" << j << " expected:" << (i-1));
j = ReadCompactSize(ss);
BOOST_CHECK_MESSAGE(i == j, "decoded:" << j << " expected:" << i);
}
}
static bool isCanonicalException(const std::ios_base::failure& ex)
{
return std::string("non-canonical ReadCompactSize()") == ex.what();
}
BOOST_AUTO_TEST_CASE(noncanonical)
{
// Write some non-canonical CompactSize encodings, and
// make sure an exception is thrown when read back.
CDataStream ss(SER_DISK, 0);
vector<char>::size_type n;
// zero encoded with three bytes:
ss.write("\xfd\x00\x00", 3);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
// 0xfc encoded with three bytes:
ss.write("\xfd\xfc\x00", 3);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
// 0xfd encoded with three bytes is OK:
ss.write("\xfd\xfd\x00", 3);
n = ReadCompactSize(ss);
BOOST_CHECK(n == 0xfd);
// zero encoded with five bytes:
ss.write("\xfe\x00\x00\x00\x00", 5);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
// 0xffff encoded with five bytes:
ss.write("\xfe\xff\xff\x00\x00", 5);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
// zero encoded with nine bytes:
ss.write("\xff\x00\x00\x00\x00\x00\x00\x00\x00", 9);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
12 years ago
Reject non-canonically-encoded sizes The length of vectors, maps, sets, etc are serialized using Write/ReadCompactSize -- which, unfortunately, do not use a unique encoding. So deserializing and then re-serializing a transaction (for example) can give you different bits than you started with. That doesn't cause any problems that we are aware of, but it is exactly the type of subtle mismatch that can lead to exploits. With this pull, reading a non-canonical CompactSize throws an exception, which means nodes will ignore 'tx' or 'block' or other messages that are not properly encoded. Please check my logic... but this change is safe with respect to causing a network split. Old clients that receive non-canonically-encoded transactions or blocks deserialize them into CTransaction/CBlock structures in memory, and then re-serialize them before relaying them to peers. And please check my logic with respect to causing a blockchain split: there are no CompactSize fields in the block header, so the block hash is always canonical. The merkle root in the block header is computed on a vector<CTransaction>, so any non-canonical encoding of the transactions in 'tx' or 'block' messages is erased as they are read into memory by old clients, and does not affect the block hash. And, as noted above, old clients re-serialize (with canonical encoding) 'tx' and 'block' messages before relaying to peers.
11 years ago
// 0x01ffffff encoded with nine bytes:
ss.write("\xff\xff\xff\xff\x01\x00\x00\x00\x00", 9);
BOOST_CHECK_EXCEPTION(ReadCompactSize(ss), std::ios_base::failure, isCanonicalException);
Compact serialization for variable-length integers Variable-length integers: bytes are a MSB base-128 encoding of the number. The high bit in each byte signifies whether another digit follows. To make the encoding is one-to-one, one is subtracted from all but the last digit. Thus, the byte sequence a[] with length len, where all but the last byte has bit 128 set, encodes the number: (a[len-1] & 0x7F) + sum(i=1..len-1, 128^i*((a[len-i-1] & 0x7F)+1)) Properties: * Very small (0-127: 1 byte, 128-16511: 2 bytes, 16512-2113663: 3 bytes) * Every integer has exactly one encoding * Encoding does not depend on size of original integer type
12 years ago
}
BOOST_AUTO_TEST_SUITE_END()