Nathan Wilcox
8 years ago
50 changed files with 32 additions and 850 deletions
@ -1,83 +0,0 @@ |
|||
Deterministic OS X Dmg Notes. |
|||
|
|||
Working OS X DMGs are created in Linux by combining a recent clang, |
|||
the Apple's binutils (ld, ar, etc), and DMG authoring tools. |
|||
|
|||
Apple uses clang extensively for development and has upstreamed the necessary |
|||
functionality so that a vanilla clang can take advantage. It supports the use |
|||
of -F, -target, -mmacosx-version-min, and --sysroot, which are all necessary |
|||
when building for OS X. A pre-compiled version of 3.2 is used because it was not |
|||
available in the Precise repositories at the time this work was started. In the |
|||
future, it can be switched to use system packages instead. |
|||
|
|||
Apple's version of binutils (called cctools) contains lots of functionality |
|||
missing in the FSF's binutils. In addition to extra linker options for |
|||
frameworks and sysroots, several other tools are needed as well such as |
|||
install_name_tool, lipo, and nmedit. These do not build under linux, so they |
|||
have been patched to do so. The work here was used as a starting point: |
|||
https://github.com/mingwandroid/toolchain4 |
|||
|
|||
In order to build a working toolchain, the following source packages are needed |
|||
from Apple: cctools, dyld, and ld64. |
|||
|
|||
These tools inject timestamps by default, which produce non-deterministic |
|||
binaries. The ZERO_AR_DATE environment variable is used to disable that. |
|||
|
|||
This version of cctools has been patched to use the current version of clang's |
|||
headers and and its libLTO.so rather than those from llvmgcc, as it was |
|||
originally done in toolchain4. |
|||
|
|||
To complicate things further, all builds must target an Apple SDK. These SDKs |
|||
are free to download, but not redistributable. |
|||
To obtain it, register for a developer account, then download the Xcode 6.1.1 dmg: |
|||
https://developer.apple.com/devcenter/download.action?path=/Developer_Tools/xcode_6.1.1/xcode_6.1.1.dmg |
|||
|
|||
This file is several gigabytes in size, but only a single directory inside is |
|||
needed: Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.9.sdk |
|||
|
|||
Unfortunately, the usual linux tools (7zip, hpmount, loopback mount) are incapable of opening this file. |
|||
To create a tarball suitable for Gitian input, mount the dmg in OS X, then create it with: |
|||
$ tar -C /Volumes/Xcode/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/ -czf MacOSX10.9.sdk.tar.gz MacOSX10.9.sdk |
|||
|
|||
|
|||
The Gitian descriptors build 2 sets of files: Linux tools, then Apple binaries |
|||
which are created using these tools. The build process has been designed to |
|||
avoid including the SDK's files in Gitian's outputs. All interim tarballs are |
|||
fully deterministic and may be freely redistributed. |
|||
|
|||
genisoimage is used to create the initial DMG. It is not deterministic as-is, |
|||
so it has been patched. A system genisoimage will work fine, but it will not |
|||
be deterministic because the file-order will change between invocations. |
|||
The patch can be seen here: |
|||
https://raw.githubusercontent.com/theuni/osx-cross-depends/master/patches/cdrtools/genisoimage.diff |
|||
No effort was made to fix this cleanly, so it likely leaks memory badly. But |
|||
it's only used for a single invocation, so that's no real concern. |
|||
|
|||
genisoimage cannot compress DMGs, so afterwards, the 'dmg' tool from the |
|||
libdmg-hfsplus project is used to compress it. There are several bugs in this |
|||
tool and its maintainer has seemingly abandoned the project. It has been forked |
|||
and is available (with fixes) here: https://github.com/theuni/libdmg-hfsplus . |
|||
|
|||
The 'dmg' tool has the ability to create DMGs from scratch as well, but this |
|||
functionality is broken. Only the compression feature is currently used. |
|||
Ideally, the creation could be fixed and genisoimage would no longer be necessary. |
|||
|
|||
Background images and other features can be added to DMG files by inserting a |
|||
.DS_Store before creation. The easiest way to create this file is to build a |
|||
DMG without one, move it to a device running OS X, customize the layout, then |
|||
grab the .DS_Store file for later use. That is the approach taken here. |
|||
|
|||
As of OS X Mavericks (10.9), using an Apple-blessed key to sign binaries is a |
|||
requirement in order to satisfy the new Gatekeeper requirements. Because this |
|||
private key cannot be shared, we'll have to be a bit creative in order for the |
|||
build process to remain somewhat deterministic. Here's how it works: |
|||
|
|||
- Builders use Gitian to create an unsigned release. This outputs an unsigned |
|||
dmg which users may choose to bless and run. It also outputs an unsigned app |
|||
structure in the form of a tarball, which also contains all of the tools |
|||
that have been previously (deterministically) built in order to create a |
|||
final dmg. |
|||
- The Apple keyholder uses this unsigned app to create a detached signature, |
|||
using the script that is also included there. |
|||
- Builders feed the unsigned app + detached signature back into Gitian. It |
|||
uses the pre-built tools to recombine the pieces into a deterministic dmg. |
|||
@ -1,23 +0,0 @@ |
|||
Bitcoin Core 0.11.2 |
|||
===================== |
|||
|
|||
Intro |
|||
----- |
|||
Bitcoin is a free open source peer-to-peer electronic cash system that is |
|||
completely decentralized, without the need for a central server or trusted |
|||
parties. Users hold the crypto keys to their own money and transact directly |
|||
with each other, with the help of a P2P network to check for double-spending. |
|||
|
|||
|
|||
Setup |
|||
----- |
|||
Unpack the files into a directory and run bitcoin-qt.exe. |
|||
|
|||
Bitcoin Core is the original Bitcoin client and it builds the backbone of the network. |
|||
However, it downloads and stores the entire history of Bitcoin transactions; |
|||
depending on the speed of your computer and network connection, the synchronization |
|||
process can take anywhere from a few hours to a day or more. |
|||
|
|||
See the bitcoin wiki at: |
|||
https://en.bitcoin.it/wiki/Main_Page |
|||
for more help and information. |
|||
@ -1,117 +0,0 @@ |
|||
Mac OS X Build Instructions and Notes |
|||
==================================== |
|||
This guide will show you how to build bitcoind (headless client) for OS X. |
|||
|
|||
Notes |
|||
----- |
|||
|
|||
* Tested on OS X 10.7 through 10.10 on 64-bit Intel processors only. |
|||
|
|||
* All of the commands should be executed in a Terminal application. The |
|||
built-in one is located in `/Applications/Utilities`. |
|||
|
|||
Preparation |
|||
----------- |
|||
|
|||
You need to install Xcode with all the options checked so that the compiler |
|||
and everything is available in /usr not just /Developer. Xcode should be |
|||
available on your OS X installation media, but if not, you can get the |
|||
current version from https://developer.apple.com/xcode/. If you install |
|||
Xcode 4.3 or later, you'll need to install its command line tools. This can |
|||
be done in `Xcode > Preferences > Downloads > Components` and generally must |
|||
be re-done or updated every time Xcode is updated. |
|||
|
|||
You will also need to install [Homebrew](http://brew.sh) in order to install library |
|||
dependencies. |
|||
|
|||
The installation of the actual dependencies is covered in the Instructions |
|||
sections below. |
|||
|
|||
Instructions: Homebrew |
|||
---------------------- |
|||
|
|||
#### Install dependencies using Homebrew |
|||
|
|||
brew install autoconf automake berkeley-db4 libtool boost miniupnpc openssl pkg-config protobuf qt5 |
|||
|
|||
NOTE: Building with Qt4 is still supported, however, could result in a broken UI. As such, building with Qt5 is recommended. |
|||
|
|||
### Building `bitcoind` |
|||
|
|||
1. Clone the GitHub tree to get the source code and go into the directory. |
|||
|
|||
git clone https://github.com/bitcoin/bitcoin.git |
|||
cd bitcoin |
|||
|
|||
2. Build bitcoind: |
|||
|
|||
./autogen.sh |
|||
./configure --with-gui=qt5 |
|||
make |
|||
|
|||
3. It is also a good idea to build and run the unit tests: |
|||
|
|||
make check |
|||
|
|||
4. (Optional) You can also install bitcoind to your path: |
|||
|
|||
make install |
|||
|
|||
Use Qt Creator as IDE |
|||
------------------------ |
|||
You can use Qt Creator as IDE, for debugging and for manipulating forms, etc. |
|||
Download Qt Creator from http://www.qt.io/download/. Download the "community edition" and only install Qt Creator (uncheck the rest during the installation process). |
|||
|
|||
1. Make sure you installed everything through Homebrew mentioned above |
|||
2. Do a proper ./configure --with-gui=qt5 --enable-debug |
|||
3. In Qt Creator do "New Project" -> Import Project -> Import Existing Project |
|||
4. Enter "bitcoin-qt" as project name, enter src/qt as location |
|||
5. Leave the file selection as it is |
|||
6. Confirm the "summary page" |
|||
7. In the "Projects" tab select "Manage Kits..." |
|||
8. Select the default "Desktop" kit and select "Clang (x86 64bit in /usr/bin)" as compiler |
|||
9. Select LLDB as debugger (you might need to set the path to your installtion) |
|||
10. Start debugging with Qt Creator |
|||
|
|||
Creating a release build |
|||
------------------------ |
|||
You can ignore this section if you are building `bitcoind` for your own use. |
|||
|
|||
bitcoind/bitcoin-cli binaries are not included in the Bitcoin-Qt.app bundle. |
|||
|
|||
If you are building `bitcoind` or `Bitcoin-Qt` for others, your build machine should be set up |
|||
as follows for maximum compatibility: |
|||
|
|||
All dependencies should be compiled with these flags: |
|||
|
|||
-mmacosx-version-min=10.7 |
|||
-arch x86_64 |
|||
-isysroot $(xcode-select --print-path)/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.7.sdk |
|||
|
|||
Once dependencies are compiled, see [doc/release-process.md](release-process.md) for how the Bitcoin-Qt.app |
|||
bundle is packaged and signed to create the .dmg disk image that is distributed. |
|||
|
|||
Running |
|||
------- |
|||
|
|||
It's now available at `./bitcoind`, provided that you are still in the `src` |
|||
directory. We have to first create the RPC configuration file, though. |
|||
|
|||
Run `./bitcoind` to get the filename where it should be put, or just try these |
|||
commands: |
|||
|
|||
echo -e "rpcuser=bitcoinrpc\nrpcpassword=$(xxd -l 16 -p /dev/urandom)" > "/Users/${USER}/Library/Application Support/Bitcoin/bitcoin.conf" |
|||
chmod 600 "/Users/${USER}/Library/Application Support/Bitcoin/bitcoin.conf" |
|||
|
|||
The next time you run it, it will start downloading the blockchain, but it won't |
|||
output anything while it's doing this. This process may take several hours; |
|||
you can monitor its process by looking at the debug.log file, like this: |
|||
|
|||
tail -f $HOME/Library/Application\ Support/Bitcoin/debug.log |
|||
|
|||
Other commands: |
|||
------- |
|||
|
|||
./bitcoind -daemon # to start the bitcoin daemon. |
|||
./bitcoin-cli --help # for a list of command-line options. |
|||
./bitcoin-cli help # When the daemon is running, to get a list of RPC commands |
|||
@ -1,396 +0,0 @@ |
|||
Gitian building |
|||
================ |
|||
|
|||
*Setup instructions for a Gitian build of Bitcoin using a Debian VM or physical system.* |
|||
|
|||
Gitian is the deterministic build process that is used to build the Bitcoin |
|||
Core executables. It provides a way to be reasonably sure that the |
|||
executables are really built from source on GitHub. It also makes sure that |
|||
the same, tested dependencies are used and statically built into the executable. |
|||
|
|||
Multiple developers build the source code by following a specific descriptor |
|||
("recipe"), cryptographically sign the result, and upload the resulting signature. |
|||
These results are compared and only if they match, the build is accepted and uploaded |
|||
to bitcoin.org. |
|||
|
|||
More independent Gitian builders are needed, which is why I wrote this |
|||
guide. It is preferred to follow these steps yourself instead of using someone else's |
|||
VM image to avoid 'contaminating' the build. |
|||
|
|||
Table of Contents |
|||
------------------ |
|||
|
|||
- [Create a new VirtualBox VM](#create-a-new-virtualbox-vm) |
|||
- [Connecting to the VM](#connecting-to-the-vm) |
|||
- [Setting up Debian for Gitian building](#setting-up-debian-for-gitian-building) |
|||
- [Installing Gitian](#installing-gitian) |
|||
- [Setting up the Gitian image](#setting-up-the-gitian-image) |
|||
- [Getting and building the inputs](#getting-and-building-the-inputs) |
|||
- [Building Bitcoin](#building-bitcoin) |
|||
- [Building an alternative repository](#building-an-alternative-repository) |
|||
- [Signing externally](#signing-externally) |
|||
- [Uploading signatures](#uploading-signatures) |
|||
|
|||
Preparing the Gitian builder host |
|||
--------------------------------- |
|||
|
|||
The first step is to prepare the host environment that will be used to perform the Gitian builds. |
|||
This guide explains how to set up the environment, and how to start the builds. |
|||
|
|||
Debian Linux was chosen as the host distribution because it has a lightweight install (in contrast to Ubuntu) and is readily available. |
|||
Any kind of virtualization can be used, for example: |
|||
- [VirtualBox](https://www.virtualbox.org/), covered by this guide |
|||
- [KVM](http://www.linux-kvm.org/page/Main_Page) |
|||
- [LXC](https://linuxcontainers.org/), see also [Gitian host docker container](https://github.com/gdm85/tenku/tree/master/docker/gitian-bitcoin-host/README.md). |
|||
|
|||
You can also install on actual hardware instead of using virtualization. |
|||
|
|||
Create a new VirtualBox VM |
|||
--------------------------- |
|||
In the VirtualBox GUI click "Create" and choose the following parameters in the wizard: |
|||
|
|||
 |
|||
|
|||
- Type: Linux, Debian (64 bit) |
|||
|
|||
 |
|||
|
|||
- Memory Size: at least 1024MB, anything lower will really slow the build down |
|||
|
|||
 |
|||
|
|||
- Hard Drive: Create a virtual hard drive now |
|||
|
|||
 |
|||
|
|||
- Hard Drive file type: Use the default, VDI (VirtualBox Disk Image) |
|||
|
|||
 |
|||
|
|||
- Storage on Physical hard drive: Dynamically Allocated |
|||
|
|||
 |
|||
|
|||
- Disk size: at least 40GB; as low as 20GB *may* be possible, but better to err on the safe side |
|||
- Push the `Create` button |
|||
|
|||
Get the [Debian 7.8 net installer](http://cdimage.debian.org/cdimage/archive/7.8.0/amd64/iso-cd/debian-7.8.0-amd64-netinst.iso) (a more recent minor version should also work, see also [Debian Network installation](https://www.debian.org/CD/netinst/)). |
|||
This DVD image can be validated using a SHA256 hashing tool, for example on |
|||
Unixy OSes by entering the following in a terminal: |
|||
|
|||
echo "e39c36d6adc0fd86c6edb0e03e22919086c883b37ca194d063b8e3e8f6ff6a3a debian-7.8.0-amd64-netinst.iso" | sha256sum -c |
|||
# (must return OK) |
|||
|
|||
After creating the VM, we need to configure it. |
|||
|
|||
- Click the `Settings` button, then go to the `Network` tab. Adapter 1 should be attached to `NAT`. |
|||
|
|||
 |
|||
|
|||
- Click `Advanced`, then `Port Forwarding`. We want to set up a port through which we can reach the VM to get files in and out. |
|||
- Create a new rule by clicking the plus icon. |
|||
|
|||
 |
|||
|
|||
- Set up the new rule the following way: |
|||
- Name: `SSH` |
|||
- Protocol: `TCP` |
|||
- Leave Host IP empty |
|||
- Host Port: `22222` |
|||
- Leave Guest IP empty |
|||
- Guest Port: `22` |
|||
|
|||
- Click `Ok` twice to save. |
|||
|
|||
Then start the VM. On the first launch you will be asked for a CD or DVD image. Choose the downloaded iso. |
|||
|
|||
 |
|||
|
|||
Installing Debian |
|||
------------------ |
|||
|
|||
This section will explain how to install Debian on the newly created VM. |
|||
|
|||
- Choose the non-graphical installer. We do not need the graphical environment; it will only increase installation time and disk usage. |
|||
|
|||
 |
|||
|
|||
**Note**: Navigation in the Debian installer: To keep a setting at the default |
|||
and proceed, just press `Enter`. To select a different button, press `Tab`. |
|||
|
|||
- Choose locale and keyboard settings (doesn't matter, you can just go with the defaults or select your own information) |
|||
|
|||
 |
|||
 |
|||
 |
|||
|
|||
- The VM will detect network settings using DHCP, this should all proceed automatically |
|||
- Configure the network: |
|||
- System name `debian`. |
|||
- Leave domain name empty. |
|||
|
|||
 |
|||
|
|||
- Choose a root password and enter it twice (remember it for later) |
|||
|
|||
 |
|||
|
|||
- Name the new user `debian` (the full name doesn't matter, you can leave it empty) |
|||
|
|||
 |
|||
 |
|||
|
|||
- Choose a user password and enter it twice (remember it for later) |
|||
|
|||
 |
|||
|
|||
- The installer will set up the clock using a time server; this process should be automatic |
|||
- Set up the clock: choose a time zone (depends on the locale settings that you picked earlier; specifics don't matter) |
|||
|
|||
 |
|||
|
|||
- Disk setup |
|||
- Partitioning method: Guided - Use the entire disk |
|||
|
|||
 |
|||
|
|||
- Select disk to partition: SCSI1 (0,0,0) |
|||
|
|||
 |
|||
|
|||
- Partitioning scheme: All files in one partition |
|||
|
|||
 |
|||
|
|||
- Finish partitioning and write changes to disk -> *Yes* (`Tab`, `Enter` to select the `Yes` button) |
|||
|
|||
 |
|||
 |
|||
|
|||
- The base system will be installed, this will take a minute or so |
|||
- Choose a mirror (any will do) |
|||
|
|||
 |
|||
|
|||
- Enter proxy information (unless you are on an intranet, you can leave this empty) |
|||
|
|||
 |
|||
|
|||
- Wait a bit while 'Select and install software' runs |
|||
- Participate in popularity contest -> *No* |
|||
- Choose software to install. We need just the base system. |
|||
|
|||
 |
|||
|
|||
- Make sure only 'SSH server' and 'Standard System Utilities' are checked |
|||
- Uncheck 'Debian Desktop Environment' and 'Print Server' |
|||
|
|||
 |
|||
|
|||
- Install the GRUB boot loader to the master boot record? -> Yes |
|||
|
|||
 |
|||
|
|||
- Installation Complete -> *Continue* |
|||
- After installation, the VM will reboot and you will have a working Debian VM. Congratulations! |
|||
|
|||
Connecting to the VM |
|||
---------------------- |
|||
|
|||
After the VM has booted you can connect to it using SSH, and files can be copied from and to the VM using a SFTP utility. |
|||
Connect to `localhost`, port `22222` (or the port configured when installing the VM). |
|||
On Windows you can use putty[1] and WinSCP[2]. |
|||
|
|||
For example to connect as `root` from a Linux command prompt use |
|||
|
|||
$ ssh root@localhost -p 22222 |
|||
The authenticity of host '[localhost]:22222 ([127.0.0.1]:22222)' can't be established. |
|||
ECDSA key fingerprint is 8e:71:f9:5b:62:46:de:44:01:da:fb:5f:34:b5:f2:18. |
|||
Are you sure you want to continue connecting (yes/no)? yes |
|||
Warning: Permanently added '[localhost]:22222' (ECDSA) to the list of known hosts. |
|||
root@localhost's password: (enter root password configured during install) |
|||
Linux debian 3.2.0-4-amd64 #1 SMP Debian 3.2.54-2 x86_64 |
|||
root@debian:~# |
|||
|
|||
Replace `root` with `debian` to log in as user. |
|||
|
|||
[1] http://www.chiark.greenend.org.uk/~sgtatham/putty/download.html |
|||
[2] http://winscp.net/eng/index.php |
|||
|
|||
Setting up Debian for Gitian building |
|||
-------------------------------------- |
|||
|
|||
In this section we will be setting up the Debian installation for Gitian building. |
|||
|
|||
First we need to log in as `root` to set up dependencies and make sure that our |
|||
user can use the sudo command. Type/paste the following in the terminal: |
|||
|
|||
```bash |
|||
apt-get install git ruby sudo apt-cacher-ng qemu-utils debootstrap lxc python-cheetah parted kpartx bridge-utils |
|||
adduser debian sudo |
|||
``` |
|||
|
|||
When you get a colorful screen with a question about the 'LXC directory', just |
|||
go with the default (`/var/lib/lxc`). |
|||
|
|||
Then set up LXC and the rest with the following, which is a complex jumble of settings and workarounds: |
|||
|
|||
```bash |
|||
# the version of lxc-start in Debian 7.4 needs to run as root, so make sure |
|||
# that the build script can execute it without providing a password |
|||
echo "%sudo ALL=NOPASSWD: /usr/bin/lxc-start" > /etc/sudoers.d/gitian-lxc |
|||
# add cgroup for LXC |
|||
echo "cgroup /sys/fs/cgroup cgroup defaults 0 0" >> /etc/fstab |
|||
# make /etc/rc.local script that sets up bridge between guest and host |
|||
echo '#!/bin/sh -e' > /etc/rc.local |
|||
echo 'brctl addbr br0' >> /etc/rc.local |
|||
echo 'ifconfig br0 10.0.3.2/24 up' >> /etc/rc.local |
|||
echo 'exit 0' >> /etc/rc.local |
|||
# make sure that USE_LXC is always set when logging in as debian, |
|||
# and configure LXC IP addresses |
|||
echo 'export USE_LXC=1' >> /home/debian/.profile |
|||
echo 'export GITIAN_HOST_IP=10.0.3.2' >> /home/debian/.profile |
|||
echo 'export LXC_GUEST_IP=10.0.3.5' >> /home/debian/.profile |
|||
reboot |
|||
``` |
|||
|
|||
At the end the VM is rebooted to make sure that the changes take effect. The steps in this |
|||
section need only to be performed once. |
|||
|
|||
Installing Gitian |
|||
------------------ |
|||
|
|||
Re-login as the user `debian` that was created during installation. |
|||
The rest of the steps in this guide will be performed as that user. |
|||
|
|||
There is no `python-vm-builder` package in Debian, so we need to install it from source ourselves, |
|||
|
|||
```bash |
|||
wget http://archive.ubuntu.com/ubuntu/pool/universe/v/vm-builder/vm-builder_0.12.4+bzr489.orig.tar.gz |
|||
echo "ec12e0070a007989561bfee5862c89a32c301992dd2771c4d5078ef1b3014f03 vm-builder_0.12.4+bzr489.orig.tar.gz" | sha256sum -c |
|||
# (verification -- must return OK) |
|||
tar -zxvf vm-builder_0.12.4+bzr489.orig.tar.gz |
|||
cd vm-builder-0.12.4+bzr489 |
|||
sudo python setup.py install |
|||
cd .. |
|||
``` |
|||
|
|||
**Note**: When sudo asks for a password, enter the password for the user *debian* not for *root*. |
|||
|
|||
Clone the git repositories for bitcoin and Gitian. |
|||
|
|||
```bash |
|||
git clone https://github.com/devrandom/gitian-builder.git |
|||
git clone https://github.com/bitcoin/bitcoin |
|||
``` |
|||
|
|||
Setting up the Gitian image |
|||
------------------------- |
|||
|
|||
Gitian needs a virtual image of the operating system to build in. |
|||
Currently this is Ubuntu Precise x86_64. |
|||
This image will be copied and used every time that a build is started to |
|||
make sure that the build is deterministic. |
|||
Creating the image will take a while, but only has to be done once. |
|||
|
|||
Execute the following as user `debian`: |
|||
|
|||
```bash |
|||
cd gitian-builder |
|||
bin/make-base-vm --lxc --arch amd64 --suite precise |
|||
``` |
|||
|
|||
There will be a lot of warnings printed during build of the image. These can be ignored. |
|||
|
|||
**Note**: When sudo asks for a password, enter the password for the user *debian* not for *root*. |
|||
|
|||
Getting and building the inputs |
|||
-------------------------------- |
|||
|
|||
Follow the instructions in [doc/release-process.md](release-process.md) in the bitcoin repository |
|||
under 'Fetch and build inputs' to install sources which require manual intervention. Also follow |
|||
the next step: 'Seed the Gitian sources cache', which will fetch all necessary source files allowing |
|||
for Gitian to work offline. |
|||
|
|||
Building Bitcoin |
|||
---------------- |
|||
|
|||
To build Bitcoin (for Linux, OS X and Windows) just follow the steps under 'perform |
|||
Gitian builds' in [doc/release-process.md](release-process.md) in the bitcoin repository. |
|||
|
|||
This may take a long time as it also builds the dependencies needed for each descriptor. |
|||
These dependencies will be cached after a successful build to avoid rebuilding them where possible. |
|||
|
|||
At any time you can check the package installation and build progress with |
|||
|
|||
```bash |
|||
tail -f var/install.log |
|||
tail -f var/build.log |
|||
``` |
|||
|
|||
Output from `gbuild` will look something like |
|||
|
|||
Initialized empty Git repository in /home/debian/gitian-builder/inputs/bitcoin/.git/ |
|||
remote: Reusing existing pack: 35606, done. |
|||
remote: Total 35606 (delta 0), reused 0 (delta 0) |
|||
Receiving objects: 100% (35606/35606), 26.52 MiB | 4.28 MiB/s, done. |
|||
Resolving deltas: 100% (25724/25724), done. |
|||
From https://github.com/bitcoin/bitcoin |
|||
... (new tags, new branch etc) |
|||
--- Building for precise x86_64 --- |
|||
Stopping target if it is up |
|||
Making a new image copy |
|||
stdin: is not a tty |
|||
Starting target |
|||
Checking if target is up |
|||
Preparing build environment |
|||
Updating apt-get repository (log in var/install.log) |
|||
Installing additional packages (log in var/install.log) |
|||
Grabbing package manifest |
|||
stdin: is not a tty |
|||
Creating build script (var/build-script) |
|||
lxc-start: Connection refused - inotify event with no name (mask 32768) |
|||
Running build script (log in var/build.log) |
|||
|
|||
Building an alternative repository |
|||
----------------------------------- |
|||
|
|||
If you want to do a test build of a pull on GitHub it can be useful to point |
|||
the Gitian builder at an alternative repository, using the same descriptors |
|||
and inputs. |
|||
|
|||
For example: |
|||
```bash |
|||
URL=https://github.com/laanwj/bitcoin.git |
|||
COMMIT=2014_03_windows_unicode_path |
|||
./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin/contrib/gitian-descriptors/gitian-linux.yml |
|||
./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin/contrib/gitian-descriptors/gitian-win.yml |
|||
./bin/gbuild --commit bitcoin=${COMMIT} --url bitcoin=${URL} ../bitcoin/contrib/gitian-descriptors/gitian-osx.yml |
|||
``` |
|||
|
|||
Signing externally |
|||
------------------- |
|||
|
|||
If you want to do the PGP signing on another device, that's also possible; just define `SIGNER` as mentioned |
|||
and follow the steps in the build process as normal. |
|||
|
|||
gpg: skipped "laanwj": secret key not available |
|||
|
|||
When you execute `gsign` you will get an error from GPG, which can be ignored. Copy the resulting `.assert` files |
|||
in `gitian.sigs` to your signing machine and do |
|||
|
|||
```bash |
|||
gpg --detach-sign ${VERSION}-linux/${SIGNER}/bitcoin-linux-build.assert |
|||
gpg --detach-sign ${VERSION}-win/${SIGNER}/bitcoin-win-build.assert |
|||
gpg --detach-sign ${VERSION}-osx-unsigned/${SIGNER}/bitcoin-osx-build.assert |
|||
``` |
|||
|
|||
This will create the `.sig` files that can be committed together with the `.assert` files to assert your |
|||
Gitian build. |
|||
|
|||
Uploading signatures |
|||
--------------------- |
|||
|
|||
After building and signing you can push your signatures (both the `.assert` and `.assert.sig` files) to the |
|||
[bitcoin/gitian.sigs](https://github.com/bitcoin/gitian.sigs/) repository, or if that's not possible create a pull |
|||
request. You can also mail the files to Wladimir (laanwj@gmail.com) and he will commit them. |
|||
|
Before Width: | Height: | Size: 70 KiB |
|
Before Width: | Height: | Size: 78 KiB |
|
Before Width: | Height: | Size: 80 KiB |
|
Before Width: | Height: | Size: 52 KiB |
|
Before Width: | Height: | Size: 128 KiB |
|
Before Width: | Height: | Size: 88 KiB |
|
Before Width: | Height: | Size: 7.7 KiB |
|
Before Width: | Height: | Size: 9.3 KiB |
|
Before Width: | Height: | Size: 6.5 KiB |
|
Before Width: | Height: | Size: 7.9 KiB |
|
Before Width: | Height: | Size: 10 KiB |
|
Before Width: | Height: | Size: 8.6 KiB |
|
Before Width: | Height: | Size: 11 KiB |
|
Before Width: | Height: | Size: 9.6 KiB |
|
Before Width: | Height: | Size: 7.4 KiB |
|
Before Width: | Height: | Size: 8.7 KiB |
|
Before Width: | Height: | Size: 74 KiB |
|
Before Width: | Height: | Size: 9.6 KiB |
|
Before Width: | Height: | Size: 6.8 KiB |
|
Before Width: | Height: | Size: 13 KiB |
|
Before Width: | Height: | Size: 9.4 KiB |
|
Before Width: | Height: | Size: 10 KiB |
|
Before Width: | Height: | Size: 6.6 KiB |
|
Before Width: | Height: | Size: 6.4 KiB |
|
Before Width: | Height: | Size: 12 KiB |
|
Before Width: | Height: | Size: 7.4 KiB |
|
Before Width: | Height: | Size: 6.2 KiB |
|
Before Width: | Height: | Size: 6.2 KiB |
|
Before Width: | Height: | Size: 59 KiB |
|
Before Width: | Height: | Size: 14 KiB |
|
Before Width: | Height: | Size: 84 KiB |
@ -1,39 +0,0 @@ |
|||
Support for using travis-ci has been added in order to automate pull-testing. |
|||
See https://travis-ci.org/ for more info |
|||
|
|||
This procedure is different than the pull-tester that came before it in a few |
|||
ways. |
|||
|
|||
There is nothing to administer. This is a major feature as it means |
|||
that builds have no local state. Because there is no ability to login to the |
|||
builders to install packages (tools, dependencies, etc), the entire build |
|||
procedure must instead be controlled by a declarative script (.travis.yml). |
|||
This script declares each build configuration, creates virtual machines as |
|||
necessary, builds, then discards the virtual machines. |
|||
|
|||
A build matrix is constructed to test a wide range of configurations, rather |
|||
than a single pass/fail. This helps to catch build failures and logic errors |
|||
that present on platforms other than the ones the author has tested. This |
|||
matrix is defined in the build script and can be changed at any time. |
|||
|
|||
All builders use the dependency-generator in the depends dir, rather than |
|||
using apt-get to install build dependencies. This guarantees that the tester |
|||
is using the same versions as Gitian, so the build results are nearly identical |
|||
to what would be found in a final release. However, this also means that builds |
|||
will fail if new dependencies are introduced without being added to the |
|||
dependency generator. |
|||
|
|||
In order to avoid rebuilding all dependencies for each build, the binaries are |
|||
cached and re-used when possible. Changes in the dependency-generator will |
|||
trigger cache-invalidation and rebuilds as necessary. |
|||
|
|||
These caches can be manually removed if necessary. This is one of the the very few |
|||
manual operations that is possible with Travis, and it can be done by the |
|||
Bitcoin Core committer via the Travis web interface. |
|||
|
|||
In some cases, secure strings may be needed for hiding sensitive info such as |
|||
private keys or URLs. The travis client may be used to create these strings: |
|||
http://docs.travis-ci.com/user/encryption-keys/ |
|||
|
|||
For the details of the build descriptor, see the official docs: |
|||
http://docs.travis-ci.com/user/build-configuration/ |
|||
@ -1,129 +0,0 @@ |
|||
# Calgary Design |
|||
|
|||
## Motivation |
|||
Calgary is a reimplementation of Zerocash in Bitcoin meant to address a number of shortcomings and design flaws in the original academic implementation. This gives us an opportunity to rework the implementation from scratch, setting out rigorous guidelines for how we interact with upstream code. |
|||
|
|||
### Bridging implementation gaps |
|||
The original implementation carried Protect and Pour operations through the transaction inputs. This required a number of hacks. Transaction inputs that carried Pours, for example, had no actual `CTxOut` to spend, so a dummy "always_spendable" transaction was created. The scripting system (opcodes in particular) had to be changed to store the zerocash operation metadata. Changes to support things like intermediate Pours or input value to the circuit had unclear interactions with the transaction system. |
|||
|
|||
The obvious advantage of the original implementation was avoiding structural changes to `CTransaction`. However, these are necessary, not least because versioning semantics must be laid out in a sensible way. |
|||
|
|||
## Goals |
|||
* We should take this opportunity to understand the implementation better, and build on top of a more recent version of Bitcoin. Upstream has made a number of changes to critical components and has begun to refactor consensus-critical code into a `libconsensus` library. |
|||
* We should rigorously practice our [design policy](https://github.com/Electric-Coin-Company/zerocashd/wiki/design). This includes avoiding changes to upstream's scripting system wherever possible, and only modifying structures with proper versioning plans. |
|||
* We should strive to preserve the semantics of our private alpha implementation such that the `zc-raw-*` RPC commands still work properly. |
|||
|
|||
## Scope |
|||
|
|||
* **Chained pours** ([#121][ticket121]) mean that we should anticipate multiple pours in a transaction, specifically because the pours may require commitments from previous pours. |
|||
* **Symmetric pours** ([#338] [ticket338]) mean that instead of separate Protect / Pour operations, a *single* Pour operation exists which takes a `vpub_in` and `vpub_out`, unifying the two operations. This requires a circuit change. |
|||
* **Versioning semantics** ([#114] [ticket114]) require us to avoid breaking upstream tests whenever possible. We need to anticipate both changes to our own structures after launch to support new features (such as circuit changes, see #152) and potential changes to upstream transaction structures we will eventually need to rebase on top of. |
|||
* **Cryptographic binding of pours** ([#366] [ticket366]) is necessary to ensure that (in the most common situation) it is not possible to move a pour from one transaction to another, or replace pours in a transaction without the authorization of its inputs. |
|||
|
|||
[ticket121]: https://github.com/Electric-Coin-Company/zerocashd/issues/121 |
|||
[ticket338]: https://github.com/Electric-Coin-Company/zerocashd/issues/338 |
|||
[ticket114]: https://github.com/Electric-Coin-Company/zerocashd/issues/114 |
|||
[ticket366]: https://github.com/Electric-Coin-Company/zerocashd/issues/366 |
|||
|
|||
### Not in Scope |
|||
|
|||
#### PoW and other block header changes |
|||
|
|||
It should not be necessary to make any block header changes yet in the design. We anticipate to change the PoW scheme, which may affect how nonces appear in coinbases. However, this doesn't appear to have a significant affect on the overall design, so we do not specify its implications here. |
|||
|
|||
#### Systemic incentives for fungibility |
|||
|
|||
It will be necessary to make a number of modifications to the fee calculation and priority system, because pours will introduce larger transactions. We must be careful to avoid a "tragedy of the commons" scenario which would make protected transactions more expensive to process, producing adverse incentives for the network. |
|||
|
|||
Upstream bitcoin appears to be phasing out `relaypriority` and other priority systems. |
|||
|
|||
#### Anonymity set partitioning (#425) |
|||
|
|||
Any place in our design where multiple structural decisions can be made, especially by wallet software, allows not only particular wallet software to be identified but potentially users as well. This is most pronounced when determining how to split and merge buckets. (Depending on when and how the buckets are merged, it may be possible to identify users.) |
|||
|
|||
Additionally, the merkle root anchor indicates to others your "view" of the network state, which could be used to identify users. |
|||
|
|||
## Design |
|||
|
|||
### CTransaction |
|||
|
|||
The heart of zerocash modifications is in `CTransaction`. The current layout is as follows: |
|||
|
|||
``` |
|||
{ |
|||
int32_t nVersion; |
|||
const std::vector<CTxIn> vin; |
|||
const std::vector<CTxOut> vout; |
|||
const uint32_t nLockTime; |
|||
} |
|||
``` |
|||
|
|||
#### Versioning |
|||
|
|||
In this design, we will increment the latest `nVersion` for transactions, adding new fields to the end to encapsulate our zerocash operations ([#114] [ticket114]). Our fields must anticipate the case that no zerocash operations occur, such as in traditional or "purely cleartext" transactions. |
|||
|
|||
##### Alternative: Use bitflags to indicate the presence of zerocash fields |
|||
In this alternative, we use bitflags in the `nVersion` field to indicate the presence of zerocash fields at the end. This would allow us to track upstream CTransaction changes seamlessly, *but* would indefinitely require that upstream `nVersion` bits are available to us for this purpose. Additionally, the absence of these bitflags would conflict in purpose with an empty vector of `PourTx` structures as outlined later. |
|||
|
|||
#### PourTx |
|||
|
|||
We add an additional field to the end of a `CTransaction`, a vector of `PourTx` structures called `vpour`. This vector is merely empty in the case of traditional transactions. Operations are allowed to reference the (commitment tree) merkle root produced by a previous operation or block for verification purposes ([#121] [ticket121]). |
|||
|
|||
The structure of a `PourTx` is as follows: |
|||
|
|||
``` |
|||
struct PourTx { |
|||
anchor, // merkle root this operation is anchored to |
|||
scriptPubKey, // CScript which is hashed and provided as an input to the snark verifier/prover |
|||
scriptSig, // CScript which is verified against scriptPubKey |
|||
vpub_in, // the value to draw from the public value pool |
|||
vpub_out, // the value to provide to the public value pool |
|||
pour, // libzerocash::PourTransaction (modified slightly) |
|||
encrypted_buckets, // two ciphertexts using pk_enc to send v, r, rho to recipient |
|||
macs, // MACs binding scriptSig to the pour |
|||
serials, // the serials "spent" by this transaction |
|||
commitments // the new commitments of buckets produced by this transaction |
|||
} |
|||
``` |
|||
|
|||
The `CTransactionSignatureSerializer` (and perhaps other components) will need to be modified so that the inputs are cryptographically bound to the `PourTx`s ([#366] [ticket366]). |
|||
|
|||
Note that this general operation has both `vpub_in` and `vpub_out` which is a generalization from the academic prototype which has separate `Protect` (with public input) and `Pour` (with public output) operations ([#338] [ticket338]). |
|||
|
|||
The `vin` and `vout` fields can be empty if there exist some well-formed `PourTx`s, *in contrast* to Bitcoin's invariants on these vectors. |
|||
|
|||
##### Chained Pour Validation |
|||
|
|||
Note that chained pours ([#121] [ticket121]) require that witnesses in a `PourTx` after the first may refer to a treestate which is not an ancestor of the block tree state after the `CTransaction` containing it is mined. For example suppose transaction `A` appears prior to transaction `B` in the same block and that each contain two `PourTx`s, `A1`, `A2`, `B1`, and `B2`. Call the tree state of the block's ancestor `T_ancestor`, and the tree state of the latest block `T_block`. The state of `T_block` will be derived from applying on top of `T_ancestor` the commits from `A1`, `A2`, `B1`, and `B2` in that order. However, `B2` may spend an output of `B1`, and if it does so, it will refer to the tree state derived by applying on top of `T_ancestor` commitments from only `B1`. Notice that a *different* output of `B1` may be spent in a later block. In *that* case, this later spend would use a proof anchored in `T_block` or some valid subsequent tree state. |
|||
|
|||
##### Script-based Signatures |
|||
|
|||
In the original academic implementation, an ephemeral ECDSA keypair is generated and used to sign the rest of the transaction, binding it to the Pour as the public key is provided as an input to the prover/verifier. We can reuse Bitcoin's scripting system to abstract this behavior further, allowing for more flexible binding behaviors between pours and the rest of the transaction. As an example of a future use of this flexibility, crowdfunding pours would be possible by allowing users to bind their pours to nothing else in the transaction. |
|||
|
|||
##### Alternative: `SIGHASH_ALL` binding |
|||
In this alternative, instead of using `CScript`s to offer flexible binding semantics to users, we force `SIGHASH_ALL`-style signature checking on the transaction. This is safer if we discover that scripts in this context have strange interactions that cannot be bridged, or that malleability issues are unavoidable. |
|||
|
|||
### CBlock |
|||
|
|||
In the previous design, blocks stored the root of the commitment merkle tree. This isn't strictly necessary for consensus as the commitment tree, like the UTXO set, is derived from the transactions in the block. It will likely be necessary to retain a mapping between merkle roots and block hashes. |
|||
|
|||
We will revisit this aspect of the design later when we incorporate improvements for light clients, "light full nodes", and some other potential use cases. |
|||
|
|||
## Implementation strategy |
|||
|
|||
Various parts of this design can be implemented concurrently and iteratively. |
|||
|
|||
### Stage 1: Foundations |
|||
|
|||
1. Integrate libzerocash into the repository. |
|||
2. Make changes to ConnectBlock and BlockUndo infrastructure to support our incremental merkle trees and commit to those trees via our transactions as appropriate. |
|||
3. Prevent double-spends of serials, including in the mempool. |
|||
4. `CTransaction` should be modified so that additional fields are supported. |
|||
|
|||
### Stage 2: Concurrent tasks |
|||
Each of the following tasks to complete the redesign can be done independently. |
|||
|
|||
* Add old RPC commands. ([#527] [ticket527]) |
|||
* `CScript` scheme to enforce cryptographic binding of the transaction to the pour. ([#529] [ticket529]) |
|||
* Chained pours which allow pours to reference merkle roots produced by other pours. ([#555] [ticket555]) |
|||
* A zerocash-specific versioning field can be added, along with upstream interaction semantics. ([#114] [ticket114]) |
|||
Loading…
Reference in new issue