In the last article of mine about SSL-related programming, the API to handle SSL transaction for the DICE was the SSPI (Security Support Provider Interface) that is one of the standard API sets provided by Microsoft Windows. Though I outlined why I chose SSPI over OpenSSL in the article, recently I replaced SSPI with OpenSSL in the latest version of the DICE that was released with HTTPS implemented. The rationale behind the switch of the SSL engine was not so straightforward.
For me, the main concern about OpenSSL had been its putative close relationship with the BSD socket architecture that is not compatible with asynchronous sockets and I/O completion ports. Another concern was about OpenSSL's vulnerabilities against security breaches. OpenSSL has been an active target by crackers and one of the most scrutinized library. Not that Microsoft's implementation is any better, but as far as I know OpenSSL gets many security advisories about it through its update history.
On the other hand, SSPI's complex nature as an abstraction layer for multiple protocols had been a growing pain. Since the DICE is built with I/O completion ports, it was pretty messy to embed SSPI negotiation in combination with asynchronous sockets. After the sour experience of implementing IRC over SSL with SSPI on asynchrounous sockets and I/O completion ports some years ago, I was not so thrilled to touch it again to implement HTTPS. I needed a better, simpler framework to get the job done.
In updating the DICE to 0.9.0.0 after a long hiatus, I took the new version as a chance to address awkward points that had been long overlooked, sometimes even breaking compatibility with older versions. One of such problems in the DICE was SSL-related implementation. That's why I took a second look at OpenSSL. Though it may contain unknown security flaws, its core cryptographic functions are actively upgraded to the latest standard spec unlike Microsoft's implementation. Also it's nice in terms of compiler/linker global optimization that the source code is available.
So what is the key to the actual usage of OpenSSL in asynchronous sockets? I browsed the sparse user manual and found the necessary tool for me: memory BIO.
BIO is an "I/O abstraction, it hides many of the underlying I/O details from an application" according to the OpenSSL document (What does B in BIO stand for anyway?). Also a "memory BIO is a source/sink BIO which uses memory for its I/O." This basically means you can do SSL negotiation without touching a BSD socket or other platform-specific stuff at all. All you need to do is feed encrypted data into a memory BIO until it's ready to dump something meaningful decrypted out of it.
While the title of this article mentions "asynchronous", the sample code below uses traditional synchronous sockets. In fact, the simple abstraction by memory BIO allows smooth transition from synchronous sockets to asynchronous sockets since buffering memory BIO is exactly the same process as parsing a plain message by waiting for a whole message packet to arrive in asynchronous callbacks. In addition, you had to manually set up an SSL handshake at the beginning of a connection by yourself in SSPI while BIO in OpenSSL completely hides it from a client code. I could replace SSPI with OpenSSL for the already implemented IRC over SSL, and then could implement HTTPS in the latest version of the DICE by making the flow described in this sample code asynchronous.
The only caveat in making it asynchronous is it's a bit difficult to support SSL renegotiation. Though it's rare, sometimes SSL resets SSL properties at the middle of connection to make it difficult for crackers to collect necessary data. When it happens, you have to save the context such as a partially decrypted message, then do handshake again, and go back to where you originally were by restoring the original state in a new SSL context. In synchronous connection all these steps can occur between 2 braces in a certain function, but in asynchronous sockets it's hard to suspend it by saving the context and resume it from the original point, since you have to juggle many more states in your session state machine to fully support renegotiation.
This test code of a synchronous HTTPS server was written and tested with VS2005 SP1 on Windows XP SP2. You have to build OpenSSL by obtaining its source code prior to building this sample. The OpenSSL version I used was 0.9.8g. After building OpenSSL, you have to add
To use OpenSSL in a multithread application you have to implement necessary synchronization primitives for OpenSSL with your platform-supplied functions. In this sample application, I used CRITICAL_SECTION and
A function to get an
A function to generate an X509-format certificate and RSA public key. It's signed by a private key.
An error report function for an SSL object.
The main function begins by setting up OpenSSL synchronization primitives, then creates a new SSL context object
The next thing to do is key-pair/certificate generation for public key en/decryption. This sample application uses a self-signed certificate to omit the need of a valid certificate.
Setting up Winsock 2 and a listener socket to create a network server.
Everything is ready, now let's get into the actual server connection loop. When it accepts a new connection, it creates a new SSL object that encapsulates a single SSL connection. Also it creates memory BIO buffers for input and output.
After it's done you have to clean up all resources. If there's a memory leak, CRYPTO_mem_leaks_fp will print it out.
When you run this sample code it listens to the standard HTTPS port 443. Start with a web browser and open "https://127.0.0.1/" to see how an SSL connection is processed. To stop this server, put "quit" in a request URL ("https://127.0.0.1/quit").
For me, the main concern about OpenSSL had been its putative close relationship with the BSD socket architecture that is not compatible with asynchronous sockets and I/O completion ports. Another concern was about OpenSSL's vulnerabilities against security breaches. OpenSSL has been an active target by crackers and one of the most scrutinized library. Not that Microsoft's implementation is any better, but as far as I know OpenSSL gets many security advisories about it through its update history.
On the other hand, SSPI's complex nature as an abstraction layer for multiple protocols had been a growing pain. Since the DICE is built with I/O completion ports, it was pretty messy to embed SSPI negotiation in combination with asynchronous sockets. After the sour experience of implementing IRC over SSL with SSPI on asynchrounous sockets and I/O completion ports some years ago, I was not so thrilled to touch it again to implement HTTPS. I needed a better, simpler framework to get the job done.
In updating the DICE to 0.9.0.0 after a long hiatus, I took the new version as a chance to address awkward points that had been long overlooked, sometimes even breaking compatibility with older versions. One of such problems in the DICE was SSL-related implementation. That's why I took a second look at OpenSSL. Though it may contain unknown security flaws, its core cryptographic functions are actively upgraded to the latest standard spec unlike Microsoft's implementation. Also it's nice in terms of compiler/linker global optimization that the source code is available.
So what is the key to the actual usage of OpenSSL in asynchronous sockets? I browsed the sparse user manual and found the necessary tool for me: memory BIO.
BIO is an "I/O abstraction, it hides many of the underlying I/O details from an application" according to the OpenSSL document (What does B in BIO stand for anyway?). Also a "memory BIO is a source/sink BIO which uses memory for its I/O." This basically means you can do SSL negotiation without touching a BSD socket or other platform-specific stuff at all. All you need to do is feed encrypted data into a memory BIO until it's ready to dump something meaningful decrypted out of it.
While the title of this article mentions "asynchronous", the sample code below uses traditional synchronous sockets. In fact, the simple abstraction by memory BIO allows smooth transition from synchronous sockets to asynchronous sockets since buffering memory BIO is exactly the same process as parsing a plain message by waiting for a whole message packet to arrive in asynchronous callbacks. In addition, you had to manually set up an SSL handshake at the beginning of a connection by yourself in SSPI while BIO in OpenSSL completely hides it from a client code. I could replace SSPI with OpenSSL for the already implemented IRC over SSL, and then could implement HTTPS in the latest version of the DICE by making the flow described in this sample code asynchronous.
The only caveat in making it asynchronous is it's a bit difficult to support SSL renegotiation. Though it's rare, sometimes SSL resets SSL properties at the middle of connection to make it difficult for crackers to collect necessary data. When it happens, you have to save the context such as a partially decrypted message, then do handshake again, and go back to where you originally were by restoring the original state in a new SSL context. In synchronous connection all these steps can occur between 2 braces in a certain function, but in asynchronous sockets it's hard to suspend it by saving the context and resume it from the original point, since you have to juggle many more states in your session state machine to fully support renegotiation.
This test code of a synchronous HTTPS server was written and tested with VS2005 SP1 on Windows XP SP2. You have to build OpenSSL by obtaining its source code prior to building this sample. The OpenSSL version I used was 0.9.8g. After building OpenSSL, you have to add
libeay32.lib and ssleay32.lib in the out32 folder under the OpenSSL source root directory to the linker input option in the VC++ project of the sample code, in addition to ws2_32.lib for Winsock 2.// openssltest.cpp
// InitializeCriticalSectionAndSpinCount
#ifndef _WIN32_WINNT
#define _WIN32_WINNT 0x0500
#endif _WIN32_WINNT
#include <windows.h>
#include <tchar.h>
#include <openssl/ssl.h>
#include <openssl/err.h>
#include <iostream>
#include <string>
using namespace std;
To use OpenSSL in a multithread application you have to implement necessary synchronization primitives for OpenSSL with your platform-supplied functions. In this sample application, I used CRITICAL_SECTION and
InitializeCriticalSectionAndSpinCount available in Windows 2000 and later. Its wrapper class Synchronizer does synchronization in various OpenSSL callbacks.class Synchronizer
{
private:
CRITICAL_SECTION lock_;
public:
__forceinline Synchronizer()
{
InitializeCriticalSectionAndSpinCount(&lock_, 4000);
}
__forceinline ~Synchronizer()
{
DeleteCriticalSection(&lock_);
}
__forceinline void acquire()
{
__try
{
EnterCriticalSection(&lock_);
}
__except (EXCEPTION_EXECUTE_HANDLER)
{
}
}
__forceinline void release()
{
LeaveCriticalSection(&lock_);
}
};
static Synchronizer* openssl_locks = 0;
struct CRYPTO_dynlock_value
{
Synchronizer s;
};
static void funcOpenSSLLockingCallback(
int mode,
int type,
const char* file,
int line
)
{
if (mode & CRYPTO_LOCK)
openssl_locks[type].acquire();
else
openssl_locks[type].release();
}
static unsigned long funcOpenSSLIDCallback(void)
{
return GetCurrentThreadId();
}
static CRYPTO_dynlock_value* funcOpenSSLDynCreateCallback(
const char* file,
int line
)
{
return new CRYPTO_dynlock_value;
}
static void funcOpenSSLDynDestroyCallback(
CRYPTO_dynlock_value* l,
const char* file,
int line
)
{
delete l;
}
static void funcOpenSSLDynLockCallback(
int mode,
CRYPTO_dynlock_value* l,
const char* file,
int line
)
{
if (mode & CRYPTO_LOCK)
l->s.acquire();
else
l->s.release();
}
A function to get an
EVP_PKEY RSA private key object.EVP_PKEY* get_evp_pkey(RSA* rsa, int priv)
{
RSA* key = (priv ? RSAPrivateKey_dup(rsa): RSAPublicKey_dup(rsa));
if (!key)
goto error;
EVP_PKEY* pkey = EVP_PKEY_new();
if (!pkey)
goto error;
if (!(EVP_PKEY_assign_RSA(pkey, key)))
goto error;
return pkey;
error:
if (pkey)
EVP_PKEY_free(pkey);
if (key)
RSA_free(key);
return NULL;
}
A function to generate an X509-format certificate and RSA public key. It's signed by a private key.
static X509* create_certificate(
RSA* rsa,
RSA* rsaSign,
const char* cname,
const char* cnameSign,
const char* pszOrgName,
unsigned int certLifetime
)
{
time_t start_time = time(NULL);
time_t end_time = start_time + certLifetime;
EVP_PKEY* sign_pkey = get_evp_pkey(rsaSign, 1);
if (!sign_pkey)
goto error;
EVP_PKEY* pkey = get_evp_pkey(rsa, 0);
if (!pkey)
goto error;
X509* x509 = X509_new();
if (!x509)
goto error;
if (!(X509_set_version(x509, 2)))
goto error;
if (!(ASN1_INTEGER_set(X509_get_serialNumber(x509), (long)start_time)))
goto error;
X509_NAME* name = X509_NAME_new();
if (!name)
goto error;
int nid = OBJ_txt2nid("organizationName");
if (nid == NID_undef)
goto error;
if (!(X509_NAME_add_entry_by_NID(name, nid, MBSTRING_ASC, (unsigned char*)pszOrgName, -1, -1, 0)))
goto error;
if ((nid = OBJ_txt2nid("commonName")) == NID_undef)
goto error;
if (!(X509_NAME_add_entry_by_NID(name, nid, MBSTRING_ASC, (unsigned char*)cname, -1, -1, 0)))
goto error;
if (!(X509_set_subject_name(x509, name)))
goto error;
X509_NAME* name_issuer = X509_NAME_new();
if (!name_issuer)
goto error;
if ((nid = OBJ_txt2nid("organizationName")) == NID_undef)
goto error;
if (!(X509_NAME_add_entry_by_NID(name_issuer, nid, MBSTRING_ASC, (unsigned char*)pszOrgName, -1, -1, 0)))
goto error;
if ((nid = OBJ_txt2nid("commonName")) == NID_undef)
goto error;
if (!(X509_NAME_add_entry_by_NID(name_issuer, nid, MBSTRING_ASC, (unsigned char*)cnameSign, -1, -1, 0)))
goto error;
if (!(X509_set_issuer_name(x509, name_issuer)))
goto error;
if (!X509_time_adj(X509_get_notBefore(x509), 0, &start_time))
goto error;
if (!X509_time_adj(X509_get_notAfter(x509), 0, &end_time))
goto error;
if (!X509_set_pubkey(x509, pkey))
goto error;
if (!X509_sign(x509, sign_pkey, EVP_sha1()))
goto error;
goto done;
error:
if (x509)
{
X509_free(x509);
x509 = NULL;
}
done:
if (sign_pkey)
EVP_PKEY_free(sign_pkey);
if (pkey)
EVP_PKEY_free(pkey);
if (name)
X509_NAME_free(name);
if (name_issuer)
X509_NAME_free(name_issuer);
return x509;
}
An error report function for an SSL object.
void reportError(SSL* ssl, int result)
{
if (result <= 0)
{
int error = SSL_get_error(ssl, result);
switch (error)
{
case SSL_ERROR_ZERO_RETURN:
cout << "SSL_ERROR_ZERO_RETURN" << endl;
break;
case SSL_ERROR_NONE:
cout << "SSL_ERROR_NONE" << endl;
break;
case SSL_ERROR_WANT_READ:
cout << "SSL_ERROR_WANT_READ" << endl;
break;
default:
{
char buffer[256];
while (error != 0)
{
ERR_error_string_n(error, buffer, sizeof(buffer));
cout << "Error: " << error << " - " << buffer << endl;
error = ERR_get_error();
}
}
break;
}
}
}
The main function begins by setting up OpenSSL synchronization primitives, then creates a new SSL context object
SSL_CTX. This global OpenSSL context should not be mistaken as the per-connection context object SSL that will be introduced later.int _tmain(int argc, _TCHAR* argv[])
{
#ifdef _DEBUG
// OpenSSL internal memory-leak checkers
CRYPTO_malloc_debug_init();
CRYPTO_dbg_set_options(V_CRYPTO_MDEBUG_ALL);
CRYPTO_mem_ctrl(CRYPTO_MEM_CHECK_ON);
#endif
openssl_locks = new Synchronizer[CRYPTO_num_locks()];
// callbacks for static lock
CRYPTO_set_locking_callback(funcOpenSSLLockingCallback);
CRYPTO_set_id_callback(funcOpenSSLIDCallback);
// callbacks for dynamic lock
CRYPTO_set_dynlock_create_callback(funcOpenSSLDynCreateCallback);
CRYPTO_set_dynlock_destroy_callback(funcOpenSSLDynDestroyCallback);
CRYPTO_set_dynlock_lock_callback(funcOpenSSLDynLockCallback);
// Load algorithms and error strings.
SSL_load_error_strings();
SSL_library_init();
// Compatible with SSLv2, SSLv3 and TLSv1
SSL_METHOD* method = SSLv23_server_method();
// Create new context from method.
SSL_CTX* ctx = SSL_CTX_new(method);
// SSL_new() creates a new SSL structure which is needed to hold the data for a TLS/SSL connection.
// The new structure inherits the settings of the underlying context ctx: connection method (SSLv2/v3/TLSv1),
// options, verification settings, timeout settings.
The next thing to do is key-pair/certificate generation for public key en/decryption. This sample application uses a self-signed certificate to omit the need of a valid certificate.
RSA* rsa = RSA_generate_key(1024, 65537, NULL, NULL);
if (!rsa)
{
cerr << "RSA_generate_key" << endl;
return 1;
}
const char* pszCommonName = "test name";
X509* cert = create_certificate(rsa, rsa, pszCommonName, pszCommonName, "DICE", 3 * 365 * 24 * 60 * 60 /* 3 years */);
if (!cert)
{
cerr << "Couldn't create a certificate" << endl;
return 1;
}
///////////////////////////////////////////////////////////////////////////
if (SSL_CTX_use_RSAPrivateKey(ctx, rsa) <= 0)
{
ERR_print_errors_fp(stderr);
return 1;
}
if (SSL_CTX_use_certificate(ctx, cert) <= 0)
{
ERR_print_errors_fp(stderr);
return 1;
}
RSA_free(rsa);
X509_free(cert);
if (!SSL_CTX_check_private_key(ctx))
{
cerr << "Private key is invalid." << endl;
return 1;
}
else
cout << "Private key is OK" << endl;
//////////////////////////////////////////////////////////////////////////////////
Setting up Winsock 2 and a listener socket to create a network server.
cout << "Preparing socket" << endl;
WORD wVersionRequested = MAKEWORD(2,1);
WSADATA wsaData;
int nRet = WSAStartup(wVersionRequested, &wsaData);
if (wsaData.wVersion != wVersionRequested)
{
cerr << "Wrong version" << endl;
return 1;
}
SOCKET listenSocket = socket(AF_INET, SOCK_STREAM, IPPROTO_TCP);
if (listenSocket == INVALID_SOCKET)
{
cerr << "socket()" << endl;
return 1;
}
SOCKADDR_IN saServer;
saServer.sin_family = AF_INET;
saServer.sin_addr.s_addr = INADDR_ANY;
saServer.sin_port = htons(443);
nRet = bind(listenSocket, (LPSOCKADDR)&saServer, sizeof(struct sockaddr));
if (nRet == SOCKET_ERROR)
{
closesocket(listenSocket);
cerr << "bind()" << endl;
return 1;
}
nRet = listen(listenSocket, SOMAXCONN);
if (nRet == SOCKET_ERROR)
{
closesocket(listenSocket);
cerr << "listen()" << endl;
return 1;
}
bool bQuit = false;
cout << "Ready" << endl;
Everything is ready, now let's get into the actual server connection loop. When it accepts a new connection, it creates a new SSL object that encapsulates a single SSL connection. Also it creates memory BIO buffers for input and output.
while (1)
{
SSL* ssl = SSL_new(ctx);
BIO* bioIn = BIO_new(BIO_s_mem());
BIO* bioOut = BIO_new(BIO_s_mem());
// SSL_set_bio() connects the BIOs rbio and wbio for the read and write operations of the TLS/SSL
// (encrypted) side of ssl.
SSL_set_bio(ssl, bioIn, bioOut);
SSL_set_accept_state(ssl);
SOCKET remoteSocket = accept(
listenSocket,
NULL,
NULL
);
if (remoteSocket == INVALID_SOCKET)
{
closesocket(listenSocket);
cerr << "accept()" << endl;
return 1;
}
cout << "Accepted" << endl;
char szBuf[128];
memset(szBuf, 0, sizeof(szBuf));
bool bHandShakeOver = false;
bool bReplyOver = false;
const char* reply = "HTTP/1.0 200 OK\r\nContent-Type: text/plain\r\nContent-Length: 14\r\nConnection: close\r\n\r\nthis is a test";
string strBuffer;
while (1)
{
nRet = recv(remoteSocket, szBuf, sizeof(szBuf), 0);
if (nRet == SOCKET_ERROR)
{
cerr << "recv() - SOCKET_ERROR" << endl;
break;
}
if (nRet > 0)
{
int bufferUsed = BIO_write(bioIn, szBuf, nRet);
if (bufferUsed == -1 || bufferUsed == -2 || bufferUsed == 0)
{
// error
}
// SSL read loop
while (1)
{
// SSL_read() tries to read num bytes from the specified ssl into the buffer buf.
int bytesOut = SSL_read(ssl, (void*)szBuf, sizeof(szBuf));
if (bytesOut > 0)
{
strBuffer.append(szBuf, bytesOut);
if ((strBuffer.size() > 4 && strBuffer.find("\r\n\r\n", strBuffer.size() - 4) != string::npos)
|| (strBuffer.size() > 2 && strBuffer.find("\n\n", strBuffer.size() - 2) != string::npos)
)
{
cout << "message size: " << strBuffer.size() << endl << "message: " << strBuffer << endl;
bReplyOver = true;
if (strBuffer.find("quit") != string::npos)
bQuit = true;
// SSL_write() writes num bytes from the buffer buf into the specified ssl connection.
int result = SSL_write(ssl, reply, strlen(reply));
if (result <= 0)
{
reportError(ssl, result);
bReplyOver = true;
}
break;
}
}
else
{
if (SSL_want_read(ssl))
{
cout << "SSL_want_read" << endl;
}
else
{
reportError(ssl, bytesOut);
bReplyOver = true;
break;
}
if (!bHandShakeOver && SSL_is_init_finished(ssl))
{
cout << "Handshake has been finished" << endl;
bHandShakeOver = true;
char cipdesc[128];
SSL_CIPHER* sc = SSL_get_current_cipher(ssl);
if (sc)
cout << "encryption: " << SSL_CIPHER_description(sc, cipdesc, sizeof(cipdesc)) << endl;
}
break;
}
}
}
else if (nRet == 0)
{ // the connection has been gracefully closed
break;
}
while (1)
{
// BIO_ctrl_pending() returns the number of bytes buffered in a BIO.
size_t pending = BIO_ctrl_pending(bioOut);
if (pending > 0)
{
cout << "BIO_ctrl_pending(bioOut) == " << pending << endl;
// BIO_read() attempts to read len bytes from BIO b and places the data in buf.
int bytesToSend = BIO_read(bioOut, (void*)szBuf, sizeof(szBuf) > pending ? pending : sizeof(szBuf));
if (bytesToSend > 0)
{
cout << "BIO_read(bioOut) == " << bytesToSend << endl;
int sent = 0;
while (1)
{
nRet = send(remoteSocket, szBuf + sent, bytesToSend, 0);
if (nRet == SOCKET_ERROR)
{
bReplyOver = true;
cerr << "send() - SOCKET_ERROR" << endl;
break;
}
else
{
sent += nRet;
bytesToSend -= nRet;
if (bytesToSend == 0)
break;
}
}
}
else if (!BIO_should_retry(bioOut))
{// BIO_should_retry() is true if the call that produced this condition should then be retried at a later time.
reportError(ssl, bytesToSend);
}
}
else
{
cout << "BIO_ctrl_pending(bioOut) == 0" << endl;
break;
}
}
if (bReplyOver)
{
cout << "post-reply" << endl;
break;
}
}
After it's done you have to clean up all resources. If there's a memory leak, CRYPTO_mem_leaks_fp will print it out.
closesocket(remoteSocket);
// this frees associated BIO too, so no need for BIO_free
SSL_free(ssl);
if (bQuit)
break;
}
closesocket(listenSocket);
WSACleanup();
//////////////////////////////////////////////////////////////////////////////////
SSL_CTX_free(ctx);
CRYPTO_set_dynlock_create_callback(NULL);
CRYPTO_set_dynlock_destroy_callback(NULL);
CRYPTO_set_dynlock_lock_callback(NULL);
CRYPTO_set_locking_callback(NULL);
CRYPTO_set_id_callback(NULL);
delete[] openssl_locks;
openssl_locks = 0;
EVP_cleanup();
CRYPTO_cleanup_all_ex_data();
ERR_remove_state(0);
ERR_free_strings();
#ifdef _DEBUG
CRYPTO_mem_leaks_fp(stdout);
#endif
return 0;
}
When you run this sample code it listens to the standard HTTPS port 443. Start with a web browser and open "https://127.0.0.1/" to see how an SSL connection is processed. To stop this server, put "quit" in a request URL ("https://127.0.0.1/quit").
コメント
コメントを投稿