About 2 years ago, I opened an IRC channel on a small IRC network. I'd been hanging out in EFnet, Undernet, or DALnet, but the accessibility to those large networks from Japan was not decent and not much improvement is seen even today. In some networks you are just kicked because you are from the .jp domain. As a consequence, I chose a small but reliable place for the new channel among many networks. It had nickserv and chanserv. One of the largest networks known in Japan, IRCnet, lacked those services.
The Japanese multi-byte encoding standards are split into 3 different encodings. The de jure is JIS and the de facto are Shift-JIS for MS Windows and EUC for Unix. While the old JIS is nearly forced by Unix-flavoured Japanese people in IRCnet, I chose Shift-JIS. My favorite client is mIRC and the version at that time couldn't handle 2 Japanese encodings at the same time, JIS and Shift-JIS (later I requested the mIRC author to add the support for multi-byte strings and Japanese encodings. It was fulfilled, thx KMB u r awesome!!). The IRC protocol itself just processes "octet-stream" and offers nothing related to i18n. Using mIRC without the support for 2 Japanese encodings means mIRC is useless in the JIS IRC world. I was so obsessed with something new and strange that I made my new channel "Shift-JIS-only". Client applications should hide annoying things that break smooth communication. Forcing JIS is not so productive when it has no advantage over the others. For example, today's standard web browsers can handle 3 Japanese encodings and Unicode correctly. I chose Shift-JIS to rebel against the preset standard.
To enforce ShiftJIS in my channel and to punish JIS use, I put an eggdrop bot in my channel. It's programmable via Tcl scripts and can establish its own overlay network over different IRC networks. I had run the channel for 8 months and had to protect it from war bots by hacking an eggdrop bot with my awkward Tcl scripts. I persisted in keeping my channel open and free, but not for abusers. Unfortunately my channel got involved in a war that lasted for a few months but the experience taught me a lot of things. I got very frustrated in IRC itself. Before running a channel I was just a casual IRC user who had no interest in the structure of IRC itself, but suddenly a new idea possessed me - the reform of IRC. I started planning in June 2001.
First of all, I investigated open-source Unix IRCd variants. Implementing pretty features such as unique hostname mask into the existing IRCd code base seemed to be the easiest and promising way. I love new things so I picked up an IRC server that called itself as the new generation or something like that. It had ports to MS Win32 and other OSs. I got its source code and browsed through a pile of C, common Unix programming language. I was daunted.
It was spaghetti! None other than spaghetti! Since I learned C++ without the experience in bare C, a large C code base itself was new to my eyes, but even with that reservation it was spaghetti! It was clear that GNU GPL had huge influence on the status of such a code base. IRC was invented in 1988 by Jarkko Oikarinen and IRCd came out. I thought "IRCd" was the general term for all IRC servers or daemons, but it was a false notion. All IRCd are descendants of the fabulous IRCd source code that is 10 years old. Even the Original IRCd is still maintained. In 10 years many people added patches and many branches were born and died. All IRCd variants represent different facets of an unnamed history. It was like looking at a human body with countless tattoos - beautiful and ugly at the same time. I was almost fascinated in the genealogy of source code. Almost - yes I had no time to be indulged in it! It seemed that reading all the source code of an IRCd variant would take a fairly large amount of time. After all I was convinced that it'd be easier to write a new one from scratch than to understand a complicated mosaic woven in many different styles without refactoring efforts.
Unfortunately (or fortunately, in hindsight) I had too little time to devote to hobby programming. IRCd variants are fruits of 10 years work produced by many skillful programmers. What can I do? I felt no urge to make my software open-source. Making your software open-source without a clear strategy is not a brilliant idea. It's like using something that should be the last resort at the beginning. I couldn't expect team development either. I want no management tasks. Anyway who will participate in developing a program that deals with the stale IRC protocol when good enough open-source implementations are available? Yes, the IRC protocol sucks hard. I'd not read many RFCs, but after reading RFC1459 and RFC 2812, I was convinced that there were good RFCs and bad RFCs. IRC itself is one of the bad protocols in today's standard, and for worse, those RFCs are merely a vague specification meant for the IRCnet IRCd.
The current standard IRC protocol, so-called IRC2, is actively developed and extended in not IRCnet but in the other big 3 networks. They are Hybrid in EFnet, IRCu in Undernet, and bahamut in DALnet. They modified the server-to-server protocol to make it possible to carry more client connections at the cost of interoperability. DALnet and Undernet are spin-offs of EFnet, though according to netsplit.de, DALnet is the largest network as of April 2002, carrying 120,000 clients on 30 servers. I can't help feeling admiration for this great achievement by DALnet people. Thus the bahamut source code became my main reference among various IRCd variants to check how an actual implementation works it out, if the RFC documents fall short. However, I had no time to scan it thoroughly to track down all of its control flow. I had interest in distributed computing and distributed objects at that time and didn't feel like adopting an existing IRCd design and its server-to-server protocol.
Finally I made up my plan.
Phase 1. Develop a robust server for client-server interaction
Phase 2. Add a server-to-server protocol and a clustering scheme
Way simple, huh? I assumed achieving the Phase 1 wouldn't take much time. I wanted something that actually works. Next, programming language. Actually I had no choice, C++ is my only one. I love C++ template, the functional programming style with template, and STL. Before planning the DICE I had read a book for C++ performance tuning, "Efficient C++" by the architect of the IBM Domino-Go Web server and was fairly influenced by it.
I hunted through the web for scarce information of IRC servers that are not IRCd variants. Several IRCd variants have Win32 ports, but I found no open-source non-IRCd-variants for the Win32 platform. (Later I found the cross-platform SIRCD. It's very unique and had an interesting implementation, but cross-platform compatibility requires huge overhead in Windows if you are not careful enough - I'll come back to this issue later in this document) For Win32, there were 3 commercial implementations, IRCPlus, Conference Room, and Microsoft Exchange Server. They are all full of special extensions, especially IRCX, a new binary/UTF-8 based protocol introduced in MS-chat and Exchange Server. Apparently they are products for people who prefer easy visual administration, but I don't share such a preference as my goal, which means I could learn little from them. Well I learned one thing from IRCPlus, it is that adding as many channel modes looks stupid.
I needed a clearer and simpler implementation model to forge my own image of a total application design. Another search found Perl IRC daemon at the Mecca of open-source projects, SourceForge. As Perl is one of favorite scripting languages of mine I examined the project with eagerness. It took the path that looked like mine, i.e. make simple server and then add server-to-server facility. Perl IRC daemon was on the way of the Phase 2 about which I explained above. Browsing it didn't give me much impression and inspiration. I searched C++ IRC server projects, but none was found (there were pre-alpha plans but no actual code). IRC server projects in C were all Unix IRCd variants that fall off my interest. I believe C++ is the best solution to build fairly large projects without spoiling application performance, but it's not so popular in Unix probably because of compatibility issues between Unix variants, and because of immature implementations of the C++ standard. However, the help I needed was found in an unexpected area - Java. There were 2 Java IRC server projects in SourceForge, they were Pastiche IRCd and Sonata IRC Network. Reading through a cute Java code with a wild application design was a pure joy if you ask me. They lacked Phase 2 server-to-server link, but it just helped their design clarity. While the former is not so good in terms of overall design that puts all server work into a single queue for synchronization which would lower its performance, the design of the latter, Sonata IRC Network was so straight that finally I could imagine my own image of a server implementation. It was perfect for me except that it used too many Java exceptions to drive non-error operations and Java ServerSocket can't handle many requests coming at a time.
On some Unix IRCd documents I found sentences that roughly stated "Unix is the platform of IRCd, forget MS Windows." Is it true? Yes on Windows 9X it must be. But how is Windows 2000? I had switched my desktop from Windows 98SE to Windows 2000 in late 2000. I was very impressed by its stability and usability. Hearing that Microsoft aimed at the server market by those NT OSes, I figured out they might add some goodies into NT to surpass Unix. I'd played with Linux and FreeBSD both on my PC by partitioning its HDD to triple-boot with Windows and had some fun, but eventually I found making a cross-platform application was not more than a source of fatigue. You must forget performance to make it compatible. You must add ugly #ifdef macros here and there. Now with Java and .NET mobile code, why consider cross-platform in low-level C++? Mozilla is great because it's a desktop application. Make binary components cross-platform and reusable, but infrastructures at lower layers of network endpoints should exploit system capability to the max without considering portability.
Since I had almost no prior experience about NT system programming I had to go through Windows 2000 reference books. Finally I found the key feature of NT to beat Unix; it's called "I/O completion ports". The explicit I/O completion ports interface was not implemented in the Linux kernel (though there's queued realtime signals) at that time. For example the Windows port of Apache version 2 takes advantage of it (see "mpm_winnt.c"). It allows OS-level adoption of the so-called "proactor pattern" by the ACE fame Douglas C. Schmidt in POSA2 though it's questionable whether it's appropriate to call it a design pattern. As it's not yet known if Windows had actually improved in the effort to catch up with the stability and performance of Unix servers, I wanted to test it by myself. The DICE internal is built fully utilizing proprietary APIs and will be 0%-compatible with Unix. I know the existence of the ACE framework for cross-platform C++, but this time it's not my choice since it's too complicated with unwanted OO overhead. Portability issues were gone and I was happy. All I/O in the DICE are constructed by I/O completion ports; there's absolutely no exception. No need for message pumping. Gradually I realized certain inclination in the APIs newly added to the Platform SDK. I/O completion ports, APC, thread-pooling... all were connected together and I could get the picture of the new world of Windows, free from POSIX APIs and BSD socket. .NET framework shares the same vision with the "delegate" feature (not by inner-class of Java). The DICE is a server application born out of the pure MS Windows world and it is one aspect of the DICE uniqueness. MS Windows has the huge install base and Windows XP is NT-based. I hoped my software would be accepted by the broader audience.
The new plan is:
Phase 1. Make a scalable Windows server for client-server interaction
Phase 2. Add a server-to-server protocol and a clustering scheme
The programming language and the target platform were fixed. How about the design of the application? I almost forgot it when searching required tools... Anyway it's not much fun to implement IRC as told by the RFC document without valuable modifications. My intention is to reform the IRC as stated above. This time I searched what was going on at the front of IRC protocol development to see the latest trend. As an ordinary IRC user I'd known nothing about it. The new revelation shocked me - there was no such major movement. Since the current state of IRC2 is stable enough, there seems to be little pressure to update it in a significant way. The new version of IRC, so-called IRC3, is planned by Andrew Church in a form of a tentative RFC. IMHO it resembles current IRC2 too much and looks a little too conservative. Another search found discussions of IRC3 at the now defunct the-project.org, but it was old and apparently dead. Sadly the fact is no one wants IRC3 and there is no actual implementation of an IRC3 client or an IRC3 server. IRC2 works fine and its server-to-server protocol has a few points to be improved like the EFnet timestamp protocol. To move to IRC3, an influential client software such as mIRC must support it while there must be a reliable software implementation of a server application. Does it happen in the near future? I'm pessimistic.
But there's another new trend in IRC. That is IRC over SSL. From a technical point of view using SSL in IRC costs so much that scalability will be totally lost. To interpret IRC messages from clients, a server has to decrypt entire messages one by one in an SSL stream. Scalability and security, 2 virtues of a network server, are already lacking in the IRC2 and are trade-off each other. In the case of IRC over SSL it's obvious that scalability will be lost forever. In other words, how much security do you need in a light-weight chat protocol? For example HTTP isn't always encrypted in SSL. Definitely it's a matter of balance. I found a good analysis on current IRC2 problems by Liam Quin, it's a good read to get a grip on the relationship between various problems of the current IRC.
I already wrote that no one would care about IRC3, but there's another candidate that may replace IRC2 one day if you don't stick to the name "IRC". SILC, Secure Internet Live Conferencing, which was apparently started in 1996, has a very similar goal to0 that of IRC3. It has its focus heavily on security, but looks like scalability is also on their minds though there ostensibly are some bottlenecks. It appears not bad - but I'm skeptical. As far as I know one of the reasons why IRC got so popular is because programming a basic IRC client was not so difficult. If you know IRC commands, you can do IRC even with a telnet terminal. Debugging a server is easy too because no special filter is required to parse a text-based message. Simplicity matters. As SILC had released its open toolkit, it may not matter, but I'll keep an eye on how it tackles with this issue.
Though I put off the server-to-server issue for the time being, I had to sort out what I should do to set up smooth transition to distributed computing. Reusable existing technologies could be examined to reduce implementation difficulties. CORBA and DCOM (COM+) are the established protocols for distributed objects, but marshalling overhead for a general RPC protocol is not trivial unless you do custom marshalling. As IRC is nearly realtime, the merit of adopting those protocols are offset by poor performance. In addition the particular experience that defines IRC is not about many objects scattered across the network, but about sharing and maintaining one integrated image of a channel in near-realtime. All distribution efforts in IRC are meant to keep channel integrity though I myself am a little skeptical about the necessity to have one channel over multiple servers at one particular point of time. The context is different between IRC and distributed object protocols. As for DCOM, secure connection between distributed objects is possible without implementing it into your own application, but you are forced to be bound by the web of complicated proprietary technologies. It's not favorable that DICE users are forced to configure their Windows servers to create a DCOM domain. A similar problem occurs in using Kerberos since it can't work without Windows Forest of ActiveDirectory. After all, these technologies were not without caveats. Instead, I turned my attention on different aspects of a network server. A hint was found while reading the discussion about IRC3. There were several different proposals for IRC3, among which the one that was most interesting was Ian's 3 Tier + DB Approach. It seemed realistic and doable. I decided to use a 128-bit UUID as an IRC component ID and decided to use an RDBMS as a backend persistence store. In the local C++ memory space of a 32-bit environment a 32-bit pointer is an object ID, but dynamic object reallocation may happen across a distributed network. Besides ID for objects that belong to different classes can't be confused. It's neither object ID nor CLSID, rather I call it "component ID".
Which RDBMS should I use? PostgreSQL had no Windows port at that time. Naturally the selection was narrowed between MySQL and MSDE / MS SQL Server. Recently a Windows port of MySQL was made free and the license was changed to the dual-license including GPL. But I felt uneasy about its stability and APIs. MS SQL Server is expensive and not affordable but it has a free version called MSDE. Programming with MS SQL Server is easy enough to save time. I heard that MSDE was 99% identical to MS SQL Server without support tools. The DICE uses MS SQL Server for the time being, probably migration to other alternatives may be examined after the DICE reaches the version 1.0. Eventually it took nearly a month to select RDBMS.
All necessary tools were in my hand, the rest was just the code. Instead of replacing all, the reform of IRC can be accomplished by fixing minor flaws too. There were many things to be fixed... Nickserv and chanserv would be incorparated into a single IRC-user shell interface. Users would have their accounts in the persistent IRC-user shell and all management tasks would be done through single authentication for their accounts. Channel members are not recognized by their nicks or hostmasks but by their accounts. Host names are not revealed in public to keep privacy and anonymity. Instead of a partial mask that only hides variable portion of hostname, all host name is hashed without losing its uniqueness to make a ban mask work. The "+p" for the private channel mode is removed because there's "+s" for a secret channel and that's enough. All commands related to IRC "services" are removed. An ordinary IRCd world is composed of IRCd itself and other services. Those services, nickserv, chanserv, or proxy-scanner, are usually executed from other machines outside of an IRCd box. It's just like a micro-kernel in the OS model with one small kernel and many services/daemons running around. Independent IRC services can be replaced without rebuild or reboot of an IRCd. An abnormal behavior of service processes doesn't have an effect on a stable IRCd process. But the communication between IRCd and services is done in IRC messages that have the same format as the protocol between IRCd and clients. There may be an implementation that uses a different channel for IRCd-service communication, but apparently the implementation described in RFC doesn't work like that.
On the contrary the DICE takes the monolithic-kernel approach to improve its performance. All user services are executed in the kernel space, in other words there's no specific "service". Instead of having services in the user space, those services are threads in the server process and can exploit a multi-processor setup because of the integrated architecture of the Windows I/O completion port and its thread-pooling model. To a user, all services are revealed in the "/shell" command as a single interface. Administration of the DICE is secure over the internet with SSL. Frustrated about the IRC packet format I made another new format and adopted it for the new binary protocol of the DICE Administration Shell. In the world of the DICE administration shell a string is UTF-16 Unicode and users are identified by its UUID. They can have aliases that may be shared by multiple users unlike the IRC based on the identification by unique ASCII-string nicknames. The protocol for the DICE administration service has a strict RPC style with which all human-readable string messages that describe what's happenning at the server-side are generated at the client-side unlike those of IRC. The DICE was my laboratory to test new protocols and my experimental hacking ground for all fields of network programming. For internationalization in IRC, the DICE can carry multiple string resources to let users switch their locales dynamically.
After planning the design for weeks, I began actual coding. First, I wrote a generic asynchronous I/O server. Then added a generic stateful session class. What's added the next was an expandable session pool/container. It's like a beehive of which core is the place for a queen bee as a server with many little bees around as the server-side representation of client connections. To interpret IRC messages sent from connected clients, a finite state machine to parse and interpret IRC messages was embedded in a session object. An interpreted message must generate a server reply or fire certain action at server-side; they are associated with message-handler methods of an IRC session object. Channel objects remember the state of the association between clients. After many, many IRC-message-handler methods were implemented, persistence of user properties and channel properties were implemented with a database connector. The GUI client for the administration shell required another class of a session object at the server-side. It took 6 months to develop this basic server implementation that satisfies the aforementioned Phase 1.
A week before the first public release of the DICE, I looked at the C source code of opennap 0.44 without particular interest. An opennap server itself has nothing P2P except for its clustering feature while clients do P2P connection. It has many things in common with an IRC server. The size of the source code for the opennap server was unexpectedly small. Suddenly an idea struck me; how about implementing opennap in the DICE? Since the DICE framework is generic enough to hold any stateful session objects, it would be able to handle opennap clients with little server-side modification such as the addition of an opennap message parser. Moreover the DICE has a backend database that can be useful to store user directory lists. I bound the opennap service to IRC by making people in "+x" mode channels possible to join in the opennap-compatible space called "VirtualDirectory". It took only one week to implement the opennap function in the DICE and it was a great proof for the applicability of my framework.
Where will the DICE go? I don't know, since the development is not active right now. It'll take at least 3 or 4 months or more to design the topology of DICE clusters. The actual implementation will take more. It may die without reaching the Phase 2. Or IRC may be completely removed from the new DICE. Please note only one thing, that the DICE is not freeware for all - it's very closely connected and restricted to my own body and mental state. It's humanware - living not with its own life, but with the life of the author. I promise you nothing, except that I see you one day again, even in a different color of hair, in a different name, or in another cycle of reincarnation... bye for now.
The Japanese multi-byte encoding standards are split into 3 different encodings. The de jure is JIS and the de facto are Shift-JIS for MS Windows and EUC for Unix. While the old JIS is nearly forced by Unix-flavoured Japanese people in IRCnet, I chose Shift-JIS. My favorite client is mIRC and the version at that time couldn't handle 2 Japanese encodings at the same time, JIS and Shift-JIS (later I requested the mIRC author to add the support for multi-byte strings and Japanese encodings. It was fulfilled, thx KMB u r awesome!!). The IRC protocol itself just processes "octet-stream" and offers nothing related to i18n. Using mIRC without the support for 2 Japanese encodings means mIRC is useless in the JIS IRC world. I was so obsessed with something new and strange that I made my new channel "Shift-JIS-only". Client applications should hide annoying things that break smooth communication. Forcing JIS is not so productive when it has no advantage over the others. For example, today's standard web browsers can handle 3 Japanese encodings and Unicode correctly. I chose Shift-JIS to rebel against the preset standard.
To enforce ShiftJIS in my channel and to punish JIS use, I put an eggdrop bot in my channel. It's programmable via Tcl scripts and can establish its own overlay network over different IRC networks. I had run the channel for 8 months and had to protect it from war bots by hacking an eggdrop bot with my awkward Tcl scripts. I persisted in keeping my channel open and free, but not for abusers. Unfortunately my channel got involved in a war that lasted for a few months but the experience taught me a lot of things. I got very frustrated in IRC itself. Before running a channel I was just a casual IRC user who had no interest in the structure of IRC itself, but suddenly a new idea possessed me - the reform of IRC. I started planning in June 2001.
First of all, I investigated open-source Unix IRCd variants. Implementing pretty features such as unique hostname mask into the existing IRCd code base seemed to be the easiest and promising way. I love new things so I picked up an IRC server that called itself as the new generation or something like that. It had ports to MS Win32 and other OSs. I got its source code and browsed through a pile of C, common Unix programming language. I was daunted.
It was spaghetti! None other than spaghetti! Since I learned C++ without the experience in bare C, a large C code base itself was new to my eyes, but even with that reservation it was spaghetti! It was clear that GNU GPL had huge influence on the status of such a code base. IRC was invented in 1988 by Jarkko Oikarinen and IRCd came out. I thought "IRCd" was the general term for all IRC servers or daemons, but it was a false notion. All IRCd are descendants of the fabulous IRCd source code that is 10 years old. Even the Original IRCd is still maintained. In 10 years many people added patches and many branches were born and died. All IRCd variants represent different facets of an unnamed history. It was like looking at a human body with countless tattoos - beautiful and ugly at the same time. I was almost fascinated in the genealogy of source code. Almost - yes I had no time to be indulged in it! It seemed that reading all the source code of an IRCd variant would take a fairly large amount of time. After all I was convinced that it'd be easier to write a new one from scratch than to understand a complicated mosaic woven in many different styles without refactoring efforts.
Unfortunately (or fortunately, in hindsight) I had too little time to devote to hobby programming. IRCd variants are fruits of 10 years work produced by many skillful programmers. What can I do? I felt no urge to make my software open-source. Making your software open-source without a clear strategy is not a brilliant idea. It's like using something that should be the last resort at the beginning. I couldn't expect team development either. I want no management tasks. Anyway who will participate in developing a program that deals with the stale IRC protocol when good enough open-source implementations are available? Yes, the IRC protocol sucks hard. I'd not read many RFCs, but after reading RFC1459 and RFC 2812, I was convinced that there were good RFCs and bad RFCs. IRC itself is one of the bad protocols in today's standard, and for worse, those RFCs are merely a vague specification meant for the IRCnet IRCd.
The current standard IRC protocol, so-called IRC2, is actively developed and extended in not IRCnet but in the other big 3 networks. They are Hybrid in EFnet, IRCu in Undernet, and bahamut in DALnet. They modified the server-to-server protocol to make it possible to carry more client connections at the cost of interoperability. DALnet and Undernet are spin-offs of EFnet, though according to netsplit.de, DALnet is the largest network as of April 2002, carrying 120,000 clients on 30 servers. I can't help feeling admiration for this great achievement by DALnet people. Thus the bahamut source code became my main reference among various IRCd variants to check how an actual implementation works it out, if the RFC documents fall short. However, I had no time to scan it thoroughly to track down all of its control flow. I had interest in distributed computing and distributed objects at that time and didn't feel like adopting an existing IRCd design and its server-to-server protocol.
Finally I made up my plan.
Phase 1. Develop a robust server for client-server interaction
Phase 2. Add a server-to-server protocol and a clustering scheme
Way simple, huh? I assumed achieving the Phase 1 wouldn't take much time. I wanted something that actually works. Next, programming language. Actually I had no choice, C++ is my only one. I love C++ template, the functional programming style with template, and STL. Before planning the DICE I had read a book for C++ performance tuning, "Efficient C++" by the architect of the IBM Domino-Go Web server and was fairly influenced by it.
I hunted through the web for scarce information of IRC servers that are not IRCd variants. Several IRCd variants have Win32 ports, but I found no open-source non-IRCd-variants for the Win32 platform. (Later I found the cross-platform SIRCD. It's very unique and had an interesting implementation, but cross-platform compatibility requires huge overhead in Windows if you are not careful enough - I'll come back to this issue later in this document) For Win32, there were 3 commercial implementations, IRCPlus, Conference Room, and Microsoft Exchange Server. They are all full of special extensions, especially IRCX, a new binary/UTF-8 based protocol introduced in MS-chat and Exchange Server. Apparently they are products for people who prefer easy visual administration, but I don't share such a preference as my goal, which means I could learn little from them. Well I learned one thing from IRCPlus, it is that adding as many channel modes looks stupid.
I needed a clearer and simpler implementation model to forge my own image of a total application design. Another search found Perl IRC daemon at the Mecca of open-source projects, SourceForge. As Perl is one of favorite scripting languages of mine I examined the project with eagerness. It took the path that looked like mine, i.e. make simple server and then add server-to-server facility. Perl IRC daemon was on the way of the Phase 2 about which I explained above. Browsing it didn't give me much impression and inspiration. I searched C++ IRC server projects, but none was found (there were pre-alpha plans but no actual code). IRC server projects in C were all Unix IRCd variants that fall off my interest. I believe C++ is the best solution to build fairly large projects without spoiling application performance, but it's not so popular in Unix probably because of compatibility issues between Unix variants, and because of immature implementations of the C++ standard. However, the help I needed was found in an unexpected area - Java. There were 2 Java IRC server projects in SourceForge, they were Pastiche IRCd and Sonata IRC Network. Reading through a cute Java code with a wild application design was a pure joy if you ask me. They lacked Phase 2 server-to-server link, but it just helped their design clarity. While the former is not so good in terms of overall design that puts all server work into a single queue for synchronization which would lower its performance, the design of the latter, Sonata IRC Network was so straight that finally I could imagine my own image of a server implementation. It was perfect for me except that it used too many Java exceptions to drive non-error operations and Java ServerSocket can't handle many requests coming at a time.
On some Unix IRCd documents I found sentences that roughly stated "Unix is the platform of IRCd, forget MS Windows." Is it true? Yes on Windows 9X it must be. But how is Windows 2000? I had switched my desktop from Windows 98SE to Windows 2000 in late 2000. I was very impressed by its stability and usability. Hearing that Microsoft aimed at the server market by those NT OSes, I figured out they might add some goodies into NT to surpass Unix. I'd played with Linux and FreeBSD both on my PC by partitioning its HDD to triple-boot with Windows and had some fun, but eventually I found making a cross-platform application was not more than a source of fatigue. You must forget performance to make it compatible. You must add ugly #ifdef macros here and there. Now with Java and .NET mobile code, why consider cross-platform in low-level C++? Mozilla is great because it's a desktop application. Make binary components cross-platform and reusable, but infrastructures at lower layers of network endpoints should exploit system capability to the max without considering portability.
Since I had almost no prior experience about NT system programming I had to go through Windows 2000 reference books. Finally I found the key feature of NT to beat Unix; it's called "I/O completion ports". The explicit I/O completion ports interface was not implemented in the Linux kernel (though there's queued realtime signals) at that time. For example the Windows port of Apache version 2 takes advantage of it (see "mpm_winnt.c"). It allows OS-level adoption of the so-called "proactor pattern" by the ACE fame Douglas C. Schmidt in POSA2 though it's questionable whether it's appropriate to call it a design pattern. As it's not yet known if Windows had actually improved in the effort to catch up with the stability and performance of Unix servers, I wanted to test it by myself. The DICE internal is built fully utilizing proprietary APIs and will be 0%-compatible with Unix. I know the existence of the ACE framework for cross-platform C++, but this time it's not my choice since it's too complicated with unwanted OO overhead. Portability issues were gone and I was happy. All I/O in the DICE are constructed by I/O completion ports; there's absolutely no exception. No need for message pumping. Gradually I realized certain inclination in the APIs newly added to the Platform SDK. I/O completion ports, APC, thread-pooling... all were connected together and I could get the picture of the new world of Windows, free from POSIX APIs and BSD socket. .NET framework shares the same vision with the "delegate" feature (not by inner-class of Java). The DICE is a server application born out of the pure MS Windows world and it is one aspect of the DICE uniqueness. MS Windows has the huge install base and Windows XP is NT-based. I hoped my software would be accepted by the broader audience.
The new plan is:
Phase 1. Make a scalable Windows server for client-server interaction
Phase 2. Add a server-to-server protocol and a clustering scheme
The programming language and the target platform were fixed. How about the design of the application? I almost forgot it when searching required tools... Anyway it's not much fun to implement IRC as told by the RFC document without valuable modifications. My intention is to reform the IRC as stated above. This time I searched what was going on at the front of IRC protocol development to see the latest trend. As an ordinary IRC user I'd known nothing about it. The new revelation shocked me - there was no such major movement. Since the current state of IRC2 is stable enough, there seems to be little pressure to update it in a significant way. The new version of IRC, so-called IRC3, is planned by Andrew Church in a form of a tentative RFC. IMHO it resembles current IRC2 too much and looks a little too conservative. Another search found discussions of IRC3 at the now defunct the-project.org, but it was old and apparently dead. Sadly the fact is no one wants IRC3 and there is no actual implementation of an IRC3 client or an IRC3 server. IRC2 works fine and its server-to-server protocol has a few points to be improved like the EFnet timestamp protocol. To move to IRC3, an influential client software such as mIRC must support it while there must be a reliable software implementation of a server application. Does it happen in the near future? I'm pessimistic.
But there's another new trend in IRC. That is IRC over SSL. From a technical point of view using SSL in IRC costs so much that scalability will be totally lost. To interpret IRC messages from clients, a server has to decrypt entire messages one by one in an SSL stream. Scalability and security, 2 virtues of a network server, are already lacking in the IRC2 and are trade-off each other. In the case of IRC over SSL it's obvious that scalability will be lost forever. In other words, how much security do you need in a light-weight chat protocol? For example HTTP isn't always encrypted in SSL. Definitely it's a matter of balance. I found a good analysis on current IRC2 problems by Liam Quin, it's a good read to get a grip on the relationship between various problems of the current IRC.
I already wrote that no one would care about IRC3, but there's another candidate that may replace IRC2 one day if you don't stick to the name "IRC". SILC, Secure Internet Live Conferencing, which was apparently started in 1996, has a very similar goal to0 that of IRC3. It has its focus heavily on security, but looks like scalability is also on their minds though there ostensibly are some bottlenecks. It appears not bad - but I'm skeptical. As far as I know one of the reasons why IRC got so popular is because programming a basic IRC client was not so difficult. If you know IRC commands, you can do IRC even with a telnet terminal. Debugging a server is easy too because no special filter is required to parse a text-based message. Simplicity matters. As SILC had released its open toolkit, it may not matter, but I'll keep an eye on how it tackles with this issue.
Though I put off the server-to-server issue for the time being, I had to sort out what I should do to set up smooth transition to distributed computing. Reusable existing technologies could be examined to reduce implementation difficulties. CORBA and DCOM (COM+) are the established protocols for distributed objects, but marshalling overhead for a general RPC protocol is not trivial unless you do custom marshalling. As IRC is nearly realtime, the merit of adopting those protocols are offset by poor performance. In addition the particular experience that defines IRC is not about many objects scattered across the network, but about sharing and maintaining one integrated image of a channel in near-realtime. All distribution efforts in IRC are meant to keep channel integrity though I myself am a little skeptical about the necessity to have one channel over multiple servers at one particular point of time. The context is different between IRC and distributed object protocols. As for DCOM, secure connection between distributed objects is possible without implementing it into your own application, but you are forced to be bound by the web of complicated proprietary technologies. It's not favorable that DICE users are forced to configure their Windows servers to create a DCOM domain. A similar problem occurs in using Kerberos since it can't work without Windows Forest of ActiveDirectory. After all, these technologies were not without caveats. Instead, I turned my attention on different aspects of a network server. A hint was found while reading the discussion about IRC3. There were several different proposals for IRC3, among which the one that was most interesting was Ian's 3 Tier + DB Approach. It seemed realistic and doable. I decided to use a 128-bit UUID as an IRC component ID and decided to use an RDBMS as a backend persistence store. In the local C++ memory space of a 32-bit environment a 32-bit pointer is an object ID, but dynamic object reallocation may happen across a distributed network. Besides ID for objects that belong to different classes can't be confused. It's neither object ID nor CLSID, rather I call it "component ID".
Which RDBMS should I use? PostgreSQL had no Windows port at that time. Naturally the selection was narrowed between MySQL and MSDE / MS SQL Server. Recently a Windows port of MySQL was made free and the license was changed to the dual-license including GPL. But I felt uneasy about its stability and APIs. MS SQL Server is expensive and not affordable but it has a free version called MSDE. Programming with MS SQL Server is easy enough to save time. I heard that MSDE was 99% identical to MS SQL Server without support tools. The DICE uses MS SQL Server for the time being, probably migration to other alternatives may be examined after the DICE reaches the version 1.0. Eventually it took nearly a month to select RDBMS.
All necessary tools were in my hand, the rest was just the code. Instead of replacing all, the reform of IRC can be accomplished by fixing minor flaws too. There were many things to be fixed... Nickserv and chanserv would be incorparated into a single IRC-user shell interface. Users would have their accounts in the persistent IRC-user shell and all management tasks would be done through single authentication for their accounts. Channel members are not recognized by their nicks or hostmasks but by their accounts. Host names are not revealed in public to keep privacy and anonymity. Instead of a partial mask that only hides variable portion of hostname, all host name is hashed without losing its uniqueness to make a ban mask work. The "+p" for the private channel mode is removed because there's "+s" for a secret channel and that's enough. All commands related to IRC "services" are removed. An ordinary IRCd world is composed of IRCd itself and other services. Those services, nickserv, chanserv, or proxy-scanner, are usually executed from other machines outside of an IRCd box. It's just like a micro-kernel in the OS model with one small kernel and many services/daemons running around. Independent IRC services can be replaced without rebuild or reboot of an IRCd. An abnormal behavior of service processes doesn't have an effect on a stable IRCd process. But the communication between IRCd and services is done in IRC messages that have the same format as the protocol between IRCd and clients. There may be an implementation that uses a different channel for IRCd-service communication, but apparently the implementation described in RFC doesn't work like that.
On the contrary the DICE takes the monolithic-kernel approach to improve its performance. All user services are executed in the kernel space, in other words there's no specific "service". Instead of having services in the user space, those services are threads in the server process and can exploit a multi-processor setup because of the integrated architecture of the Windows I/O completion port and its thread-pooling model. To a user, all services are revealed in the "/shell" command as a single interface. Administration of the DICE is secure over the internet with SSL. Frustrated about the IRC packet format I made another new format and adopted it for the new binary protocol of the DICE Administration Shell. In the world of the DICE administration shell a string is UTF-16 Unicode and users are identified by its UUID. They can have aliases that may be shared by multiple users unlike the IRC based on the identification by unique ASCII-string nicknames. The protocol for the DICE administration service has a strict RPC style with which all human-readable string messages that describe what's happenning at the server-side are generated at the client-side unlike those of IRC. The DICE was my laboratory to test new protocols and my experimental hacking ground for all fields of network programming. For internationalization in IRC, the DICE can carry multiple string resources to let users switch their locales dynamically.
After planning the design for weeks, I began actual coding. First, I wrote a generic asynchronous I/O server. Then added a generic stateful session class. What's added the next was an expandable session pool/container. It's like a beehive of which core is the place for a queen bee as a server with many little bees around as the server-side representation of client connections. To interpret IRC messages sent from connected clients, a finite state machine to parse and interpret IRC messages was embedded in a session object. An interpreted message must generate a server reply or fire certain action at server-side; they are associated with message-handler methods of an IRC session object. Channel objects remember the state of the association between clients. After many, many IRC-message-handler methods were implemented, persistence of user properties and channel properties were implemented with a database connector. The GUI client for the administration shell required another class of a session object at the server-side. It took 6 months to develop this basic server implementation that satisfies the aforementioned Phase 1.
A week before the first public release of the DICE, I looked at the C source code of opennap 0.44 without particular interest. An opennap server itself has nothing P2P except for its clustering feature while clients do P2P connection. It has many things in common with an IRC server. The size of the source code for the opennap server was unexpectedly small. Suddenly an idea struck me; how about implementing opennap in the DICE? Since the DICE framework is generic enough to hold any stateful session objects, it would be able to handle opennap clients with little server-side modification such as the addition of an opennap message parser. Moreover the DICE has a backend database that can be useful to store user directory lists. I bound the opennap service to IRC by making people in "+x" mode channels possible to join in the opennap-compatible space called "VirtualDirectory". It took only one week to implement the opennap function in the DICE and it was a great proof for the applicability of my framework.
Where will the DICE go? I don't know, since the development is not active right now. It'll take at least 3 or 4 months or more to design the topology of DICE clusters. The actual implementation will take more. It may die without reaching the Phase 2. Or IRC may be completely removed from the new DICE. Please note only one thing, that the DICE is not freeware for all - it's very closely connected and restricted to my own body and mental state. It's humanware - living not with its own life, but with the life of the author. I promise you nothing, except that I see you one day again, even in a different color of hair, in a different name, or in another cycle of reincarnation... bye for now.
コメント
コメントを投稿