Wednesday, June 28, 2017

TCP / UDP Gateway

I work on UDP networking code near daily for my day job, but I have some use cases for TCP networking in my Orbital Aero among other projects. The Orbital Aero code actually broadcasts UDP states of all its bodies now, and I'll go over its networking soon. However I want the Orbital Aero to have comprehensive networking capabilities, so both TCP and UDP are on its development track. The most fundamental difference between UDP and TCP is whether a connection is maintained or not.


UDP (User Datagram Protocol) is a connectionless protocol. In UDP, messages are sent to a unicast address (one computer), to a broadcast address (all computers on the network), or to a multicast address (all computers subscribed to those messages on the network). The sender has no knowledge if the packet was received by any recipients (unless code is specifically written by the receiver to confirm receipt, but even then that receipt packet may not make it to the sender!). The sender also doesn't even know what recipients are out there (again unless code is specifically written by the receivers to announce themselves, but even then those packets may not make it to the sender!). Although the design of a UDP network may specify a server and clients, that's entirely up to how it was created by the programmer. A large UDP network can just as easily have no single server.


TCP (Transmission Control Protocol) is a connection based protocol. In TCP, a connection is established and maintained between two computers. If a message is sent from one computer to another, the underlying network infrastructure confirms that the message made it through. TCP setups typically have a server and one or many clients that may connect to it.

So if TCP (basically) guarantees delivery and UDP doesn't, why would you choose UDP? It all depends on the use-case. If I was creating a chat program between two people, it would be very important that every message made it through. If a message failed to arrive, I'd want the sending software to know that so it can gracefully handle the situation (try connecting / sending again later, report that the message failed to arrive, try an alternate means of delivery, etc.). But consider simulation software that is broadcasting the positions of hundreds of simulated bodies orbiting each other at real time in space. If the location of a satellite in orbit fails to make it to its destination for any reason (packet dropped, network momentarily interrupted, etc.), there is no need for the sender to re-send that message because the satellite has already moved. And likely the sender has already sent a new location for that satellite that is now more accurate than the lost one anyways. We wouldn't want the overhead of TCP to know an old value, we'd rather just have the latest UDP state.

In the following code I created a simple network gateway for UDP traffic using a TCP server to cross between separate networks. Say network A has a UDP network sharing simulation bodies, network B has a UDP network sharing simulation bodies, and network C has a UDP network sharing simulation bodies. Those networks may be separated down the hallway or across the country, but they all have access to one another, such as via the Internet. I can create a server (at A, B, C, or a completely separate location) to share those UDP messages as though they were all on the same network. Basically a VERY simple VPN (Virtual Private Network).


There are abundant examples online of a TCP server with a single client so no need for a tutorial on that. There are a few examples online of a TCP server with multiple clients, but they were typically way overkill and/or far too much for a teaching aid. I think the following is a fairly straightforward example of a multi-client TCP server. It also includes code for sending and receiving UDP messages so it doubles as a tutorial for that.

Source Code


Below is my explanation of the source code, but you can find the complete files at:
http://kamoly.space/projects/tcp_udp_gateway/

Gateway Server



Headers


This example was written for Windows, but I'll revisit it later to add Linux support. In the mean time, here are the includes you need to compile everything in Windows.


#define WIN32_LEAN_AND_MEAN

#include <process.h>
#include <windows.h>
#include <winsock2.h>
#include <ws2tcpip.h>
#include <stdlib.h>
#include <stdio.h>
#include <vector>
#include <thread>
#include <mutex>
#include <iostream>

#pragma comment (lib, "Ws2_32.lib")

Constants


// Constant denoting the maximum packet size that can be handled.


const int maximumPacketLength_bytes = 8192;

Connection Class


I created a ConnectionModel that can hold any sorts of information I want to keep associated with a connection. For now it's very basic - only keeping track of the socket the client is connected on. However it could be expanded to include a username of who connected, what time they connected, what part of the service they're using, etc.


// Class that stores information related to each of the established connections.
class ConnectionModel
{
public:
    SOCKET clientSocket;
    // Add other variables as desired associated with this client connection.

    // Constructor defining the client socket.
    ConnectionModel(SOCKET newClientSocket)
    {
        this->clientSocket = newClientSocket;
    }
};

Prototypes


I don't have a separate header file to include for the Gateway Server (it's a very small tutorial app), so all the function prototypes are just near the top.


bool sendPacketToClients(ConnectionModel* sourceConnection, char packetData[maximumPacketLength_bytes], int packetLength_bytes);
ConnectionModel* addClient(SOCKET clientSocket);
bool removeClient(ConnectionModel* clientConnection);
void tcpClientListener(SOCKET* listenSocket);
void tcpPacketListener(ConnectionModel* clientConnection);
bool disconnectAllClients();

Global Variables


The Gateway Server has 2 global variables: One is vector (list) that contains pointers to all the client connections. The second is a mutex lock to prevent modifications to the connection list while another part of the server is iterating through it.


// Multithreading lock to protect changes to connections list while iterating through it.
std::mutex clientConnectionsMutex;

// List that holds all client connections.
std::vector<ConnectionModel*> clientConnections;

Main


Now that we're passed the infrastructure setup, we'll begin with the main function! This is a console based application that takes in a single command line argument for the listen port. If no port is given, then it listens on port 4000.


// Main takes a single command line argument for the listen port.
int main(int argc, char *argv[])
{
    // Determine the TCP port to listen for connections.
    // Default to 4000 unless a command line argument is given with another value.
    int tcpPort = 4000;
    if (argc >= 2)
    {
        // Port given, attempt to use it.
        tcpPort = atoi(argv[1]);
    }
    else
    {
        // If no arguments are given, show the format.
        std::cout << "Command line arguments:" << std::endl;
        std::cout << "<TCP Port>" << std::endl;
    }

    // Bound checking for given TCP port.
    if (tcpPort <= 0 || tcpPort >= 65535)
    {
        tcpPort = 4000;
    }
    std::cout << "Listening on port: " << tcpPort << std::endl;


For Windows, we'll need to initialize Winsock.


    // Initialize Winsock.
    WSADATA wsaData;
    int wsaStartupReturn = WSAStartup(MAKEWORD(2, 2), &wsaData);
    if (wsaStartupReturn != 0)
    {
        std::cout << "WSAStartup failed." << std::endl;
        return 1;
    }


Next we'll define an addrinfo with our hints for the TCP server.


    // Setup address info hints.
    struct addrinfo addressInfoHints;
    memset(&addressInfoHints, 0, sizeof(addressInfoHints));
    addressInfoHints.ai_family = AF_INET;
    addressInfoHints.ai_socktype = SOCK_STREAM;
    addressInfoHints.ai_protocol = IPPROTO_TCP;
    addressInfoHints.ai_flags = AI_PASSIVE;

    // Get address TCP port string.
    char portString[20];
    sprintf(portString, "%d", tcpPort);

    // Resolve the server address and port.
    struct addrinfo* addressInfoResult = NULL;
    int getaddrinfoReturn = getaddrinfo(NULL, portString, &addressInfoHints, &addressInfoResult);
    if (getaddrinfoReturn != 0)
    {
        std::cout << "getaddrinfo failed." << std::endl;
        WSACleanup();
        return 1;
    }


The addressInfoResult from getaddrinfo is used to create and bind the listenSocket.


    // Create a TCP socket to listen for clients.
    SOCKET listenSocket = socket(addressInfoResult->ai_family, addressInfoResult->ai_socktype, addressInfoResult->ai_protocol);
    if (listenSocket == INVALID_SOCKET)
    {
        std::cout << "listen socket failed." << std::endl;
        freeaddrinfo(addressInfoResult);
        WSACleanup();
        return 1;
    }

    // Bind the TCP socket to the address.
    int bindReturn = bind(listenSocket, addressInfoResult->ai_addr, (int)addressInfoResult->ai_addrlen);
    if (bindReturn == SOCKET_ERROR)
    {
        std::cout << "bind socket failed." << std::endl;
        freeaddrinfo(addressInfoResult);
        closesocket(listenSocket);
        WSACleanup();
        return 1;
    }

    freeaddrinfo(addressInfoResult);


At this point our server has a listen socket defined and is ready to accept connections. Let's ensure our connections list is clear and kickoff a thread to listen for connections.


    // Initialize / clear the connections list.
    clientConnections.clear();

    // Start the TCP Client Listener thread to listen for connections.
    std::thread{ tcpClientListener, &listenSocket }.detach();


The real work of this gateway is done on the tcpClientListener thread and the connection threads that it spools off. This main thread just needs to sit here and look pretty until the user wants to close the gateway (by pushing a key on the keyboard).


    // All client listening and packet relaying is done on other threads.
    // Continue the program until the user presses a key on the main thread.
    std::cout << "Press any key to quit." << std::endl;
    getchar();


Instead of just waiting on a getchar(), I could make a whole menu with server relevant options for the user. Perhaps an option to list active connections or another to see how much data has been sent. Since the connection management is done on other threads, there is lots of freedom here.

For now, once the user has pressed a key on the server signifying they want to close the app, go ahead and close the listener socket (so we stop receiving new connections) and disconnect all the clients.


    // Close the listen socket to stop receiving new connections.
    closesocket(listenSocket);

    // Disconnect all clients.
    disconnectAllClients();


The listen and client threads have wait locks where they wait for either a new client connection or a new messages from one of their clients in a continuous loop. If their socket is closed, then they get kicked out of that wait lock and are able to cleanly exit their thread. We'll sleep here for 100 milliseconds to let all those threads cleanly exit, but honestly it's way more time than we need to delay.


    // Wait for socket threads to conclude. (Disconnecting above causes them to break out of their socket waits.)
    Sleep(100);


For Windows we'll call the Winsock cleanup, but that's it for our main!


    WSACleanup();

    return 0;
}



TCP Client Listener Thread


The tcpClientListener thread is spooled off from the main function; its duty is to listen for and establish new connections.

// Thread function listening for new clients.
void tcpClientListener(SOCKET* listenSocket)
{


The thread work is done in a while (1) as to allow any number of new clients to connect. As soon as a new connection is established, the loop repeats and listens for another client.


    // Listen for new connections until the listenSocket is closed.
    while (1)
    {


The listen() call will cause the thread to wait until either a new connection is requested or until the socket closes down. The downside is this operation must be done in its own thread (hence why we created the tcpClientListener thread). The upside is the operation does not waste CPU cycles waiting for a connection, and we don't need a hackish Sleep(1) to keep our CPU from spinning.


        // listen will wait until a connection is requested on the socket. If the socket is closed
        // while waiting, then listen will break from its wait and throw a socket error.
        int listenResult = listen(*listenSocket, SOMAXCONN);
        if (listenResult != SOCKET_ERROR)
        {


A new client is attempting to connect! accept() the connection.


            // Accept a client socket
            SOCKET clientSocket = accept(*listenSocket, NULL, NULL);
            if (clientSocket != INVALID_SOCKET)
            {


Now that we have the new client, we need a way to listen for data from that client. addClient() is described later on, but it essentially creates a ConnectionModel and returns a pointer to that data. For now the ConnectionModel just contains the socket the client is on.

A new tcpPacketListener thread is created with the clientConnection passed in as an argument so it knows what to listen from. Calling .detach() has the new thread execute independently so we can get back to work listening for new clients.


                // Create a connection to the client and begin a listener thread.
                ConnectionModel* clientConnection = addClient(clientSocket);
                std::thread{ tcpPacketListener, clientConnection }.detach();
            }
            else
            {


If we fell into here, we were unable to accept the connection so break out of the while (1) loop. This will cause us to stop receiving any new connections. You may not wait to break out here and instead let the thread continue to attempt to make other connections.


                std::cout << "TCP accept failed." << std::endl;
                break;
            }
        }
        else
        {


If we fell into here, we were unable to listen for a new connection so break out of the while (1) loop. This will cause us to stop receiving any new connections. This error is caused when the listenSocket is closed, and it is how we end this thread at shutdown.


            std::cout << "TCP listen failed." << std::endl;
            break;
        }
    }

    return;
}



TCP Packet Listener Thread


The tcpPacketListener thread is spooled off from the tcpClientListener thread; its duty is to listen for and packets from a single connection.

// Thread function for listening for packets from clients to relay.
void tcpPacketListener(ConnectionModel* clientConnection)
{
    char receiveBuffer[maximumPacketLength_bytes];


The thread work is done in a while (1) as to continuously receive new packets. As soon as a new packet is received, the loop repeats and listens for another packet.


    // Receive until the client connection is closed.
    while (1)
    {


The recv() call will cause the thread to wait until either a new packet arrives or until the socket closes down. The downside is this operation must be done in its own thread (hence why we created the tcpPacketListener thread). The upside is the operation does not waste CPU cycles waiting for a packet, and we don't need a hackish Sleep(1) to keep our CPU from spinning.


        // recv will wait until a packet is received on the socket. If the socket is closed
        // while waiting, then recv will break from its wait and throw a socket error.
        int packetLength_bytes = recv(clientConnection->clientSocket, receiveBuffer, maximumPacketLength_bytes, 0);
        if (packetLength_bytes > 0)
        {


New packet received! In this TCP/UDP gateway example, I'm relaying the packets to other clients that have connected to the server. You may want to do the same or something completely different. It's entirely possible your clients never needs packets relayed to each other and just communicate with the server.


            std::cout << "Bytes received: " << packetLength_bytes << std::endl;

            // Relay the packet back to the other clients.
            sendPacketToClients(clientConnection, receiveBuffer, packetLength_bytes);
        }
        else if (packetLength_bytes == 0)
        {


If the number of bytes received is 0, then the connection cleanly closed. We should break out of the while (1) loop which will remove the client and end the thread.


            std::cout << "TCP connection closed." << std::endl;
            break;
        }
        else
        {


If the number of bytes received is less than 0, then the connection abruptly closed. We should break out of the while (1) loop which will remove the client and end the thread.


            std::cout << "TCP socket receive error." << std::endl;
            break;
        }
    };


We are now outside the while (1) loop! Either the connection cleanly closed, the connection had an error, or the socket was closed (we are shutting down). In all cases, remove the client from our clients list and let the thread close out.


    // Client connection was closed, remove it from our connection list.
    removeClient(clientConnection);

    return;
}



Client Management Functions



Client connections are maintained in the clientConnections std::vector. The following functions just add and remove from that vector.

When a new connection is established, the clientSocket established in the tcpClientListener thread is passed to addClient. Here it creates a new ConnectionModel to store parameters of the client and then adds them to the clientConnections std::vector. A Mutex lock prevents the clientConnections from being modified while we're iterating through it elsewhere.

// Add a new client to the maintained connections.
ConnectionModel* addClient(SOCKET clientSocket)
{
    // Create a new connection for the socket.
    ConnectionModel* clientConnection = new ConnectionModel(clientSocket);

    clientConnectionsMutex.lock();

    // Add the client to the connections list.
    clientConnections.push_back(clientConnection);

    std::cout << "Client added!" << std::endl;
    std::cout << "Active Connections: " << clientConnections.size() << std::endl;

    clientConnectionsMutex.unlock();

    return clientConnection;
}


When a client connection is closed (or told to close), the closing clientConnection is passed to removeClient. Here it ensures the socket is closed and the client is removed from the clientConnections std::vector. A Mutex lock prevents the clientConnections form being modified while we're iterating through it elsewhere.


// Remove a client from the maintained connections.
bool removeClient(ConnectionModel* clientConnection)
{
    clientConnectionsMutex.lock();

    // Shutdown and close the socket.
    shutdown(clientConnection->clientSocket, SD_BOTH);
    closesocket(clientConnection->clientSocket);

    // Remove the client from our connection list.
    auto removedConnection = std::find(clientConnections.begin(), clientConnections.end(), clientConnection);
    if (removedConnection != clientConnections.end())
    {
        clientConnections.erase(removedConnection);
    }

    // Free the connection memory.
    delete clientConnection;

    std::cout << "Client removed!" << std::endl;
    std::cout << "Active Connections: " << clientConnections.size() << std::endl;

    clientConnectionsMutex.unlock();

    return true;
}


And finally as a cleanup function for when the application is closing, we have a disconnectAllClients. It iterates through our clientConnections std::vector to close every client sockets.

Each client has its own thread that listens for packets, and the threads waits on a recv() for new packets. By closing the socket, the recv() will stop waiting and throw an error. The thread knows when it receives a recv() error to break out of the loop, remove the clientConnection from our std::vector, and end the thread.


// Disconnect all clients. (This does not prevent new clients from connecting later.)
bool disconnectAllClients()
{
    clientConnectionsMutex.lock();

    // Loop through the connections and close each of their connections.
    std::vector<ConnectionModel*>::iterator clientConnectionIterator;
    for (clientConnectionIterator = clientConnections.begin(); clientConnectionIterator != clientConnections.end(); clientConnectionIterator++)
    {
        ConnectionModel* loopClientConnection = *clientConnectionIterator;

        // Closing the socket will cause the listener threads for those sockets to close.
        shutdown(loopClientConnection->clientSocket, SD_BOTH);
        closesocket(loopClientConnection->clientSocket);
    }

    clientConnectionsMutex.unlock();

    return true;
}



TCP Sending



The last function we have sends packets back to the clients. In this example, it is relaying a packet receive from one client (sourceConnection) to all other clients. Your use-case for a TCP server may be completely different, and your send function look nothing like this. However, it is a good example of how to loop through clientConnections and send data to them.

// Relay a packet to all clients except to the client that originated the packet (sourceConnection).
bool sendPacketToClients(ConnectionModel* sourceConnection, char packetData[maximumPacketLength_bytes], int packetLength_bytes)
{
    clientConnectionsMutex.lock();

    // Loop through the connections and send the packet to each.
    std::vector<ConnectionModel*>::iterator clientConnectionIterator;
    for (clientConnectionIterator = clientConnections.begin(); clientConnectionIterator != clientConnections.end(); clientConnectionIterator++)
    {
        ConnectionModel* loopClientConnection = *clientConnectionIterator;

        // Skip sending the packet if it's to the source.
        if (loopClientConnection->clientSocket == sourceConnection->clientSocket)
        {
            std::cout << "Skipped ID: " << loopClientConnection->clientSocket << std::endl;
            continue;
        }
        std::cout << "Sent to: " << loopClientConnection->clientSocket << std::endl;

        // Send the packet on the socket!
        int bytesSent = send(loopClientConnection->clientSocket, packetData, packetLength_bytes, 0);
        if (bytesSent == SOCKET_ERROR)
        {
            std::cout << "Send failed." << std::endl;
        }
    }

    clientConnectionsMutex.unlock();

    return true;
}




TCP Client



Again this example was written for Windows, but I'll eventually revisit to add Linux support. Overall the following will be helpful with how to connect and communicate with a TCP server. The UDP code is specific for my use-case (relaying UDP packets), so likely your own software won't have any UDP code. You could use the following as an example on how to do UDP communication and in that case you'll likely ignore the TCP stuff.

I'm using separate sockets for UDP sending and receiving. Depending on your network design, that may not be necessary. I'm using separate sockets so I can identify the source of the UDP messages so I don't receive & rebroadcast messages I just sent.

Headers


#define WIN32_LEAN_AND_MEAN

#include <process.h>
#include <windows.h>
#include <winsock2.h>
#include <ws2tcpip.h>
#include <stdio.h>
#include <map>
#include <string>
#include <iostream>
#include <sstream>
#include <time.h>
#include <thread>

#pragma comment (lib, "Ws2_32.lib")


Constants


// Constant denoting the maximum packet size that can be handled.
const int maximumPacketLength_bytes = 8192;


Prototypes


Again I don't have a separate header file for the client code. The function prototypes are just at the top.


// Function Prototypes
bool udpConnect(char udpLocalIpAddress[16], char udpSendIpAddress[16], int udpPort);
void udpListener();
bool udpDisconnect();
bool tcpConnect(char serverAddress[16], int tcpPort);
bool tcpDisconnect();
void tcpListener();
bool sendPacketToUdp(char packetData[maximumPacketLength_bytes], int packetLength_bytes);
bool sendPacketToTcp(char packetData[maximumPacketLength_bytes], int packetLength_bytes);
bool sendInitSelfPacket();


Global Variables


There are 3 global variables to keep track of the network sockets (one for TCP, one for receiving UDP, and one for sending UDP).


// UDP and TCP Sockets
SOCKET udpReceiveSocket; // Socket to receive UDP messages on.
SOCKET udpSendSocket; // Socket to send UDP messages on.
SOCKET tcpSocket; // Socket of the TCP connection to the server.


And there are 3 socket addresses (one for the address to listen for UDP, one for the address to send UDP, and one for our own UDP address).


// UDP Socket Addresses
SOCKADDR_IN udpSocketAddressIncoming; // Address for incoming packets.
SOCKADDR_IN udpSocketAddressBroadcast; // Address for broadcasting packets.
SOCKADDR_IN udpSocketAddressSelf; // Address to identify packets that originated from ourself.


To identify our sending UDP address, we'll be sending an initial packet with a random value. These 2 global variables are to keep track of if we've found our own address and the value to check in that first packet.


// Variables to keep track of initialize the self address.
int initSelfValue = 0; // Random value sent to identify our own address.
bool initSelfComplete = false; // Flag to know if we've identified our own address.


Main


On to the actual functions! The client takes in far more arguments than the server. Here we need the TCP IP Address, the TCP Port, the UDP Local Address (that we're binding to), the UDP Send Address, and the UDP Port.


// Main takes several command line arguments for the TCP server address and the local addresses for UDP traffic.
int main(int argc, char *argv[])
{
    // If insufficient arguments are given, show the format.
    if (argc < 6)
{
        std::cout << "Command line arguments:" << std::endl;
        std::cout << "<TCP Address> <TCP Port> <UDP Local Address> <UDP Send Address> <UDP Port>" << std::endl;
return 0;
}

    char serverAddress[16];
strcpy(serverAddress, argv[1]);

    int tcpPort = atoi(argv[2]);

    char udpLocalIpAddress[16];
strcpy(udpLocalIpAddress, argv[3]);

    char udpSendIpAddress[16];
strcpy(udpSendIpAddress, argv[4]);

    int udpPort = atoi(argv[5]);


Since we're on Windows, initialize Winsock.


 // Initialize Winsock.
 WSADATA wsd;
if (WSAStartup(MAKEWORD(2, 2), &wsd) != 0)
{
        std::cout << "WSAStartup failed." << std::endl;
return false;
}


Call udpConnect with the arguments passed in to bind our UDP socket. This will also start the UDP listener thread.


 // Connect to sending and receiving UDP sockets.
if (!udpConnect(udpLocalIpAddress, udpSendIpAddress, udpPort))
{
        std::cout << "UDP connect failed." << std::endl;
WSACleanup();
return 0;
}


Then call tcpConnect with the arguments passed in to connect to the remote TCP server. This will also start the TCP listener thread.


 // Connect to the TCP server.
if (!tcpConnect(serverAddress, tcpPort))
{
        std::cout << "TCP connect failed." << std::endl;
WSACleanup();
return 0;
}


Now that the network sockets are ready to go, we can send an self-identification packet. We send it to our UDP broadcast address where it will get received via the UDP listener thread. That way we can identify packets that originated from ourselves.


    // Send a single packet over UDP so we can receive it and know how to filter our own messages.
    sendInitSelfPacket();


Similar to the server, the real work of this application is done on the UDP and TCP listener threads that were spooled off in udpConnect and tcpConnect respectively. All we need to do on this main thread is have some kind of stop until the user wants to close the application.

To keep things simple, I'm just using a getchar() and when the user presses a key the app will close down.


    // All packet communication is done on other threads.
    // Continue the program until the user presses a key on the main thread.
    std::cout << "Press any key to quit." << std::endl;
getchar();


At this point, the user had signaled he or she wants the program to close. Disconnect both the UDP and TCP sockets. By doing this, we'll throw errors in our socket receive functions which will kick us out of our thread while loops which will allow those threads to cleanly close.


// Disconnect both the UDP and TCP sockets.
udpDisconnect();
tcpDisconnect();

    // Wait for threads to conclude. (Disconnecting above will cause them to break out of their socket waits.)
    Sleep(100);


On Windows, so cleanup Winsock on exit.


WSACleanup();

return 0;
}


UDP Socket


The udpConnect function binds a UDP socket based on the command line arguments given.


// Create sending and receiving UDP sockets, and create a thread to listen for UDP packets.
bool udpConnect(char udpLocalIpAddress[16], char udpSendIpAddress[16], int udpPort)
{
    std::cout << "Connecting to socket..." << std::endl;


Two sockets are created. The first is the socket that will receive packets.


    // Create the UDP receive socket.
    udpReceiveSocket = socket(AF_INET, SOCK_DGRAM, 0); // IPPROTO_UDP
    if (udpReceiveSocket == INVALID_SOCKET)
{
        std::cout << "receive socket failed." << std::endl;
return false;
}



The second is the socket that will send packets.


    // Create the UDP send socket.
    udpSendSocket = socket(AF_INET, SOCK_DGRAM, IPPROTO_UDP);
    if (udpSendSocket == INVALID_SOCKET)
    {
        std::cout << "send socket failed." << std::endl;
        return false;
    }


We want to allow reuse for both sockets - basically allow other applications to bind to the same port to also listen for UDP messages. I've never encountered a situation where I didn't want to allow socket reuse. I have encountered other software that prevented reuse and I hated it. So as a general rule, please don't prevent other software from reading off the same port as you are!


    // Enable reuse for the UDP receive socket.
bool receiveSocketReuse = 1;
    if (setsockopt(udpReceiveSocket, SOL_SOCKET, SO_REUSEADDR, (char *)&receiveSocketReuse, sizeof(receiveSocketReuse)) == SOCKET_ERROR)
{
        std::cout << "receive setsockopt reuse failed." << std::endl;
return false;
}

    // Enable reuse for the UDP send socket.
    bool sendSocketReuse = 1;
    if (setsockopt(udpSendSocket, SOL_SOCKET, SO_REUSEADDR, (char *)&sendSocketReuse, sizeof(sendSocketReuse)) == SOCKET_ERROR)
    {
        std::cout << "send setsockopt reuse failed." << std::endl;
        return false;
    }


When receiving packets on a very congested network, I have seen messages inexplicably get dropped. Bumping up the receive buffer size solved the problem. I have not encountered any downsides to doing this.


    // Increase the receive buffer size. This can help when many packets are being transmitted simultaneously.
    int receiveBufferSize = 65535;
    if (setsockopt(udpReceiveSocket, SOL_SOCKET, SO_RCVBUF, (char *)&receiveBufferSize, sizeof(receiveBufferSize)) == SOCKET_ERROR)
    {
        std::cout << "receive setsockopt failed buffer size." << std::endl;
    }


Next configure the socket addresses.


    // Configure the UDP broadcast address.
udpSocketAddressBroadcast.sin_family = AF_INET;
udpSocketAddressBroadcast.sin_port = htons(udpPort);
udpSocketAddressBroadcast.sin_addr.S_un.S_addr = inet_addr(udpSendIpAddress);

    // Configure the UDP local address.
    SOCKADDR_IN udpSocketAddressLocal;
    udpSocketAddressLocal.sin_family = AF_INET;
udpSocketAddressLocal.sin_port = htons(udpPort);
udpSocketAddressLocal.sin_addr.S_un.S_addr = inet_addr(udpLocalIpAddress); // htonl(INADDR_ANY);


And bind the UDP receive socket to the local address.

The UDP local address could be set to INADDR_ANY as opposed to a specific IP. If the computer happened to have multiple network connections (multiple IP addresses) and it was bound to INADDR_ANY, then it could receive packets from any of those IP addresses with a single socket. There are pros and cons to doing this. Perhaps you want to monitor all traffic regardless of its network source. Perhaps instead you want to specifically only monitor traffic from a single network source. Both use cases are common and easy to configure just by changing udpSocketAddressLocal.sin_addr.S_un.S_addr between: inet_addr(udpLocalIpAddress) and htonl(INADDRY_ANY).


    // Bind the UDP receiving socket to the local address.
    if (bind(udpReceiveSocket, (SOCKADDR *)&udpSocketAddressLocal, sizeof(udpSocketAddressLocal)) == SOCKET_ERROR)
{
        std::cout << "bind() failed." << std::endl;
return false;
}


Next initialize a variable that will keep track of the source address of incoming messages and start the thread.


    // Prepare the UDP incoming address.
udpSocketAddressIncoming.sin_family = AF_INET;
udpSocketAddressIncoming.sin_port = htons(udpPort);

    // Start the UDP Listener thread.
    std::thread{ udpListener }.detach();

return true;
}


udpDisconnect just has to close the send and receive sockets. The UDP listener thread will automatically cleanly exit when the udpReceiveSocket is closed.


// Close the UDP sending and receiving sockets.
bool udpDisconnect()
{
// Close the socket.
 closesocket(udpReceiveSocket);
 closesocket(udpSendSocket);

return true;
}


UDP Listener Thread


The bottom of udpConnect creates a thread using the udpListener function to listen for incoming packets. The function remains in a continuous while (1) loop to process those incoming packets until the socket is closed.


// Thread function listening for UDP packets.
void udpListener()
{


Define a receiveBuffer that will hold the incoming data.


char receiveBuffer[maximumPacketLength_bytes];
int dwSenderSize = sizeof(udpSocketAddressIncoming);


Begin the loop!


    // Listen for packets until the UDP socket is closed.
    while (1)
    {


Similar to the TCP recv() function described in the server code above, the UDP recvfrom() will cause the thread to wait until either a new packet arrives or until the socket closes down. The downside is this operation must be done in its own thread (hence why we created the udpListener thread). The upside is the operation does not waste CPU cycles waiting for a packet, and we don't need a hackish Sleep(1) to keep our CPU from spinning.


        // recvfrom will wait until a packet is received on the socket. If the socket is closed
        // while waiting, then recvfrom will break from its wait and throw a socket error.
        int packetLength_bytes = recvfrom(udpReceiveSocket, &receiveBuffer[0], maximumPacketLength_bytes, 0, (SOCKADDR *)&udpSocketAddressIncoming, &dwSenderSize);


At this point, either recvfrom successfully received an incoming UDP packet or there was an error with the socket.

If there was an error with the socket, bail out of the thread now. This is also caused when the application is closing - we close the receiving socket which in turn kicks us out of the recvfrom when falls into this socket error code which allows this thread to exit nicely..


        if (packetLength_bytes < 0)
        {
// Socket error.
            std::cout << "UDP socket receive error." << std::endl;
            break;
        }


At startup, we send an initial UDP packet out to broadcast that loops back and gets received by our gateway. That is for identifying our sending address. If we haven't yet received that init packet, check to see if the incoming packet is the correct size (just 4 bytes for a single Int variable). If the incoming packet is 4 bytes and its value matches the random value we had sent in the packet, then this packet likely originated from us. Record the source address as our udpSocketAddressSelf and set the flag that initSelfComplete is done.


        else if (!initSelfComplete)
        {
            // We have not yet recieved the initialization packet, is this it?
            if (packetLength_bytes == 4)
            {
                // This packet is the correct size for the initialization, does its value match what we sent?
                int* packetValue = (int*)receiveBuffer;
                if (*packetValue == initSelfValue)
                {
                    // It does! This is our initialization packet; we now know our own broadcasting address.
                    // With udpSocketAddressSelf, we can ignore packets that we originated.
                    udpSocketAddressSelf = udpSocketAddressIncoming;
                    initSelfComplete = true;
                    std::cout << "Received initialization packet!" << std::endl;
                }
            }
        }
        else if (udpSocketAddressSelf.sin_port == udpSocketAddressIncoming.sin_port &&
                 udpSocketAddressSelf.sin_addr.S_un.S_addr == udpSocketAddressIncoming.sin_addr.S_un.S_addr &&
                 udpSocketAddressSelf.sin_family == udpSocketAddressIncoming.sin_family)
        {
            // We originated this packet, ignore it.
            std::cout << "Skipped." << std::endl;
        }


If we received a good packet and initSelfComplete has completed, then we can process the incoming packet! For this gateway example, I'm relaying the packets to the TCP server (so it can rebroadcast it out to all of its clients). For your use-case, you'll likely do something very different here.


else if (packetLength_bytes >= 0)
{
// Packet received, relay it to the TCP socket to be shared to other clients.
            std::cout << "UDP bytes received: " << packetLength_bytes << std::endl;
sendPacketToTcp(receiveBuffer, packetLength_bytes);
}

return;
}


TCP Socket


At startup we connect to the given TCP server.


// Connect to the TCP server.
bool tcpConnect(char serverAddress[16], int tcpPort)
{


Define an addrinfo with our hints for the TCP server.


    // Setup address info hints.
    struct addrinfo addressInfoHints;
    memset(&addressInfoHints, 0, sizeof(addressInfoHints));
    addressInfoHints.ai_family = AF_UNSPEC;
    addressInfoHints.ai_socktype = SOCK_STREAM;
    addressInfoHints.ai_protocol = IPPROTO_TCP;


With the addressInfoHints, resolve the server IP address and port from what was given in the command line arguments.


// Resolve the server address.
char portString[20];
sprintf(portString, "%d", tcpPort);
    struct addrinfo* addressInfoResult = NULL;
    int getaddrinfoReturn = getaddrinfo(serverAddress, portString, &addressInfoHints, &addressInfoResult);
    if (getaddrinfoReturn != 0)
{
        std::cout << "getaddrinfo failed." << std::endl;
return false;
}


Next up, create the TCP socket.


// Loop through potential server addresses to attempt connection.
    for (struct addrinfo *ptr = addressInfoResult; ptr != NULL; ptr = ptr->ai_next)
{
        // Create the TCP socket.
tcpSocket = socket(ptr->ai_family, ptr->ai_socktype, ptr->ai_protocol);
if (tcpSocket == INVALID_SOCKET)
{
            std::cout << "TCP socket failed." << std::endl;
return false;
}


And connect to the server!


// Connect to the server.
        int connectReturn = connect(tcpSocket, ptr->ai_addr, (int)ptr->ai_addrlen);
        if (connectReturn == SOCKET_ERROR)
{
closesocket(tcpSocket);
tcpSocket = INVALID_SOCKET;
continue;
}
break;
}

    freeaddrinfo(addressInfoResult);

if (tcpSocket == INVALID_SOCKET)
{
        std::cout << "Unable to connect to server!" << std::endl;
return false;
}


Now that our server connection is established, start a TCP listener thread that will wait for and process incoming TCP packets.


    // Start the TCP Listener thread.
    std::thread{ tcpListener }.detach();

return true;
}


tcpDisconnect just has to close the TCP socket. The TCP listener thread will automatically cleanly exit when the tcpReceiveSocket is closed.


// Close the TCP connection with the server.
bool tcpDisconnect()

    // Shutdown and close the socket.
    shutdown(tcpSocket, SD_BOTH);
closesocket(tcpSocket);

return true;
}


TCP Listener Thread


The bottom of tcpConnect creates a thread using the tcpListener function to listen for incoming packets. The function remains in a continuous while (1) loop  to process those incoming packets until the socket is closed.


// Thread function listening for TCP packets.
void tcpListener()
{
char receiveBuffer[maximumPacketLength_bytes];

    // Receive until the connection is closed.
    while (1)
{


The recv() call will cause the thread to wait until either a new packet arrives or until the socket closes down. The downside is this operation must be done in its own thread (hence why we created the tcpListener thread). The upside is the operation does not waste CPU cycles waiting for a packet, and we don't need a hackish Sleep(1) to keep our CPU from spinning.


        // recv will wait until a packet is received on the socket. If the socket is closed
        // while waiting, then recv will break from its wait and throw a socket error.
        int packetLength_bytes = recv(tcpSocket, receiveBuffer, maximumPacketLength_bytes, 0);


If we get a valid packet, relay it back to our local UDP network! You will likely do something different for your own use-case with the incoming packets.


if (packetLength_bytes > 0)
{
            std::cout << "TCP bytes received: " << packetLength_bytes << std::endl;

            // Relay the packet to the local UDP network.
            sendPacketToUdp(receiveBuffer, packetLength_bytes);
}


If the number of bytes is 0, then the connection cleanly closed. We should break out of the while (1) loop and end the thread.


else if (packetLength_bytes == 0)
{
            std::cout << "TCP connection closed." << std::endl;
            break;
}


If the number of bytes received is less than 0, then the connection abruptly closed (possibly by our shutdown process closing the thread). We should break out of the while (1) loop which will end the thread.


else
{
            std::cout << "TCP socket receive error." << std::endl;
            break;
} // receiveSize_bytes
}

return;
}


Packet Sending


Sending packets for both TCP and UDP is very straightforward. Both functions just take in an array of the packet data and the number of bytes from that array to send.

The TCP version of sending...


// Send the packet received from the UDP address to the TCP server.
bool sendPacketToTcp(char packetData[maximumPacketLength_bytes], int packetLength_bytes)
{
    // Send the packet.
int sendResult = send(tcpSocket, packetData, packetLength_bytes, 0);
if (sendResult == packetLength_bytes)
{
        std::cout << "TCP bytes sent: " << packetLength_bytes << std::endl;
return true;
}
else if (sendResult < 0)
{
        std::cout << "TCP send failed at socket level." << std::endl;
return false;
}
else
{
        std::cout << "TCP failed to send entire message." << std::endl;
return false;
}
}


The UDP version of sending...


// Send the packet received from the TCP server to the UDP address.
bool sendPacketToUdp(char packetData[maximumPacketLength_bytes], int packetLength_bytes)
{
    // Send the packet.
    int sendResult = sendto(udpSendSocket, packetData, packetLength_bytes, 0, (const struct sockaddr *)&udpSocketAddressBroadcast, sizeof(struct sockaddr));
if (sendResult == packetLength_bytes)
{
        std::cout << "UDP bytes sent: " << packetLength_bytes << std::endl;
return true;
}
else if (sendResult < 0)
{
        std::cout << "UDP send failed at socket level." << std::endl;
return false;
}
else
{
        std::cout << "UDP failed to send entire message." << std::endl;
return false;
}
}


Above I mentioned that we send an init packet at startup that we can use to self-identify our own UDP sending address. The following function takes a random integer and sends it as the init packet. This is basically an example of how to send a single integer over the network. When I move this into the orbital aero code, I'll have examples of how to send entire structures over a network.


// Send an initilization packet to identify our own sending address.
bool sendInitSelfPacket()
{
    std::cout << "Sending initialization packet to find own address." << std::endl;

    // Pick a random value to be put in the packet that we'll compare against upon receipt.
    srand((unsigned int)time(NULL));
    initSelfValue = rand();

    // Create a packet and stuff the 4 byte random value into it.
    char packetData[maximumPacketLength_bytes];
    memcpy(packetData, &initSelfValue, 4);

    // Send the packet.
    return sendPacketToUdp(packetData, 4);
}






The code within this post is released into the public domain. You're free to use it however you wish, but it is provided "as-is" without warranty of any kind. In no event shall the author be liable for any claims or damages in connection with this software.

Friday, May 26, 2017

Office / Shop: Data Wiring

Electrical is all done, but now's the perfect opportunity to hide all my data cables in the walls before insulation/drywall goes up.

Since I'm planning on mounting a projector in the ceiling of the office, might as well wire up 5.1 surround sound as well! (I debated going with a 7.1 solution, but decided against it as the office is relatively small - only 12x24'.)

Rough sketch of shop layout...


I went with 14 Gauge CL2 (approved for in-wall) speaker cable.

One of the shop audio jacks...




One of the office audio jacks...


Offices typically have an in-wall telecommunication/media panel (similar to an electrical panel) that houses the termination points of all the data lines and the switch. I'm going to buy a half-rack instead that contains the switch, audio/visual equipment, and other hardware, so I'll just terminate all the wires to at the wall.


There will be quite a few wires from the rack to this part of the wall, but they should be fairly tidy and hidden behind the rack. There are six network ports here; five of them go to other ports along the office/shop walls but the blue one is routed back to the house for Internet.

I also added the T shape 2x4s to add stability for the 3 gang box.


Wiring up the 5.1 surround audio and HDMI cable...





Looking good!


The Home Theater panel is for 7.1 surround sound, but I only have 5.1 for the office. The two rear channels of the 7.1 panel are to the speaker jacks in the shop.

I didn't think insulation deserved its own post, so here is the progress late into the night...


My helper.


All done!



Cost:
12x 1 Gang Box ($10)
3 Gang Box ($3)
100' Indoor/Outdoor Cat5E ($30)
100' Indoor Cat5E ($20)
250' 14 Gauge CL2 Speaker Cable ($60)
3-Gang 7.1 Home Theatre Cover ($17)
25' HDMI Cable ($16)
2x 5 Pack 2 Connector Banana Wall Plate ($34)
11x Cat5E Jacks ($30)
6 Port Wall Plate ($1)
5x 1 Port Wall Plate ($5)
Cable Mounting Hardware ($15)
Other ($10)
========
Total: $251

Lots of my free time has been on this construction project, but I'm hoping to get back to Orbital Aero soon.

Saturday, April 8, 2017

Office / Shop: Electrical

At some point in the past, the shop building had a single 120V 20 Amp circuit for outlets along the wall (power from the house's main panel). Then later someone came in and cut the wire at the end of the conduit and at each outlet... to probably make $2 from copper recycling and screw over the next homeowner. Makes sense...

Rather than try and get the woefully inadequate 20 Amp circuit legal (that panel was fugly), I went all out:
New panel at the shop.
240V 60 Amp rated wiring from the house.
Overhead light in office.
Two dedicated 20 Amp in the office supplying power to 6 outlets (one of which is in the ceiling for mounted projectors).
Overhead 20 Amp outlet in shop (primarily for a light).
One dedicated 20 Amp in the shop supplying power to 3 outlets.
One dedicated 20 Amp mounted outside near the roof line for security camera and Christmas lights.
NEMA 14-50 240V 50 Amp Outlet for the electric car charger.

I lost some of my early progress pictures, but there was more than enough work to have plenty left to share...

Outlets along the wall...


Close up of the wiring - it needs to be secured at least every 4.5 feet, within 12" of boxes, and within 8" inches of the single gang boxes. Please see NEC 334.30 or 314.17 for more info.


Please note I attached my wires behind my studs, but if you're going through studs be sure the hole is at least 1 1/4" from the edge. Please see NEC 300.4 for more info.


Outlet for projectors! And Star Trek light. :-)


Mounting the exterior 240V outlet for the car charger...


New panel with shop wiring (excluding the supply)!


Ground rods are freaking 8' long, and you need two of them! (Spaced > 6 ft apart.)


The first 7' go in easy enough, and then not so much...


The T-Post driver actually worked well for awhile, but towards the ground I had to resort to the sledgehammer. Slightly crooked, but close enough!


A photo of one of the wall outlets. You can see the ground wire being routed through the siding out to the ground rod outside.


Using the electric car to pick up the 6/3 cable...


It's always scary pulling off drywall to see what's underneath. And this house was no different... How much f'd up stuff do you see?


The wires tied together behind the wall? The air vent blowing cold wet air directly into the wall? The inexplicable reason they couldn't have a continuous stud? I removed the pitiful amount of insulation in there to do the wire work, but it wasn't much.

Here is my progress on bringing the mess up to code... And my new 6/3 cable being routed from the panel into the crawl space. (In the photo I attached the wires to the plywood, but I moved those to the studs to minimize chance of nails/screws going in them from the outside.)


Got the insulation in and new drywall up! I hate that conduit sticking out, but that will be something to clean up later.


The new 6/3 cable into the house panel!


Stripping back that UF-B cable is a gigantic pain. The best approach I found was to use my snips to slowly cut between the wires, then it was easier to get the UF-B jacket off the individual wires. But it still sucked.


The wire dropped down from the house electrical panel into the crawl space, then through the crawl space nearly the entire length of the house to one of the vents. By code the cable can't be laying on the ground in the crawl space, so I used 1 1/4" EMT 1-Hole straps to attach it to the supports underneath.


2" electrical conduit brings the wire from the crawl space into my trench to the shop.


Trench needed to be minimum 18" to the top of the cable.


I was going to use the existing 2" conduit at the shop to bring the UF-B in, but apparently it was 2" water pipe and thus illegal. I replaced it with a 2" LB Conduit Body and actual electrical conduit. It looks better at least...


The final wiring within the shop! The UF-B cable has a straight shot from the LB Conduit Body into the panel - which is a good thing because that cable is hard to bend. You can also see my two 6 gauge ground wires going out the bottom of the panel on their way to the ground rods.


Office...


Shop...


 The car charger connected to the new outlet, so beautiful!


And it's so much nicer that I can take a drive to town and back and charge in 1 hour as opposed to 6!

I haven't been keeping track of costs, so these are estimates:
Permit ($97.40)
Electrical Panel ($60)
Panel Cover ($20)
5x 120V 20 Amp Breakers ($25)
240V 50 Amp Breaker ($10)
2x 240V 60 Amp Breaker ($20)
Main Breaker Tiedown Clip ($8)
6x 3/8" Cable Clamp Connectors ($5)
1 1/4" Cable Clamp Connector ($5)
1 1/2" Cable Clamp Connector ($5)
20 Amp Switch ($3)
11x 1-Gang Box ($10)
Round Box ($1)
125 ft 6/3 with ground UF-B ($307)
250 ft 12/2 Solid NM-B wire (only had 2 feet of wire extra at the end) ($61.47)
6 ft 6/3 Romex wire ($15)
25 ft 6 ga Solid Copper Wire ($18)
4x 20 Amp GFCI Tamper Resistant ($80)
1x 20 Amp GFCI Tamper Resistant Weather Resistant ($20)
6x 20 Amp Outlet Tamper Resistant ($36)
Exterior Enclosure for 120V outlet ($30)
Exterior Enclosure with NEMA 14-50 outlet ($40)
Ground Rod ($11)
Ground Rod Clip ($5)
Wire Staples ($5)
30x 1 1/4" EMT 1-Hole straps ($60)
2" x 10' Electrical Conduit ($7)
2x 2" Electrical 90 degree Elbows ($7)
2" LB Conduit Body ($35)
2" Conduit Clamps ($10)
========
Total: $1016.87
Let's add 10% for some sales tax I'm missing and stuff I forgot:
New Total: $1118.56
Let's add about $100 for stuff I bought that was a complete waste.
Actual cost: $1218.56

Not especially cheap, but it's a fraction of the cost of the quote I got to hire an electrician to do the work and I'm very happy with the final product.

Lessons learned...

Even though this is an exterior shop - it is at a residence, I made the expensive mistake of initially not buying all Tamper Resistance outlets. If the electrical is at your home, you must use the Tamper Resistant outlets. (At least if you have the same code requirements as me...)

The panel I bought did not have a main cutoff, but it was required for my situation. The solution was to put in a 60A breaker to backfeed the electricity into the panel. And then use a main breaker tiedown clip to "doubly" hold it down. Next time I would have bought a panel with a built-in main cutoff.

I already mentioned this, but the one piece I tried to re-use from the existing setup was the conduit in the shop. Make sure you use electrical conduit, not plumbing!

(As I've said in previous electrical work posts, I'm not an electrician. What I'm sharing was permitted and approved by an inspector and it hasn't burned my house or shop down, but I don't guarantee anything I say. Use this post as nothing more than another data point on your quest to do your own electrical work.)

Totally unrelated, but if you leave Doom 2 idle for an hour and then come back and finish the level, it mocks you for your time.


Hilarious!