Freeciv Hacker's Guide This guide is intended to be a help for developers, wanting to mess with Freeciv program. Here and there, you'll see some comments marked as [...], containing more personal thoughts on the design, why it looks like it does, and sometimes what went wrong. I hope developers will find that interesting too. To read about the AI, see README.AI =========================================================================== Basic =========================================================================== Freeciv is a client/server civilization style of game, written in C. The client is pretty dumb. Almost all calculations is performed on the server. [It wasn't like this always. Originally more code was placed in the common/ dir, allowing the client to do some of the world updates itself. The end_of_turn city-refresh was for example performed both on the server and on the client. However things got quite complex, more and more info was needed on the client-side(security problem). Little by little we moved more code to the server, and as of 1.5 the client is quite dumb -PU] The source code has the following important directories: common: data structures and code used by both the client and server. server: (duh) client: common client code client/* (fx gui-gtk): a specific gui implementation of the client. data: graphics, rulesets and stuff po: translations ai: the ai, later linked into the server. =========================================================================== Server =========================================================================== General: The server main loop basically looks like: while(server_state==RUN_GAME_STATE) { /* looped once per turn */ do_ai_stuff(); /* do the ai controlled players */ sniff_packets(); /* get player requests and handle them */ end_turn(); /* main turn update */ game_next_year(); } Most time is spend in the sniff_packets() function, where a select() call waits for packets or input on stdin(server-op commands). =========================================================================== Server Autogame Testing =========================================================================== Code changes should always be tested before submission for inclusion into the CVS source tree. It is useful to run the client and server as autogames to verify either a particular savegame no longer shows a fixed bug, or as a random sequence of games in a while loop overnight. To start a server game with all AI players, create a file (below named civ.serv) with lines such as the following: # set randseed 42 # repeat a particular game (random) sequence # set seed 42 # repeat a particular map generation sequence # set timeout 3 # run a client/server autogame set timeout -1 # run a server only autogame set aifill 7 # fill to 7 players hard # make the AI do complex things create Caesar # first player (with known name) created and # toggled to AI mode start Note: After the start command the server prompt is unusable in autogame mode. The commandline to run server-only games can be typed as variations of: $ while( time server/civserver -r civ.serv ); do date; done --- or --- $ server/civserver -r civ.serv -f buggy1534.sav.gz To attach one or more clients to an autogame, remove the "start" command, start the server program and attach clients to created AI players. Or type "aitoggle " at the server command prompt for each player that connects. Finally, type "start" when you are ready to watch the show. Note, that the server will eventually flood a client with updates faster than they can be drawn to the screen, thus it should always be throttled by setting a timeout value high enough to allow processing of the large update loads near the end of the game. The autogame mode with timeout -1 is only available in DEBUG versions and should not be used with clients as it removes virtually all the server gating controls. =========================================================================== Data Structures =========================================================================== For variable length list of fx units and cities freeciv uses a genlist, which is implemented in common/genlist.c. By some macro magic type specific macros have been defined, avoiding much trouble. For example a tile struct (the pointer to it we call ptile) has a unit list, ptile->units; to iterate though all the units on the tile you would do the following: unit_list_iterate(ptile->units, punit) { /* In here we could do something with punit, which is a pointer to a unit struct */ } unit_list_iterate_end; Note that the macro itself declares the variable punit. Similarly there is a city_list_iterate(pplayer->cities, pcity) { /* Do something with pcity, the pointer to a city struct */ } city_list_iterate_end; There are other operations than iterating that can be performed on a list; inserting, deleting, sorting etc. See common/speclist.h Note that the way the *_list_iterate macro is implemented means you can use "continue" and "break" in the usual manner. One thing you should keep in the back of your mind: Say you are iterating through a unit list, and then somewhere inside the iteration decide to disband a unit. In the server you would do this by calling wipe_unit(punit), which would then remove the unit node from all the relevant unit lists. But by the way unit_list_iterate works, if the removed unit was the following node unit_list_iterate will already have saved the pointer, and use it in a moment, with a segfault as the result. The way to avoid this is to use the function void wipe_unit_spec_safe(struct unit *punit, struct genlist_iterator *iter, int wipe_cargo) and as the second argument write "myiter", a variable declared in the macro. The unit list will then automatically be kept sane. You can also define your own lists with operations like iterating; read how in common/speclist.h. [There was a time before the macros were developed, and raw genlists were used. Then a unit iteration would look like { struct unit_list *punit_list; struct genlist_iterator myiter; ..... genlist_iterator_init(&myiter, &punit_list->list, 0); for(; ITERATOR_PTR(myiter); ITERATOR_NEXT(myiter)) { punit=(struct unit *)ITERATOR_PTR(myiter); /* do something with the unit */ } } There are still examples of this in the code] ========================================================================= Network and Packets ========================================================================= The basic netcode is located in server/sernet.c and client/clinet.c. All information passed between the server and clients, must be sent through the network as serialized packet structures. These are defined in common/packets.h. For each 'foo' packet structure, there is one send and one receive function: int send_packet_foo (struct connection *pc, struct packet_foo *packet); struct packet_foo * receive_packet_foo (struct connection *pc); The send_packet_foo() function serializes a structure into a bytestream and adds this to the send buffer in the connection struct. The receive_packet_foo() function de-serializes a bytestream into a structure and removes the bytestream from the input buffer in the connection struct. The connection struct is defined in common/connection.h. Each structure field in a structure is serialized using architecture independent functions such as put_int32() and de-serialized with functions like get_int32(). A packet is constituted by header followed by the serialized structure data. The header contains the following fields: uint16 : length (the length of the entire packet) uint8 : type (e.g. PACKET_TILE_INFO) To demonstrate the route for a packet through the system, here's how a unit disband is performed: 1) A player disbands a unit. 2) The client initializes a packet_unit_request structure, and calls the packet layer function send_packet_unit_request() with this structure and packet type: PACKET_UNIT_DISBAND. 3) The packet layer serializes the structure, wraps it up in a packet containing the packetlength, type and the serialized data. Finally the data is send to the server. 4) On the server the packet is read. Based on the type, the corresponding de-serialize function is called is called by get_packet_from_connection(). 5) A packet_unit_request is initialized with the bytestream. 6) Since the incoming packet is a request (a request in this context is every packet sent from the client to the server) the server sends a PACKET_PROCESSING_STARTED packet to the client. 7) Finally the corresponding packet-handler, handle_unit_disband() function, is called with the newly constructed structure. 8) The handler function checks if the disband request is legal (is the sender really the owner of the unit) etc. 9) The unit is disbanded => wipe_unit() => send_remove_unit(). 10) Now an integer, containing the id of the disbanded unit is wrapped into a packet along with the type PACKET_REMOVE_UNIT: send_packet_generic_integer(). 11) The packet is serialized and send across the network. 12) The packet-handler returns and the end of the processing is announced to the client with a PACKET_PROCESSING_FINISHED packet. 13) On the client the PACKET_REMOVE_UNIT packet is deserialized into a packet_generic_integer structure. 14) The corresponding client handler function is now called handle_remove_unit(), and finally the unit is disbanded. Notice that the two packets (PACKET_UNIT_DISBAND and PACKET_REMOVE_UNIT) were generic packets. That means the packet structures involved, are used by various requests. The packet_unit_request() is for example also used for the packets PACKET_UNIT_BUILD_CITY and PACKET_UNIT_CHANGE_HOMECITY. When adding a new packet type, check to see if you can reuse some of the existing packet types. This saves you the trouble of writing new serialize/deserialize functions. The PACKET_PROCESSING_STARTED and PACKET_PROCESSING_FINISHED packets from above serve two main purposes: - they allow the client to identify what causes a certain packet the client receives. If the packet is framed by PACKET_PROCESSING_STARTED and PACKET_PROCESSING_FINISHED packets it is the causes of the request. If not the received packet was not caused by this client (server operator, other clients, server at a new turn) - after a PACKET_PROCESSING_FINISHED packet the client can test if the requested action was performed by the server. If the server has sent some updates the client data structure will now hold other values. The Xaw client uses XtAppAddInput() to tell Xt to call the callback functions, when something happens on the client socket. The Gtk+ client uses a similar gdk_input_add() call. [The packet system started out as a pain and it stayed that way all the time. After using Java's object serialize system, this handcoded serialize code really feels like the stoneage. That said, I fear it requires some advanced pre-processing system etc to automate the task for C structures. -PU] ========================================================================= Network Improvements ========================================================================= In previous versions when a connection send buffer in the server got full we emptied the buffer contents and continued processing. Unfortunately this caused incomplete packets to be sent to the client, which caused crashes in either the client or the server, since the client cannot detect this situation. This has been fixed by closing the client connection when the buffer is emptied. We also had (and still have) several problems related to flow control. Basically the problem is the server can send packets much faster than the client can process them. This is especially true when in the end of the turn the AIs move all their units. Unit moves in particular take a long time for the client to process since by default smooth unit moves is on. There are 3 ways to solve this problem: 1) We wait for the send buffers to drain before continuing processing. 2) We cut the player's connection and empty the send buffer. 3) We lose packets (this is similar to 2) but can cause an incoherent state in the client). We mitigated the problem by increasing the send buffer size on the server and making it dynamic. We also added in strategic places in the code calls to a new flush_packets() function that makes the server stall for some time draining the send buffers. Strategic places include whenever we send the whole map. The maximum amount of time spent per flush_packets() call is specified by the 'netwait' variable. To disconnect unreachable clients we added two other features: the server terminates a client connection if it doesn't accept writes for a period of time (set using the 'tcptimeout' variable). It also pings the client after a certain time elapses (set using the 'pingtimeout' variable). If the client doesn't reply its connection is closed. ========================================================================= Graphics ========================================================================= Currently the graphics is stored in the xpm fileformat, and accordingly loaded using libxpm. If you alter the graphics, then notice that after saving the new xpm file, you must manually mark the first color as transparent. Look at the original xpm-files, to see how this is done. Failing to do this results in crash(the mask-pixmaps will not be generated). Each terrain tile is drawn in 16 versions, all the combinations with with a green border in one of the main directions. Hills, mountains, forests and rivers are treated in special cases. [IMO we should have gotten rid of libxpm long ago. The graphics should be drawn in 24bit, and then quantized runtime, if the client is displayed on a pseudocolor display. The current tile system should be converted to something like civ2's. They get away with drawing way less tiles, and it looks better. -PU] ========================================================================= Diplomacy ========================================================================= A few words about the diplomacy system. When a diplomacy meeting is established, a Treaty structure is created on both of the clients and on the server. All these structures are updated concurrently as clauses are added and removed. ========================================================================= Map structure ========================================================================= The map is maintained in a pretty straightforward C array, containing X*Y tiles. You can use the function struct tile *map_get_tile(x, y) to find a pointer to a specific tile. A tile has various fields; see the struct in common/map.h When operating on tiles you normally iterate over x and y and maybe use map_get_tile() to get the tile struct. When fx iterating in a square around a map position (x,y) the naive method for (x1 = x-1; x1 <= x+1; x1++) for (y1 = y-1; y1 <= y+1; y1++) /* do something */ would sometimes, fx if (x,y) = (0,0) , give tiles like (-1,0) or (0,-1) Because the map wraps in the x direction the first position should be [assuming the map has the size (map.xsize,map.ysize)] (map.xsize-1, 0), while the second tile (0,-1) is not a real time at all! This could be solved by the following: for (x1 = x-1; x1 <= x+1; x1++) { for (y1 = y-1; y1 <= y+1; y1++) { int abs_x = x1, abs_y = y1; if (!normalize_map_pos(&abs_x, &abs_y)) continue; /* do something with abs_x, abs_y */ } } normalize_map_pos() will adjust the values of abs_x, abs_y to fix within the map, and if the original pos does not correspond to a tile, as (0, -1), it will return 0. Alternatively this could have all been done via the macro call square_iterate(x, y, 1/*radius*/, x1, y1) { /* do something */ } square_iterate_end; which automatically adjust the tile values. The defined macros should be used whenever possible, the example of a "manual" square iterate was only included to give people the knowledge of how things work. Also available is the function real_map_distance(), which takes wrap and such things into account. It does not however take terrain and roads, etc. into account. For that you would use a move_cost_map, which is generated via the function generate_warmap(), and calculates the distances from a tile to all other tiles on the map. See server/gotohand.c. Note that if all the code that operate on x and y values used macros, it would be very simple to convert these, to allow fx a flat world or a map structure like in civ2 for isometric view. Almost all functions expect that any passed map positions are normal (is_normal_map_pos returns true). Currently there is no known exception of this rule. To assert this the CHECK_MAP_POS macro should be used. ========================================================================= Unknown tiles and Fog of War ========================================================================= In the tile struct there is a field struct tile { ... unsigned int known; ... }; On the server the known fields is considered to be a bitvector, one bit for each player, 0==tile unknown, 1==tile known. On the client this field contains one of the following 3 values: enum known_type { TILE_UNKNOWN, TILE_KNOWN_FOGGED, TILE_KNOWN }; The values TILE_UNKNOWN, TILE_KNOWN are straightforward. TILE_FOGGED is a tile of which the user knows the terrain (inclusive cities, roads, etc...). Because of how the tile-drawing system works, the client has to know the terrain and specials of tiles adjacent to TILE_KNOWN and TILE_KNOWN_FOGGED tiles. This is done by sending the adjacent tiles with the value TILE_UNKNOWN. To keep track of which tiles have been sent and for fast lookup the server has a bitvector sent similar to known. [You could do this by seeing if the tile in question already have a known tile adjacent, in which case the value would already would have been sent.] Fog of war is the fact that even when you have seen a tile once you are not sent updates unless it is inside the sight range of one of your units or cities. We keep track of fog of war by counting the number of units and cities [and nifty future things like radar outposts] of each client that can see the tile. This requires a number per player, per tile, so each tile has a short[]. Every time a unit/city/miscellaneous can observe a tile 1 is added to its player's number at the tile, and when it can't observe any more (killed/moved/pillaged) 1 is subtracted. In addition to the initialization/loading of a game this array is manipulated with the void unfog_area(struct player *pplayer, int x, int y, int len) and void fog_area(struct player *pplayer, int x, int y, int len) functions. "int len" is the radius of the area that should be fogged/unfogged, i.e. a len of 1 is a normal unit. In addition to keeping track of fog of war, these functions also make sure to reveal TILE_UNKNOWN tiles you get near, and send info about TILE_UNKNOWN tiles near that the client needs for drawing. They then send the tiles to void send_tile_info(struct player *dest, int x, int y) which then sets the correct known_type and sends the tile to the client. [Beware of using map_adjust_y. If you have a 80x50 map and then iterate around the tile 40,49 you can get values like 40,50, which is not a tile. if you map_adjust the y value it will return 40,49. If you then think this should count as a unit seeing tile 40,49 and add a sight point for it at 40,49 fog of war gets out of sync. This will usually result in a wrap error on some point, as you unfog a square twice and it wraps around to 65536. This should get caught by the server and the error reported. I suggest using the is_real_tile function to test the tiles. unfog_area and fog_area tests their tiles this way, or using a macro like square_iterate that does the checking for you. Remember to recode the is_real_tile function when we switch to isometric view.] If you want to just show the terrain and cities of the square the function show_area does this. The tiles remain fogged. If you play without fog of war all the values of the seen arrays are initialized to 1. So you are using the exact same code, you just never get down to 0. As changes in the "fogginess" of the tiles are only sent to the client when the value shifts between zero and non-zero, no redundant packages are sent. You can even switch fog of war on/off in game just by adding/subtracting 1 to all the tiles. We only send city and terrain updates to the players who can see the tile. So a city (or improvement) can exist in a square that is known and fogged and not be shown on the map. Likewise, you can see a city in a fogged square even if the city doesn't exist (it will be removed when you see the tile again). This is done by 1) only sending info to players who can see a tile 2) keeping track of what info has been sent so the game can be saved. For the purpose of 2) each player has a map on the server (consisting of player_tile's and dumb_city's) where the relevant information is kept. The case where a player p1 gives map info to another player p2: This requires some extra info. Imagine a tile that neither player sees, but which p1 have the most recent info on. In that case the age of the players' info should be compared which is why the player tile has a last_updated field. This field is not kept up to date as long as the player can see the tile and it is unfogged, but when the tile gets fogged the date is updated. [An alternative solution would be to give each tile a list of the units and cities that observe it. IMO this would not be any easier than just counting, and would have no benefits. The current solution also gives the possibility to reveal squares as you like, say near a radar tower tile special. Very flexible.] [The barbarians and the ai take their map info directly from the server, so they can currently ignore fog of war, and they do so. I really think that the ideal AI wouldn't be cheating like this.] There is now a shared vision feature, meaning that if p1 gives shared vision to p2, every time a function like show_area, fog_area, unfog_area or give_tile_info_from_player_to_player is called on p1 p2 also gets the info. Note that if p2 gives shared info to p3, p3 also gets the info. This is controlled by p1's really_gives_vision bitvector, where the dependencies will be kept. If there is anything I have explained inadequately in this section you can ask me on . -Thue ========================================================================= Client GUI- Athena ========================================================================= One client GUI is written using athena-widgets. A few comments on this could prove useful for anyone wishing to write new dialogs or improve on the current ones. Widgets: -------- When you create new widgets for a dialog, like: players_form = XtVaCreateManagedWidget("playersform", formWidgetClass, players_dialog_shell, NULL); then put the widget properties in the app-default file 'Freeciv', instead of hardcoding them. For the widget created above, the following entries in the app-default file applies: *playersform.background: lightblue *playersform.resizable: true *playersform.top: chainTop *playersform.bottom: chainBottom *playersform.left: chainLeft *playersform.right: chainRight Pixcomm and Canvas: ------------------- The Pixcomm is a subclassed Command-widget, which can displays a Pixmap instead of a string, on top of a button(command). The Pixcomm widget should be used all places where this kind of high-level functionality is required. The Canvas widget is more low-level. One have to write an expose(redraw) event-handler for each widget. The widget generates events on resize and mousebuttons. [Reading any Xt documentation, will tell you how powerful widget subclassing is. So I went trough great troubles subclassing the command widget. It was not before long I got mails from unhappy Xaw3d (and derives) users, that the client keeps crashing on them. Turns out that subclassing from any widgets but Core, chains the new widgets to libXaw. In hindsight I should just subclassed the Canvas widget and add more highlevel functionality. -PU] =========================================================================== Misc - The idea trashcan =========================================================================== [Currently all of the major entities - units, cities, players, contains an unique id. This id is really only required when a reference to an entity is to be serialized(saved or distributed over the net). However in the current code, the id is also used for comparing, looking up and in general referencing entities. This results in a lot of mess and unnecessary duplicate functions. Often when programming, one wonders if some function needs the id or a pointer when referring to an entity. -PU] The paragraph above isn't true anymore for player, units and cities. -RF =========================================================================== Player-related entities in Freeciv - by Reinier Post + by dwp@mso.anu.edu.au Freeciv is confused about the meaning of 'player'. As a participant in Freeciv games, you'll notice that the participants are called 'players'. At the same time, players seem to be identified with civilizations. On the other hand, civilizations seem to be identified by 'nation': every player chooses a nation at the start of the game. In the data structures, a 'player' identifies a civilization, not a user. ---- THE PLAN: There are four player-related entities: + player A civilization, with a capital, cities, units, an income, etc. + nation A type of civilization (except that very little actually depends on nation, and the oddity exists that each player must be of different nation) + user Identifies 'someone out there', usually a human user running a civclient (this is the plan; it is not yet implemented). + connection Records a client connection; like a user, but disappears when the user disconnects, whereas for real users we may want to remember them between connections. See Connections section below. Where do these entities exist? Nations aren't actually used for annything that matters; for them, so the question isn't very interesting. Players (more aptly named, 'civilizations'), exist in games. Except in the context of a running game, the entity makes no sense. Players and their status are part of savefiles. A game can be saved and restarted on a different server; the players will be the same. A new game will have new players. Players exist in common/ (even games do) but a client only has one instantiated player. The reason to introduce users is client-side server commands. It must be possible to assign different levels of access to commands to different users. Attaching it to players is not good enough: the information must survive the addition and removal of other players, the start or restart of a game, reconnections by the same user even from different computers, or transferral of the game to a different server. However, a user may have different levels of access in different games. While they last, connections are sufficient identification for users. The user entity will allow users to be identified when they reconnect. Ideally, users would be identified with unique global ids, handed out by a 'registry service' similar to the metaserver, but this would be too cumbersome in practice. So the plan is to make users persist in a server session (even whan a game is started, or restarted when that option is added) and make them persist across games (when a saved game is loaded in a different server session). Users will be created when they first connect to a server, remembered by the running server and in savefiles. Upon connecting, the client will be sent a unique session id, generated when the server started, plus a fresh user id; it will store them in a ~/.civcookie file, and send it back when trying to reconnect. This will allow the identity of users to be protected. 'Protected' players will only allow the same user to reconnect; 'unprotected' ones allow anyone to connect; server commands and probably client options will be available to control this. Player names will be assigned to users, not players. The server maintains a default access level, which is used for new users and unprotected ones. ---- THE PRESENT IMPLEMENTATION: Currently access levels are stored in the connection struct. This allows access levels to be assigned to each individual connected player, which would not be the case if they were directly assigned to the player struct (due to the fact that the players array changes when players are added or removed). But that's it. Players are still created before the game is started, and player names still belong to players. Access levels exist in client and server, but only the server uses them. User ids are not yet implemented; Server ids do not exist at all. Commands to protect/unprotect users do not yet exist; they would serve no useful purpose. Access levels can set for individual users, including AI players with a connected observer, but only while someone is connected; they will not be remembered when the user disconnects. =========================================================================== Connections =========================================================================== The code is currently transitioning from 1 or 0 connections per player only, to allowing multiple connections for each player (recall 'player' means a civilization, see above), where each connection may be either an "observer" or "controller". This discussion is mostly about connection in the server. The client only has one real connection: 'aconnection' - its connection to the server, though it does use some other connection structs (currently pplayer->conn) to store information about other connected clients (eg, capability strings). In the old paradigm, server code would usually send information to a single player, or to all connected players (usually represented by destination being a NULL player pointer). With multiple connections per player things become more complicated. Sometimes information should be sent to a single connection, or to all connections for a single player, or to all (established) connections, etc. To handle this, "destinations" should now be specified as a pointer to a struct conn_list (list of connections). For convenience the following commonly applicable lists are maintained: game.all_connections - all connections game.est_connections - established connections game.game_connections - connections observing and/or involved in game pplayer->connections - connections for specific player pconn->self - single connection (as list) Connections can be classified as follows: (first match applies) 1. (pconn->used == 0) Not a real connection (closed/unused), should not exist in any list of have any information sent to it. (All following cases exist in game.all_connections.) 2. (pconn->established == 0) TCP connection has been made, but initial Freeciv packets have not yet been negotiated (join_game etc). Exists in game.all_connections only. Should not be sent any information except directly as result of join_game etc packets, or server shutdown, or connection close, etc. (All following cases exist in game.est_connections.) 3. (pconn->player == NULL) Connection has been established, but is not yet associated with a player. Currently this is not possible, but the plan is to allow this in future, so clients can connect and then see list of players to choose from, or just control the server or observe etc. Two subcases: 3a. (pconn->observer == 0) Not observing the game. Should receive information about other clients, game status etc, but not map, units, cities, etc. (All following cases exist in game.game_connections.) 3b. (pconn->observer == 1) Observing the game. Exists in game.game_connections. Should receive game information about map, units, cities, etc. 4. (pconn->player != NULL) Connected to specific player, either as "observer" or "controller". (But observers not yet implemented.) Exists in game.game_connections, and in pconn->player->connections. =========================================================================== Style Guide =========================================================================== See README.coding_style in this directory. - If you send patches, use "diff -u" (or "diff -r -u"). For further information, see . Also, name patch files descriptively (e.g. "fix-foo-bug-0.diff" is good, but "freeciv.diff" is not). - When doing a "diff" for a patch, be sure to exclude unnecessary files by using the "-X" argument to "diff". E.g.: % diff -ruN -Xdiff_ignore freeciv_cvs freeciv >/tmp/fix-foo-bug-0.diff A suggested "diff_ignore" file is included in the Freeciv distribution. ===========================================================================