WaterWiki.net:Development/Spring 2009 Report

From WaterWiki.net

Jump to: navigation, search


WaterWiki 2.0 Phase 1 is 10-4 over and out!

Six months ago, the WaterWiki project consisted of MediaWiki with an aging collection of extensions and hacks, a loving but scattered development and management process, and an untapped potential. With a major push on content, organization, and the technicals, we relaunched WaterWiki with a fresh look, comprehensive scope, and improved software. WaterWiki is poised to be _the_ cooperative resource to serve knowledge needs of the global community of international organizations working in water governance. Phew! We've all put in a tremendous amount of work. In particular, I was impressed with the growing technical sophistication and understanding of very geeky concepts of the entire team.

It's not all boosterism. My honest appraisal of the work must reflect delays and unfulfilled goals. All members of the team had many time constraints and competitions for their attention. And coordinating was a challenge, with a team distributed across several countries, constantly on the move anyway, and a number of big deadline events. On the technical side, my estimates for the possible scale of work hit a different reality, and in some cases, desired features had to be pushed aside for later phases of development. At this late date, there are still a few remaining bugs in the new platform we're scrambling to sort out. Much appreciation to the entire team for their patience and understanding of the somewhat unpredictable world of software. We've improved weekday visits from around 150 to 500, but there's still a long way to go, and content creation is still centered in the core team. We have the challenge of inviting an ever larger audience WaterWiki to use and contribute to WaterWiki, and technical development is key to success.

This report will briefly summarize my work over Phase 1, with an overview of our entire technical approach. And then I'll make a number of technical recommendations for further phases of development. The sections are divided into "Structured Wiki Management", "Social Spaces", "Inter-Agency Collaborations", "Maps", "Dynamic Data", "Presentation", "Organization and Reaching Out", and "The Rest".


At the start, a technical decision was made to continue with MediaWiki as the basis of the WaterWiki platform. A great deal had been invested already by Waterwiki in MediaWiki; the core codebase had improved, extensions proliferated, and understanding of the both the user and developer angles of MediaWiki grown in the wider community. It's the number one choice for deploying wikis. On the minus side, code quality among extensions remains varied, and PHP remains an easy language for the creation of ugly code. The technical work began with an update to the latest MediaWiki codebase, a process which involved a somewhat painful data migration to the new schema.

From two years of ideas and brainstorms, we had to boil down raw exploration into a doable work plan. I instituted a "digestive process" of wiki pages (WaterWiki.net:Development), which takes ideas through "Investigations" into "Solutions" with associated "Tasks". "Bugs" are problems with existing functionality, and presently out of scope features are archived to "MaybeSomeday". We also kept a seperate section for discussing "UI Design", and to keep tab on our "Partnerships".

I spent a good deal of time on support. At the start, we hashed through a number of information architecture options. I helped support the user side of content creation, with much in depth investigation into MediaWiki functions. I assisted Petr's work on styling with introductions to the code development process. Developed projects for volunteer developers and thinkers. And reached out to developers and thinkers at other international organizations, and in the wider community.

The core technical challenge was to impose semi-structure on the unstructured medium of the wiki, a not always easy fit. This solution was dubbed "Layouts". A combination of extensions, especially the Uniwiki extensions, and conventions have helped to develop a flexible architecture that can be both completely open and helpfully structured. The core element of the wiki is identified as an Article Section, which can be specified in layout templates, and transcluded across different pages of the site. Technical work involved migrating prior templating system, adding new section directives (@oldtitles, @insert-above, @sysoponly), and resolving existing bugs in the extensions. In a sense, we're giving some database like concepts to a wiki, but not requiring strict schemas.

We've also made heavy use of "CategoriesTags" and Templates, in ways that test the boundaries of what these builtin features can offer. Categories and subcategories are utilised to designate geographic relationships between regions, countries, and water basins. The editing tool was updated to use both a tag cloud, and a structured tree of commonly used categories (including Organizations).

"DynamicPageList" is an extension that's progressed far since originally deployed with WaterWiki. A number of existing bugs were fixed, and its capabilities explored in great detail. It was majorly extended to support "Maps" .. a Google map can now be output directly from DPL, in contrast to the prior cumbersome GeoRSS extension, 2 page solution. The ParserFunctions and Variables extension also increased the builtin range of MediaWiki operators.

"UserManagement" has been improved, with a proper system for requesting accounts (ConfirmAccount extension), and the SocialProfile extension adding rich profiles and greater connection possibilities between users. SocialProfile in particular was heavily rewritten to support local modifications, and support more changes in the future.

Besides these five "Solutions", many small and big bugs were squashed.

Structured Wiki Management

Use of layout, categories, DPL, and templates are now extensive, and maintenance is going to be complex. There's a need for tools for administrators and users to keep on top of things.

  • Better creation and editing of layouts. Configuration should be easy, with UI options rather than directives. Renaming and re-ordering sections should be intuitive. Control of the presentation on edit pages is needed -- such as size and kind of edit control.
  • When layouts have changed, admins need a web accessible way to apply layouts to existing pages, which does not rely on developer resources.
  • InfoBox templates contain a great amount of content, in a hard to maintain format. Maintenance could be done in specially constructed screens, or through data integration points (see below).
  • Some content in the wiki is more likely to go "stale" than others. For instance, the chair of UN-Water will change in 2009, and the UN-Water article will need to be updated. Possible solutions may include tagging articles with a category ("MayChange") which are reviewed periodically; or extension of watch features to notify admins to check articles on specific dates.
  • On backend side, we must insure that heavy use of transclusion and DPL, etc can scale; this will require careful application of caching.

Social Spaces

WaterWiki must be a place where other people are evident. A lively connective space, this is key for continuous engagement of the wide, dispersed community of water governance. If it's evident that people are present, people will contribute and collaborate more.

  • More evidence of user contributions. Most recent contributions, popular pages, top users, new users, highly rated, should be cleanly presented up front.
  • Pages need some facility for discussion. This might be best handled as specially displayed Sections.
  • The SocialUser plugin needs further cleanup, with better admin management of the profile question options.
  • Users should be taggable with taxonomy .. particular country and organization. This leads to better, searchable user directories.
  • Integrate user profiles with other Social Platforms, like LinkedIn and UNDP Workspaces. We don't want users to have to replicate information across multiple systems.

Inter-Agency Collaboration

Across the UN system, and beyond, systems and projects have been developed for collecting knowledge on water governance. WaterWiki.net does not wish to replace anything systems that are doing their job perfectly well, but rather, provide linkages between databases and documents in a sensible way. We promote open data sharing, using open standards, developer apis and open licenses, so that WaterWiki and the web at large can take best advantage of the tremendous resources being developed.

Integrations and collaborations are generally going to need to be taken on a case by case basis, but certain patterns will emerge. There are degrees and directions of linkages. At the simplest, links are manually and systematically collected in WaterWiki. If portions of the partner content are relevant to summarize or highlight, and their content can be exported in some structured form, import scripts can be produced to add that content into WaterWiki; typically, this content would be locked from direct editing, with wiki space provided for users to contribute: for instance, the WHO Lexicon official definitions are locked in WaterWiki, with space for contributions below. It is hoped that user contributed content would eventually be seen as valuable and useful for partners, so WaterWiki would provide export content, most likely in RSS/Atom feeds. If content on either side is updating directly, and a standard, reliable method of transfer has been proven, we can look at automated the process.

There may be opportunities for collaboration beyond the direct remit of WaterWiki. For instance, the "Maps" concept above, or the UNESCO water basin community workspaces. In these cases, another technical platform may be better suited. WaterWiki's role, at the least, should be to advocate good web practice -- open source, standards, and data.

Suggestions for the current possible integrations.

  • UN Water Presence document. The relationships in this spreadsheet were entered by hand, converting the spreadsheet into categories and subcategories. This could be an automated process, if we expect it to change substantially and/or frequently.
  • The WHO Lexicon has been updated. We should endeavor to make this an automated periodic process. The current output from WHO is simply a database dump; we should lobby for a more reliable transfer format. And we should lobby for feedback of user defintions back to WHO.
  • GEF projects database. GEF has a comprehensive reporting system, and a good amount of geographic data. Good opportunity here for integration.
  • UNESCO Encyclopedia of Life. A "just do it" example may convince them to open up.
  • FAO Aquastat. A huge resource on legal regimes, that could be integrated into country pages.


In discussion with partners, it's evident there's a great amount of geographic data on water governance distributed about the UN system: the location of projects, the location of infrastructure, and the precise boundaries and lines or water systems and the overlapping jurisdictions governing them. There's a real opportunity for promoting geo data sharing and collaboration between organizations, a congruence but somewhat beyond WaterWiki's current mission and technical reach. In spirit, Wiki principles should apply to maps and GIS, it's just that the particulars look a little different. UNGIWG is the inter-agency working group tasked with promoting geographic data sharing, there's currently little representation on water themes; this is a charge WaterWiki could lead for UN-Water.

  • WaterWiki currently indicates relationships between water basins, countries and regions using tags. If the polygon boundaries of these entities were collected and stored in a PostGIS database, these relationships could be automatically inferred. Further, the lat/long location of an Article could automatically be placed in a country, water basin, etc.
  • Many static map images have been uploaded to WaterWiki. These can be georeferenced using open source tools like MapWarper. They're then available for display as togglable layers through OpenLayers, etc.
  • Export WaterWiki geotagged content in a number of formats: GeoRSS, KML, Shapefile.

In the current system of simply tagging articles, there's a number of recommended improvements.

  • custom markers for different types of content
  • choice of cartographic background (emphasizing water basins, currents, etc)
  • ability to tag an article with multiple points.

Besides point based mapping, there's a need for country thematic mapping. Many datasets are organized on a country basis. We need to investigate how to import such data, how to store country based statistics, and the best method to display.

Dynamic Data

The nature of wiki content is that it constantly changes. The task is to make this evident and simple to comprehend and manage. There's conceptual overlap here with other sections, but some particular issues will gather under this section.

  • Integrate RSS feeds, delicious links, you tube videos, automatically through tags and specified accounts. Potentially use widget extension.
  • Enhance watch functionality. Email alerts on watched pages. Timers for reminder about stale content.
  • Enhance DPL. Make the "categories of this article" an option available to DPL. Tag cloud output from DPL.


The look and feel is way way better, but there's still a ways to go. Here are some opinionated recommendations.

  • Front page. Front page itself can make use of a grid layout to present a more comprehensive view of the scope of WaterWiki. Each major "component" of WaterWiki should have some clear presentation on the home page.
  • Make text and content bigger, and use more of the screen real estate. Small fonts help cram more in, but crowds the page.
  • Clearer UI distinction for different types content. Subtle visual cues should help contextualize the content. Colors could be employed.
  • Low bandwidth/offline. A mode that only presents the essentials, minus lots of javascript, images, maps. An offline CD could be developed for distribution to field offices with little bandwidth. Automatic synchronization could update those installs.
  • More flexibility with the sidebar .. it's not needed on every page.
  • Breadcrumbs need better positioning.

Organization and Reaching Out

WaterWiki has received numerous commendations for its open approach to its own development. All our thoughts, scribbles, and work, and lessons, are open for the world to see and learn from. We've actively sought out connection with organizations working in similar capacity in different domains, and with developers working on similar technologies. This capacity needs care as WaterWiki continues to scale up.

  • Using the wiki for the development process has worked decently well, but it's still a strain to keep it up to date and gardened. Consider moving development to an issue tracking system (redmine looks great these days).
  • Domain ownership and responsibility. Consider forming a new organization to hold WaterWiki assets and liability.
  • A great many improvements have been made to MediaWiki extensions. Effort should be made to submit patches. Future developments should reach out to the wide MediaWiki development community to coordinate efforts.
  • Work towards formalizing the community of forward looking international organization web development. Encourage another entity to take the lead (OpenDevelopment ?)
  • The main page should highlight this aspect of WaterWiki.
  • Active editing on Wikipedia, linking to relevant content in WaterWiki

--> LinkedIn WaterWiki-Group (sub-group development & SYSOP)

The Rest

The rest don't fit any where else. The kitchen sink.

  • Search needs additional improvement. Consider deploying a seperate indexer (sphinx?). Search page results should be visually richer, showing page type, and some taxonomic classifications.
  • Continue to investigate wysiwyg. Or at least make the editing process even easier.
  • Upload needs better integration.
  • New content types: events.