Posts tagged "Opendata"

Dubai Government Treating Data as a Corporation

On Tuesday, December 20th I visited the Dubai Government Achievements Exhibition at Dubai International Convention & Exhibition Center. Imagine a traditional exhibition, for example of tech companies with all that fancy booths where they show-off their achievements and plans for the future. However instead of the companies there were governmental departments: from healthcare (with healthy breakfast demonstration) through justice, customs to business development and statistics. There were around 30 of them. Just amazing.

As a data brewmaster, the one I was most interested in was the Dubai Statistics Centre (DSC). Big flat screen with dashboard full of charts caught my eye immediately. I was greeted by Mr. Hassaan Mashmoushi, IT Specialist at DSC, who gave me a very nice overview of something amazing: governmental dashboards. It looked like traditional corporate dashboard portal, nothing special you would say. Read again: it was about the government by the government!

From Population to Real Estate

They have developed reporting portal where users can browse data from population, financials, through real estate to passenger flows in a particular metro station. Users - analysts can view wide variety of subject reports, they can drill down the data by various dimensions to get better insight.

I was told, that they treat government as a company and the residents are customers. Sounds very reasonable, from political point of view it is worth another discussion, from data point of view it is excellent concept. The big corporations already refined all the tools and processes necessary to understand what is going on in their business. They know how to collect data from various systems, put them together, clean them, refine them and visualize them for people to use them and make decisions based on them.

Where the Data Come From?

Data are currently being collected from “only” 25 departments. The DSC is quite liberal when it comes to the format in which the data might be provided: it might be excel spreadsheet or a database, the tech guys will take care of that. I consider this very good way of doing it, mostly at the beginning of the project. The portal would be useless without data, so it is necessary to get as many datasets as possible with as little effort from the data owners as possible. Departments do not seem to be against giving the data.

If a department has data in a transactional system (non aggregated, detailed transactions, such as expenses or metro passengers passing), they take data snapshots, as it is usual in data warehouses.

I didn’t learned much about data quality assurance in the process. However, I was told that the DSC is using set of business rules for some checks. For example they check the trend of certain fields and if the increase/decrease is much bigger than in previous couple of months, then analysts are warned about the fact.

Who is Using it?

Intended users of the system are analysts in the government departments. The system is not open and is currently available only to all governmental departments (not only those providing data). They plan to open it to the businesses in the near future.

Openness and The Yearly Book CD

At the booth they were giving away a CD “Yearly Book for the Dubai emirate”:

The CD sleeve reads:

The full set of annual statistical yearly book for Dubai emirate, enclosing the first annual book of 1988 up to the last one 2010.

The library includes series of statistical reports regarding various demographic, social and economic sectors for 30 years starting from 1979 up to 2010. These reports reveals Dubai success story as well as its sectors’ enormous development in figures.

Unfortunately, I was not able to browse the yearly book as it is MS Windows only.

I am big proponent of open data, and of opening the data to the public. I did couple of projects myself. However, there is little bit different situation in the region of UAE compared to the parts of our western world. We, in the west, are used to the government which hides data to hide its not-so-pleasant decisions towards its citizens. Here, in Dubai, where governments acts like a social company, you hear almost no complaints and journalists seem not to be very interested in digging into causes - because they do not need to. Therefore they do not need data to show to public: “hey! here is the proof that you are doing it wrong”. Journalists in western world are using open governmental data mostly for calling names and showing inefficiencies which are mostly ignored back by the government (at least in my experience). Dubai uses data for what data should be used for: improvement.

On the other hand, opening the data to the public might attract more analysts and give unexpected findings and suggestions. Also the actual set of reports might seem to be limited to some analysts and they might be missing raw data to play with.


For people with corporate BI experience the concept is definitely not new, but the environment where it is being used is. Dubai Statistics Centre is an example how government should use data: from collection through analysis to decision making. Having such tool gives much better understanding of what is happening and the understanding is supported by real data. It is a joy to make improvements and better decisions with all this at hand.

I think that the DSC and Dubai Government set a very challenging goal for other governments to achieve.

I would like to thank Mr. Hassaan Mashmoushi for his time and exciting presentation.

Open question: How do you think that Dubai government can benefit from having the data fully open? What are the pros and cons?

P.S.: Oh, did you know that you can follow Dubai Statistics Centre on Twitter? In arabic mostly, but anyway.

Tags: opendata opengov data

Open Data: Open Public Procurements of Slovakia launched

I am happy to announce that Transparency International Slovakia launched a project I’ve been working on: Open Public Procurements of Slovakia.

The challenge was to get something understandable from many not-so-nice documents:

Visitors can browse and see graphical aggregations of the public procurement contracts. They can see either graphical reports for comparison, list of contracts or a contract details with reference to the contract data source document. The portal answers questions that Public Procurement Office answered with “we do not have enough human resources and time to go through all the contracts to provide you the number”.

The data are scraped daily from public procurement office pages. They are processed, cleansed and changed into useable analytical form.

My part of the work is the whole invisible extraction, transformation, loading (ETL) process behind and the online analytical processing (OLAP).

For technically curious: the project is built on top of my light-weight OLAP framework Cubes (github).

I hope this helps to increase transparency in our country and that other NGOs will follow the example of TIS.

Tags: opendata data