How healthy is your data?

Posted on February 21, 2017

One of the biggest misconceptions in business is believing your data is accurate and useful for basing decisions on.

In reality you are looking at a set which has been collated from a range of inputs that have been collected together.

The 'data' is never right or wrong – it is the set that you have assembled (or that someone else has assembled for you) and it is only as good as the assembly process.

Set theory was first developed by Geog Cantor in 1874 but languished without a practical application. The first such application was provided by Ted Codd at the IBM Research lab in 1970 with his seminal paper that described the rules for how relational databases would work.

Some far-sighted educationalists started introducing set theory into the math curriculum as early as 1967, which was controversial at the time with most parents failing to understand how important and pervasive set theory would become.

There are many things that can go wrong when joining data components together. Having a healthy database will ensure fully connected data will allow for the creation of robust sets. The content of the set is controlled by filtering, the assumptions of the collator, and the presence of conditional data - did the person constructing understand all the nuances hidden in the data?

Data can become unhealthy in many ways for example where data entry forms allow ambiguous data entry or allow data that should be stored in one field to be stored in another. Does your system allow records to be saved with incomplete data? How often do your business processes and incentives discourage front line staff from entering all the data that you will need to construct robust sets for reporting and analysis?

Do your report writers and designers ever calculate what the final set should look like before constructing their sets and then compare actual vs predicted, or do they just build a report and hope that it's 'right'?

Does your system store spatial data in a poor relational schema which encourages unhealthy set building? How valid are the datasets that you are relying on? Data only makes it through if it is fully connected - how much of your data is simply hidden from you when you are making your decisions?

There is plenty that can go wrong when compiling a set of data even when the data is in a healthy condition. Much more can and will go wrong when it is not. To make sound decisions for your business you need to have healthy data.

What processes do you have in place to understand the health of your data? Chances are, if you haven't got these processes in place already, then it's not going to happen without outside help. You need to be able to quantify the health of the data in your system. You need to be able to identify problem data areas so that preventative strategies can be put in place. This stops the problem growing. You then have the option of rectifying existing problems should you choose, prioritising on data that affects critical decisions and processes.

Finally you need to be able to measure whether the health of your data is improving or deteriorating over time. By being able to link unhealthy data to the actual costs, losses and poor KPI performance that the business is exposed to, you can demonstrate a clear business case to take positive action.

Big vs Boutique

Posted on February 20, 2017

Which should you choose?, ...when?, ...and why?

'No-one ever got fired for buying IBM'. A perennial risk-management mantra. The other side of this coin is the implication that you'll be obtaining the very best mediocrity that money can buy. You'll be pleasantly surprised if you receive excellence but you're not really expecting it.

There are Pro's and Con's – and the choice depends on the fit you need.

When you buy big, you're basing your purchase on reputation. You are tapping into an organization with large resources that will be able to deliver. You are engaging an organization that can offer you a service that they have done many times before, and they know just what to do to 'crank the handle' to churn it out. The level of comfort this provides is certainty.

When you buy boutique, you're going to be engaging a specialist. Someone with a penchant for a particular technology, and/or someone with significant experience in a single vertical market segment. You're engaging someone with more intensive knowledge in who is going to produce the right results, quickly and effectively. They are not going to apply the one size fits all solution to your business problems.

More to the point they are a specialist. With the larger corporates you are likely to have the actual work undertaken by a more junior person. The senior partners are more often doing the sales work, engaging the client and overseeing the account rather than using their extensive expertise to deliver the results. The extra distance between the specialist and the coal-face means that insights that they might have detected may be obscured through the inexperience of the practitioner doing the work combined with the routinized approach to the analysis.

The boutique specialist will likely understand your industry already and you aren't going to be paying the consultant top hourly rates for you to educate them! They will know what questions to ask, and they come with a point of view. A boutique consultant should tell the client exactly what the problem is, without sugar-coating, or worrying about the egos of the people involved. You're paying to have the problem fixed, not massaged out of view.

Where a consultant discovers that part of the problem domain is outside of their expertise, a good consultant will engage others to augment the knowledge gap. The difference here is that a boutique consultant will engage another specialist whereas a corporate entity is more likely to look in-house for those skills and find the closest match.

People engage consultants for a variety of reasons, but the most common are:

  • They lack the expertise to undertake the work
  • They lack the resources to do the work
  • They require external assistance to make a tough decision

Before you engage a consultant, whether that be big or boutique:

  • Decide whether you are committed to fixing the problem
  • Don't set overtight deadlines if you don't really need them

If you know the answer you want already, but you want to instruct an 'independent' company to make the recommendation, then buy big. No one is going to argue with their reputation, but more importantly boutique companies aren't there to rubber stamp solutions, particularly if the solution doesn't match their point of view. Boutique companies stand and fall on their integrity and they are unlikely to take you on as a customer.

Often companies seeking the services of consultants set very tight timeframes, both to respond to proposals and to undertake the work. The tight timeframes are usually related to meeting internal KPIs only. Big consultancies are geared up to meet these tight timeframes and boutique ones will struggle for such a timely response. The reality is, that once a recommendation is delivered the customer rarely acts on it promptly. Setting tight timeframes might tick your boxes and keep your boss happy, but if your goal is to get to the root of the problem and generate custom solutions in areas where specialist expertise is an essential component, then give yourself and your preferred consultant time to generate the best outcomes for you.

How Self-Driving Cars, Will Drive The Spatial Industry

Posted on October 13, 2016

The race for self-driving cars began in the 1920’s. At least that’s what Wikipedia leads us to believe with its broken links and citations. But it is likely to be true on some level. Autonomous cars are separated into 5 classifications from level 0 - issuing warnings, like a reversing proximity beep, to level 5, no human interaction except setting the destination and starting the system. We have been aware of the approach of the class 3-4 self-driving car for half a decade, but it seems that the time is nigh for those in the driver’s seat to get a product to market. End of bad car puns. 

Some manufacturers like GM are looking to have their automated cars on the road and fully functional by 2016 but they will only be available to hapless employees willing to put their lives on the line for their employer. Some companies are focusing on a 2018 for release of their cars, while most are pushing this out to 2020 and even 2030. Perhaps there is some reluctance to be the first company to release the tech for fear of being the industry scapegoat. No-one wants to see their brand included in the first autonomous death headline regardless of who’s at fault.

Good for the environment bad for the wallet

Shared Autonomous Vehicles (SAV’s) for carpooling are some of the driverless vehicles set to replace conventional cars. At just 5% penetration it is estimated that one SAV would replace 11 cars on the road. To put this into layman terms, if 200,000 SAV’s existed in NZ the rate of car ownership could drop from 3,100,000 to 900,000. This might be generous at best however the environmental implications are positive nonetheless. Unfortunately the outlook for public and commercial transportation is bleak, especially if you are a driver. The demand for tech personnel is almost certainly going to increase, but it will be disproportionate to jobs lost. In a fashion similar to automated checkouts, the number of staff needed to service, upgrade and manage these units will be significantly less than the number of people displaced by the technology.

If the demand for cars and ownership drops, then the cost to produce them would certainly increase. So too might the cost of servicing vehicles as the number of privately owned vehicles declines. So how is the automotive industry going to prevent self-cannibalisation? While there is a lot of talk of SAV’s and driverless cars replacing driven ones, the truth is that the economic model suggests that they will only be competitive at best. And by 2032 it is expected that 50% of new cars, not all cars, will be autonomous. So there is still a long future ahead for current conventional cars, drivers and oil.

Another factor that will reduce uptake of these vehicles is cost. One source talks about the cost of the self-driving Prius from the YouTube advert being greater than the price of a Ferrari 599. Why? A Prius is $24,000 USD, add LIDAR, sensors and a GPS array at roughly $75,000, $10,000 and $200,000 respectively and you begin to understand how limited the first round buyers’ market might be. Although a quick google shows that these costs are reducing, with some GPS arrays starting at only $8,000, once you add on the cost of development you begin to understand that self-driving cars may be out of the question for many of us in the foreseeable future. It might also be the primary reason this tech is taking so long to reach the market.

NZ’s View

Realising that autonomous cars will arrive somewhat soon, the NZ government quickly noted that law in NZ has not been written with these brain-less chaperones in mind. What this means is a change in legislation around the requirements for autonomous vehicles and the rules which govern them. Ministry staff have already been sold hosted by Nissan and Google and shown their driverless vehicles and the Ministry of Transport has basically said, perhaps more eloquently, ‘NZ is the perfect test ground for driverless cars, send the NZTA an email if you wish to test drive yours’. While changes to legislation are currently only occurring for testing driverless cars, we can still expect a change in the law surrounding the ownership, importation and use of autonomous cars soon.

Implications for GIS

As the government does what it does best and gets buried in paperwork, hardware and software are being developed within the driverless vehicle realm that embrace geospatial technologies. It is the development of this GIS-ware that could help continue the industry growth that has been occurring and predicted to continue for some time.

Peugeot Hybrid Cutaway

Routing efficiency, traffic management, electric energy and other technological breakthroughs will be the source points for innovation and competitiveness for the driverless car market. Infrastructural changes will also be determined by the adoption of these new technologies. And the vast volumes of spatial data being collected by these new mobile sources will provide more sources for collecting big-data. Sorry for using the b-word. Google’s self-driving car fleet has racked up an impressive number of automated miles (700,000 by 2014), during which it has been collecting vast amounts of data. As more autonomous cars hit the road more data will be captured, perhaps simply to keep google street view updated, but probably for so much more.

As well as innovation within the geospatial-driverless car sector, we can expect these breakthroughs to transfer directly into other GIS related industries. Development of cheaper, more reliable, smarter, source and sensor technology will have an effect on the number of business who can enter the geospatial market and the types of analysis that can be performed. And continual growth of the industry will ensure that new business's find niche untouched positions. 

- Cody Kinzett

Game Of Drones: Innovation Or Security Threat

Posted on August 29, 2016

In recent years, drone innovation has been accelerating at such an exponential rate that regulations have struggled to keep up.

The capabilities of this technology is limitless - from the positives such as: filming athletes in a race; delivering medicines to remote places; mapping terrains; or checking the condition of a bridge - to the negatives such as: spying over private property; sneaking drugs or electronics into a prison; or risking lives by flying dangerously close to airplanes.

Image: The DHL test "microdrone md4-1000" for the delivery medicine

Many of these innovations are already providing amazing scientific, economic, and social benefits. But if standards and guidelines are ignored, or if the technology gets in the hands of those who want to cause harm, will the rules be enough to protect ourselves with confidence?

Ethical, privacy and safety breaches are providing strong reason for society to be concerned about the vulnerabilities.

Already in New Zealand there are 2645 registered drone users, 968 commercial drone operators, and 400 registered drone companies. In the US, the commercial drone industry is burgeoning, with researchers predicting it will generate more than USD$82 billion for the U.S. economy by 2025.

A report on "beyond-line-of-site" drones (also called Unmanned Aerial Vehicles or UAV, and Unmanned Aircraft Systems or UAS), estimates that drones will benefit New Zealand by up to NZD$190 million per year across just three sectors studied - if we can get the regulatory environment and technology right.

The questions are, will the risks outweigh the advantages of drone technology, and do we have sufficient capability and resource to manage infringements?

Drones – the opportunities

Originally used for target practice by the Royal Marines in the 1930’s and 1940’s, drones today are commercially available on a large and increasing scale. For a few years now, they have been helping a number of government, research and industry organisations to gain access to new and valuable data more easily, quickly and efficiently than ever before.

There is still huge potential of this new technology to be explored. And, with our vast multi-level terrain, multiple climates, weather conditions, and low population, New Zealand is one the world’s most active hot beds of drone development. What might have been considered science fiction just a couple years ago, is fast becoming a reality today. For example, in agriculture, farmers are already using drones to monitor their stock and pasture cover remotely - saving them considerable time and fuel.

The average small drone can be launched within minutes and fly over a range of five miles for up to 90 minutes. They can also be fitted with other technology, like high resolution infrared cameras that can zero in on suspected criminals, wildlife, poachers, missing people, or monitor events, and more.

Drones are also being used to help with emergency response and disaster recovery, improving security, helping with pest control and erosive monitoring. There are even trials underway to discover if drones can be used safely for delivering freight direct to the buyer’s door, transporting medicines, and more.

Even in the GIS industry, a low altitude small drone could provide surveyors and GIS professionals with a more cost-efficient alternative to the georeferenced photographs taken by manned aircraft or satellites. More so if the drone is kitted with LiDAR and camera equipment to create a turnkey remotely piloted flying LiDAR scanner that can capture rich and accurate images more frequently and cheaply.

In fact, GIS is predicted to be the second biggest commercial drone market behind aerial photography and cinema, and ahead of precision agriculture.

Image: Top commercial drone industry trends based on DroneDeploy usage data.

One example of this is the research team at Auckland University’s School of Science which specialises in unmanned aerial low altitude sensing and geospatial analysis for ecological and environmental monitoring.

Using a swarm of flying drones, the AUT UAV team is creating high resolution maps of habitats and landscapes, monitoring wildlife behaviour, and examining the human impact on the environment. They are also working with a Swiss company to create 3D mosaic landscapes and turning the images into a virtual reality experience. This allows people to visualise and understand their environment in a way they’ve never experienced before.

Disruptive innovation, or a potentially catastrophic disruption?

Because drones can dramatically lower the cost of data collection and analysis, the pace of development has been phenomenal.

The challenge for policy makers however has ranged from determining how to protect public safety and personal rights, such as privacy and land ownership, as well as areas of national, historical, or natural importance from potential harm caused by drones – either intentional or accidental.

It was only last year when the Civil Aviation Authority in New Zealand announced new rules to improve aviation safety for drone operators, other airspace users, and for the general public and their property - eight years after the first known incident was reported in the country.

And it was just this month that the White House Office of Science and Technology Policy in the US announced new steps to promote the safe integration and innovative adoption of drones across the country.

While the rules may prescribe more explicit constraints upon the use of drones than was previously the case, understanding and abiding by them is another matter. And as more and more drones enter the market, understanding and enforcing compliance will be challenging for both users and the authorities. For example, Amazon and Google's drone package delivery plans have already hit a stumbling block with the new US regulations.

What next?

By 2018 there will be an integration of airspace in New Zealand, where drones will be part of the transport grid. With increasing drones in our skies, the next challenge will be innovating a way to manage the airspace and avoid collision.

Meanwhile, the race is on with innovation in drone technology happening at warp speed around the world, and everyone is joining in from hobbyists to global brands. Amazon has been trialling its Prime Air drone delivery service in the UK, Dominos has been working on various air and land based drones to deliver pizzas, and last but by no means least Facebook is now racing against Google to deliver 5G to unconnected parts of the world with its solar powered Aquila drone that can fly for months.

To ensure their economies don’t miss out on a slice of the lucrative drone pie, governments are providing incentives for innovators to research, develop and commercialise drone technology.

This month the US announced USD$35m in research funding for the National Science Foundation to accelerate the understanding of how to intelligently and effectively design, control, and apply drones to beneficial applications - such as monitoring and inspection of physical infrastructure, smart disaster response, agricultural monitoring, the study of severe storms, and more.


Image: C-Prize UAV Challenge by Callaghan Innovation.

Last year Callaghan Innovation in New Zealand launched the first C-PRIZE UAV challenge - a NZD$50,000 incentivised challenge that aims to advance the commercialisation of innovative drone technology for the screen industry. Team VorTech won the prize for their Gyroscope UAV that uses an innovative propeller design that allows thrust in any direction, helping it hold position in gusty winds.

Like other countries in this drone innovation race, New Zealand has been ploughing funding into firms like Aeronavics to ensure the pace of research and development stays lightning fast.

If users can work around and abide by the new rules and regulations, then the future looks bright for both the drone industry and for society. But if rules and resources fail to protect your personal safety and privacy, what would you do to protect your rights?

- Shelley Grell @ Communicate IT

GeoBlog posts

Posted on February 21, 2017
Posted on February 20, 2017
Posted on August 23, 2016
Posted on January 26, 2016
Posted on May 27, 2015
Posted on May 27, 2015
Posted on November 2, 2014
Posted on July 2, 2014
Posted on July 2, 2014
Posted on July 2, 2014
Posted on April 8, 2014
Posted on April 8, 2014
Posted on December 19, 2013
Posted on December 19, 2013
Posted on December 22, 2011
Posted on November 13, 2011
Posted on September 1, 2010