Data Architecture Form Ever Follows Data Intelligence Function

Louis Sullivan’s pronouncement, “form ever follows function,” clarifies one of architecture’s most essential truths: function first.

Although Sullivan’s specialization with building architecture inspired his insight, data architecture, done well, reflects the same principle. Today, we often say, “let the use case drive the bus.” The meaning is the same. Data architecture must form to serve the function of the use case and the business challenge.

Within the world of data architecture, we may become distracted, focusing on data platform wars and the elusive capture of ideal data ecosystems. Our paths through the myriad of tools and platforms can easily catch us wasting resources seeking form for form’s sake.

We might invest heavily in distraction due to the aggressive competition among suppliers of the various data architecture components. All data professionals have experienced the capability leapfrogging among their choice of suppliers and the difficulty of making technology selections endure. The best choice today may be the second choice tomorrow. Wait longer, and the order of preference may change again. Many have experienced the pain of realizing a multimillion-dollar data warehouse investment is destined to consume capital expense longer than it provides optimal value as we recognize a new option — perhaps a data lake or cloud data warehouse — promises to serve the business better. We might have developed hundreds of reports and dashboards in one technology and then learn the business prefers a new supplier’s offering for flexibility or capability. We navigate these alternatives with the risk of accumulating technical debt and business dissatisfaction until it becomes easy to view our tool and platform selection challenge as the primary concern.

Another cause of distraction is the organizational separation in large enterprises between IT, or data analytics shops, and the business functional areas. The former focuses on enablers while the latter supplies the use cases necessary for finding value in all the data and technology. When these separate focuses are not tightly woven together, we risk losing the “ever follows” of our principle, with form and function wandering distinct paths, intersecting less than we hope.

To avoid these distractions, great architectures must be built on great use cases. This can be illustrated by an analogous example from the physical world, an architectural and engineering marvel.

A bridge connecting San Francisco to Marin County, California, spanning the Golden Gate Strait’s treacherous currents, was first proposed in 1872 by a railroad executive named Charles Croker. In 1920, Joseph B. Straus finally convinced San Francisco’s city engineer Michael O’Shaughnessy that his design, a rather complex “symmetrical cantilever-suspension hybrid span,” could somehow complete the mission. Yet, when construction began in 1933, the project utilized a more elegant suspension bridge design from Leon S. Moisseff, the designer of New York City’s Manhattan Bridge.

There were no competing vendors with offerings of bridge accelerators or consultants who anticipated the need for building bridges across 1.7 miles of deep waters, in an area prone to earthquakes, and then constructed templates for what was required.

An ambitious use case presented distinctive challenges and countless innovations. The complexity of the engineering work made this accomplishment one of the wonders of the world. Advances included cables capable of swaying 27 feet in high winds while supporting 887,000 tons of weight, and the blasting of rock under rapidly running tides to enable an earthquake-proof foundation for the towers that rise 746 feet above the water – towers filled with enough concrete for a five-foot-wide sidewalk from San Francisco to New York.

This was truly a physical world example of a great function inspiring great form.

Looking to the digital world, in 1996, two grad students named Larry Page and Sergey Brin were inspired by one of the most ambitious digital use cases. The challenge: “organize the world’s information and make it universally accessible and useful.” This must have sounded more visionary than likely at the time, similar to the concept of a Golden Gate Bridge in the 1920s. Search engines then struggled to provide useful links without the need to wade through an exhausting number of irrelevant possibilities. But the Page and Brin research thesis, that ranked relevance based on frequency and quality of linkages within a graph representing the Internet’s content, turned out to be brilliant.

Their method was immediately challenged by scale. The initial version used more than half the network bandwidth on the Stanford campus, setting in motion decades of innovation in data architecture and artificial intelligence and business models. The escalating challenge of organizing the world’s information as it grows exponentially, has been a scaling journey that never reaches its destination, a function that never stops stretching form.

No vendors with search engine accelerators supplied a ready-made path to Larry and Sergey; no consultants anticipated the technology their ever-increasing scale challenges would require. Actually, the reverse occurred. Google’s efforts have been spinning out data architecture innovations at every step in their journey.

For example, Google’s MapReduce paper, detailing the Google file system that their operation required them to invent, was the genesis of the Apache Hadoop project and eventually led to business enterprises implementing the now common standard of data lakes.

In the realm of AI, Google’s search bar has pushed for enhanced natural language processing, providing unceasing improvement in the answers to our online questions. When the search bar’s customer experience was converted to a Google Home assistant orb with a microphone we wake with the words “Hey Google,” the AI applied machine learning to recognize accents and a complicated new set of speech nuances.

Throughout all of these many years of innovations, the function has remained relatively constant. But the changing nature of the world’s information, in scale and possibility, has challenged the form of supporting technology architecture to be reinvented and enriched at a dizzying pace.

The Golden Gate Bridge and the Google search engine examples are complex, but no large enterprise should think their situation is less applicable to the same principle. Form must ever follow ambitious, audacious function. Every business should be searching for the differentiating use case that innovates. All must be stretching for unique ways to discover value and intelligence in their data. When they succeed in imagining the new use case, a new data architecture must then accommodate.

This does not mean data foundation preparations are unnecessary or always destined to be distractions. The design of the Golden Gate Bridge benefited from the engineering experimentations of prior suspension bridges. Google’s invention benefited from the flourish of web crawler and web analytic technology that focused on indexing the Internet in the late nineties. Innovators must always stand on some shoulders.

Good architecture should be well understood, categorized within instructive reference architectures, prepared for enabling future acceleration. An unfortunate tendency exists in data architecture for organizations to overvalue platform solutions. A team will solve some data challenges by integrating various technologies and then claiming they have the perfect data platform, a “Swiss Army knife” for data projects ready to be applied to any future use cases. But all of them face limitations. Keep in mind:

1. Discovering the use case that provides a competitive advantage is the priority.
2. Once discovered, that use case will likely require data architecture advances to implement.

We must be careful how much we productize form in anticipation of future, unknown function because Louis Sullivan will never be wrong about this: Form ever follows function.

Learn more about NTT DATA's solutions for Data Architecture Modernization.