Matching Big Data Projects with Development Teams

Quick Summary - What kind of data science team do you need for your Big Data project? Here we take a look at development team requirements for several types of projects ranging from very simple DIYs to creating your own Big Data Platform.

Matching Big Data Projects with Development Teams

Mid-sized organizations and enterprises can spend years acquiring their data in a variety of formats and millions of dollars in efforts trying to put it to good use. All of the big tech companies are essentially data companies at their heart. You likely interact with several of them every day, like Google, Facebook, Twitter, or Amazon. One reason why they are so big is because everyone is sharing their data with them… often for free. It follows that the better you are able to capitalize on the data that you have, the easier it will be for your business to grow, too.

In our last post, we talked about the types of development teams needed for Big Data Projects and the various roles and salaries for the developers and data science specialists in those teams. It seems like a good idea then to discuss the requirements of specific types of Big Data Projects – so feel free to keep that link open in a second window.

Third-Party DaaS and MLaaS Platforms

Third-Party Data as a Service (DaaS) and Machine Learning as a Service (MLaaS) platforms can help your business leapfrog to the cutting edge of Big Data – and at a reasonable monthly cost. Services like this (Azure ML, Amazon ML, Rapid Miner, IBM Watson, others) already have data on-demand capabilities or ML WYSIWYGs usually backed with advanced analytics. Some platforms tout low-code or no-code requirements. However, it may be necessary for you to plug in your own internal or proprietary data.

As an example, Revenue-AI is a promising startup that’s attracting a lot of attention in the Revenue Management (RM) arena. Its focus is providing data on-demand backed by an AI-enabled digital assistant for consumer packaged goods, pharmaceuticals, as well as the travel and hospitality industries. For CPG companies, RM teams can access market data by country/region, per channel, per retailer, and per SKU-basis for entire product portfolios matched against competitor data. That’s the tip of the iceberg, but it alone enables precision pricing and promotions based upon today’s (not last year’s or last quarter’s) data.

Team Needed: In some cases, you may not need a developer at all. Realistically, if you need to add your data, create models, or customize reports using different data sets, a developer or two with strong mathematical skills will enable you to do exponentially more, much faster than a non-coder. If you do have a data scientist, they can focus on gathering insights from the data and help to develop our team’s overall data science capabilities.

API’s for websites and mobile apps

Case 1. If you already have a website or mobile app, you can begin monetizing data with minimal effort – potentially by just adding an API. Savvy website developers could potentially do this on their own. If not, it’s likely a small task for a single web or mobile developer. The effort is minimal but it sets you up to better monetize on-site advertising and/or generate a supplemental revenue stream selling anonymized user data. For advertisement, Google AdSense API or the more complex Google AdMob SDK are good examples.

Case 2. The next step up involves a substantial jump in complexity and requires calling upon a small team of developers to create an API for your app or website that you can distribute to others. This depends on your app or site providing a valuable feature – on a subscription basis and/or for access to the anonymized data of their traffic. If you don’t have such a feature, then you’re looking at a much greater effort to create an app from scratch.

However, for a startup or website without a lot of traffic, creating a distributable app is likely to be the “path of least resistance” for ramping up your ability to monetize data. Instead of attracting 6-7+ digit visitors to your site, you only need to offer a feature that would be valuable to other sites already attracting a large volume of traffic. Examples Skyscanner Flight Search, Yahoo Finance, Hearthstone – here’s 50 of the most popular APIs. As people use your feature, you collect data that you can in turn sell to data exchanges like Lotame or use to produce your own actual data products.

Team Needed: In the first case, adding an API to your website is a trivial effort; but would likely require a developer to add one to an existing mobile app and make sure it fits to your look-n-feel and UI. Creating an API to distribute to other sites (B2B) will require a small development team experienced with data servers and cloud services. However, between having a feature and an api promoting it, software development becomes an inherent part of your business. If your API performs well, you will likely want to continue, if not expand, development efforts to expand your data collection efforts. You may also find it valuable to add data scientists to your team to better exploit the value of the data you are collecting.

How much does it cost to build a remote team in Ukraine?

If you decided to hire a development team in Ukraine or even open an R&D center, this calculator helps you figure out how much it would cost.

Calculate now

Mobile or Digital Wallets

This could just be a feature of a mobile app, but it’s a very important feature for retailers. Customers use their preferred method of payment (credit card, Paypal, Google Pay, etc.) for purchases via their account with your business. It gives you the ability to track and influence purchases on a per-user basis – products by SKU, average purchase size per visit, promotion effectiveness, shopping frequency, promote social sharing, and more.

Digital wallets can also handshake with your inventory control to automate replenishment. Anonymized customer data (spending habits) can be sold to a wide range of data brokers. Starbucks’ app almost single-handedly pushed mobile apps and digital wallets into prominence by driving $1.5 billion in sales via their mobile wallet-style app in 2010.

Simply optimizing inventory levels to avoid out of stock items, offering promotions with push notifications, and selling user data is the low-hanging fruit. Analyzing sales data in conjunction with floor plans, product placement, shelf spacing, and promo displays can help boost average purchase size per visit.

Team Needed: Creating your own internal digital wallet app is a complex endeavor requiring a full development team. Otherwise, you could just use services like Paypal. Getting maximum value from your mobile wallet data is likely to require strong data science and business analyst skills.

Internet of Things

There are so many possibilities in digital transformation involving IoT devices and companion apps to make it impossible to cover all possibilities. In its simplest form, there’s the creation of “skills” for the likes of Alexa, Siri, and Google Assistant. These are trivial efforts that even non-coders could create on their own. But, IoT devices make use of a wide range of sensors to collect data for every industry on nearly everything imaginable. Pushing the limits, when a lot of IoT devices work together, it’s possible to automate the end-to-end process of an entire grocery store.

Amazon Go has used eWallets in conjunction with a huge array of IoT sensors so customers can walk in, pick up what they want without ever standing in line for a cashier. Not only are they rolling this out Whole Foods and selling the Amazon Go technology to retailers. This video shows the Amazon Go technology in action for a full-sized supermarket (vs. small convenience store). It’s appropriate to bear in mind that the so-called “experts” said just a few years ago that they wouldn’t be able to make that financially viable. They were laughing at Amazon back in 2000, too.

Team Needed: If you’re developing an IoT device, you will need your own hardware engineers – whether local or in an R&D center. You will also need the firmware and companion software to go with it, requiring a development team with strong technical skills. While it’s possible to sell the raw data, data scientists can help you create higher-value data products and services, while exploring relationships with other datasets to discover more valuable insights.

Big Data Platforms

Going all out to create any kind of platform like Facebook, Credit Karma, Robin Hood, or Revenue.AI is about as sophisticated of a project as any business could take on. It means bringing together numerous technologies, from the individual software features, the advanced analytics, and a high level of automation. Increasingly, ML/AI elements with strong predictive capabilities are also needed to assist decision-makers and end-users observe all organizational best practices and objectives based upon historical and up-to-the-minute market data.

Credit Karma makes most of its money via commissions on the products and services it recommends to users based on their data. Other credit reporting agencies make selling credit reports that Credit Karma offers for free. Robinhood makes most of its money selling order flow and other information about its customer’s transactions.

Team Needed: This kind of project can only be accomplished by an enhanced integrated development team, or more likely, a specialized data science team. In the former case, you will still need a few data scientists, but your developers will need open access to SMEs throughout development.

When incorporating enhanced analytics, ML/AI for enhanced self-service or the creation of AI-enabled digital assistants, a specialized data science team is essential. Their task is not only the development of very sophisticated software, but maintaining the integrity of data quality applicable to a tremendous amount of data – and how that data is queried, answered, and capable of being manipulated into reports the way the end-user wants to see it. Data integrity is critical because even one wrong number can throw a lot of customer data out of whack.

Need help assembling a data science team?

That’s what we do! PerceptionBox is an IT staffing agency for software development, R&D centers, and Big Data projects. Developer wages are the single largest expense in software development, so we invite you to compare wages with our calculator. Select the roles you need to fill, how many, the term of the project, and email address and our cost report will be on its way to you.


Tell us about what you are trying to build

  • This field is for validation purposes and should be left unchanged.

Subscribe to our newsletter

  • This field is for validation purposes and should be left unchanged.