# Why I became a Solo Founder

By _Christian Prokopp_ on _2022-06-13_

I never wanted to be a solo founder. Yet, in 2021, I quit my job and started Bold Data to mine the Internet single-handedly. Trust me, it sounds as insane to write as to read. What on earth possessed me, and more importantly, would I do it again?

![Lonely Lego figure walking](/content/blog/images/why-i-became-a-solo-founder.webp "Lonely Lego figure walking")

## First, unwitting entrepreneurial steps

Since high school, for close to thirty years, I have wanted to start something of my own. Then, with a few friends, we tried to develop a computer game. This was long before Unity, Youtube guides, online courses, and other tools. We had no money or experience, and we started head-first into it. Unsurprisingly, it failed, but we learned a few things. It was more complex than we imagined; Dunning-Kruger effect, anyone? We spent too much time discussing how we would distribute our imaginary earnings, which sounds probably familiar to many startups and VCs. Still, we managed to develop a bespoke 3D engine with decent performance, 3D models and level designs before the project collapsed. Most importantly, we stayed friends and still are today.

A bit later, I did some self-employed work during and after my undergraduate studies. Still, the steps to a sustainable business were unclear, and I focused solely on technical problems. For example, I started a PHP framework to simplify development work in PHP's early days. I built the basic framework but had no idea what to do next. Coming from a working-class background, I knew no entrepreneurs nor how a business works. Also, this was just after the Dot-come bubble burst around 2000, so the timing was not on my side.

## Academia

I moved from Germany to the UK, Asia, and Australia. In Australia, I did a Master of Commerce (MCom). Firstly to get a visa and stay in the country with my girlfriend, and secondly, as a compromise. I had a BSc and wanted to learn more about business, but I did not want an MBA. As a young engineer, MBAs seemed too detached from the "real world". Not that I had quantifiable evidence, merely my unfounded opinion. The MCom was worthwhile despite the strain of working to pay for the study in parallel. It taught me about accounting, finance, economics, and scientific research, leading to my PhD. If the latter was a good or bad choice is a topic for another post.

I added a research semester to my MCom and worked with SAP Research and a professor on some Natural Language Processing (NLP) research. I enjoyed research and wanted to challenge myself. Looking for a PhD placement, I remember a conversation with a professor. One of my ideas was to research globally distributed databases with fewer consistency and data model restrictions. She specialised in databases but found the idea not interesting. In the following years, I watched how the NoSQL market exploded with new products and funding flooding the headlines. I learned, again, that ideas are one thing, but timing and support are another.

I did receive a scholarship and did my PhD in information retrieval with a bit of machine learning. You could call it Data Science today. There was a brief interlude where I found myself in a country on the verge of a civil war with my newborn daughter without a passport and a backpack full of diapers instead of trying to finish my PhD thesis. But that too is a topic for another post. Ultimately, I decided academia was not for me and moved to the UK to work in startups.

## Startup life

The second startup I worked at is particularly relevant to my story. It was what people envision when they hear startup. We were housed in an old office building, scavenging for abandoned, usable furniture from deserted rooms. The engineering team was a handful of bright people working on laptops, continuously releasing software to production in the public cloud, which was still novel at the time. Our critical infrastructure was our router, which was unreachable when someone used the dodgy microwave placed in the same room as the engineering team. So no releases at lunchtime. Still, we had a contingency for emergencies, a 3G USB stick. One for a team of six. It was awesome and some of the best fun I have had in my professional life.

At this startup, my work focused on crawling and scraping Amazon's product data to enrich our product catalogue for our B2B E-commerce platform. We use newfangled technology like Hadoop and Hive in the cloud, no less. But the data mining activity was not a core activity or differentiator. Yet, when we tried to outsource it, we learned that we outperformed the crawling and scraping market in quality, speed and cost by a considerable margin. At that point, we collected over 250 million products. So we had to keep it in-house.

We also learned that our dataset was valuable to many retailers competing with Amazon. They did not have and still do not have the deep insight Amazon gets from its marketplace. Amazon can dip into 100s of millions of products and their sales history and billions of customer interactions and use it to analyse and optimise its product catalogue. It is a data monopoly of tremendous competitive value. For example, how do product category manager who manages hundreds, maybe thousands of products compete with that insight into pricing, demand and supply, and customer insight? That was where our dataset and ability to use it for analytics on pricing, new product development, categories, availability, etc., sparked an immense interest.

Eventually, Google bought the startup. For a moment, I seriously considered taking my know-how and rebuilding the data mining aspect of the startup as a new business. But while I learned a lot more about startups and data. I was only experienced as an individual contributor and had no co-founder or money. However, I did have a family and bills to pay by now. And being bought by Google meant I could choose from many exciting and well-paying job offers that materialised overnight.

## Expanding my horizon

At the time, I chose to join a startup in big data consulting as the principal consultant. It was a huge gamble and intentionally yanked me out of my comfort zone from behind a screen. I was thrown immediately into the deep end, having to work face to face with a customer's team from day one. True to what you expect from an early startup, it had little to no collateral or processes. It was challenging, substantial and invaluable learning.

I had no knowledge or appreciation for sales, for example. But having to architect, sell and manage major technical programmes in large organisations was an eye-opener on interpersonal, political and business aspects. I was lucky to work with an experienced, intelligent, technically competent sales director from whom I learned much. We became a well-tuned team managing technical and commercial challenges in complex conversations and negotiations with numerous stakeholders anticipating the concerns and needs of our customers, trusting and always supporting each other.

As the startup grew, I became responsible for the technical teams and elements of the business, helping shape the organisational structure, collateral, processes and intellectual property. Eventually, we exited to Teradata, where I helped scale the business further for a bit longer. I took away a few learnings. I decided not to scale a business with billable hours again. Scaling with people is hard and gets more complex with every magnitude scale. Also, I did not want to shoulder large parts of the senior responsibility without being a founder again. You get less strategic say, have no place at the table on the most critical decisions, and you won't get a comparable payout on exits. (Cue the smallest violin in the world. I know that I live a privileged life; there is no need to shed a tear for me.)

## Life after startups is the life before the next startup

After Teradata, I did some interims positions at small and global organisations in various industries. I even pitched at a friend's startup accelerator an idea for a Data Science as a Service company with the twist on using consulting only as an enabler and accelerator with deployed ML services as a recurring revenue stream. It did not get picked up. The idea was timely and appropriate, looking at companies like DataRobot and the development in ML advances and automation today. But I dropped it, not wanting to do it alone and being busy making money elsewhere. It is hard to walk away from a comfortable life and risk everything. Notably, you make that risky choice not only for yourself but also for your family and potential co-founders. And the people I considered for co-founding had the same issue, very comfortable positions in market-leading companies. There is a reason why many people do startups early in life with less to lose, less experience of the complexities and more to gain.

My last job was at a health and wellness retailer, creating a data platform including Data Engineering, Business Intelligence, and Data Science teams and capabilities. I joined because they hired a seasoned, excellent CTO. His mission was to take them from low-tech and fully outsourced to high-tech, engineering and data-driven business in the envelope of a significant strategic effort to reinvent the company from a brick and mortar retailer to a modern enterprise with unified channels and novel services. Business transformation is a worn-out term, but it applies here. It was a tremendous challenge, and the CTO was assembling a great team. What went well and did not and the learnings are material for future posts.

After one to two years in the role, it struck me. I was looking into sourcing data once again. Category buyers, marketing, supply chain and others could benefit from various market insights and already acquired bits and pieces here and there. But I found that the data mining industry had not moved much in nearly a decade. Web crawling and scraping with proxies is still the standard method, and the prices and complexity remain high. Curated datasets are expensive, less timely, and custom-made or limited. Inexpensive, accurate, timely data and analysis were as hard to come by as before.

## Bold Data's motivation

Importantly, no one seems to have taken a step back and tackled the problem as a whole. It is not a retail or e-commerce problem only. It is not a single data source issue either. The public Internet contains hard-to-reach data that can provide or enhance Analytics to support decisions across countless industries. But it is only valuable if there is timely, flexible, easy and inexpensive access. Today, you have to build your own data mining with inadequate services or spend huge sums on cumbersome contracts and datasets with many providers. Then, you must invest on top to wrestle something of value from the disparate datasets. It is slow, costly, complex and cumbersome.

Why not help people and systems make better decisions with publicly available but hard-to-reach data by collecting, cleaning, structuring and analysing the data for them with automated, modern, inexpensive methods? In simple terms, data mine the public Internet, wrap an API, data stores and analytics services around it. Customers can use it for countless decisions like pricing, marketing, insurance risk calculations, predicting residual car lease valuations, new product development, category selections, inventory and supply management, finding copyright infringements, and a million other decisions.

The market for data and better decisions exists. It did ten years ago, and it does so even more today. You can witness it in the surge of Data Science, Machine Learning (ML) and Artificial Intelligence (AI) startups that focus on the analytics aspect. But we all know good, big data is more effective than complex algorithms. Why has a global data platform not been done? The best explanation I have is that it is challenging, slow to break in and not sexy or cutting edge on the surface. All the money and brains go after the year's hype, be it ML, AI, crypto, NFT or something else.

## Solo founder

Here I was twenty to thirty years later. I had another idea, a little money and some business-relevant experience, but no co-founders. The list of people I would love to partner with are still doing too well to consider this level of risk-taking a reasonable option. So hubris took hold of me. Maybe, just maybe, I could pull the first steps off myself? It came with the bonus of being guilt-free if things went pear-shaped instead of feeling responsible for someone betting their time and money on my idea. Lastly, I could run it my way, i.e. a self-funded shoestring operation with cost as a critical differentiator and survival factor.

In June 2021, I left my job and founded Bold Data Ltd on the mission to build a scale-out platform to mine the valuable parts of the public Internet myself. It sounds ludicrous now. At the time, I engrossed myself in work and long days, which helped me not dwell on the overwhelming scale or indulge in crippling self-doubt. I started with familiar territory, Amazon data.

## Bold Data today

So was it a good idea, and how am I doing today? I will dive into some details in future posts, but here are the key points one year after solo founding a startup taking on internet-scale data mining problems.

So far, it has cost me money and has not made me any. It is demanding with many long hours and no one to lean on but yourself. I do not tend to be lonely and have a network of colleagues and friends I can talk to occasionally. But they have jobs, so you forgo the watercooler talk or the chance to blow some steam or get some ideas over coffee or lunch from your colleagues. Luckily, I have a family, and while they could not be less interested in my day-to-day technical work, they are a great source of stability and comfort. Also, I found it valuable to reach out to people in my extended network to get feedback. It is incredible how helpful people you worked with a decade ago or strangers are if you just ask nicely.

On the technical side, I will not share the details to protect my intellectual property, but I found the world moved on in some ways and is stuck in others. Most importantly, the supposedly best scaling technology is not always suitable if you look at the problem diligently from a get-stuff-done fast, cost and scale perspective. To be fair, I also have an insanely low budget because I think of the following one or two magnitudes of scale, which excludes some tech and traditional data mining approaches right from the start.

A year on, I developed a robust global multi-cloud solution that runs and self-heals 24/7 without me handholding it, a crucial requirement. It mined over 200 million products from Amazon US, UK, and GER in detail and collected over a billion updates. I could make the numbers bigger by spending more money on mining or by including other aspects of the operation. The point is that the platform works and scales.

Next, I could add more countries, other e-commerce websites or other industries' data altogether. But I took a big gamble by building so much without a customer. It is time to pivot in front of users as soon as possible. Now I focus on the website, outreach to my network, and creating content for SEO and LinkedIn to create inbound leads for either the existing Amazon data or additional datasets to capture interest and convert it into revenue. It meant a drastic switch away from the backend engineering to frontend and more of a marketing and sales focus, with plenty of challenges and opportunities to learn.

## Patience and willingness to adapt

I was hesitant to switch gears from building to marketing and selling because there is always another thing that you should, could or must do first, especially if there is no one else to hand over to. But I knew it was time. Yet when I did switch, nothing happened, expectedly but still daunting. All can fail if no one shows up to try what you got. Getting traction is another marathon of its own. To begin with, it does feel like shouting in the wind on a vast bazaar of ideas.

It is still very early days, and I have seen some inbound requests slowly. Initially, from suspect individuals who ghosted me on the slightest question on their use-case, some students, researchers, and some small businesses started looking for some free datasets to get started. Recently, I picked up some interest from one of the best-known global FMCG brands and a renowned consultancy. While this is still a way to go to ARR or MRR, the responses are positive, and people are willing to invest time to explore Bold Data, which is a great start.

## Would I do it again?

A year on, absolutely. I have hit many hurdles, and I overcame them all. I loved switching back from meetings all day to developing all day in various engineering roles and now changing again into marketing and selling. I know it is temporary, though. Success or failure, my role will change again.

Would I do it again if you asked me in a year? We will see.

Meanwhile, contact [me](mailto:christian@bolddata.biz) if you are interested in Bold Data. Subscribe to the mailing list below, [LinkedIn](https://www.linkedin.com/company/bold-data-ltd/), or [Twitter](https://twitter.com/bolddata_org) if you want to stay updated. And read more [blog posts](https://www.bolddata.org/blog/).

* * *

_By Christian Prokopp (_[About](https://www.bolddata.org/about/)_,_ [LinkedIn](https://www.linkedin.com/in/prokopp/)_,_ [Twitter](https://twitter.com/prokopp)_), Founder of Bold Data_

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

I never wanted to be a solo founder. Yet, in 2021, I quit my job and started Bold Data to mine the Internet single-handedly. Trust me, it sounds as...

Lonely Lego figure walking

I never wanted to be a solo founder. Yet, in 2021, I quit my job and started Bold Data to mine the Internet single-handedly. Trust me, it sounds as insane to write as to read. What on earth possessed me, and more importantly, would I do it again?

Why I became a Solo Founder

# Hard work is not appreciated!?

By _Christian Prokopp_ on _2022-05-30_

Your hard work is not appreciated. So why should you still do it? There is a good reason.

![It can be frustrating to not be appreciated](/content/blog/images/hard-work-is-not-appreciated.jpg "It can be frustrating to not be appreciated")

People, especially those who do not have a close relationship with you, do not care how hard you worked on something. Case in point, I remember situations when someone tried to rush through demonstrating a minimal effort piece of work and got stopped, asked many questions, and figuratively padded on the back multiple times to their astonishment. Why did they get praise for trivialities?
Why is hard work not appreciated?

If you are a bit more seasoned, you might know the answer already. Only the impact on the audience matters and drives its interest and appreciation more than heroic late nights and technical brilliance. You should not expect much enthusiasm from others if the impact is
    
- Internal to your team mostly
- Removing technical debt
- Strategic long-term beyond quarterly horizons
- Adding security or removing risk

These have no immediate, measurable ROI or grateful audience, or their absence is more impactful than their presence, e.g. security and risk management.

Other examples are where the effort greatly outweighed the impact, but it needed doing for good or bad reasons. Good reasons can be foundational work, and the first value drop is minimal, and the impact is strategic, enabling future work and value delivery. Bad reasons can be poor planning or non-data-driven decision-making, e.g. based on gut feeling or tradition or because everyone else is doing it.
Do not despair; hard work is still worth doing!

Once you understand the above, you can use it to your and your organisation's advantage. Break with the expectation that people appreciate your effort and become comfortable with celebrating value delivery. Sometimes you put in a lot of work for little immediate value, but it pays back later when you occasionally get a lot of value for little effort. It should average over time, and hopefully, overall, you get the appreciation you deserve. But it takes a little learning and attention to 'sell' your work according to its value. I advise juniors to promote and savour the easy wins, get the attention and appreciation, and not dwell on the unappreciated low-impact hard wins. 

There is another significant benefit attached to working this way. It starts pivoting your view. I used to look at the world through the eyes of an engineer at the beginning of my career. Every problem had an elegant technical solution, and complex problems were more interesting than simple ones. That is divorced from what happens in reality. No one outside the team building the solution cares about its implementation or sophistication. 

Businesses care about impact, e.g. is something possible and feasible in resources and time? Quickly you learn that sometimes good is good enough or some problems are not worth solving no matter how irresistible they are. For a data scientist, it can mean using a simple, robust, boring algorithm instead of the latest complex ones or developing your own. For engineers, it can mean using existing tools, libraries or services instead of building your own.

If enough individuals in a business realise this, good things happen. It achieves valuable outcomes faster and more effectively. Poorly motivated goals may still exist, but they might be starved of resources or achieved with absolutely minimal effort since there is no or little value in them. Importantly, you will save your hard work for worthwhile goals, which spares you disappointment and the business wastage. Businesses either learn and reduce poorly motivated goals or lose their best staff in the long term.
Life is too short, relax whenever possible
Life is too short, relax whenever possible
Do not fight it; embrace it.

Looking at how your work is perceived and how it adds or does not add value to your organisation can help you avoid disappointment. And it can make you better at your job by avoiding some unappreciated work that is unnecessary. But there will always be necessary thankless work. Luckily there will be easy wins where you change one line of code and have people cheering you because it made their (work) lives better.

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

Your hard work is not appreciated. So why should you still do it? There is a good reason.

It can be frustrating to not be appreciated

Hard work is not appreciated!?

# 4 career tips I wish I knew

By _Christian Prokopp_ on _2022-05-25_

When I mentor university students or discuss careers with the people I lead, I often draw from four pieces of advice. I wish I had known these when I started, but they come with experience, perspective and confidence. Three things most of us lack at the beginning.

![You didn't come this far to only come this far](/content/blog/images/the-four-career-tips-i-wish-i-knew/you-have-come-800.webp "You didn't come this far to only come this far")

Why would you care? [I have worked in some form in tech for nearly 30 years now](https://www.linkedin.com/in/prokopp/). I started by putting together computers in a small shop while still in school before focusing on the new thing, the Internet. I worked in different industries, countries, and cultures, in startups and global corporations. I worked as a labourer to pay for my studies and as a leader of other tech leaders less than ten years later. Now, I am building Bold Data, a data mining and Analytics startup. If you are interested in what I learned along the way, read on.

## #1 A job is not a career

Especially university students are anxious about their first position. They are trying to land the perfect company, industry and role on the first go believing it has a deciding impact on their career. If you find the right spot on the first try, you would be an outlier, especially in tech. Think of it more as a learning opportunity.

![Change into a butterfly](/content/blog/images/the-four-career-tips-i-wish-i-knew/change-into-butterfly-800.webp "Change into a butterfly")

The tech industry moves fast, and you grow as a person. The technology and roles you start your career with may not be around a few decades later. Look at how fast Hadoop came and left the scene as a must-have skill. Data Science, Data Engineering, Machine Learning, and Data Analytics are other fields that evolve rapidly with automation, tools and better processes.

But most importantly, you and your life change. One day you may love working late in a tiny startup for close to nothing to learn the latest tech, and the next day you wake up with a family, a mortgage, and worry about your pension, not your next night out. You may develop from a technophile to a leader or from an introvert to a presenter. It happens.

So do not kill yourself with the choice of the first role. Pick something you assume you can live with; more on that below. If it is a great fit, ride the wave, learn, grow, and enjoy it until you outgrow the opportunities at the place. If it is a bad fit, figure out why and if it is worth sticking for a little while to learn or build up your CV. If it makes you unhappy, move on. But in either case, good or bad, learn **why**! What did (not) work from people, culture, company size, industry, role, and responsibilities? Reflect on everything regularly.

## #2 Know yourself as well as your opportunities

That brings us to knowing yourself. Everything you do is learning, especially when things go wrong. It is painful, but you will learn a lot if you are willing to reflect on it honestly. Are you driven by money, status, freedom, creativity, technology, self-empowerment or something else? Do you love the uncertainty, risk, and opportunity small startups give you or large organisations' reliability, structure, and hierarchy? What kind of people and culture do you thrive in, and what makes you dread going to work in the morning?

![Life is too short, relax whenever possible](/content/blog/images/the-four-career-tips-i-wish-i-knew/meditating-gnome-with-bird-800.webp "Life is too short, relax whenever possible")

Reflect on these regularly with a friend, a mentor, your manager or yourself. Whatever is comfortable, and whoever supports you. Note that your manager might only be a good choice if you can be open about failing and ambitions beyond your current role and organisation. The right person, a leader, will look after you, not after themselves. But do not use this to unload frustrations onto someone else constantly. That is putting you both in a downward spiral. Occasional, cathartic lamenting is ok and even needed, though. Check what you can change and achieve and ignore what you can not change. Also, keep an eye on what may have changed for you since what makes you happy evolves too.

People early in their career or who are not used to reflecting are often unsure about what they want, in which case choosing the right next step is impossible. Worse, you might pursue the wrong thing. How many blue soles are out there chasing money as a false proxy to knowing what they want and feeling burned out and empty the further they get in their career. If you get a handle on what makes you happy, you can evaluate new opportunities better or adjust your role and career in your current job to make you more fulfilled. Yet, life is full of compromises and not black and white. Few people find jobs that make them jump out of bed in the morning running to the office in anticipation. But you should not hate the sight of your desk, colleagues or Mondays either. If you do, change things.

## #3 Create l̶u̶c̶k̶ opportunities

People often are in despair when they feel they have no agency. I remember a young woman who felt stuck in her first job. It was a terrible environment with long hours, only negative feedback from uncaring managers, and few opportunities to learn or grow. She was worried that she was not performing well and that she could not find something else so early in her career.

![Maneki-neko (招き猫, lit. beckoning cat) brings luck](/content/blog/images/the-four-career-tips-i-wish-i-knew/beckoning-cat-800.webp "Maneki-neko (招き猫, lit. beckoning cat) brings luck")

Firstly, she had to give herself a break from the pressure and gain the perspective that she was treated poorly; that's where a friend or mentor helps. Secondly, I encouraged her to look for new jobs despite her concerns. The change was rapid even though the job did not change. But she identified unreasonable expectations and distanced herself from them, and she felt agency by looking for the next role and not feeling helpless. A few months later, she had two offers in places she was eager to work at and left soon after. Most importantly, the next time she had an issue, she felt empowered to go and make a change instead of feeling at the mercy of the circumstances.

Another common problem people face is feeling stuck in their role but liking their environment otherwise. The solution is to "do the job you want, not the job you have". It assumes a positive environment where you have at least some flexibility and goodwill around you, but if you do not, why would you stay? A good leader will enable, support and help you, but the drive has to come from within the person.

For example, I had people wanting to be promoted or become a leader merely because they had been around a long time in their current role and were competent in it. That is often insufficient. The best people already do small parts of the job they want. They naturally gravitate to taking on responsibilities or pitching in where their new role would be. That demonstrates their ability and motivation in one swoop. They will consistently outperform people waiting to be given opportunities.

## #4 Assess opportunities and find your pattern

People, especially early in their careers, have a survivorship bias when they scrutinise others' careers and think their destination was a given, looking at the chain of roles and positions that led up to it. For many, the final destination was not intentional and unknown for most of the journey, i.e. they did not plan to end up where they did, but that is ok.

![Upside down black and white tree symbolising a decision tree](/content/blog/images/the-four-career-tips-i-wish-i-knew/upside-down-black-and-white-tree-800.webp "Upside down black and white tree symbolising a decision tree")

Instead, think of a career as a vast decision tree that reveals itself only one step at a time. After a few decades, it seems like a sensible linear path when you look back. Instead, it merely shows the route taken through countless decisions, hiding the untaken paths. Knowing this, you should focus more on making good decisions than worrying about an elusive endpoint that may or may not be in your tree.

I found it helpful to create scenarios for decisions outlining worst, likely and best outcomes. For example, you consider joining a promising startup with a year's runway in a new role. I have been there a few times. The worst outcome could be that it closes down in a year. A likely outcome is that it goes through a few ups and downs and founding rounds before being acquired, and you can move through a few roles or positions thanks to the growth and flexibility. The best outcome is it turns into a unicorn, and you have a stellar career.

How does that help your decision-making? Firstly, ignore the best case; you will benefit greatly, but it is also highly improbable. Secondly, is the likely outcome desirable? If it happens, would you look back and think it was the best choice given all other options you may have? If so, good. Lastly, the worst and most important scenario. If it happens, would you look back and think it was worthwhile despite having to find a new role, maybe being paid below the market rate? It is an interesting decision support point. Some may say yes because the ride is thrilling, the learning is great, and the opportunity to break into a new role or industry may be worth it when looking for the next role. Others may say no because the opportunity cost is too high in lost earnings in a more secure place or advancing their careers in larger organisations with more well-known names.

It is surprising how we get lured into dwelling on best-case scenarios that rarely matter and are more akin to playing the lottery than the likely and worst cases, which are critical to your decision. But everyone is different. If you like a good gamble, maybe play the unicorn lottery. It comes back to knowing yourself.

## Learning

I hope some of the above is helpful to you. Especially mentoring students allowed me to appreciate how much experience I took for granted after a while and how useful it can be to give people perspective and context from another point of view. It is challenging to start your career or change it. Sometimes people even get far down the track in their career before realising they drifted without a supporting decision pattern or goal. Lastly, you may disagree with everything I wrote. That's fine. I am still learning too.

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

When I mentor university students or discuss careers with the people I lead, I often draw from four pieces of advice. I wish I had known these when...

You didn't come this far to only come this far

When I mentor university students or discuss careers with the people I lead, I often draw from four pieces of advice. I wish I had known these when I started, but they come with experience, perspective and confidence. Three things most of us lack at the beginning.

4 career tips I wish I knew

# Amazon bestsellers are big data

By _Christian Prokopp_ on _2022-05-18_

According to an adage, big data is anything too big for Excel, i.e. more than 1,048,576 rows. It is a bit cheek-in-tongue, but, as with many jokes, it is grounded in truth. Many business processes run on Excel to this day. That is an issue when analysing datasets like Amazon product data for valuable insight on pricing, production and supply planning, and new product or category development. Excel cannot load a single country's Amazon bestseller list. Even if you use more scalable systems, many will struggle to analyse the more comprehensive product catalogue, complex product and category relationships, or changes over time.

![Your Data's size matters](/content/blog/images/amazon-bestsellers-are-big-data/your-datas-size-matters-hd-800.webp "Your Data's size matters")

For context, the Amazon product catalogue has 100s of millions of products in each country. The bestsellers are a small subset of that, with more than 2 million on Amazon.com and approximately 1.3 to 1.4 million on Amazon.co.uk and Amazon.de. Millions of products would be a sizable catalogue for most businesses but are merely the bestsellers for Amazon. Yet that is only a snapshot in time and insufficient for Analytics. Continuously updating this dataset to create a time series of changes increases its size to billions and billions of rows quickly.

## How many active Amazon bestseller categories are there?

Categories are helpful to organise the many products for customers as well as for analysts, operations, supply chain, marketing and other business functions. But even they become numerous on this scale. We counted only the Amazon categories with bestsellers to demonstrate this, ignoring empty categories.

-   Amazon.co.uk has 27,894 categories
-   Amazon.de has 29,790 categories
-   Amazon.com has 42,795 categories

That is the size or even larger than many businesses' product catalogues. Of course, that kind of data would load in Excel easily. However, without the product network graph associated with the categories, the utility of the data is limited.

## A taste of the Amazon bestseller data complexity

The bestseller category frequency is a simple example to demonstrate this challenge. How frequently do products become bestsellers in more than one category? For example, a product can be a bestseller in a leaf category like `clarinet` and its parent category like `woodwind`. Bestsellers can also appear in different parts of the category graph, e.g. a product may sell well in the fashion and sports part of the category graph. It may even be a bestseller multiple times in the same category via product variations like size or colour.

A detailed category network graph analysis gains insights on product placement to achieve more than one bestseller spot and maximise its sale. It can identify inter and intra-cluster relationships to identify product and category similarities to invest into or identify and discontinue orphaned product lines.

A quick overview shows us that Amazon.co.uk lists 1,377,192 unique products and 2,076,485 bestsellers, which means each best selling product made it into the top 100 in 1.5 categories on average. Amazon.com has 2,090,907 unique products and 3,367,975 bestsellers or 1.61 on average. Amazon.de has 1,431,524 unique products and 2,122,573 or 1.48 on average. Another way to look at it is as a network, e.g., Amazon.com has a bestseller network of 2+ million nodes and 3.3+ million edges.

![Amazon US, UK, DE bestseller category frequency](/content/blog/images/amazon-bestsellers-are-big-data/bestsellers-f-ranking-800.webp "Amazon US, UK, DE bestseller category frequency: A bestseller is a product that ranks in at least one category in the top 100 selling. 75% of the bestsellers appear only in one category in the top 100. Some make it into more than one category's top 100. A small number of products make it into dozens of categories as bestsellers. The most frequent product made it into the top 100 of 84 categories. (Based on data collect in May 2022)")

A more detailed look at the frequency distribution reveals that most products are single category bestsellers (note the log scale on the Y-axes). The remaining products list in multiple categories, and some longtail outliers have surprisingly high category frequency. A closer look reveals many to be clothing with a large number of variations and books available in many formats and categories. The most frequent bestseller ranking in 84 categories is on the Amazon US website, a popular book with many product variants and relevant categories: `Sisters Behaving Badly`.

![Most frequent bestseller product ranking in 89 categories (source: Amazon.com)](/content/blog/images/amazon-bestsellers-are-big-data/most-frequent-amazon_com-bestseller-800.webp "Most frequent bestseller product ranking in 89 categories (source: Amazon.com)")

Note that if you look at the product page, you will only find a tiny subset of its categories. Bold Data collects a broader array of source data and integrates it and can therefore identify dozens of relationships unavailable from the product page.

![Most frequent bestseller product page ranks (source: Amazon.com)](/content/blog/images/amazon-bestsellers-are-big-data/most-frequent-amazon_com-bestseller-product-page-ranks.png "Most frequent bestseller product page ranks (source: Amazon.com)")

## Amazon bestsellers graph analysis

These simple insights already require unique datasets and sophisticated processing capabilities. But once in place, i.e. thanks to Bold Data, such analyses are done quickly and cost-effective. While it exceeds the scope of this post, we can share a peek at the complex network graph underlying the Amazon bestsellers. It can answer some of the questions raised early in the post.

![Amazon.com bestseller categories network graph analysis](/content/blog/images/amazon-bestsellers-are-big-data/amazon.com-bs_tl_matrix-800.webp "Amazon.com bestseller categories network graph analysis: Network graph analysis of Amazon.com top-level categories. Aggregated from 42,795 categories top 100 consisting of 3,367,975 bestsellers made up of 2,090,907 unique products. The thickness of a link shows how many bestsellers sold across two top-level categories. The colour (blue to red) and size of a label show how central, i.e. connected a category is. For example, there is a strong connection between the books, digital-texts and audible categories, or between sporting-goods and fashion. (Based on data collected in early May 2022.)")

The thickness of a link shows how many bestsellers sold across two categories. The colour (blue to red) and size of a label show how central, i.e. connected a category is. Hence, the graph shows the connectivity and degree of centrality of bestselling top-level categories based on products selling in more than one category, which we aggregated to the top-level categories here. The view is highly simplified since tens of thousands of categories across millions of bestsellers and connections are involved in the underlying network.

Yet the simplified view already reveals facts that intuitively make sense. For example, there is a strong connection between the `books,` `digital-texts` and `audible` categories, or between `sporting-goods` and `fashion`. It also shows loosely coupled or disconnected categories on the top level like `mobile-apps`, `gift-cards` or `coins`. We may share more detailed insights on the network analysis in future posts, so be sure to subscribe to our email list (links are at the end of the post and at the top right in the menu).

## The Amazon product catalogue is big data

There should be no more question that Amazon level data requires sophisticated data mining, processing and analysis capabilities. Bold Data collects all this data: the product details, the categories, the bestsellers and more. We do it for multiple countries and track changes in the bestsellers, continuously growing the dataset. Consequently, the bestsellers dataset size alone quickly increases to billions of rows, which is big data by anyone's definition.

Most companies are not staffed or equipped to deal with data mining and processing in the billions of products and updates outlined here. It also requires decades of experience, deep specialised know-how and intellectual property. Yet they would benefit significantly from the resulting datasets and analytics. If you sell on Amazon or not, you compete with them and their marketplace sellers on price, availability, category and product development.

Importantly, Bold Data can augment your abilities and turbocharge your insights, whether you use Excel, databases, or more sophisticated data storage and analytics tools. We operate a 24/7 multi-cloud data mining and analysis platform that provides accessible, affordable, quality datasets and analysis, ready to help you make better decisions daily regardless of your tools, industry, and needs. To close the gap to Amazon's data advantage and leapfrog your competitors on insight, [contact christian@bolddata.biz](mailto:christian@bolddata.biz?subject=Let's%20talk&body=%3CPlease%20describe%20your%20business%20proble%2C%20data%20and%20analytics%20needs%20briefly.%3E).

* * *

_Note that the data used in this post was collected and analysed in early May 2022 with the Bold Data platform._

---
## UPDATE 2024-04-11

Find the latest and biggest dataset yet at [Free Amazon Product and Bestseller Data](free-amazon-us-gb-de-product-bestseller-data).


---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

According to an adage, big data is anything too big for Excel, i.e. more than 1,048,576 rows. It is a bit cheek-in-tongue, but, as with many jokes,...

Your Data's size matters

According to an adage, big data is anything too big for Excel, i.e. more than 1,048,576 rows. It is a bit cheek-in-tongue, but, as with many jokes, it is grounded in truth. Many business processes run on Excel to this day. That is an issue when analysing datasets like Amazon product data for valuable insight on pricing, production and supply planning, and new product or category development. Excel cannot load a single country's Amazon bestseller list. Even if you use more scalable systems, ma...

Amazon bestsellers are big data

# Free Amazon bestsellers datasets (May 8th 2022)

By _Christian Prokopp_ on _2022-05-10_

Get huge, valuable datasets with 4.9 million Amazon bestsellers for free. No payment, registration or credit card is needed. 

---
## UPDATE 2024-04-11

Find the latest and biggest dataset yet at [Free Amazon Product and Bestseller Data](free-amazon-us-gb-de-product-bestseller-data).

---

![Free Amazon bestseller Data](/content/blog/images/free-amazon-bestsellers-datasets-2022-05-08/free-data-550.jpg "Free Amazon bestseller Data")

Download the bestsellers of Amazon.com (2,090,907 products), Amazon.de (1,431,524 products), and Amazon.co.uk (1,377,192 products). The products list all categories in which they rank in the top 100, the product name, reviews, review average, offer price, number of offers, and extra tag data like author, type, brand, etc.

The datasets\* contain all Amazon Germany, Uk and US bestsellers, i.e. top 100 products of all categories. Bold Data retrieved them on the 8th and the 9th of May 2022. For a detailed list of data attributes (column names) and their description go to the end of the post.

Note, Bold Data can provide this data updated with any frequency on request and as time-series data including trends on categories, price, reviews, ranking, etc. or any other data. [Get in touch by email to start a conversation](mailto:christian@bolddata.biz?subject=I%20want%20to%20know%20more%20about%20your%20services&body=%3CPlease%20describe%20which%20dataset%20or%20analysis%20you%20are%20interested%20in%20and%20what%20you%20try%20to%20achieve.%3E).

## Who uses Amazon bestseller data?

Amazon bestseller data is being used by startups and established businesses to analyse their pricing, supply, new product and category strategies, for example. Other uses include students studying data analytics, business intelligence, data science or machine learning, or doing online learning or competitions with Coursera or Kaggle, for example. Other use cases include researchers at universities who analyse market changes, retail and e-commerce.

## Amazon.com bestsellers dataset

### One thousand products

A random sample of one thousand products from Amazon.com.

-   [bestsellers\_amazon\_com\_1k\_sample.xlsx](https://docs.google.com/spreadsheets/d/103RdG7JA4oKERvbdYF1RvE5GwHOZ-lji/edit?usp=sharing&ouid=105392296502140495464&rtpof=true&sd=true) (Excel file)
-   [bestsellers\_amazon\_com\_1k\_sample.csv.gz](https://drive.google.com/file/d/1-L1KEqW6jyYtaOxt6Zu7SoBZkl-Khc4y/view?usp=sharing) (Gzipped CSV file)

### One million products

A random sample of one million products from Amazon.com.

-   [bestsellers\_amazon\_com\_1m\_sample.xlsx](https://docs.google.com/spreadsheets/d/1-RmU825ULWI9HFejBeFTwjoo36bopBhz/edit?usp=sharing&ouid=105392296502140495464&rtpof=true&sd=true) (Excel file)
-   [bestsellers\_amazon\_com\_1m\_sample.csv.gz](https://drive.google.com/file/d/1-mOClRl3FdBkd7fEQxkwcKZVOnkfPrM_/view?usp=sharing) (Gzipped CSV file)

### Full dataset

The full bestseller dataset consisting of 2,090,907 products was retrieved from Amazon.com and stored as a gzipped CSV file. Please send [an email request](mailto:christian@bolddata.biz?subject=Amazon%20bestsellers%20full%20dataset&body=%3CPlease%20describe%20which%20country%20dataset%20you%20would%20like%20and%20briefly%20what%20you%20will%20use%20it%20for.%3E) to receive free access.

## Amazon.co.uk bestsellers dataset

### One thousand products

A random sample of one thousand products from Amazon.co.uk.

-   [bestsellers\_amazon\_co\_uk\_1k\_sample.xlsx](https://docs.google.com/spreadsheets/d/100NkyFTXgaAbs4n5dLVd1FLTiwDKZhZu) (Excel file)
-   [bestsellers\_amazon\_co\_uk\_1k\_sample.csv.gz](https://drive.google.com/open?id=1-ZaVawER4ZoOSGQQmHMdPLR3_i9E3WPC) (Gzipped CSV file)

### One million products

A random sample of one million products from Amazon.co.uk.

-   [bestsellers\_amazon\_co\_uk\_1m\_sample.xlsx](https://docs.google.com/spreadsheets/d/1-OglvR0NRPzNMTApyneVeCEhwuVmThF7/edit?usp=sharing&ouid=105392296502140495464&rtpof=true&sd=true) (Excel file)
-   [bestsellers\_amazon\_co\_uk\_1m\_sample.csv.gz](https://drive.google.com/open?id=1-K2uFbJg54H7w2FMe8ImatpG94UGdnYD) (Gzipped CSV file)

### Full dataset

The full bestseller dataset consisting of 1,377,192 products was retrieved from Amazon.co.uk and stored as a gzipped CSV file. Please send [an email request](mailto:christian@bolddata.biz?subject=Amazon%20bestsellers%20full%20dataset&body=%3CPlease%20describe%20which%20country%20dataset%20you%20would%20like%20and%20briefly%20what%20you%20will%20use%20it%20for.%3E) to receive free access.

## Amazon.de bestsellers dataset

### One thousand products

A random sample of one thousand products from Amazon.de.

-   [bestsellers\_amazon\_de\_1k\_sample.xlsx](https://docs.google.com/spreadsheets/d/1-ogVB8nIkfC5tV-quRGfui4gEwe8owlK) (Excel file)
-   [bestsellers\_amazon\_de\_1k\_sample.csv.gz](https://drive.google.com/open?id=1-Q4OBvvzJfh7d2QrWNn-kIPH1xGGjFTr) (Gzipped CSV file)

### One million products

A random sample of one million products from Amazon.de.

-   [bestsellers\_amazon\_de\_1m\_sample.xlsx](https://docs.google.com/spreadsheets/d/1-TPj8rZ-BoMTeGtXIY1NpHXGFOm78LrP/edit?usp=sharing&ouid=105392296502140495464&rtpof=true&sd=true) (Excel file)
-   [bestsellers\_amazon\_de\_1m\_sample.csv.gz](https://drive.google.com/open?id=1-mI9av_Hd6sOJ_uAfkqQXHouty0tAmub) (Gzipped CSV file)

### Full dataset

The full bestseller dataset consisting of 1,431,524 products was retrieved from Amazon.de and stored as a gzipped CSV file. Please send [an email request](mailto:christian@bolddata.biz?subject=Amazon%20bestsellers%20full%20dataset&body=%3CPlease%20describe%20which%20country%20dataset%20you%20would%20like%20and%20briefly%20what%20you%20will%20use%20it%20for.%3E) to receive free access.

## Need more data or help?

If you need help with the data contact [Christian](mailto:christian@bolddata.biz?subject=Free%20Amazon%20bestsellers%20data%20questions&body=%3CPlease%20describe%20which%20dataset%20you%20work%20on%2C%20what%20you%20try%20to%20achieve%20and%20your%20questions.%3E) the founder of Bold Data. If you want to stay up to date with information on the datasets and future datasets subscribe to the email list (see the bottom or top right for links).

If you have specific dataset needs and want to inquire about Bold Data's services do contact [Christian](mailto:christian@bolddata.biz?subject=I%20want%20to%20know%20more%20about%20your%20services&body=%3CPlease%20describe%20which%20dataset%20or%20analysis%20you%20are%20interested%20in%20and%20what%20you%20try%20to%20achieve.%3E). This can be specific to Amazon, e.g. frequent updates, detailed product data or different countries. It can also be completely different websites, datasets or analyses you are interested in.

* * *

## Data attributes

Below are the dataset column names and their meaning.

**sku:** The unique product identifier (ASINs in these datasets).

**name:** The product name.

**review\_avg:** The average review rating.

**review\_count:** The total number of reviews.

**ranks:** All category identifiers and associated best selling rank for the category.

**min\_rank:** Best (smallest) rank across bestseller categories.

**max\_rank:** Worst (highest) rank across bestseller categories.

**ranks\_count:** Number of bestseller categories the product was found in.

**offer:** Best offer price in local currency, e.g. GBP, USD, EUR.

**offers:** Number of offers (may include used or warehouse offers).

**tag1:** Additional data like product type, author, brand, etc.

**tag2:** Additional data like product type, author, brand, etc.

**request\_date:** Date the data was retrieved.

* * *

_\*Note that_ _**no guarantees**_ _are made about the completeness or accuracy of the data and no liabilities arise from the download or use of the data. The data was collected from public sources as is and may contain restricted data like trademarks, language deemed inappropriate in certain circumstances, erroneous data or other unforeseen limitations._

_The data may be used for private or commercial analysis and decision-making use only. Redistribution or resale of the data is prohibited unless explicitly agreed in writing by Bold Data Ltd. Where the data is used, e.g. for analysis, diagrams, charts or otherwise, attribution to the source, e.g. "Bold Data, https://www.bolddata.org" or equivalent, must be made._

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

Get huge, valuable datasets with 4.9 million Amazon bestsellers for free. No payment, registration or credit card is needed. 

Free Amazon bestseller Data

Free Amazon bestsellers datasets (May 8th 2022)

# Bad Data: Nameless Amazon bestsellers

By _Christian Prokopp_ on _2022-05-03_

Many Amazon marketplace customers know that its huge product catalogue has data quality issues. However, they might expect its top sellers, which they frequently see and buy, to be accurate. Bold Data, which is processing 100s of millions of products daily, has a unique ability to find hidden insights and issues. For example, active Amazon bestsellers with names resulting from data processing errors.

![Bad Amazon data example](/content/blog/images/bad-data-nameless-amazon-bestsellers/bad-amazon-names.webp "Bad Amazon data. Example missing names.")

Amazon serves data of 100s of millions of products on its websites, often provided by marketplace sellers and many being seen and sold rarely. Over the last decade, [Bold Data's founder](https://www.bolddata.org/about) has seen a lot of poor data in this long tail. This included gems like Amazon test data that somehow made it onto the public website.

Last week, while processing the bestsellers of amazon.co.uk, amazon.de, and amazon.com, something peculiar surfaced. Bestsellers are products ranked in the top 100 in at least one product category. We found that four bestsellers had no names. Based on how the Amazon bestseller website presents its data, that should not be possible. What happened?

## Error in error out

Looking at the Amazon web pages for the nameless products, what happened quickly becomes apparent to people familiar with data or software engineering.

![NULL product name](/content/blog/images/bad-data-nameless-amazon-bestsellers/bestsellers-data-quality-issue-ant-killer.webp "NULL product name")

But even if you are not an engineer, you can see the names in the images sound strange. Computer systems use NULL, NaN, NA, and similar outputs to indicate no data for a field or attribute to a human user. In simple terms and making some inferences, the upload into Amazon contained a message of no data, e.g. `NULL`. Instead of failing, the message was converted into text and was stored wrongly as the product name in Amazon's product catalogue. When Bold Data analysed the data, the error reappeared.

![NA product name](/content/blog/images/bad-data-nameless-amazon-bestsellers/bestsellers-data-quality-issue-pantyhoses.webp "NA product name")

You may know the saying "garbage in, garbage out", which computer scientists use. In particular, data engineers and data scientists use it to highlight that if your foundation, the data, is imperfect, then so will be the outcome. Therefore, experienced data professionals prioritise sourcing accurate data and its processing instead of applying increasingly complex analytics or machine learning algorithms.

![NULL product name](/content/blog/images/bad-data-nameless-amazon-bestsellers/bestsellers-data-quality-issue-dog-food.webp "NULL product name")

## One in a million error

While this is unfortunate and surprising that these items made it into the bestsellers without names, it is not as bad as it seems. The dataset analysed comprised 4.88 million products from three Amazon websites, the United States, Great Britain, and Germany. So close to one in a million was wrong, a small number. However, it demonstrates that our systems have to expect and accommodate data errors. Mined data is only as good as its source system.

While the error rate is low, the product's name is prominent, and other attributes have not as much scrutiny. We will publish datasets, analyses and more findings in the future. Be sure to subscribe to our email list so as not to miss these updates.

## Bold Data

The described challenge in this post is one of many that data mining and data engineering face daily. The Internet has an abundance of valuable data. Mining and processing data at scale, with low cost, high confidence and quality are complex, requiring decades of experience. This is precisely what Bold Data has focused on for our customers. Create affordable, reliable datasets and decision support Analytics so you can make better decisions daily.

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

Many Amazon marketplace customers know that its huge product catalogue has data quality issues. However, they might expect its top sellers, which t...

Bad Amazon data. Example missing names.

Many Amazon marketplace customers know that its huge product catalogue has data quality issues. However, they might expect its top sellers, which they frequently see and buy, to be accurate. Bold Data, which is processing 100s of millions of products daily, has a unique ability to find hidden insights and issues. For example, active Amazon bestsellers with names resulting from data processing errors.

Bad Data: Nameless Amazon bestsellers

# Public data is of public interest

By _Christian Prokopp_ on _2022-04-25_

Public data has an enormous commercial and social impact. For example, in Ukraine, it affects war and peace, and with the Coronavirus, it involves life and death. We must keep public data accessible for the public good.

![Data monopolies](/content/blog/images/web-scraping-and-public-data-is-of-public-interest/fortress-logos-major-tech-companies-like-800.webp "Data monopolies: The big tech companies have a monopoly on their users' data. They lock it away and make it hard to access.")

As [Techcrunch reported](https://techcrunch.com/2022/04/18/web-scraping-legal-court/), the recent challenge by LinkedIn against web scraping of its public content received a blow in the courts. While this is a commercial battle between two companies, the impact of the ruling reaches much further. Besides commercial interests, academia, journalism, research and archiving depend on the ability to scrape public data without limitations.

## Data mining and data scraping in the public interest

An example of how access to information is essential is the impact [Open-source intelligence (OSINT)](https://en.wikipedia.org/wiki/Open-source_intelligence) has. The war in Ukraine demonstrates how [amateurs can collect and assemble impressive intelligence](https://restofworld.org/2022/osint-viral-ukraine/) informing reporting and even the defence of the country. Notably, the legal accessibility of data is the foundation of OSINT. It may not have prevented the war, but it certainly helps journalists report it, and investigators [analyse possible war crimes](https://www.nytimes.com/2022/04/04/world/europe/bucha-ukraine-bodies.html). Data has become so essential for reporting that it coined the term [Data Journalism](https://www.theguardian.com/membership/datablog/2021/sep/13/numbers-you-can-tell-stories-with-a-decade-of-guardian-data-journalism). Who could imagine the COVID reporting of the last years without the detailed Analytics? They require vast datasets coming from data mining and web scraping, for example, ["[t]he New York Times has made more than 9.98 million programmatic requests for Covid-19 data from websites around the world"](https://open.nytimes.com/tracking-covid-19-from-hundreds-of-sources-one-extracted-record-at-a-time-dd8cbd31f9b4).

## Data help markets function

At Bold Data, we believe public data also plays a vital role in the functioning of markets. Public data can improve decision-making dramatically by helping companies, for example, with competitive pricing, reduced stock and capital costs, better forecasting models predicting demand and much more. Customers and the environment benefit from more efficient markets providing better products and services with less waste.

However, to retrieve public data at scale, making it easy to process, interpret, and analyse is complex. Public data is hard to retrieve, and its quality varies widely. For example, source system availability and data format change, the data has errors, is rarely complete, may change and often has duplication. Consequently, data mining and wrangling are out of the reach of most companies either because of technological or financial hurdles. Moreover, combining the data mining efforts across companies and industries has enormous scale and cost benefits for all participants. That is the idea behind Bold Data to share expertise, technology and economies of scale with our customers for affordable, accessible, data-driven decision-making.

## Data monopolies

The most challenging hurdle is the data monopolies by platforms like Google, Amazon, and Facebook, locking their users' and customers' data away. Bold Data does not provide copyrighted or restricted non-public data. However, wherever public data is legally available, we collect, clean and analyse it to give our customers the best possible insight to compete on a more level playing field. Over time, we aim to reduce the data monopolies. That is why the continued lawfulness of scraping public data is of importance for you, our customers and their customers, to create a competitive landscape.

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

Public data has an enormous commercial and social impact. For example, in Ukraine, it affects war and peace, and with the Coronavirus, it involves...

Data monopolies: The big tech companies have a monopoly on their users' data. They lock it away and make it hard to access.

Public data has an enormous commercial and social impact. For example, in Ukraine, it affects war and peace, and with the Coronavirus, it involves life and death. We must keep public data accessible for the public good.

Blog