# How to code Python with ChatGPT

By _Christian Prokopp_ on _2023-01-29_

Can ChatGPT help you develop software in Python? Let us ask ChatGPT to write code to query AWS Athena to test if and how we can do it step-by-step.

![A robot meditating](/content/blog/images/how-to-code-python-with-chatgpt/meditating-robot.jpg "A robot meditating")

We start with a brief introduction to ChaptGPT and code generation. Then we attempt a real-world code problem with ChaptGPT. This will highlight its abilities and limits.

## OpenAI ChatGPT for Code

If you have not done so, go to [OpenAI's ChatGPT website](https://openai.com/blog/chatgpt/), register for an account and login. You will see a screen similar to the one below after logging in.

![ChatGPT empty dialog screen](/content/blog/images/how-to-code-python-with-chatgpt/1.png)

The ease of use of ChatGPT is one of its success factors. You type your question or command in natural language, and it will answer human-like with generated text.

`> What is ChatGPT?`

![Example of using ChatGPT](/content/blog/images/how-to-code-python-with-chatgpt/2.png)

One amazing feature of ChatGPT is its ability to respond to programming questions with code.

`> Write a hello world program in Python.`

![ChatGPT > Write a hello world program in Python.](/content/blog/images/how-to-code-python-with-chatgpt/3.png)

A useful capability of ChatGPT is its ability to respond to follow-up questions and modifications to previous questions.

`> Move it into a function.`

![ChatGPT > Move it into a function.](/content/blog/images/how-to-code-python-with-chatgpt/4.png)

The model understood the reference of 'it' and what a Python function is and named it aptly. And if you are unhappy with a response, you can edit the question asked by clicking the edit icon to the right of your question. Or you can click the 'Regenerate response' button to give ChatGPT a chance to come up with a different answer.

While we will stick with Python for this example, it is noteworthy that ChatGPT can also generate other code in other languages.

`> Turn it into JS code.`

![ChatGPT > Turn it into JS code.](/content/blog/images/how-to-code-python-with-chatgpt/5.png)

Apparently, twelve or more, according to the model itself.

`> List the programming languages for which you can produce code.`

![ChatGPT > List the programming languages for which you can produce code for.](/content/blog/images/how-to-code-python-with-chatgpt/6.png)


Having the right approach and avoiding some pitfalls is essential to using ChatGPT. Otherwise, you may waste your time or end up with bad code.

## How to start coding in ChatGPT

You will need your usual tools like an IDE, terminal, code version control like git, etc. If you are new, a simple text editor or any IDE will do to start. Create your Python project and add an empty Python file.

Trying to get all aspects of the code into one question that ChatGPT will understand correctly is tricky. A good strategy for using ChatGPT is to ask it to solve a simple problem and then iterate with additional improvements in follow-up questions. The starting point can be a simple requirement you work on or an idea you want to explore a new technology or feature.

## Query Athena with Python using ChatGPT

Let us use something that could come up in everyday development work. Write some reusable code we can use to query Athena. Athena is an AWS NoSQL store using Trino as a query engine and S3 as storage. Such code could be useful for Data Engineers, Data Scientists or Software Engineers. Importantly, it is achievable for ChatGPT and more challenging than reproducing a Hello World problem.

Click on the '+ New chat' button on the top left to open a clean context. Note your output will vary from mine.

`> Write a Python function to query Athena.`

![ChatGPT > Write a Python function to query Athena.](/content/blog/images/how-to-code-python-with-chatgpt/7.png)


Besides the code, it provides helpful commentary to explain it. So far, we could easily get similar code using Google and Stack Overflow.

![ChatGPT > Write a Python function to query Athena. (Commentary)](/content/blog/images/how-to-code-python-with-chatgpt/8.png)

It becomes interesting as we refine the work. Let us move the database into a parameter instead of hardcoding it in the query.

`> Pass the database as a parameter for the function and the query execution.`

![ChatGPT > Pass the database as a parameter for the function and the query execution.](/content/blog/images/how-to-code-python-with-chatgpt/9.png)

Of course, this will not work as an independent script, so we cannot test if the generated code works. Let us fix that so we can sense-check the output. We will get to unit testing later.

`> Add a main function. Read the database, query and s3_output as named command line arguments.`


![ChatGPT Add a main function. Read the database, query and s3_output as named command line arguments. 1/2](/content/blog/images/how-to-code-python-with-chatgpt/10.png)


![ChatGPT > Add a main function. Read the database, query and s3_output as named command line arguments. 2/2](/content/blog/images/how-to-code-python-with-chatgpt/11.png)

Assuming you have your AWS credentials set up, you could now copy the code into a Python file to try and query Athena from the CLI.

`CLI > python scripts/chatgpt.py --database default --query "SELECT 1+1" --s3_output s3://somewhere`

`Traceback (most recent call last): ...`

`botocore.exceptions.NoRegionError: You must specify a region.`

Clearly, ChatGPT missed the region parameter. Let us ask it to fix the error.

`> Your code creates the following error:   botocore.exceptions.NoRegionError: You must specify a region.   Fix it.`

![ChatGPT > Fix botocore.exceptions.NoRegionError](/content/blog/images/how-to-code-python-with-chatgpt/12.png)

The code works. Adding the region allows the query to execute. But instead of the result, we only get the query execution ID printed.

`> Let query_athena return the query result as a dictionary and print the returned result in the main function.`

![ChatGPT > Let query_athena return the query result as a dictionary and print the returned result in the main function.](/content/blog/images/how-to-code-python-with-chatgpt/13.png)

When we copy the code and run it, we have a new error, so let us ask it to be fixed too.

`> Your code throws the error:   botocore.errorfactory.InvalidRequestException: An error occurred (InvalidRequestException) when calling the GetQueryResults operation: Query has not yet finished. Current state: QUEUED   Fix it.`

![ChatGPT > Fix botocore.errorfactory.InvalidRequestException](/content/blog/images/how-to-code-python-with-chatgpt/14.png)

Experienced developers will spot a bug here. If we run the ChatGPT code, we will get the same error because the while loop tests for a condition that can never be true since the variable is set to None before the loop.

`> Find the bug in query_athena and fix it.`

![ChatGPT > Find the bug in query_athena and fix it.](/content/blog/images/how-to-code-python-with-chatgpt/15.png)

While it has some good suggestions, none are relevant, and it needs to understand what it did wrong without giving it a helping hand.

`> There is a bug in the query_status while loop. It can never be true because it is set to None before the loop. Fix it.`

![ChatGPT > There is a bug in the query_status while loop. It can never be true because it is set to None before the loop. Fix it.](/content/blog/images/how-to-code-python-with-chatgpt/16.png)

Better, the code works again. But the output is a list of dictionaries, e.g. each row is a separate dictionary. That is not easy to work with or store. Let us use a dataframe instead.

`> Convert the query result in query_athena into a dataframe instead of a dictionary.`

![ChatGPT > Convert the query result in query_athena into a dataframe instead of a dictionary.](/content/blog/images/how-to-code-python-with-chatgpt/17.png)

![ChatGPT > Convert the query result in query_athena into a dataframe instead of a dictionary. (main function)](/content/blog/images/how-to-code-python-with-chatgpt/18.png)

Unprompted, it also provided a 'new' main function. ChatGPT does reimagine the code every time, which allows for errors, resurfacing bugs or duplications. That can be confusing and break code. We will edit the main function as needed manually in our code instead.

We can see a few opportunities to move forward by playing around and taking a step back. Results of 1,000 or more rows only download partially. Large Athena results need pagination to download the whole. Furthermore, there is no unit testing, no type annotation, and no easy reuse if we want to use the code for multiple queries in an existing code base. Let us quickly try and fix some of these without going into each in detail.

`> Add type annotation to query_athena.`

`[...]`

`> Add pagination to the query result handling to deal with results larger than 1000 rows.`

`[...]`

`> In the while NextToken loop only ignore the first row of the query result for the first result page but not for any of the following.`

`[...]`

`> Change query_athena to a singleton class called AthenaQuery that takes s3_output and region as parameters to the constructor. Let the constructor create and store a reusable Athena client connection. Put all the query execution and result transformation into a method called execute that takes the query and database as parameters.`

![ChatGPT > Change query_athena ... (1/2)](/content/blog/images/how-to-code-python-with-chatgpt/19.png)

![ChatGPT > Change query_athena ... (2/2)](/content/blog/images/how-to-code-python-with-chatgpt/20.png)

At this point, I asked ChatGPT a few times to refactor the execute method to break out some private methods. It led to the reintroduction of previous bugs and inefficient in-memory data movement. So I abandoned that step.

Looking at the execute method, we find it reintroduced storing data in an in-memory list of dictionaries before converting it into a dataframe. And it reintroduced storing the header as the first row of data. When I tried to prompt it to remove these, it started breaking further, moving around parameter order in the method signature and changing variable names.

Similarly, when I attempted to ask for a refactored \_\_new\_\_ method to accommodate an optional mockable client for easier unit testing, it started breaking the code in various ways. I have had luck with ChatGPT providing some decent unit tests in some circumstances but not this time.

![ChatGPT > An error occurred. If this issue persists please contact us through our help center at help.openai.com.](/content/blog/images/how-to-code-python-with-chatgpt/21.png)

Sadly, eventually, the service stopped working and returned only error messages. So the how-to has to stop here. Hopefully, when it becomes a paid service, it will be more reliable.

## ChatGPT code limitations

In a way, the issues this how-to ran into are an excellent example of the current limitations.

1.  Timeliness - the model's data is not as up-to-date as a search engine, so recent tools, libraries, releases, etc, may need to be added and not suggested.
2.  Hallucinations and inconsistency - the model may plug gaps with sensible sounding words or code. That could be data structures, libraries or keys that do not exist. Or it may just change names, ordering or anything else with every new step.
3.  Duplication - the model may combine (in isolation) reasonable steps or libraries, but that have duplicate effort, e.g. it might suggest compression of already compressed data. Also, it tends to repeat errors after a while despite prompts that corrected it previously.
4.  Simplistic - the model often needs to be prompted to improve or harden code and is biased towards simple examples without edge cases or production aspects considered.
5.  Superfluous and wrong - the model may make duplicate, unnecessary or plain harmful suggestions or code. It does not understand the logical dependency of code and may give generally good advice but to unsuitable specific circumstances.
6.  Size and complexity - the input and output of ChatGPT are limited. The exact details are confidential, but you run into them. For example, the output might cut off mid-code or sentence when it runs out of 'space'.
7.  Availability - while the service is in public trials, it is unreliable.

## Summary

Today, I find correctness, complexity and length to be the key issue. If you can limit your interaction to a small function and scope, you might get something helpful or inspiring. But check every line and every statement.

Overall, it is a fascinating technology and gives us a glimpse into future software development tooling. Like documentation lookups on function signatures in IDEs, LLMs like ChatGPT could provide intelligent suggestions or code completions. But the output's usefulness and correctness need addressing first.

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

Can ChatGPT help you develop software in Python? Let us ask ChatGPT to write code to query AWS Athena to test if and how we can do it step-by-step.

A robot meditating

How to code Python with ChatGPT

# No, ChatGPT is not 10x-ing developer performance

By _Christian Prokopp_ on _2023-01-25_

ChatGPT and similar language models have recently been gaining attention for their potential to revolutionise code generation and enhance developer productivity. I was curious to see what all the hype was about, so I decided to try it out for some development work.

![A robot staring at you](/content/blog/images/chatgpt-is-not-10x-developer-performance.webp "A robot staring at you")

After conducting my own research, it became clear that ChatGPT and similar language models are powerful tools that can help developers address various issues. Their ability to solve problems in code, self-optimise and find alternatives to given solutions is particularly impressive. Additionally, ChatGPT can introduce developers to new libraries and tools they may not have been aware of before.

However, it is noteworthy that these tools have severe limitations. The model sometimes generates subtle errors, such as comparisons without conditions or accessing non-existing parts in data structures. Additionally, it may offer pointless improvements or duplicate steps like applying multiple compressions.

Is the speed at which machine learning advances solving these issues soon? Maybe, maybe not. For example, self-driving cars have been just around the corner for many years. Replacing human capabilities is not a linear problem. While car manufacturers have solved most of the problem swiftly, the last bit has been intractable and requires further research and development of an unknown timeframe.

Similarly, ChatGPT code and other generations have matured to impressive abilities quickly. But to make it reliable and trustworthy, the last bit of maturity could be many, many years away. As developers are already exploring the technology, it has the potential to make its way into critical systems such as financial, infrastructure, defence, or healthcare systems. This is why it is important to be cautious and aware of the limitations of these tools.

Developers need to remember to treat output from ChatGPT with caution. While it can be intriguing and helpful, developers should always double-check and adapt the code. This is why colleagues or an interactive community like Stack Overflow, where developers validate, share experiences and discuss the pros and cons of the code, remain superior.

In conclusion, ChatGPT and similar language models are powerful tools that can be valuable for developers. However, they come with limitations which negate many of their benefits. These tools can be a great way to inspire, explore, and learn, but they require experienced developers and good practices to be used safely. They are not tools that will replace developers and make them unnecessary but instead means to help experienced developers do their work more efficiently in the future.

---

_Note, I used ChatGPT with some bullet points to generate the first draft of this post. That was helpful but the amount of edits needed to get to something worth posting was surprisingly high. This experience feels similar to trying to use it as a code copilot. It looks good on first glance but the devil is in the detail._

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

ChatGPT and similar language models have recently been gaining attention for their potential to revolutionise code generation and enhance developer...

A robot staring at you

ChatGPT and similar language models have recently been gaining attention for their potential to revolutionise code generation and enhance developer productivity. I was curious to see what all the hype was about, so I decided to try it out for some development work.

No, ChatGPT is not 10x-ing developer performance

# Faster and Cheaper: ARM Graviton vs Intel and AMD x86 AWS EC2

By _Christian Prokopp_ on _2023-01-20_

How Bold Data achieved an astonishing 2.3x improvement by switching from x86 to ARM.

![A graph showing performance change](/content/blog/images/faster-and-cheaper-arm-graviton-vs-intel-and-amd-x86-aws-ec2.webp "A graph showing performance change")

Measure, measure, measure, and do the math. It pays.

## ARM vs x86 performance

Generally, AWS ARM EC2 instances perform similarly to their x86 equivalents. AWS prices them that way with a bit of favour towards ARM. But they are different architectures, and sometimes, one excels over the other in specific tasks.

I recently switched an EC2 fleet from x86 to ARM as a sense check, expecting similar performance. Occasional experiments allow you to (re)check assumptions if you always measure and have baselines.

To my surprise, there was a substantial performance difference of approximately 70% for the specific workload on this fleet, i.e. ARM performed 170% throughput of the x86 equivalents for both Intel and AMD instances (see above image).

## ARM vs x86 cost

While an impressive improvement, it does not yet account for the price difference. Always watch spot prices, as they can spike and exceed on-demand costs. Sometimes ARM, while having a lower initial cost, can exceed x86 spot prices. However, in this case, the ARM instances cost 75% of the x86 equivalents I used before.

## Savings

In summary, **we achieved 2.3x per $ by switching from x86 to ARM** when combining the higher throughput and lower cost per instance. The caveat from the post should be clear. Improvements depend entirely on your workload and instance (spot) prices and are not a general rule. It is a reminder that measuring, observing and experimenting with your performance and cost is essential.

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

How Bold Data achieved an astonishing 2.3x improvement by switching from x86 to ARM.

A graph showing performance change

Faster and Cheaper: ARM Graviton vs Intel and AMD x86 AWS EC2

# A poem about Data

By _Christian Prokopp_ on _2022-12-05_

Data is the root of all my worries ...

![A humanoid robot staring into the distance](/content/blog/images/a-poem-about-data.webp "A humanoid robot staring into the distance")

_Data is the root of all my worries
A complex system of numbers and hurry
It holds the secrets to success
A source of knowledge, so I confess_

_Data can make or break a business
A tool to measure what’s true, or a mess
It can offer insights and trends
Or simply lead to dead ends_

_Data can be a blessing or a curse
It can fill us with joy or make us feel worse
It can help us make informed decisions
Or add unnecessary revisions_

_Data can be a powerful tool
It can help us reach our goals
We must use it wisely and with care
It’s an asset that we must share_

* * *

Poem generated using GPT-3 🤖

Image generated using DALL·E 🤖

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

A humanoid robot staring into the distance

A poem about Data

# OpenAI GPT-3: Content spam or more?

By _Christian Prokopp_ on _2022-12-04_

OpenAI's ChatGPT has made the news recently as a next-generation conversational agent. It has a surprising breadth which made me wonder, could OpenAI generate specific technology content good enough to post, and what would that imply for the future?

![A robot at a typewriter in a library](/content/blog/images/openai-content-spam-problem.webp "A robot at a typewriter in a library")

First, I tried the research preview of ChatGPT.

---

**Me:** _What is AWS Athena?_

**ChatGPT:** _Amazon Athena is a serverless query service that allows you to analyze data in Amazon S3 using SQL. Athena is easy to use, and you can get started with it quickly. It allows you to query data stored in Amazon S3 using SQL, and can be used to analyze a wide variety of data such as logs, clickstream data, and IoT data. With Athena, there is no need to set up, manage, or scale a data warehouse, which makes it a cost-effective and convenient option for querying data stored in S3._

---

Nice. Immediately, ChatGPT could be useful querying documentation or even the internet (hey Google) in a better way. But how about something more complex, like writing technical content? If you read Medium or other sources, you know that there is a wide range of human-written content, from the novice and incomprehensible to simple tutorials and analysis to deep, insightful technical domain expertise.

Last week, I wrote about [AWS Athena](https://www.bolddata.org/blog/delta-lake-vs-data-lake/), and in the past, teams I worked with discussed the pros and cons between Athena and Redshift. They are different tools for different purposes but have some overlap. It is a particular technical discussion. Surely, a general ML text model will be unable to generate anything meaningful about the topic.

## Enter OpenAI

I used the OpenAI GTP-3 (text-davinci-003) website's beta and prompted it to write a blog post about the subject.

---

**Prompt:** _A blog post on AWS Athena and why it is better than AWS Redshift._

**OpenAI content:** _As businesses increasingly move their data and tracking systems to the cloud, there is an increasing need for powerful and efficient ways of querying this stored data. This is where AWS Athena comes in._

_AWS Athena is a query service built on top of Amazon’s cloud infrastructure that allows users to easily query and analyze data stored in the cloud in a variety of different formats. With Athena, users can run complex queries against their data without having to manage a separate cluster or rely on managing their own hardware._

_One of the primary advantages that sets Athena apart from other cloud query services is its ability to quickly query data stored in the cloud with low latency. Athena queries are able to execute in a matter of seconds, which makes it much faster than other services like AWS Redshift, which can take minutes or even hours to execute. This means that users can quickly access the data they need without having to wait for long periods of time._

_Another benefit is that Athena provides users with a wide range of data sources that can be queried, including Amazon S3, Amazon RDS, and other cloud storage services. This makes it much easier to query data from multiple sources in one place._

_Finally, Athena is also much cheaper to use than AWS Redshift. With Athena, users only pay for the queries they run, allowing them to save significant amounts of money. By contrast, with Redshift, users have to pay for the cluster as well as the queries, which can add up to significant costs._

_In conclusion, AWS Athena provides businesses with a powerful and efficient way of querying data stored in the cloud in a variety of different formats. With its ability to execute query quickly with low latency, support for a range of data sources, and lower costs than AWS Redshift, Athena is a great choice for businesses looking to efficiently query and analyze their cloud data._

---

Impressive. Some details are either inaccurate or depend on circumstance, but this was mostly correct and had some valuable insight. It is as good (or bad) as some posts I have read thrown together by the odd human, but also, it is far from what an eloquent domain expert could write. It does not come across as artificial or nonsensical, importantly.

## And the reverse?

As I wrote, it is an open argument, so let's see what happens if we favour Redshift in the prompt as the better solution.

---

**Prompt:** _A blog post about AWS Redshift and why it is better than AWS Athena._

**OpenAI content:** _AWS Redshift is a cloud-based data warehouse service that is part of Amazon's larger suite of cloud-based products. It provides businesses with a scalable, secure, and cost-effective way to store and analyze their data. It offers features such as columnar storage, automatic compression, advanced query optimization, and much more._

_Compared to AWS Athena, Redshift offers a number of advantages that make it a better choice for businesses looking to store and analyze their data._

_First, Redshift is much faster than Athena. With Redshift, it is easy to quickly analyze large amounts of data. It offers advanced query optimization, allowing queries to be processed faster and more efficiently. Additionally, Redshift offers accelerated databases, which can increase query performance even further._

_Second, Redshift is more secure than Athena. It offers improved security measures such as encryption at rest and in transit, role-based access control, and data auditing. With these features, businesses can rest assured that their data is safe and protected._

_Third, Redshift offers much more scalability than Athena. Redshift's elasticity and scalability allows businesses to easily scale up or down their data analysis needs as the business grows. This makes it easier to manage and control costs associated with storing and analyzing data._

_Finally, Redshift is much more cost-effective than Athena. It offers businesses flexible pricing options, making it easy to select the best plan for their particular needs._

---

We can see an inconsistency in the argument, e.g. both scenarios claim to be the faster and cheaper one. But you could argue with enough know-how and given the right scenarios either way. Is OpenAI that smart, or is it simply repeating common unsubstantiated claims from mediocre articles it learned? I doubt it is that clever. Still, the point is that it is good enough that you can argue about it and not immediately dismiss it as false or artificial.

## AI content: Spam or Gold?

Sticking with OpenAI, I created the image of the robot with the typewriter at the beginning using Dall-E. Here is the kicker, the two blog posts and image generation cost only 20 cents and minutes of machine time. How much money and time would it cost to hire a mediocre technical writer and artist to create equivalent content?

Highly qualified humans can still create much more insightful and correct content. However, things are about to change on the lower end of content creation, which is about the art of good enough, cost and scale. And where that lower end begins will move rapidly with the quality of the machine learning models improving.

As a pessimist, I imagine the advent of a new wave of low-cost spam content indistinguishable from mediocre human content regurgitating existing content learned in new variations.

As an optimist, I imagine the advent of new content generation that synthesises unique insight from multiple data sources or more accessible information like translating documentation into compelling articles and posts.

Whatever the future holds, visual and text content generation will change dramatically.

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

OpenAI's ChatGPT has made the news recently as a next-generation conversational agent. It has a surprising breadth which made me wonder, could Open...

A robot at a typewriter in a library

OpenAI's ChatGPT has made the news recently as a next-generation conversational agent. It has a surprising breadth which made me wonder, could OpenAI generate specific technology content good enough to post, and what would that imply for the future?

OpenAI GPT-3: Content spam or more?

# Is Athena Spark a Delta Lake alternative to Databricks?

By _Christian Prokopp_ on _2022-12-02_

Finally. AWS re:Invent 2022 brought the answer to both Databricks and Athena's worst limitations. Athena Spark promises to bring Delta Lake scale-out processing effortlessly and inexpensively.

![Beautiful lake](/content/blog/images/athena-spark-delta-lake-alternative-to-databricks.webp "Beautiful lake")

## Athena Data Lake

Athena started as a serverless Presto engine six years ago, in 2016. Its key selling point was to do away with the cost and complexity of Hadoop at a fixed price per TB of Data scanned. As an experienced and avid user of Hadoop and Hive at the time, I fell in love with Athena. It took away the pain and complexity of Hadoop, and with S3, it had the perfect durable, scalable, performant storage layer.

AWS Glue joined as its metastore in the 2017 product lineup. It allowed later on the option to access and process the same data from Athena or Spark via EMR and Hadoop. An important addition since Athena's Presto core has limits, and Athena queries timeout at a hard limit of 15 minutes. So Athena could not execute complex, extremely memory-intensive or long-running processing. Of course, with enough creativity, breaking down processes and data into more tables or steps, you can workaround the limit to a degree but at the cost of complexity, stability and your sanity.

## Athena Delta Lake

Athena grew further with the addition of Iceberg and the Trino engine update. These promised ACID compliance and performance improvements. It is noteworthy that the underlying concept of storing the data in S3 with serverless processing means Athena never intended to compete with the access patterns of RDBMS or low latency NoSQL stores. ACID compliance, in this case, is more aimed at helping with the overlapping or continuous landing of data that previously needed to be reconciled before analysis or loading into low-latency stores.

I have yet to explore these features in depth. Do check out how Athena engine v2 vs v3 performs for you, though. I was surprised by how much slower instead of faster it was in a specific circumstance of mine. That may be resolved with future updates, or I may have to change my data processing strategy a bit, e.g. embracing Iceberg over my (now) slow legacy workarounds.

## Athena Spark

While I am a fan of Athena, I had the opportunity to work with Databricks for a while. Its ease of use, performance and accessibility to Data Engineers, Data Scientists and Data Analysts/BI make it a valuable part of a company's data stack. However, that value comes at a cost in licensing and computing. Athena was always mind-blowingly cheaper but limited in applicable use cases. Extending the use cases with less expensive EMR/Hadoop adds so much complexity that it removes the cost benefits compared to Databricks in many scenarios.

That is the sweet spot that Athena Spark could fill and make a huge impact. If Athena Spark turns out to be nearly as easy to use and as cost-effective as Athena was so far, then it could cover a massive range of use cases requiring something like Hadoop with Spark or Databricks today.

'Legacy' Athena with Trino is excellent at scale-out SQL for BI analytics. The addition of Athena Spark promises to break into advanced analytics, ML, Data Science and extensive data processing and movement. The parts are here; it is up to AWS to prove the ease of use and cost.

## Athena Spark Pricing

Anyone familiar with Databricks pricing will find Athena Spark pricing oddly familiar. Instead of DBU (Databricks Unit) AWS uses DPU (Data Processing Units), an hourly rate (e.g. $0.35 in us-east-1) for the machines involved doing the processing normalised at 4 vCPUs and 16GB RAM per DPU and hour. The cost depends on the length of sessions, the driver machine and the number and time of machines processing data.

Sessions are abstracted away from you with transparent starts and timed out shutdowns making it less complicated than DIY Hadoop on EC2s or EMR. However, it is less predictable and straightforward than the Athena Trino version's $5 per TB scanned. It is understandable why AWS chose this model to tie pricing to costs, but from a user perspective, it is as complicated as Databrick's model. With one significant benefit. Databricks does encourage contracts and pre-buying DBUs with its pricing and discounting model, which is a severe headache when you budget and try to predict the future. The AWS pay-as-you-go one/best price option is preferable.

Lastly, I have yet to try the service. It is not available in the regions I need it most. AWS promised to roll it out over the following months in more regions. I also have no illusions that its feature set will be significantly smaller than Databrick's. However, when I attempted to start an Athena Spark session in a supported region, it failed with a permission error. Teething errors, I hope. The promise of an accessible, cost-effective Athena Spark is exciting but has to prove itself in the real-world now.

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

Finally. AWS re:Invent 2022 brought the answer to both Databricks and Athena's worst limitations. Athena Spark promises to bring Delta Lake scale-o...

Beautiful lake

Finally. AWS re:Invent 2022 brought the answer to both Databricks and Athena's worst limitations. Athena Spark promises to bring Delta Lake scale-out processing effortlessly and inexpensively.

Is Athena Spark a Delta Lake alternative to Databricks?

# Delta Lake vs Data Lake

By _Christian Prokopp_ on _2022-11-02_

Should you switch your Data Lake to a Delta Lake? At first glance, Delta Lakes offer benefits and features like ACID transactions. But at what cost?

![Beautiful lake](/content/blog/images/delta-lake-vs-data-lake.webp "Beautiful lake")

## Data Lake

The Data Lake traces its origin back to the heyday of Hadoop and the emergence of cloud computing. The idea was intriguing. Store all of your raw data, e.g. log files, CSV, JSON, and so forth, on commodity scale-out storage and compute and make sense of it at read time, for example, with MapReduce.

That was great for emerging schema, scalable write, and utilising comparatively inexpensive computing compared to Oracle or similar. However, MapReduce is expensive in engineering time and excludes most users from directly using the data.

Innovations like Apache Hive and Apache Parquet addressed some of these shortcomings. Hive provides familiar SQL interfaces to the data, and Parquet makes storing and retrieving data more efficient. A key aspect of Data Lakes remained the same. They usually are append-only and optimised for large-scale processing making small data retrieval slow compared to RDBMS or indexed NoSQL stores.

## Delta Lake

Delta Lake offers several improvements intended to make it more usable for users and in more scenarios than a Data Lake. It sometimes is framed as an additional layer on top or an evolution of the Data Lake. However, as expert readers might note, some features and improvements made their way into the ever-evolving Data Lake ecosystems or could be achieved with effort, making the lines between them blurry.

Something most RDBMS users miss in a Data Lake are ACID transactions which Delta Lake provides as a powerful feature, including the ability to version and time travel back and forth between versions of the data. Delta Lake improves metadata management with more scalability, consistency, and flexibility. It also integrates well with batching and streaming with SQL, Python, Scala and Java APIs.

Ideally, Delta Lakes offer more usability to engineers and savvy data end-users, reducing or eliminating the need for landing data aggregates in specialised stores for use cases with differing access patterns and data reconciliation from multiple stores.

## Data Lake vs Delta Lake

While Delta Lake provides significant improvements and features, it comes at a cost. Firstly, Delta Lakes are more expensive in money, time, infrastructure and complexity than Data Lakes. Data Lakes and technology like Hive, Trino and Athena are cost-efficient for their ideal use cases. AWS Athena, for example, is priced at $5 per TB scanned, which can go a long way with efficient Parquet storage.

You should only consider a Delta Lake if a Data Lake proves insufficient. The connectivity and integration option of Delta Lakes allows this as an evolution. Upgrading to a Delta Lake from a Data Lake when needed is a viable option. Use Hive, Athena, Trino, Flink or a similar Data Lake if it works for your use case, especially when you are cost sensitive.

Secondly, the underlying technologies and the architecture of Delta Lakes dictate constraints. Limits are natural when you run on distributed computing and storage. You can lessen them with cutting-edge and expensive machines, local storage, or optimise and tune things in private clouds. But you will not beat systems built for a purpose. Do not expect to outperform RDBMS in their domain or specialised NoSQL stores in theirs.

Depending on the use case, you could choose a simple Data Lake and optimised storage. However, Delta Lakes can be an appealing option where performance expectations allow some flexibility to address multiple access patterns and needs in one architecture.

* * *

## Update

AWS released Athena Spark at re:invent 2022. [Is Athena Spark a Delta Lake alternative to Databricks](https://www.bolddata.org/blog/delta-lake-vs-data-lake/)?

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

Should you switch your Data Lake to a Delta Lake? At first glance, Delta Lakes offer benefits and features like ACID transactions. But at what cost?

Delta Lake vs Data Lake

# One simple thing companies miss about their Data

By _Christian Prokopp_ on _2022-08-08_

There is one simple thing most companies miss about their data. It has been instrumental in my work as a data professional ever since.

![Can't see the trees for the wood](/content/blog/images/one-simple-thing-companies-miss-on-data.webp "Can't see the trees for the wood")

Regarding data, we regularly cannot see the wood for the trees. It happened to me when I worked at a startup a decade ago when the brilliant [Ted Dunning](https://www.linkedin.com/in/teddunning/) visited us. We did the horse and pony show to impress him with the cool stuff we built. He listened, and the comment that stuck with me was, "all the successful startups I have seen measure themselves".

## Know thyself

It was powerful. Ted did not say much about what we did wrong or what we should do. At least not directly, but we all got the message, and it hit home.

We were too comfortable with our work, i.e. we knew every line of code and datastore. But humans are terrible at looking at data and finding patterns without some (graphical) reporting. I mined millions and millions of products daily but had only basic observability built in because I knew every process and system involved. Yet that is not good enough and not sustainable.

We heard Ted loud and clear. We started collecting and visualising operational data and metadata to help us manage our platform and processes. Eventually, we exited to Google, so things did work out.

## Define and measure data quality and processing

The one simple thing most companies miss on their data is measuring and measuring the right thing, which is tough. Many companies use data to measure (other) things, but few measure the data itself! The data quality, processes and metadata, i.e. is the data you use trustworthy? What are the trends regarding data quality, size, speed, etc.?

Since Ted's memorable visit, I have developed this further in every role. It keeps paying dividends. Recently, for example, for a good week, I have been trying to figure out a weird shape in the curve measuring some operational metrics on processing throughput and queuing in my staging data platform at Bold Data. The platform's overall performance was good, and so was the data quality. Still, it looked wonky with jumps where there should be smooth lines. After extensive digging and thinking, I figured out an overlapping, interfering behaviour between multiple data mining processes that only occurs under specific circumstances that could impact subtly but significantly the SLA and cost in the long term. Something hard to predict or test for proactively or even detect later on.

## Do not trust what you have not checked yourself

In previous roles, I always used a two-pronged approach when I had to deal with legacy systems and data. Quality and quantity, i.e. manually dive into the raw data and systems to understand what is what and automate metrics at a larger scale. Firstly, do not believe ancient data dictionary documents, third-party consultants, or system integrators with underpaid and overworked staff.

I found plenty of cautionary tales. There were innocent columns in tables containing PII data because of a broken process or a user manually entered it in the wrong field. Debug logs that spilt sensitive financial data from secured processes into unsecured areas. Unicode characters break ancient SQL and processing scripts, leading to strange behaviour. Data synchronisation processes missed a few financial records every time they ran. Just small enough to slip through as reconciliation errors for the finance department to deal with but monetary errors nonetheless. Or the supply chain system that dropped records when overloaded and never reconciled them with the source system, mysteriously losing packages.

Secondly, start measuring things. It can be simple, like how many records go in and come out of a pipeline. What defines good for a field, the ratio of good vs bad, and how does it change over time? Look at basic stats, e.g. NULL might be acceptable in your schema, but is it acceptable that all records are NULL? How long do processes take? How are they interconnected? You get the idea.

## "We do not know" is not good enough

The interesting part is that in the examples I mentioned, the risk either was not known or realised, e.g. leaking financial data. Or the error was small enough that people and processes started to work around it, e.g. expect losing X% of shipped packages or Y% of monetary error. People started accepting that these systems and processes are too complex and demanding to fix instead of addressing the issues.

However, processes are deterministic in an organisation. They may not be understood because they are complex or not measured, but they have no magic. Not knowing why things go wrong should be an alarm bell. You may ignore an issue because it is economical to do so but not measuring and understanding it is dangerous.

My version of Ted's advice today is "Companies that do well measure, understand and can trust their data all the way from origin to consumption".

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

There is one simple thing most companies miss about their data. It has been instrumental in my work as a data professional ever since.

Can't see the trees for the wood

One simple thing companies miss about their Data

# Your Data lies: Be a data-driven Luddite

By _Christian Prokopp_ on _2022-06-29_

I have worked with data for decades. There are the two key lessons I share with every customer, stakeholder and beginner in the field. Firstly, follow the data, not (only) your gut or experience. Secondly, do not trust the data; never. Why and what has this to do with Luddites?

![Shadow figure of people working](/content/blog/images/your-data-lies-be-a-datadriven-luddite.jpg "Shadow figure of people working")

## Data validates ideas

The first point is well-examined and self-evident to anyone who has worked with data, so I will not dwell on it. In short, experience and gut are helpful to spark ideas, explore new avenues or unearth causalities, but metrics and hard data validate and steer us to valuable, measurable outcomes. So why should you not trust the data?

## Everybody lies, including data

The problem with data is that it is rarely correct if you deal with data mining or data sourced from external parties, containing user input or legacy and complex systems. They suffer from broken processes, late data, fat finger input, broken Unicode characters, localisation issues, and sometimes simple lies.

For example, if you mine a marketplace like Amazon, you can expect innocent and malicious inputs from users. Which reviews are genuine or bought? Are the GTINs (identifiers) correct or merely copied from another (original?) product, or is it faked to fill a required field?

## Be a Luddite, occasionally

I have countless stories of assumed correct data only to find it untrue. Big data from legacy or complex systems are a common source. I remember building with a team a new analysis on top of a data source only to discover the transaction data contained was partially missing. It was a small mistake in how the batch processes synchronised at night overlooked by inexperienced staff, a silent error.

The true source of the issue here was that the people in charge of the data movement were not the same as those preparing the analysis and again not the same as those using the data. It is also an excellent example in favour of cross-functional agile product development, but that is another topic. Only when we started checking on the data and the results in detail, diving into the data 'manually' slicing it in all possible ways and going up and down the processing chain became the issue and pattern obvious. A small number of transactions got lost and underreported sales; every day!

Another example is the product feed from a retailer to a service company to create a data feed to advertise on a search platform. As part of the work at Bold Data, I dug through the final feed 'manually' and found some issues immediately. The retailer was paying another company to ingest and prepare their data, yet the result was broken in various ways, likely impacting the final advertising performance, a potentially costly error from a paid-for service.

But before judging these companies, think about the number of processes, data sources, feeds, stores and humans involved in your data processes. Can you confidently say that they all have automated metrics and robust checks to validate them technically and logically? Do you have competent staff with the time and incentives to check these data flows to harden them or find issues?

## Get your hands dirty

Here are a few tips to improve your situation. Nowadays, it is tempting with the plethora of tools and services with shiny UIs and automation to _plug and play_, but it is more _plug and pray_ if you can't pop the hood and check yourself. Pay extra to get skilled staff at all levels. People who can drop onto a console or a query engine and dig through logs, databases, and feeds, or who can slice and dice the data in varying ways.

Have clear communication and collaboration between technical and business staff (again, cross-functional teams are fantastic) to ensure data is also logically correct. Most importantly, do not trust data that a data-driven Luddite has not validated. Someone who is not just watching automated tools or high-level metrics, which are essential for smooth operations, but also gets in there and occasionally digs around.

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

I have worked with data for decades. There are the two key lessons I share with every customer, stakeholder and beginner in the field. Firstly, fol...

Shadow figure of people working

I have worked with data for decades. There are the two key lessons I share with every customer, stakeholder and beginner in the field. Firstly, follow the data, not (only) your gut or experience. Secondly, do not trust the data; never. Why and what has this to do with Luddites?

Your Data lies: Be a data-driven Luddite

# Will Tesla's AI break the insurance market?

By _Christian Prokopp_ on _2022-06-20_

Insurance works because it shares costs in the face of uncertainty. What happens when Tesla removes uncertainty and distributes cost seemingly more fairly? First partially and eventually wholly? Will insurance fail, doing more harm than good?

![Car with accident damage](/content/blog/images/will-tesla-ai-break-the-insurance-market.jpg "Car with accident damage")

## Predict accidents better

An [Insurance Journal article on Tesla's insurance program](https://www.insurancejournal.com/news/national/2022/04/22/664161.htm) talks about how Tesla collects a safety score, `"a score that captures how safely the policyholder drives by monitoring forward collision warnings per 1,000 miles, hard braking, aggressive turning, unsafe following time and forced Autopilot disengagement [...]".`In simple terms, Tesla is collecting Data to predict how likely you are to have an accident to adjust your rate accordingly. This is not a new concept. You may have heard of the black boxes some insurances ask policyholders to install for the same purpose. It is seemingly fair. But a leading AI company like Tesla could ultimately take this to another level.

## Getting personal with data

Tesla could improve forecasts with more data and models about location, traffic and weather. Then personalise the predictions further with increasingly advanced machine learning algorithms, enhancing them with cues about emotions from car cameras and your social media. Eventually, creating digital twins, individual models, per customer with a highly accurate forecast on who and when they will have an accident. A bit of an uncomfortable thought, but we are happily sharing our shopping and viewing behaviour today. That seemed invasive and awkward a decade ago, but no more.

## Your digital twin works for the insurance

What will the outcome be? Will your car or insurance deny you its service because you are likely to have an accident in an individual trip or defined period? Or will it rack up your premium by thousands of percentage points in anticipation of a crash?

Imagine monthly adjusting personal car insurance that employs your digital twin and highly accurate forecasting. Say it is offered at $10 for a few months because you have no incidents forecast, and it is a great way to sell the insurance. Nice, right? Next, imagine it jumps to $10,000 in a subsequent month because your insurance predicts an incident with $6,000 to $10,000 costs. Why have insurance at all?

## Predictive insurance makes society worse off?

While this exaggerates the development, it shows a trend towards a problematic outcome. Other insurances like health or home insurance may follow similarly. How would insurance products have to adjust to deal with this? This problem is not theoretical, and the development is underway. Insurance markets that have the flexibility push 'risky' groups and individuals into higher rate plans or exclude them from insurance completely.

You could argue it helps encourage better behaviour, like better driving. But what if you are poor, living in riskier areas, or disabled, having some impairment. Is it fair for you to pay more? How much predictability is desirable for customers to encourage better behaviour and fairer pricing? At what point does it negate insurance's purpose and become a social problem or discrimination?

---

[Christian Prokopp, PhD](https://www.linkedin.com/in/prokopp/), is an experienced data and AI advisor and founder who has worked with Cloud Computing, Data and AI for decades, from hands-on engineering in startups to senior executive positions in global corporations. You can contact him at christian@bolddata.biz for inquiries.

Insurance works because it shares costs in the face of uncertainty. What happens when Tesla removes uncertainty and distributes cost seemingly more...

Car with accident damage

Insurance works because it shares costs in the face of uncertainty. What happens when Tesla removes uncertainty and distributes cost seemingly more fairly? First partially and eventually wholly? Will insurance fail, doing more harm than good?

Blog