Nowadays, everybody is talking about GPT-3, San Francisco based OpenAI’s new language model. GPT stands for “generative pre-training transformer” and GPT-3 is the third iteration of this model which has 175 billion parameters — a big jump from its predecessor GPT-2 that has 1.5 billion parameters. Beta users are exploring use-cases to understand GPT-3’s capabilities and many use-cases have already gone viral on social media.
Duygu Oktem Clark, our Venture Partner and founder of DO Venture Partners spoke with Yigit Ihlamur, cofounder and General Partner of Vela Partners about GPT-3 including its effects on our lives and the worries about harmful biases.
Duygu Oktem Clark: I know that you specialized in Machine Learning during your graduate study in 2009. How do you see the evolution of AI and machine learning since then?
Yigit Ihlamur: When I started my studies in 2009, academia was mainly concentrated on math-focused algorithms. Towards the end of 2010, access to storage and compute-power improved the performance of data-intensive algorithms. Academia started experimenting with those algorithms, specifically neural networks, and testing them against benchmarks such as object recognition accuracy in images. In 2012, neural networks outperformed every other algorithm in object recognition. This success motivated more people to experiment with 30+ years old neural network algorithms and iterate on them. Machine learning experts realized that more data and compute-power improve accuracy without changing the fundamentals of neural networks. Hence, the machine learning expertise moved towards data science, statistics, and custom chip design making data consumption and computation faster and easier.
Over time, neural networks became more sophisticated. Easy-to-use developer libraries emerged. More models were trained with large datasets with cheap compute power. Thanks to the concept of ‘Transfer Learning’, experts started to build on top of other experts’ models. This spiraled an exponential growth and high collaboration within the community.
Around 2016, neural networks were able to detect objects in images better than humans. This was similar to the time when computers were able to do math better than humans. Since this revolution, computers can see as good as humans, if not better in some use-cases. Unlike people, computers can scale and never get tired.
Around that time, I got my hands on a computer vision algorithm. I was blown away as I was able to train my algorithm to detect objects in less than 2 hours. This innovation caused large industries such as automotive, robotics, security, and manufacturing to evolve rapidly.
While computer vision reached an important milestone, other popular applications of machine learning such as speech recognition, text-to-speech, and natural language understanding were also making significant progress. However, they did not reach the same level of human-level accuracy as computer vision algorithms until recently.
In 2018, a new generation of neural networks, called Transformers, emerged and triggered the next growth area for machine learning, specifically in natural language understanding. GPT-3 (Generative Pretrained Transformers) is one of the iterations that helped the developer community to see the same thing they saw with computer vision. Now, machines are close to surpassing human-level accuracy in some areas of natural language understanding. For example, early results indicate that GPT-3 writes news articles and performs better in the SAT.
The next five years will be the same as what we have seen in computer vision. Academia and technologists will iterate on GPT-3, mix and match with other algorithms, and build innovative applications to help computers understand and generate text better than humans in various areas.
Duygu: GPT-3 has been a very popular topic since it was released in June. Beta users have been publishing about their experiences. How has been your experience with GPT-3 so far?
Yigit: The experience is no different than when one gets her first PC, connects to the internet, accesses her email from a beach on a mobile phone or social networks spotting her face on images.
We’re currently in the era of pattern matching thanks to the immense production of data by billions of devices and people. At Vela, we have a variety of in-house machine learning algorithms to facilitate sourcing and evaluating startups. One of our algorithms helps us extract entities such as companies from news articles. We built this algorithm in a week as we have significant in-house deep learning expertise. Thanks to GPT-3, we built the same entity extraction algorithm much faster.
Duygu: What is the most surprising thing about GPT-3 for you?
Yigit: Any developer will have the capacity to extract and find information independently and efficiently in the near future. Any developer will soon have a search power similar to Google’s.
Duygu: It seems like use-cases are endless with GPT-3. What are the use-cases that you have seen that you find interesting?
Yigit: Search and auto-complete is coming to coding, which will speed things up exponentially.
Due to our interest in the market, we extended Vela Partners’ AI algorithm to categorize tweets using GPT-3. You can see all use-cases compiled in this sheet.
Duygu: We have seen use-cases that lead us to think about the future of coding. As a Computer Engineer and investor, I’m excited to explore how GPT-3 will impact programming. Do you think GPT-3 will replace the coders?
Yigit: No, the work that most engineers don’t want to do will be replaced. When I copy and paste code from Stackoverflow to build an authentication flow for Google accounts, I always question why I am doing this in 2020. Millions of people do the same thing.
What will happen is that 90% of the boring tasks will be done very fast. However, like any engineer, who reads this, knows that the devil is in the details and in the last 10%.
Soon engineers will focus on the real details and do more creative work. As a result of this trend, general software will be further commoditized and subfields will continue to emerge.
This trend of developers concentrating on more value-added work started about a decade ago. The number of low-code and no-code products have increased significantly in recent years. Their software production capability has also improved. We expect GPT-3 to accelerate this trend.
We, as Vela Partners, have invested in low-code startups and this field is an important component of ‘maker tools’ investment thesis.
Duygu: Some people including Jeremo Pesenti, Head of AI at Facebook, and Prof. Anima Anandkumar, a Bren Professor at Caltech, raised concerns about GPT-3 due to harmful biases. What do you think about this? Do you think this issue is solvable?
Yigit: I’m deeply worried about this for our society and for my family as a father of a daughter.
Machine learning is all about data. If you put garbage in, you’ll get garbage out. If you put biased data in, then algorithms will make biased decisions.
I am not sure if the bias overall is solvable, but obvious biases that we can clearly articulate are solvable.
I have never met a person in my life that is not biased. And I am not surprised about that. Our brains are pattern matchers like computers. We’re biologically coded to avoid danger. Our biases exist because we’re afraid of the unknown. Our brains must make quick decisions to offload energy through biases, and spare the rest for compute-intensive tasks. For more detailed thoughts on this subject, I suggest reading Daniel Kahneman’s famous ‘Thinking, Fast and Slow’.
That being said, since we haven’t solved the bias among humans, how can we solve that with algorithms? Whose values are right? Isn’t ethics subjective and change from community to community?
On the other hand, some common biases of our communities are clearly solvable. I still can’t believe that we’re living in a world where some people think some races should be incentivized more than others or women are not given the same chances as men. This is unacceptable. What we can do is to solve these obvious problems and use algorithms as augmentation tools for humans to overcome these biases.
Academia, innovative companies, entrepreneurs, and venture capitalists are constantly thinking about this problem. The key question is how governments can scale these policies to the whole society. We need proper processes, laws, and regulations to address this. This is yesterday’s problem and we are already late to solve it.
Duygu: Have you tried to use GPT-3 in other languages? Maybe in Turkish?
Yigit: I have tried. It works, but in many cases, the quality is not as good as English yet. Since GPT-3 doesn’t have any semantic logic behind the algorithm, it depends on which web content of other languages OpenAI feeds. Apparently, the OpenAI team feeds data that they can easily find through open-source and their own crawling algorithms.
The right step for fellow non-English speaking engineers is to contribute to the open-source data crawling projects that academia is using. One such example is http://commoncrawl.org.
Duygu: At this point, it is obvious that every area of work and life will be affected by the latest generation of AI. In your opinion which areas will be transformed primarily?
Yigit: Technology touches all aspects of our lives. And, we’ve been in this transition for many years since the introduction of the PC. This is similar to the industrialization process that took many decades and changed how people lived.
Things are just going at a much more rapid pace now. The art is in the sequence. Selling pet food online in 1998 was a good idea, but consumers were not ready to make that shift. However, starting with selling books online worked really well. The value was much clearer. It unleashed the benefit of buying long-tail books, which were not available in physical stores before.
The key principle is not only to provide efficiency but also to provide access to opportunities that were not economically feasible before. Constructing a framework around this idea would help us think more methodically. For example, let’s reflect on history. Many people in many countries live a better life now than a king or a sultan two centuries ago thanks to the efficiency and accessibility to education, healthcare, products, and services provided or facilitated by innovation and technology. How can we make products and services more affordable and accessible thanks to the power of personalization through data and machine learning algorithms?
Products and services will be cheaper, better, and faster. Any enterprise product that costs millions of dollars will be cheaper. Any personal service that costs hundreds of dollars per hour will be more affordable. These products and services have been expensive because they require human experts to do customization and fit them for each customer. For example, wealthy individuals and large companies have accountants, doctors, and lawyers, and the rest of the population and firms are under-served and over-pay for these services. Thanks to machine learning and data, I expect that most people will have a personalized lawyer, doctor, and accountant in the coming years.
Duygu: You are investing in AI startups. Has your investment thesis changed due to GPT-3? What kind of AI startups do you expect to see in the mid to long term?
Yigit: It made our investment thesis stronger. GPT-3 (natural language understanding, adjacent technologies, and its future iterations) is now a sub-thesis of our AI-focus.
I expect that we’ll interact with strangers a lot less and live outside the cities more. There will be more robots, self-driving vehicles, new living and work environments, no-touch physical experiences, and personalized assistants.
AI is changing how we find, filter, and process information. The most important disruption will be to information aggregators such as Google, Facebook, and Twitter. Millions of developers will have the search power of Google.
Duygu: Thank you Yigit! It was a pleasure to talk to you about GPT-3.
Yigit: Pleasure is mine. Thank you for having me.
— —
We thank Duygu and Yigit for this interesting and enlightening conversation. As MaxiTech, we are keeping a close eye on GPT-3 and looking forward to exploring its capabilities to transform our lives (hopefully) for the better :)
