Interesting Content in AI, Software, Business, and Tech- 7/26/2023[Updates]

Content to help you keep up with Machine Learning, Deep Learning, Data Science, Software Engineering, Finance, Business, and more

Jul 27, 2023

Hey, it’s Devansh 👋👋

In issues of Updates, I will share interesting content I came across. While the focus will be on AI and Tech, the ideas might range from business, philosophy, ethics, and much more. The goal is to share interesting content with y’all so that you can get a peek behind the scenes into my research process.

If you’d like to support my writing, please consider buying and rating my 1 Dollar Ebook on Amazon or becoming a premium subscriber to my sister publication Tech Made Simple using the button below.

Help me buy chocolate milk

p.s. you can learn more about the paid plan here.

A lot of people reach out to me for reading recommendations. I figured I’d start sharing whatever AI Papers/Publications, interesting books, videos, etc I came across each week. Some will be technical, others not really. I will add whatever content I found really informative (and I remembered throughout the week). These won’t always be the most recent publications- just the ones I’m paying attention to this week. Without further ado, here are interesting readings/viewings for 7/26/2023. If you missed last week’s readings, you can find it here.

Community Spotlight- Emre Safak

I’m a nomad. Emre makes remote work easier for people. Thus, Emre is now my best friend. Remote work has multiple benefits- lower costs, better for the environment, increased employee retention, and more. His company, Archipelago AI, creates the next generation of communication tools for remote work so no one can say “People work better at the office”. You can find their website over here.

If you’re doing interesting work and would like to be featured in the spotlight section, just drop your introduction in the comments/by reaching out to me. There are no rules- you could talk about a paper you’ve written, an interesting project you’ve worked on, some personal challenge you’re working on, your content platform, or anything else you consider important. The goal is to get to know you better, and possibly connect you with interesting people in the community. No costs/obligations attached.

AI Papers/Writeups

Beware the Binary

Michael Woudenberg

Why nuance matters in AI. Need I say more?

We don’t need AI to replace doctors or make diagnoses, nor do we need it to make fully autonomous decisions. What we do need is AI to fill the gaps in our healthcare systems, not widen them. By moving beyond the binary, we can focus on fostering system collaboration — a third dimension where AI and humans can work together.

Building software with a LLM is not what they say

Santiago Valdarrama

Santiago goes over some challenges that you would face when developing Software with LLMs. It’s a fairly approachable look into the topic, while still being insightful.

The future of software development is not natural language (at least, I don’t think it is.) We need an unambiguous way to make computers do what we need. Programming languages are fantastic for this; natural language isn’t. Ask an LLM the same thing in 12 ways, and you’ll get 12 different answers.Even worse: you can’t predict what that answer will be. It’s always a surprise.

If you’re looking for more advanced looks into the subject- I would suggest reading Serious Problems With ChatGPT And Other LLMs That Internet Influencers Won’t Tell You About and The Ethics of GitHub Copilot.

Inflection AI: The Story of the Unforeseen and Secretive Unicorn with a $4 Billion Valuation

Ksenia Se

A look into a secretive AI Unicorn. Ksenia is one of the best in the game when it comes to covering AI businesses, and this was no exception. She’s a hawk when it comes to finding relevant media surrounding figures, and putting the pieces together.

There are rumors, that Inflection AI is working on a cutting-edge supercomputer equipped with 22,000 NVIDIA H100 GPUs that is expected to be one of the largest In the industry, right behind AMD’s Frontier. However, there’s some debate regarding the true computational power of this setup. While Inflection AI claims that these GPUs could provide 22 exaflops in 16-bit precision mode, calculations by The Next Platform suggest that the actual performance might not meet these expectations.

Efficient rematerialization for deep networks

When training complex neural networks, memory usage can be an important bottleneck. The question of when to rematerialize, i.e., to recompute intermediate values rather than retaining them in memory, becomes critical to achieving the best time and space efficiency. In this work we consider the rematerialization problem and devise efficient algorithms that use structural characterizations of computation graphs — treewidth and pathwidth — to obtain provably efficient rematerialization schedules. Our experiments demonstrate the performance of these algorithms on many common deep learning models.

How to improve Neural Networks by exploiting asymmetry

In case you missed it, I covered how we could take some inspiration from our brains and create a new kind of neural network- one that is bilaterally asymmetrical. This asymmetry gives the neural network better feature extraction and higher performance, matching the performance attained by architectures with wayy more parameters. The article also covers how this idea can be taken forward.

REALM: Retrieval-Augmented Language Model Pre-Training

Language model pre-training has been shown to capture a surprising amount of world knowledge, crucial for NLP tasks such as question answering. However, this knowledge is stored implicitly in the parameters of a neural network, requiring ever-larger networks to cover more facts.
To capture knowledge in a more modular and interpretable way, we augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia, used during pre-training, fine-tuning and inference. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner, using masked language modeling as the learning signal and backpropagating through a retrieval step that considers millions of documents.
We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA). We compare against state-of-the-art models for both explicit and implicit knowledge storage on three popular Open-QA benchmarks, and find that we outperform all previous methods by a significant margin (4–16% absolute accuracy), while also providing qualitative benefits such as interpretability and modularity.

Interesting Tech/Business Writeups

Remote employees work longer and harder, studies show

Remote employees are working more, in part, because they are commuting less. Another landmark study, based on data from 27 countries, found that remote workers saved 72 minutes in daily commuting time. On average, employees spent about half an hour of that extra time engaged in daily work: more than two hours a week.

When and how to run group-targeted A/B tests

A fairly good introduction to the topic. Would recommend it if you’re looking for a quick but useful intro/refresher into the topic.

How to Stop Being Rate Limited: Best Practices for Making API Calls At Scale

A great writeup on the topic.

The main factor in your decision is going to be the time frame of the limit. Sleeping a process is a waste of resources: if a rate limit is configured with a time period of several hours or a day, you’ll want to give those resources back to the machine and schedule a time later to try. But another consideration is that “starting from where I left off” will either not be possible for some 3rd Party APIs, or will be more computationally expensive than just sleeping for a few seconds and trying again.