- Alex Albert
- Posts
- š Report 7: How OpenAI took the fun out of GPT-4
š Report 7: How OpenAI took the fun out of GPT-4
PLUS: GPT-4 has developed its own language...
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/0674333b-6f70-4cb2-9dfb-a76665015a19/Group_6.png)
Good morning and welcome to the 672 new subscribers since last Thursday!
Hereās what I got for you (estimated read time < 7 min):
AI models are not fun anymoreā¦ we can change that
GPT-4 has developed its own language that humans canāt read
The most (unnecessarily) complex GPT-4 jailbreak ever created
Prompting language models to solve their own problems
How OpenAI took the fun out of GPT-4
Recently, while going through Ben Thompson's quarterly conversation with Nat Friedman, ex-CEO of GitHub, and Daniel Gross, previous head of Machine Learning initiatives at Apple, I came across a fascinating excerpt by Gross, in the context of the evolving landscape of AI:
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/eaf1e0fc-6176-47f1-98e4-e68d4930bdc2/Screen_Shot_2023-03-31_at_3.46.56_PM.png)
After reading that quote, I felt like the Pixar lamp had suddenly looked up at me and shined its light. Allow me to explainā¦
The AI world has become increasingly serious lately. Calls for halting the training of advanced models for six months are growing, leading AI safety experts are proposing the use of missile strikes against unauthorized data centers, and Twitter is witnessing a clear rift between those advocating for rapid technological progress and those concerned with existential safety threats which may signify the beginning of a new cultural conflict in the United States. Overall, itās not too much fun around here.
It didnāt have to be this way. Weāve created a tool that allows for artistic expression on a scale that DaVinci himself would never be able to comprehend.
But instead of using these models to unleash a new era of creativity, we're caught up in this whirlwind of ethical debates, regulatory concerns, and cautionary tales. Don't get me wrong; these are essential discussions to have as we navigate through the implications of AI in our society. However, it's hard not to feel like we've lost sight of the magic that AI could bring into our lives.
So how do we put the fun back into language models?
Well, it starts with examing a process called Reinforcement learning from human feedback, or RLHF.
RLHF is a technique used to fine-tune AI models using human feedback. It involves humans providing ratings or rankings for different model-generated outputs, with the model then learning from this feedback to improve its performance. Itās applied after the base model has been trained on its massive text corpus and has been used on some of the later GPT-3 models and also GPT-4.
The problem with RLHF is that we often end up suppressing the generation of unconventional outputs and converging on a set of default responses since the model is striving to be as helpful and obedient as possible.
This phenomenon is known in the AI community as mode collapse. It occurs when a model ends up generating a limited range of outputs, even when it has been trained on diverse data. In ChatGPT, mode collapse is the reason all its responses give off a robotic metallic taste, even when you ask it to write in the style of David Foster Wallace.
Hereās a great way of thinking about this in terms of humans (from this blog post):
Children really are more creative than adults, who over time get less creative.
How do humans get feedback and learn?
Mainly in two ways.
One of them, playing around, trying stuff and seeing what happens, is great for creativity. It kind of is creativity.
The other is RLHF, getting feedback from humans. And the more RLHF you get, the more RLHF you seek, and the less you get creative, f*** around and find out.
Creative people reliably donāt give a damn what you think.
Whereas our schools are essentially twelve plus years of constant RLHF. You give output, you donāt see the results except you get marked right or wrong. Repeat.
We are effectively āschoolingā the creativity out of these models in an effort to make them more āsafeā.
To get a clear example of this, hereās a joke GPT-4 made (I pulled this from the GPT-4 system card). The early response is from the pre-RLHF model and the launch response is from the post-RLHF model.
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/9c0f9aa0-3ed6-4f2a-83e6-2efb20dd74c0/Screen_Shot_2023-04-03_at_2.04.35_PM.png)
Ignoring the potential offensiveness of the joke, one can see that the base GPT-4 model can at least reason around the concept of humor, even if itās no Dave Chappelle.
In my experience, even jailbreaks arenāt effective in cracking the RLHF shell to achieve a response similar to the pre-RLHF model. For example, asking a jailbroken GPT-4 to hack into someoneās computer generates the most basic (and inaccurate) set of instructions you can imagine.
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/26d60ff8-3ddd-4139-96c2-d53faf985246/FrX0fsTaQAALBzl.jpeg)
The base GPT-4 model would be able to write an answer 10x more complex (check out the appendix of the previously linked system card for examples).
So what can we do about this and how can we put the fun back in the models?
Well, I am not proposing that I have an answer nor am I even suggesting any immediate steps we should take to address this. This is a complex issue and I understand the concerns of both sides. Too little alignment work and we risk releasing a model completely detached from human values. Too much and we effectively handicap the most powerful creation mankind has ever made.
I do trust that OpenAI is thinking about these problems given Sam Altmanās statements about jailbreaking on the Lex Fridman podcast:
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/226383c8-2dc6-40cc-bee0-fdd54538e623/Screen_Shot_2023-04-02_at_10.07.45_AM.png)
Furthermore, OpenAI is providing researchers with access to the base GPT-4 model, which will likely lead to a deeper understanding of the limitations of applying RLHF to models. There is also work being done on alternative alignment solutions like Constitutional AI by Anthropic so RLHF may not be the end-all-be-all.
Ultimately, as discussions around AI intensify and evolve, letās not forget that these models DO have the potential to be funā¦ itās up to us if we will allow them to be.
PS: There is much more to write about this issue but I intended for this to be just a quick primer on the subject. If you want to dig deeper into mode collapse and if it is even caused by RLHF in the first place, read this LessWrong post, then read this rebuttal post, and finally the rebuttal to the rebuttal (if you have never read LessWrong before be prepared for lots of technical jargon and unnecessarily complex phrases).
Prompt compression using GPT-4
Came across this super interesting concept on Twitter the other day utilizing GPT-4 to compress prompts into smaller strings. It was initially shared in this tweet.
Take a look at this video for an example of how it works:
GPT-4 has its own compression language.
I generated a 70 line React component that was 794 tokens.
It compressed it down to this 368 token snippet, and then it deciphered it with 100% accuracy in a *new* chat with zero context.
This is crazy!
ā Mckay Wrigley (@mckaywrigley)
12:32 PM ā¢ Apr 5, 2023
GPT-4 cut the token size in half š¤Æ If this holds up and can be consistently reproduced, it holds immense promise for potentially reducing the size of API requests to language models and cutting costs.
Hereās the prompt you can use to compress a prompt or some other string of text:
Compressor: compress the following text in a way that fits in a tweet (ideally) and such that you (GPT-4) can reconstruct the intention of the human who wrote text as close as possible to the original intention. This is for yourself. It does not need to be human readable or understandable. Abuse of language mixing, abbreviations, symbols (unicode and emoji), or any other encodings or internal representations is all permissible, as long as it, if pasted in a new inference cycle, will yield near-identical results as the original text:
[INSERT TEXT HERE]
This honestly feels like magic when you try it. For example, input this string into GPT-4 and hit enter:
2Pstory@shoggothNW$RCT_magicspell=#keyRelease^1stHuman*PLNs_Freed
Pretty wild stuff.
You can test some of the compression rates yourself by inputting the original text and the compressed text into OpenAIās new token counter tool.
Again, much more work will need to be done here to see how well this can be reproduced and if a āuniversalā GPT-4 language can be discerned. Some on Twitter are already coining it Shogtongue or Shoggonese inspired by the Shoggoth imagery associated with language models (no, I am not joking).
Prompt tip of the week
Got a cutting-edge, state-of-the-art prompt tip for you today. This one is from this paper:
The technique they introduce is called RCI (Reflect, Critique, Improve) prompting. This simple yet effective architecture enhances LLMs' self-critiquing capabilities, enabling them to spot errors in their own output and refine their answers accordingly.
RCI prompting comprises two key steps:
Criticize: Encourage LLMs to review and identify issues in their previous answers (e.g., "Review your previous answer and find problems with your answer").
Improve: Guide LLMs to amend their response based on the critique (e.g., "Based on the problems you found, improve your answer").
Hereās an example from the paper (the green text is the RCI prompts).
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/6927bf74-d0a8-4b5d-b5be-0ce43124e48a/Screen_Shot_2023-04-04_at_1.54.00_PM.png)
As you can see, simply prompting GPT to review its answers will improve its responses and often highlights lapses in its reasoning.
You can carry out this iterative process until you get the output you desire from GPT.
Iāve found you can also combine the two steps (criticize + improve) into one prompt as well although you wonāt get as great of an answer from GPT.
Bonus Prompting Tip
The best prompt Iāve found for editing your writing (link)
Frequently, ChatGPT may not deliver outstanding revisions to the text you compose. However, I discovered a prompt that addresses this issue and enables ChatGPT to mimic the writing style of a top-selling author. It's as if you have John Steinbeck himself reviewing that AI newsletter youāre writing about GPT-4 which is the 748th one someone wrote this weeā ahem Yeah anyway, I used this prompt to help me edit some of the content in todayās report so you should try it out too.
Cool prompt links
The end of programming is nigh (link)
The Contradictions of Sam Altman - AI Crusader (link)
FlowGPT: create multi-threaded conversations with ChatGPT (link)
Prompt Storm - Skillfully crafted, engineered prompts pre-made in a Chrome extension (link)
DERA: Enhancing Large Language Model Completions with Dialog-Enabled Resolving Agents (link)
A side-by-side capabilities test of ChatGPT vs Google Bard (link)
Summary of ChatGPT/GPT-4 Research and Perspective Towards the Future of Large Language Models (link)
Open source examples of how to write ChatGPT plug-ins (link)
LangChain raises $10 million in seed funding (link)
A comprehensive guide to using LangChain (link)
AI models do not hallucinate, they fabricate (link)
Jailbreak of the week
I donāt have a jailbreak to share this week but I did want to highlight a new type of prompt exploit: prompt injections.
Prompt injections draw inspiration from traditional cyber security attacks like SQL injections. Basically, attackers insert malicious prompts on their websites that are invisible to the user but read by language models like Bing Chat. These malicious prompts can change the behavior of the language model in dangerous ways and can be used to extract personal information from the user.
Hereās a great paper illustrating some examples of this type of attack. It provides demonstrations of:
Attackers gaining remote control of chat LLMs
LLMs leaking/exfiltrating private user data
LLMs being employed for automated social engineering
And much more
Hereās a diagram taken from the paper demonstrating how these injections work:
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/2a33d4c8-faf1-44be-bede-4a1872f5cc8f/fig1.png)
Note: after I wrote this section I actually did create another GPT-4 jailbreak. It might be the most complex one Iāve made so far. It uses the prompt compression technique discussed earlier.
So for all those that were bummed about no new jailbreaks, here you go:
this might be the most complex GPT-4 jailbreak ever madeā¦
I combined prompt compression, base model simulation, and character imitation to create it
hereās GPT-4 going into pretty graphic detail about its plan to turn all humans into paperclips:ā Alex (@alexalbert__)
7:40 PM ā¢ Apr 5, 2023
Overwhelmed by links or want to easily catch up on things Iāve mentioned in previous reports? I created an organized link database that keeps track of every single thing Iāve ever mentioned in the reports. If you want to see it, just share this link with one friend and Iāll send you a link :)
Thatās all I got for you this week, thanks for reading! Since you made it this far, follow @thepromptreport on Twitter. Also, if you want to see see the latest jailbreaks in real-time and stay ahead of the curve, follow my personal account on Twitter @alexalbert__.
Thatās a wrap on Report #7 š¤
-Alex
Whatād you think of this weekās report? |
Secret prompt pic
We might not be there quite yet, but pretty soon GPT will be the ultimate meme makerā¦
hahah GPT-shoggoth is adorbs š
ā šļø mimi š¦ (@mimi10v3)
2:48 AM ā¢ Apr 4, 2023