Xu Hao makes use of chain of thought and basic data prompting with ChatGPT when writing self-testing code
My account of an inside chat with Xu Hao, the place he exhibits how he
drives ChatGPT to provide helpful self-tested code. His preliminary immediate primes
the LLM with an implementation technique (chain of thought prompting). His
immediate additionally asks for an implementation plan reasonably than code (basic
data prompting). As soon as he has the plan he makes use of it to refine the
implementation and generate helpful sections of code.
Not too long ago I watched an enchanting name on Zoom. Xu Hao, Thoughtworks’s
Head of Know-how in China, defined his explorations in utilizing ChatGPT to
assist construct Self Testing Code. He
went by means of the form of interplay that works for him.
He begins with a immediate that units the context for the applying and the way
he desires the code to be structured.
The present system is a web-based whiteboard system. Tech stack:
typescript, react, redux, konvajs and react-konva. And vitest, react
testing library for mannequin, view mannequin and associated hooks, cypress part
exams for view.
All codes ought to be written within the tech stack talked about above.
Necessities ought to be carried out as react elements within the MVVM
structure sample.
There are 2 sorts
of view mannequin within the system.
-
Shared view mannequin. View mannequin that represents states shared amongst
native and distant customers. -
Native view mannequin. View mannequin that represents states solely relevant
to native consumer
Listed here are the frequent implementation technique:
-
Shared view mannequin is carried out as Redux retailer slice. Examined in
vitest. -
Native view mannequin is carried out as React part props or states(by
useState hook), except for international native view mannequin, which can be
carried out as Redux retailer slice. Examined in vitest. -
Hooks are used as the most important view helpers to retrieve knowledge from shared
view mannequin. For many the case, it is going to use ‘createSelector’ and
‘useSelector’ for memorization. Examined in vitest and react testing
library. -
Don’t dispatch motion immediately to vary the states of shared view
mannequin, use an encapsulated view mannequin interface as a substitute. Within the interface,
every redux motion is mapped to a way. Examined in vitest. -
View is include konva shapes, and carried out as react part through
react-konva. Examined in cypress part exams
Listed here are sure patterns ought to be adopted when implement and check
the part
-
When write check, use
describe
as a substitute ofcheck
-
Knowledge-driven exams are most popular.
-
When check the view part, pretend view mannequin through the view mannequin
interface
Consciousness Layer
Requirement:
Show different customers’ consciousness data(cursor, identify and on-line
data) on the whiteboard.
AC1: Don’t show native consumer
AC2: When distant consumer modifications cursor location, show the change in
animation.
Present an total resolution following the steerage talked about above.
Trace, hold all consciousness data in a Konva layer, and an consciousness
data part to render cursor, and identify. Don’t generate code. Describe
the answer, and breaking the answer down as a job checklist based mostly on the
steerage talked about above. And we are going to refer this job checklist as our grasp
plan.
There’s so much occurring with this immediate, so he highlighted a couple of
factors.
He is utilizing a generic utility instance in right here: one factor to be
cautious of when interacting with ChatGPT and the like is that we should always by no means
put something that could be confidential into the immediate, as that will be a
safety threat. Enterprise guidelines, any code from an actual undertaking – all these should
not enter the interplay with ChatGPT.
Many of the immediate is setting out the design pointers that he desires
ChatGPT’s generated code to comply with. He refined this by placing prompts into
ChatGPT, wanting on the outcome, and crafting the immediate to make it generate
within the fashion he needed. As soon as he has context working, he can paste it into
each session he has with ChatGPT, and share it with the remainder of the
workforce.

Xu Hao begins his session by prompting the
ChatGPT LLM with context in addition to the instruction to generate a plan
Although the ultimate objective of this session is to generate code and its
exams, at this stage he particularly mentioned Don’t generate code
.
At this level he desires ChatGPT to stipulate its strategy earlier than we glance to any
code era. This can be a immediate method referred to as Generated Data.
A standard means of interacting with an LLM is to offer it an issue and ask
it to generate some closing product, whether or not than be prose or (on this case)
code. With Generated Data we break up our interplay into separate steps:
first ask the LLM to generate some helpful details about the issue, and
then we feed that data again into the LLM to generate the ultimate
product.
Probably the most attention-grabbing a part of the immediate is the “implementation technique”
part which mixes two types of immediate:
- Instruction prompting supplies directions for the LLM to comply with
- Chain of Thought prompting encourages the LLM to clarify its
reasoning.
Within the Implementation Technique part, Xu Hao described the specified
structure sample as an anticipated “Chain of Thought directions” for
ChatGPT to comply with. Then he instructed ChatGPT to construct a job checklist (the
generated data) based mostly on this chain of thought.
Resolution:
To implement the attention layer function, we will comply with the rules talked about in the issue assertion and use the prevailing tech stack.
Job Checklist:
-
Create a brand new part named “AwarenessInfo” to render cursor and identify for distant customers.
-
Create a brand new Konva layer to carry all of the “AwarenessInfo” elements.
-
Add the Konva layer to the principle whiteboard part.
-
Create a brand new Redux slice to retailer the attention data for distant customers.
-
Create an encapsulated view mannequin interface for the attention Redux slice.
-
Implement the “AwarenessInfo” part to retrieve and show the attention data for every distant consumer from the view mannequin.
-
Implement a way within the encapsulated view mannequin interface to replace the attention data for every distant consumer.
-
When a distant consumer modifications the cursor location, dispatch the replace technique within the encapsulated view mannequin interface to replace the corresponding consciousness data.
-
Within the “AwarenessInfo” part, use the brand new location to animate the cursor change.
-
Write exams for the brand new “AwarenessInfo” part, together with data-driven exams to make sure it correctly shows the attention data for all distant customers.
-
Write exams for the encapsulated view mannequin interface to make sure it correctly updates the attention data for distant customers.
-
Write end-to-end Cypress exams to make sure the attention layer is correctly displayed and up to date when distant customers change their cursor location.
By following these duties, we will implement the attention layer function within the on-line whiteboard system.
The response from ChatGPT is non-deterministic, so this result’s simply
one run. In case you (or I) put this immediate in once more, we’ll get a unique response.
The important thing aspect of the response is an ordered job checklist, with numbers for
every step. We will use these numbers to refer to those duties for the remainder of
the session.

The LLM responds with a plan.
Xu Hao can now take a look at the grasp plan and see if it is smart. If
there are issues with it, he can ask ChatGPT to re-write a few of the duties
or all the grasp plan by offering extra data.
For instance, the present grasp plan was not grouped by differing types
of elements. One potential immediate is likely to be:
Rewrite the grasp plan, and group duties as mannequin, view mannequin and examine.
This may very well be helpful to confirm the alignment between the structure
imaginative and prescient and grasp plan.
These tweaks may very well be added to the preliminary immediate to get an total
higher response.For this specific immediate, the next instruction was
not within the preliminary immediate, however have been added in later iteration:
4. Don’t dispatch motion immediately to vary the states of shared view mannequin, use an encapsulated view mannequin interface as a substitute. Within the interface, every redux motion is mapped to a way. Examined in vitest.
As soon as the grasp plan appears to be like cheap, Xu Hao asks to regenerate it with
names for this system parts.
Rewrite the grasp plan, this time with detailed part names,
strategies, and props
Positive, here is a revised grasp plan with detailed part names, strategies, and props:
Job Checklist:
-
Create a brand new part named “AwarenessInfo” to render cursor and identify for distant customers. This part will obtain props akin to consumer identify, cursor place, and on-line standing.
-
Create a brand new Konva layer named “AwarenessLayer” to carry all of the “AwarenessInfo” elements.
-
Add the “AwarenessLayer” to the principle whiteboard part.
-
Create a brand new Redux slice named “consciousness” to retailer the attention data for distant customers.
-
Create an encapsulated view mannequin interface named “AwarenessViewModel” for the “consciousness” Redux slice. This interface will include a way named “updateUserAwarenessInfo” to replace the attention data for every distant consumer.
-
Implement the “AwarenessInfo” part to retrieve and show the attention data for every distant consumer from the view mannequin. The part will use the “useSelector” hook to retrieve the related data.
-
Implement the “updateUserAwarenessInfo” technique within the “AwarenessViewModel” interface to dispatch a Redux motion to replace the attention data for every distant consumer. This technique shall be referred to as by the “MainWhiteboard” part.
-
When a distant consumer modifications the cursor location, dispatch the “updateUserAwarenessInfo” technique within the “AwarenessViewModel” interface to replace the corresponding consciousness data.
-
Within the “AwarenessInfo” part, use the brand new location to animate the cursor change. This may be completed utilizing the Konva “Tween” animation.
-
Write exams for the brand new “AwarenessInfo” part, together with data-driven exams to make sure it correctly shows the attention data for all distant customers. The exams shall be written utilizing the React Testing Library.
-
Write exams for the encapsulated view mannequin interface to make sure it correctly updates the attention data for distant customers. These exams will use the Vitest framework.
-
Write end-to-end Cypress exams to make sure the attention layer is correctly displayed and up to date when distant customers change their cursor location. The exams will simulate consumer interactions with the whiteboard and confirm the state of the “AwarenessLayer” and “AwarenessInfo” elements.
At this level, Xu Hao is able to ask ChatGPT to offer code for one among
the steps

present me instance exams and implementation for the attention slice
and AwarenessViewModel. Check first.
Positive, let's begin with writing exams for the attention slice and AwarenessViewModel: Check for Consciousness Slice: describe('consciousness slice', () => { let retailer: MockStore; beforeEach(() => { retailer = configureMockStore()(initialState); }); it('ought to set the cursor location', () => { retailer.dispatch(setCursorLocation({ x: 100, y: 200 })); const state = retailer.getState().consciousness; count on(state.cursorLocation).toEqual({ x: 100, y: 200 }); }); it('ought to set the identify', () => { retailer.dispatch(setName('John Doe')); const state = retailer.getState().consciousness; count on(state.identify).toBe('John Doe'); }); it('ought to set the net standing', () => { retailer.dispatch(setOnline(true)); const state = retailer.getState().consciousness; count on(state.isOnline).toBe(true); }); }); Check for AwarenessViewModel: describe('AwarenessViewModel', () => { let retailer: MockStore ; let awarenessViewModel: AwarenessViewModel; beforeEach(() => { retailer = configureMockStore()(initialState); awarenessViewModel = new AwarenessViewModel(retailer); }); it('ought to set the cursor location', () => { awarenessViewModel.setCursorLocation({ x: 100, y: 200 }); const state = retailer.getState().consciousness; count on(state.cursorLocation).toEqual({ x: 100, y: 200 }); }); it('ought to set the identify', () => { awarenessViewModel.setName('John Doe'); const state = retailer.getState().consciousness; count on(state.identify).toBe('John Doe'); }); it('ought to set the net standing', () => { awarenessViewModel.setOnline(true); const state = retailer.getState().consciousness; count on(state.isOnline).toBe(true); }); }); Now, let's implement the Consciousness slice and AwarenessViewModel: Consciousness slice: interface AwarenessState { cursorLocation: { x: quantity; y: quantity }; identify: string; isOnline: boolean; } const initialState: AwarenessState = { cursorLocation: { x: 0, y: 0 }, identify: '', isOnline: false, }; const awarenessSlice = createSlice({ identify: 'consciousness', initialState, reducers: { setCursorLocation: (state, motion: PayloadAction<{ x: quantity; y: quantity }>) => { state.cursorLocation = motion.payload; }, setName: (state, motion: PayloadAction ) => { state.identify = motion.payload; }, setOnline: (state, motion: PayloadAction ) => { state.isOnline = motion.payload; }, }, }); export const { setCursorLocation, setName, setOnline } = awarenessSlice.actions; export default awarenessSlice.reducer; AwarenessViewModel: class AwarenessViewModel { personal retailer: Retailer ; constructor(retailer: Retailer ) { this.retailer = retailer; } setCursorLocation(cursorLocation: { x: quantity; y: quantity }) { this.retailer.dispatch(setCursorLocation(cursorLocation)); } setName(identify: string) { this.retailer.dispatch(setName(identify)); } setOnline(isOnline: boolean) { this.retailer.dispatch(setOnline(isOnline)); } } Be aware: On this implementation, I assumed that the AwarenessViewModel is used to set the values within the retailer. If additionally it is used to retrieve the values from the retailer, we will add corresponding strategies to the AwarenessViewModel class
(I’ve eliminated the styling from ChatGPT’s response.)

Total this code is neither unhealthy nor excellent. It will not run because it stands.
It is lacking a configureMockStore
and assumes it makes use of
redux-mock-store, which is not a part of the tech stack. Xu Hao can repair these
issues by asking the LLM to progressively tweak the code by means of a
sequence of rewrites. Here is one to get rig of redux-mock-store.
rewrite the exams with out redux-mock-store
Positive, here is an instance implementation of the exams for the AwarenessViewModel and the attention slice with out utilizing redux-mock-store
:
import { createAwarenessSlice, AwarenessState, updateAwareness } from './awarenessSlice';
import { AwarenessViewModel } from './AwarenessViewModel';
describe('AwarenessViewModel', () => {
const mockDispatch = jest.fn();
const initialState: AwarenessState = {
customers: {
'user1': {
identify: 'Consumer 1',
cursorPosition: { x: 0, y: 0 },
on-line: true,
},
'user2': {
identify: 'Consumer 2',
cursorPosition: { x: 100, y: 100 },
on-line: true,
},
},
};
let viewModel: AwarenessViewModel;
beforeEach(() => {
viewModel = new AwarenessViewModel(initialState, mockDispatch);
});
describe('getUserNames', () => {
it('returns the checklist of consumer names', () => {
const names = viewModel.getUserNames();
count on(names).toEqual(['User 1', 'User 2']);
});
});
describe('getCursorPosition', () => {
it('returns the cursor place for a given consumer ID', () => {
const cursorPosition = viewModel.getCursorPosition('user1');
count on(cursorPosition).toEqual({ x: 0, y: 0 });
});
it('returns undefined for an unknown consumer ID', () => {
const cursorPosition = viewModel.getCursorPosition('unknown');
count on(cursorPosition).toBeUndefined();
});
});
describe('updateCursorPosition', () => {
it('dispatches the updateAwareness motion with the brand new cursor place', () => {
viewModel.updateCursorPosition('user1', { x: 50, y: 50 });
count on(mockDispatch).toHaveBeenCalledWith(updateAwareness({
consumer: 'user1',
cursorPosition: { x: 50, y: 50 },
}));
});
});
});
And here is an instance implementation of the awarenessSlice
with out utilizing redux-mock-store
:
import { createSlice, PayloadAction } from '@reduxjs/toolkit';
export sort CursorPosition = { x: quantity, y: quantity };
export sort UserAwareness = {
identify: string;
cursorPosition: CursorPosition;
on-line: boolean;
};
export sort AwarenessState = {
customers: File<string, UserAwareness>;
};
const initialState: AwarenessState = {
customers: {},
};
const awarenessSlice = createSlice({
identify: 'consciousness',
initialState,
reducers: {
updateAwareness: (state, motion: PayloadAction<{ consumer: string, cursorPosition: CursorPosition }>) => {
const { consumer, cursorPosition } = motion.payload;
state.customers[user].cursorPosition = cursorPosition;
},
setOnline: (state, motion: PayloadAction<{ consumer: string, on-line: boolean }>) => {
const { consumer, on-line } = motion.payload;
state.customers[user].on-line = on-line;
},
},
});
export const { updateAwareness, setOnline } = awarenessSlice.actions;
export const awarenessReducer = awarenessSlice.reducer;
Be aware that since we’re not utilizing redux-mock-store
, we have to cross a mock dispatch operate to the view mannequin constructor, and within the check for updateCursorPosition
we have to test that the proper motion was dispatched utilizing jest.fn()
.
Additional rewrites can match in additional changes, akin to asking it to
Use `configureTestingStore(‘consciousness’, initialState)` to initialize
can get it to slot in with the workforce’s check idioms. Then
the check retailer
with some path changes and a small quantity of glue for the reducers, the
code can run simply positive.
As soon as that is working, Xu Hao can repeat the method for the remainder of the
duties within the grasp plan.
One of many issues working with ChatGPT in workout routines like that is the
restricted context window (additionally referred to as “token restrict”). As soon as it receives sufficient
phrases (or extra strictly tokens) within the dialog, it begins forgetting
the earliest materials, an impact that makes it curiously forgetful. That is
not noticeable for brief chats, nevertheless it does matter when working with giant
quantities of context as on this train. ChatGPT has been steadily rising
its context window, GPT-4 has a token restrict of 8,192, with one other variant
elevated to 32,768.
Xu Hao finds the token restrict has three predominant impacts on his work. Firstly
ChatGPT might cease producing content material: that is pretty simple to repair, as we will immediate
it with one thing like “you aren’t completed” or “go on”. Secondly, if the general
immediate is simply too huge, then ChatGPT will merely return a deadly error and we now have
to start out a brand new dialog.
The third is the extra difficult drawback, the place ChatGPT begins forgetting and
dropping context. When this occurs we have to reset the context. That is
the place the strategy of getting a grasp plan and breaking down the duty into
separate parts helps. We will begin a brand new dialog with the unique
technique and the grasp plan, and ask it to generate the code for a unique
merchandise on the plan.
Xu Hao discovered that the chain of thought within the context was essential to
making the generated code match collectively regardless of being generated in separate
periods. Every now and then he wanted so as to add a touch to the immediate to generate
cohesive code, however discovered he received higher outcomes from revising the chain of
thought as a substitute.
My take away from this dialogue was that utilizing chain of thought and
generated data prompting approaches could be a considerably great tool
for programming. Particularly it exhibits that to make use of LLMs properly, we have to
learn to assemble prompts to get the very best outcomes. This expertise
means that it is helpful to work together with the LLM like a junior accomplice,
beginning them with architectural pointers, asking them to indicate their
reasoning, and tweaking their outputs as we go.