
When it comes to architecture, we often think about MVC, microservices, monoliths, etc. Traditionally, a monolith setup would take you very far, take a look at Shopify for example. Most frameworks such as Laravel, Ruby On Rails have been battle tested and have proven itself as tools that will scale. As recommended by Martin Fowler, we should start with monolith and only start decoupling code by services when the complexity increases. This blog explains how I used my software engineering knowledge to maintain a 150k+ LOC Nextjs application.

When setting up Nextjs, you would just run the commands from docs in the terminal and it would setup a full stack application. Other parts of SDLC, such as testing, deployment would be automated using CICD tools such as Github Actions. This would go a long way if you are using the api route for backend or server actions which is code that runs on the server side similar to PHP. For database ORMs usually Prisma or Drizzle is used. Trpc can also be used where you get type safe client out of the box but it always seemed limiting as it requires a lot of common code to be written such fetching data by id, list etc. In Spring Boot for example, you can use the CRUD repository to generate CRUD operations. Testing can be done using Cypress, Playwright or Jest.

The tooling of Nextjs made me think of only using its ecosystem which are Next Auth, server actions, api route, and Prisma/Drizzle. But this made me realize a lot of stuff had to be reinvented, such as authentication with RBAC and also ABAC, database triggers, and a custom CMS. Previously the client was using Directus CMS to manage the database so I recommended that we use it as API and internal management while the Nextjs application will be used for public access and some role specific operations. This cut down a lot of time for development as I had to only worry about database design and the API query logic for the directus API and the database triggers. The app currently has 5 roles. Organizing code based on role is the most time consuming task I have ever done especially when there is column level access control. Thankfully Directus has a library to generate types similar to Supabase. From the base types, I created role specific types for each collection. But this was not enough. Each page had fields specific to a role so I had to create data fetching logic per role and also do checks in the UI to show or hide fields based on role, giving rise to spaghetti code. All the design patterns I had known came to no use in this case. If there were different views per role, then it would be easily separated. But since the views were the same, I had to use a lot of conditional rendering.

Then I discovered Refine.dev, a framework built on top of React. This is an opinionated framework that uses provider pattern to modularize the codebase as well as abstractions for data fetching hooks using Tanstack Query. This was a goldmine for me. With Refine.dev, the auth, access control, data, live and notification providers are easily replaceable. Need to change an Express API client to Firebase? Just add a new provider! If there are too many conditional checks for roles, Refine provides CanAccess Higher Order Component as well as usePermissions and useCan hooks. These can be used to reduce the spaghetti code as all the checks are made in the access control provider, for which I used Casbin. The clear separation of concerns out of the box is what makes Refine.dev powerful.

But with great power comes great responsibility, and in this case, manual typing of responses, the second most time consuming task I have done for this project. I have used GraphQL extensively in the past years and got used to autogenerated typings. But in Refine, due to the data provider being separate, you cannot even pass the client SDK types back to the response in the hook (maybe someday this becomes possible). REST APIs are a pain, but thanks to Directus API, I decided to use the good old typing the JSON response (thanks to Maximiliano Firtman for teaching this on FrontendMasters Flutter course). You may think why I didn't just use the types generated by Directus. There are somewhere around 20-100 fields among each page, this means if I were to reuse the types for each role, I would have to use Omit and Pick just to get the correct schema. I also got this motivation to manually type the data from one of Brian Holt's courses where he mentioned he found it easier to manually maintain the types.

This workflow lasted for a long time and still is in use. I have always preferred to use Test Driven Development in mobile and web apps. But when there is a lot of code to write, I would prefer to focus on the use cases and test that first followed by any core logic not part of the use cases. Snapshot testing can also be kept as an option. In my case, I decided to carry out end to end tests from the beginning with a Directus docker container, covering the use cases only. Another tool I recommend is Storybook. This will force you to think in components instead of business logic, a common mistake of mine. You can also develop the components in isolation without having to run the entire application.


How I architected an enterprise web application using Next.js, Directus and Refine.dev

Architecting enterprise Nextjs web applications with Directus and Refine.dev


It has been a quite a while since I wrote a blog on this site. Recently I have worked on a few projects that changed my view on how software should be developed. Based on my experience in startups, initially the main goal is to release the app as long as it works due to tight deadlines. The project was a fullstack Flutter app with Express backend developed in 2021, where I was tasked with fixing Typescript errors and versioning issues.

The codebase was huge, with a lot of spaghetti code and uncommented code blocks. It was a clear sign of tech debt. But the client's idea was to reuse the same widgets and backend code for the new app, meaning even more tech debt. I tried to find out which parts of the code can be reused by creating 87 use case diagrams, with ChatGPT's help and PlantUML. This helped me understand how the codebase works while also documenting it. Then I looked at the architecture diagram and realised the whole backend is dependent on Azure. All the code was written while only thinking about Azure services and did not follow any good design patterns. The client claimed that the app was very slow, and I told him it was because of Node.js single thread nature while being non-blocking for IO operations. Video files were being uploaded to a single instance of the server and then ffmpeg was used to format it, in the same thread. All of this processing meant the server will be busy handling all the IO operations and then processing the videos which is a CPU intensive task. I knew setting a job queue with Redis or BullMQ would kind of fix this issue, where the processing is done in a background worker and using signed uploads to directly upload to Azure bucket as well as load balancers with multiple instances of the server. Or even a worker thread may work but I didn't have time to figure it out. Then came the loading of the videos in the app. It was downloading one MP4 video at a time as you scroll xD. The way to fix this is to preload videos and using HLS encoding to stream the videos, which meant another service had to be developed. The backend had to be ditched as there was no time to migrate it from Azure to another cloud provider and a lot of code had to be modified (didn't follow Open Closed Principle). 

I was doing my Software Design and Architecture course at that time and tried to see if there were any patterns used from the course such as adapter, factory, builder patterns etc. But there were none except singleton for database connections and this made realise the hard truth. Unopinionated backend frameworks such as Express are more likely to have tech debt in the long run due to its flexibility in allowing developers to make their own decisions when structuring their code and also when they are inexperienced. Laravel, Spring Boot or even Ruby On Rails enforce a certain structure, meaning no matter how experienced the developer is, they will follow the same convensions provided by these opinionated frameworks. It is easier to get started with Express but in the long run, if there were poor initial planning, it will be very difficult to maintain.

The Flutter app had a lot of patterns which helped me practise the design patterns I learned from the course. However the app was three years old with no version upgrades. Upgrading the Dart and Flutter version was a nightmare. A lot of the packages became deprecated and there were no alternatives. This meant the new app had to be rewritten from scratch. So I started recruiting developers and interviewed over ten candidates on Upwork. In the end I selected three, one of whom now works at Google in Silicon Valley. They chose the Clean Architecture with a two weeks deadline, and I got assigned to setup Firebase for backend with Sports API. I setup the database after creating UML diagrams then spent the rest of the days on studies.

The app did not meet all the requirements, and I blame it on Clean Architecture. Its a good way to structure your code but not when you are on a deadline. I think a simpler pattern such as bloc or MVP would have been a lot easier.

When designing projects, we often think of how many design patterns or architecture we can include, focusing on DRY, SOLID and all the other terms you find in books, often leading to overengineered project designs. But in reality, these are guidelines, not laws. It is not always possible to follow them, but thinking about it when writing the code or even passing the terms as prompts can greatly improve your code quality.

My experience of aligning the learnings from my Software Engineering degree with real job requirements.

Aligning learnings from Software Engineering Degree with real job


For the past 4 months, I have been working in a legacy MERN stack project that is a booking system for a sports complex in Malaysia where I developed the website, backend API and an admin panel using AdminJS. It was all functional and allowed rapid iterations for requirements until a new one was added: export data as CSV.

The API and the admin panel are all using CommonJS modules. If you have ever written code using `require()` and `module.exports`, then you have already been using CommonJS. However, this is no longer the standard approach to developing modern Node.js applications and you need to use ESModules. So instead of `require()` and `module.exports`, you need to use `import` and `export` statements. Unfortunately, many npm packages are being migrated to ESModules and this is an issue for legacy projects built with CommonJS. ESModules are backwards compatible with CommonJS but its not the same way around. This has been a big issue in the Node.js ecosystem and it can be quite difficult to migrate large codebases to ESModules.

The admin panel built with AdminJS was in version 6.x which still has support for CommonJS. As I tried to implement the import/export plugin, there was this issue I couldnt resolve at all. I knew this was something to do with the AdminJS version so I upgraded to version 7.x, only to realize that this version dropped support for CommonJS completely. So the only option I had was to use the unmaintained [cjs-to-es6](https://www.npmjs.com/package/cjs-to-es6) package to rewrite all CommonJS code with ESM. Then I saw the bun's announcement: [It got released on Windows!](https://www.youtube.com/watch?v=yXTFOeGly9o).

When I saw the release of bun six months ago, I was not that hyped as I saw a tool similar to this that tried to do pretty much the the same, [Rome](https://github.com/rome/tools), and dissapointed many. But it was different this time. It really is a drop in replacement for Node.js so you can start using it by replacing the npm and node commands in your package.json file. The main feature that captured my interest was the ability to use require and import statements in the same file. This allows you to keep using CommonJS modules and use import statements for any new modules that drop support for it. The only catch I could find so far is that if you decide to mix import and require statements, you cannot use module.exports but instead use export statement. I did exactly that and now I have a fully functional backend with admin panel that won't make your head scratch fighting with CommonJS and ESModules.


How I migrated a legacy Node.js project from CommonJS to ESModules using Bun

Saving legacy Node.js projects with Bun


For the past couple of months, one thing has worried me quite a lot: the C drive was taking up too much space. I have a 240GB SSD and was left with only 20GB of free space. I was wondering what was taking up so much space.

During that time, I was learning Docker by doing Brian Holt's Complete Intro to Containers course on [FrontendMasters](https://frontendmasters.com/courses/complete-intro-containers/) as well as [Part 12](https://fullstackopen.com/en/part12/introduction_to_containers) on Full Stack Open and blamed the Docker images for taking up so much space, as well as WSL. I was wrong. It was the cache of Yarn and NPM.

Just by running the following commands, I was able to free up 27GB of space:

```javascript
yarn cache clean

npm cache clean --f
```

I couldn't believe my eyes when I saw the amount of space I was able to free up. It is now a habit of mine to run these commands every now and then.

As I am focusing on containerizing everything these days, I find it inconvenient to install anything locally. I have begun to appreciate the under-the-hood workings of Docker. To my understanding, the essence of containers consists merely of a few Linux kernel capabilities duct-taped together. It is merely a way to combine a few Linux features to create isolation.


How I freed up 27GB of disk space on my C drive by removing Yarn and NPM cache

Got Yarn and NPM installed? Remove cache!


So I just tried [ChatGPT](https://chat.openai.com/) and I was shocked! It was able to solve all the problems that I gave it, be it a C++ problem or even a full stack [React](https://reactjs.org/) project with [Firebase](https://firebase.google.com). No matter how tough the problems were for me, it was solving it right after I hit enter on my keyboard. Although I saw many being afraid of it taking our jobs in the future, I see it as a way to develop applications way faster than it is humanly possible, currently.

Whenever I face a problem that I cannot solve, I spend about 15-30 minutes to figure it out. If the problem still persists, I go to [Stackoverflow](https://stackoverflow.com/users/14602088/sadman-yasar-sayem) to find out the solution. This is a five step process - 1) Search on browser 2) Click on the stackoverflow link 3) Go through all the possible answers 4) Understand the code 5) Implement it on the codebase.

But imagine this: you can ask an AI system about the problem and it gives you not only a very accurate answer, but also a brief explanation behind it. This is a 2 step process. This is [ChatGPT](https://chat.openai.com/)!

As I was exploring this game changer, I was also learning about [Firebase Authentication integration with NextJS](https://github.com/SadmanYasar/react-auth-learn) by following [developedbyed's](https://www.youtube.com/@developedbyed) tutorial. During that time, I opened up Firebase console, created a new app and copied the credentials on the project. It is a good practice to store the credentials as environment variables. Thus, I began to copy each field on the .env file.

Suddenly, I realised that there must be someone who made a tool to convert Javascript objects to environment variables. Unfortunately, I could not find a working solution. So I knew I had to make one! Let's see how I made it using ChatGPT and Svelte. You can try it [here](https://svelte.dev/repl/187f2c56f22c428fa14c7043906878d3?version=3.55.0)

---

## Initial Setup

To get started, go to Svelte [REPL](https://svelte.dev/repl/hello-world) and name your project. Setup App.svelte as follows:

```html
<script></script>

<main></main>

<style></style>
```

For the application, we need one textarea for pasting the javascript object and another to show the output as .env variables.

Let's now add two textareas in App.svelte:

```js
<script>
    let text = '';
    let output = '';
</script>

<main>
    <textarea
        bind:value={text}
        placeholder='Enter JS object'>
    </textarea>
    <textarea
        bind:value={output}
        placeholder='ENV Output'
        disabled={true}
        readonly={false}>
    </textarea>
    <button>
		Convert
	</button>
</main>

<style>
    main{
		width: 100%;
		display: flex;
		flex-direction: column;
	}
    textarea {
		flex-grow: 1;
		height: 300px;
		padding: 20px;
		margin: 10px;
	}
    button {
		width: 200px;
	}
</style>
```

Here I have added styles for the main as well as for the textareas so that these two are next to each other using <code>flex-direction: column</code>. I also added a button that we will later use to convert the object.

## Using ChatGPT to create the function

Now comes the fun part! On ChatGPT, I asked the following questions:

- algorithm to convert javascript object to .env values
- javascript convert string to object where keys do not have double quotes in string
- regex delete text and everything after it until a character is reached
- how to add double quotes to keys in a string then convert to object in Javascript?
- javascript convert each key in an object to have underscore after each capital letter then convert to uppercase

Using the answers provided by ChatGPT, I was able to come up with a function that did the conversion:

```js
const objectToEnvString = () => {
  try {
    errorMsg = "";
    output = "";
    // adds double quotes to each key
    // removes semicolon
    const str = text
      .trim()
      .replace(/([\{,]\s*)([A-Za-z_$][A-Za-z_$0-9]*)\s*:/g, '$1"$2":')
      .replace(/;/g, "");
    const obj = JSON.parse(str);

    // adds underscore after each capital letter
    // converts each key to uppercase
    const formattedObj = Object.keys(obj).reduce((acc, key) => {
      const modifiedKey = key.replace(/([A-Z])/g, "_$1").toUpperCase();
      acc[modifiedKey] = obj[key];
      return acc;
    }, {});

    for (const [key, value] of Object.entries(formattedObj)) {
      output += `${key}=${
        typeof value === "string" ? value : JSON.stringify(value)
      } \n`;
    }
  } catch (error) {
    errorMsg = "Invalid Object";
  }
};
```

I asked ChatGPT to explain this code. Here is what it said:

- The function starts by defining an empty string called errorMsg and another empty string called output.
- It then uses the trim and replace methods to modify the input string (stored in the variable text). The trim method removes leading and trailing white spaces from the string, and the replace method adds double quotes to each key and removes semicolons from the string.
- The modified string is then parsed into a JavaScript object using the JSON.parse method.
- The object is then transformed into a new object called formattedObj using the Object.keys and reduce methods. The Object.keys method returns an array of the object's keys, and the reduce method iterates over the array and modifies the keys by adding an underscore after each capital letter and converting them to uppercase.
- Finally, the function uses a for-of loop to iterate over the key-value pairs of the formattedObj object and build a string in the format of an environment variable. The string is stored in the output variable and returned by the function.

Well, this explained it all. Without ChatGPT, it would have taken me quite a while to come up with this function.


How I used ChatGPT to create a JS object to .env variables converter in Svelte

Thoughts on ChatGPT/ how to make a JS object to .env variables converter in Svelte


After 3 months of waiting, I finally attended my first lecture at UTM. Even though the classes started at 8am MYT, I had to wake up at 6am due to time zone difference. You may wonder why time zone can be an issue if my university is in Malaysia. Well, the whole semester was in online mode due to the pandemic and travel restrictions. It was not so difficult to adapt, after all, as during my gap year after finishing my ALevels, I was an online math tutor for OLevels.

<br />
As time went by, I got to meet students from many different countries like UK, Egypt,
Yemen, Sudan, Saudi Arabia, China and of course, Malaysia. We worked together in
group projects, prepared slides for presentation, designed an elevator system using
Deeds software for our Digital Logic course. I learned to use Matlab in our Linear
Algebra course. The language is syntactically similar to Python but the best feature
I enjoyed was how easy it is to write matrices and assign it to a variable. For other
languages like C++, it can get cumbersome to use multidimentional arrays as matrices.
The programming course was the easiest for me as I already knew the concepts that
were taught.
<br />
Overall, I will say the experience was great. The professors provided guidance whenever
we got stuck in a problem. The students were very friendly and we helped each other
by doing group discussions to prepare well for examinations.
<br />I got my results at the beginning of this August. With a CGPA of 3.89, I
am now in the Dean's list. I hope I can keep this result in the next semester.


Sadman Yasar Sayem

Blog