I am a hybrid, I used to remember most of the linux commands and GTA Cheatcodes but I don't have them byheart anymore

Here is why

Introducing askcmd

askcmd is a command line tool which takes natural language as input and outputs commands specific to your OS.

1. npm i askcmd -g

2. Go to Groq.com and get an API KEY

3. askcmd -s <APIKEY>

4. Start getting commands for anything like askcmd show me the last 5 commitids with the message

5. Forget Linux Commands like you have forgotten routes after GPS arrived.

It's open source. Would appreciate it if you star it.

I had a similar requirement a few months back for an LMS product. I had to publish reports of students at the end of the term. (A specific date and time)

When I googled how to schedule date-based jobs,

I found this package called node-schedule.

(Why not just use event bridge? - My Answer)

Node Schedule was super easy to use, You just pass it a name, callback and the date on which the callback function should be executed.

But it had 2 major and 1 minor problem

• Jobs weren't persistent (If I restarted my express app, the jobs were gone)

• Jobs weren't Atomic - If the same job was scheduled on multiple machines, then it would execute them on every machine leading to duplicate data, so you could only vertically scale your machines where jobs are running. Also cancelling the job on every machine was not possible.

• No Inbuilt Retry Mechanism, if a job failed, I had to catch it and then reschedule the job.

To fix these things, I had previously built a class on top of node-schedule in my code base that made sure the jobs were atomic and had the rescheduling ability.

A similar requirement popped up in a different project recently and I decided to bundle this into a new package and that's how building job-stash began.

Introducing Job Stash - Alternative to Node Schedule and Agenda

Job Stash is an npm package to schedule tasks which takes a callback function and the date on which the function has to be executed.

• Persists task metadata in DB - Reschedule all the jobs using this on your app restart.

• Abstracts DB Implementation - Give it a Mongo DB Address(URI) and it forms a new connection. Give it a MongoDBClient and it reuses the connection.

• It's Atomic - If you have multiple machines running, only one of them will execute your callback function.

• Has an Inbuilt Retry Mechanism - if something goes wrong while executing the job, they are retried after a retry window(customisable).

• Unpacked size less than 30KB.

For persistence storage I am using MongoDB here since our app already used MongoDB and the idea was I could reuse the same connection.

Tutorial on How to Schedule Jobs in Node.js using Job Stash

Step 1 - Install it on your project

npm i job-stash

Step 2- Initialise the Scheduler

The scheduler has to be initialised with db credentials and other options

I am here reusing the db connection from Mongoose and setting useLock to true.

Making useLock to true ensures that if you are running a service on multiple machines, the job is executed only in one of them.

import { Scheduler } from 'job-stash';

await Scheduler.init({ mongo: mongoose.connection.db }, { useLock: true });

/* 
If you don't already have a mongodb connection running
use the below snippet.

await Scheduler.init(
  {
    db: {
      address: mongoConnectionString,
      collection: 'jobs',
      name: 'evaluation_service',
    },
  },
  { useLock: true }
);
*/

Let's talk about the callback function that decides what to do when the job is executed.

I have a common function that decides which function to call / what to do based on a field called operation.

Example:

If my operation is equal to publishReport I want to call a publishReport().

async function decideCallback(job) {
  const { jobId, operation, activityType } = job;
  if (operation === "publishReport") {
    await publishReport(jobId);
  }
  else if (operation === "gradeZero" && activityType === "quiz") {
    await gradeZero(jobId);
    await createQuizReport(jobId);
  }
  else if (operation === "gradeZero") {
    await gradeZero(jobId);
  }
}

If you notice, the decideCallback function has access to a job object which includes jobId and other metadata that you store.

The Scheduler binds the job data inside any callback you pass to it when scheduling or rescheduling jobs.

Step 3 - Reschedule Jobs From Disk to Memory on App Start

I will now use the decideCallback function to reschedule all my tasks on the restart of my express app after initialising the scheduler class.

await Scheduler.init({ mongo: mongoose.connection.db }, { useLock: true });
await Scheduler.rescheduleJobs(decideCallback)

the Scheduler.rescheduleJobs will internally fetch all the active jobs and will pass all the job data inside the decideCallback function.

You can use the same decideCallback function to schedule a new job too.

How to Schedule A New Job

It's a little weird to cover rescheduling before scheduling a new job, but if you look at how you use this package in your app, You initialise it, you Reschedule all the current jobs from Disk to Memory and then based on the API Request / Event you schedule a new job.

Here is how you schedule a new job

// decideCallback has access to jobId, dateToRunOn and metadata if any.
// the Scheduler binds this data when scheduling or rescheduling jobs.
async function decideCallback(job) {
  const { jobId, dateToRunOn, operation } = job;
  if (operation === "publishReport") {
    await publishReport(jobId, dateToRunOn);
  }
}

async function scheduleJob(jobId, dateToRun, metadata) {
  await Scheduler.scheduleJob(decideCallback, dateToRunOn, jobId, metadata);
}

scheduleJob(
  jobId: '123',
  dateToRunOn: '2023-11-13T07:44:06.191Z',
  { operation: 'publishReport' },
);

When the callback function is executed(In our case decideCallback), it calls the function with the jobId, dateToRunOn and all the other metadata that means the jobId, dateToRunOn and all the other metadata are accessible inside decideCallback through the first argument.

The Scheduler.scheduleJob function expects a callback, dateToRunOn, jobId and metadata.

metadata and jobId are optional

1. metadata - This is any information you want to save in the database or use to choose which function or API to call when the job runs.

   In our case, I am storing a field called operation that will help me decide which function to call when the job is executed.

2. jobId - If you don't pass a jobId then the package creates a unique UUID.

   The Scheduler.scheduleJob returns a job object with a method called get getJobId

   You can use that to store a jobId that can be further used to cancel a job.

   Here is how you cancel a job

await Scheduler.cancelJob(jobId);

How to Update a Job

Maybe you want to change the date of execution or add some new metadata to a job.

There is an inbuilt method that lets you update a job.

It accepts the same things as a Scheduler.scheduleJob but the only difference is that it cancels the current running job and reschedules a new one.

await Scheduler.updateJob(jobId, callback, dateToRunOn, metadata)

Why not use a Managed Service like Event Bridge?

It's not like I don't want to use Event Bridge. In fact, I use it for a couple of cron-style APIs that have to be run on specific intervals like recalculating analytics for all the links every 10 minutes for tapthe.link etc.

I also agree that managed services are great.

However, the use case for scheduling reports was a little different in my case.

Here are my requirements

• The Reports of students are usually published by the end of the term - meaning long-lived jobs.

• Wanted to be accurate within 5 minutes of max delay.

• I won't have more than 5000 active jobs at a given time

I just didn't want to poll my DB every 2-5 minutes through a cron expression because I know that there won't be any new job the majority of the time.

The job should be like an event.

an event occurred -> timestamp was hit -> Job has to be executed.

That's why I liked the idea of using a date-based scheduler instead of polling the DB. There is an internal clock in the app that knows when to execute a job.

Current Limitations of Job Stash

• It requires your tech stack to be MongoDB. If you don't use MongoDB as your primary database, it doesn't make sense to introduce another DB just to support a scheduler. If you plan to make your scheduler service separate then it's still okay to have mongoDB. (Even the Free plan is more than enough)

• Doesn't Support Cron Job yet - It's not that hard enabling this, I just haven't worked on this yet.

  For now, all my cron-related things just work on Event Bridge. There is a hacky way of making interval-based things work in the current setup too, you just have to schedule a new job whenever the current job is executed but yeah , I haven't built it. will build it soon or you can build and contribute too.

Why does Job Stash even exist?

While there exists a plethora of packages and ways to schedule jobs. The Hackernews Community loves using systemd but that involves a lot of boilerplate code and research.

it's just pain.

Job Stash exists for developers who have a Node.js codebase with MongoDB as their database and want to get a scheduler up and running within 5 minutes.

Job Stash exists for developers

1. Who have a Node.js codebase

2. Who have MongoDB as their database

3. Who like their machines to be scaled horizontally and want only one of them to execute that job

4. Who wants easy rescheduling of jobs on restart

5. Who schedules a medium scale of 50K-100K jobs

6. Who wants to schedule a job on a specific date and time instead of polling the database every X minutes.

Do let me know what you think in the comment section.

Shout out to Prajwal who reviewed the code and motivated me into building this.

I remembered things I contributed last year.

made them into points and started expanding on how anyone can implement/learn that.

Then realised the blog post is getting long :)

Anyways, Hey everyone, I am Hrithwik, I am a Software Developer From Bangalore working at an Indian EdTech Company.

I started my journey as a Software Developer professionally last year and these are some of the things I learnt last year.

If you are someone who is going to work with MongoDB and Javascript, This blog post can help you learn a thing or two, just skim through the Table of Contents and choose any heading and you will learn something.

Learnt to use Debugger in VSCode

For a guy who used to write a lot of console logs

Learning to use a debugger, was the first coolest thing I learnt.

Initially, I had set up ts-node and had done a bunch of things to set up a debugger in every project I worked on VSCode.

Then discovered that there is an easier way to run a debugger in VSCode.

That's it :)

Working with Typescript

I was a developer who started coding backend with python(flask) and then got introduced to Node.js with javascript

While I thought this is good, I then discovered Typescript.

Typescript is javascript but with types, it's pretty easy to get started with the basics, but once you start writing code, you realise how much more is there to learn.

Here is what I did

• Learnt some basics like data types and interfaces to contribute to the codebase in my first week.

• Started using some advanced concepts like Generics and keyof in the coming weeks

• Then got busy with shipping more instead of wasting time with types. (haha jk. I actively use TS and explore different open source repos like excalidraw, zod etc to get better)

The amount of run time errors typescript can save you from is enormous.

But do note that once the codebase get's huge, your type server starts acting weird, You might not get type recommendations immediately, basically, the IDE experience becomes slower, I manually restart the server most of the time to get the type get reflected immediately

GitHub 2 Branch Strategy

Well, I knew how to use GitHub and some basic commands, but learnt a new thing called GitHub 2 branch strategy.

Initially, when I joined my first company, the approach was like this, I used to create a branch from develop and push all my changes to develop. Then every X weeks once, all the code from develop used to go to main.

Life was simple when we did this, but this has a major flaw. At one point your master branch will have all the code which is in develop and some code which is in develop is not usually stable.

So our Architect came up with a new strategy which we follow to raise PRs.

The idea is simple, we have 2 Branches, Develop and Main.

Main has verified non-buggy code.

Develop has new features which are yet to be tested fully.

You take a branch from Main and create a Feature branch, Push all the code to your feature branch and raise a PR to develop. If it has no merge conflicts, good :)

If it does have merge conflicts, you don't take a pull from develop to your feature branch but

• You create a branch from develop

• Name it like dev/{featureBranchName}

• You merge your feature branch to the new develop branch

• You will get merge conflicts, resolve them and raise a PR to develop and delete the source branch(dev/{featureBranchName})

Then test the feature thoroughly in develop and then raise a PR to main, If you get merge conflicts, take a pull from main and resolve them.

The idea is simple, Your feature branch should only have the code which you have written and the code which is in main.

Atomic Updates using Transactions in Mongo

I will have an entire section on MongoDB Learnings but wanted to share about this before I started sharing other things

When I was working on personal projects, I had never thought about transactions or Atomic Updates. I had just read the theory in my engineering and never thought about it in the NoSQL context.

Although update and findAndModify Queries are Document Level Atomic, they by default aren't collection level atomic.

What's an Atomic Update ?

Let's say you have an API which is changing the data in two different collections, If the update in one collection fails, the other one should not be stored as well.

For Atomic Updates to happen through different collections we have to use transactions.

How do we make this work in Mongoose?

const cls = require('cls-hooked');
const mongoose = require('mongoose');

const ns = cls.createNamespace('session'); // any name works

async function initialiseTransaction() {
  const session = await mongoose.startSession();
  session.startTransaction();
  ns.set('MONGOOSE_SESSION', session);
  return session;
}

The initialiseTransaction Function will create a session object. You can run this while starting the app and use something like cls-hooked , which is local-storage for nodejs to share this session object to any part of your app and have several util functions like this

function getSession(){
    const session = ns.get('MONGOOSE_SESSION');
    return session;
}

async function commitTransaction() {
  const session = ns.get('MONGOOSE_SESSION');
  try {
    await session.commitTransaction();
    console.log('Transaction successful!');
  } catch (err) {
    console.log(err);
  }
}

async function abortTransaction() {
  const session = ns.get('MONGOOSE_SESSION');
  try {
    await session.abortTransaction();
  } catch (err) {
    console.log(err);
  }
}

here is how you use transactions in your queries

const result1 = await Collection1.findOneAndUpdate(filter, updateData, {
  session: getSession(),
});
// updates collection 1
const result2 = await Collection2.findOneAndUpdate(filter, updateData2, {
  session: getSession(),
});
// updates something in collection 2
await Collection3.deleteMany(filter).session(getSession());
// deletes something in collection 3
await commitTransaction();
/* commits all the values to the DB if it reaches here, if there is an 
error , you can call the abortTransaction function in your global
error handler 
*/

For every query you use a .session() or pass it as an option and then at last you commit the data.

Although transactions might make the API response a little slower than usual, having a transaction manager for all your DB calls guarantees that your writes to different collections are Atomic.

Folks at MongoDB always recommend you to keep related data at one collection since at a document level, things are atomic by default, but what happens most of the times is , you don't have a choice but to keep your data at different collection, that's when transactions can help you

Built a scheduler feature which can schedule tasks on a particular DateTime

There was this requirement in the product which I am building where we had to publish a report for students on a particular day.

This can be anywhere from 3 weeks - 4 months.

The simplest approach I thought initially was to create an API which polls the DB every minute using a CRON expression. This API could have been called via AWS EventBridge.

But for my use case,

• The Reports are usually published after 1 month or in the semester's end (5th month)

• We won't have more than 300 JOBS per batch(every 6 months)

so calling the API every minute doesn't make sense.

Although there are many ways of building a Scheduler,

My idea was simple, why not schedule a different job for every task(report-publish) instead of polling the DB every min.

I was searching for something which can just plug into the express server and something which is not complex.

I started looking for packages which support date-based scheduling and found node-schedule. This can run a task on a specific date.

It's this easy to schedule a job

const schedule = require('node-schedule');
const date = new Date();
date.setMinutes(date.getMinutes() + 2);
// runs after 2 minutes after this script is exucuted

const job = schedule.scheduleJob(date, function(){
  console.log('The world is going to end today.');
});

This might not fit everyone's requirements, let me list down some limitations which will help you make a decision

• Limitation 1: If the express app stops running or restarts, all the tasks scheduled will be flushed. (can be solved)

• Limitation 2: You can't upscale the service horizontally, because there may be a case where the same task can be scheduled in two different systems. (probably can be solved)

• Limitation 3: There is always a limit on how many jobs you can set up. If you plan to have millions of jobs, this might not be ideal.

Let's address how these limitations weren’t a problem for me

Fix Limitation 1: Store the Tasks scheduled in a collection

Whenever you schedule a job, store the information required in a collection.

In your express app, there is a place called app.listen which accepts a callback function which is executed when your server is ready to handle requests. Add a function which will get all the jobs from

app.listen(port, async () => {
    /* reschedules all the jobs which are in the DB 
    when restarting the express app */
    await Scheduler.reScheduleAllJobs();
    console.log(`Listening: http://localhost:${port}`)
})

Fix Limitation 2: Scaling Issues

If you have multiple instances running for the same service(codebase) then you will have the same tasks spawned in all the services.

This might be a problem especially if you are dealing with sending a notification to users. You don't want to send multiple notifications to your users for the same action.

Here are some possible fixes.

Fix 1: Handle the multiple calls scenario in your code. if 2 requests are calling to publish the report for the user at the same time, you will publish the report only if it was not published before. Your code should use some kind of lock mechanism,

In MongoDB, the update queries like updateOne and findAndModify use a lock, what that means is, if two requests are trying to update the same resource, only one can update at a time.

so you can use an updateOne query with an if a condition like, if isReportPublished is false then make it true. updateOne returns an object with a property called nModified , if this value is 0 that means the document was not updated. So we send notification and other related logic only if the nModified!=0.

Fix 2: If your code can't handle multiple jobs calling the same API at the same time for the same task, then the simple fix is to have only one job running for a particular task.

Don't scale the Service Horizontally but do it vertically. - hahaha

The chances of your service having downtime may be high, but since we restart the jobs, it should be alright (if you are okay with your jobs running a little later than expected due to downtime).

In this package node-schedule when scheduling the jobs, if the current date is greater than the scheduled date, it won't schedule the task, but in our case here, we need it to run immediately after restarting the service, if it failed. So you have to handle this and retry mechanisms, error notifications etc in your code.

You can have a separate service which just schedules jobs and runs them.

Limitation 3: There is always a cap on the number of jobs you can run

I want to share some numbers, depending on your requirements and scale, you can decide.

As I mentioned earlier, our app will not have more than 2000-3000 active jobs that are scheduled. Since we are a B2B EdTech SAAS, we usually are pre-aware of the scale.

This is mostly not the case with most of the SAAS products. But I ran some benchmarks.

Here is what I tested on a t3a.medium (AMD EPYC 7000 - 4GB RAM 2Core) EC2 Machine

90K Jobs to be executed at a different time in the future and 10K Jobs to be executed at the same date and time.

The RAM consumption went from 60MB to 140MB and the CPU Utilisation was at 95% at a point since I scheduled all the 90K jobs at once.

All the 10K Jobs ran within a minute. (Running 10K jobs at once took 49 seconds ).

This is 10X more jobs than my requirement, so I am good.

Working with Microservices

Microservices are a way of building applications as a set of small, independent services. Each microservice has a specific function and can communicate with other microservices through REST APIs to perform tasks and accomplish the overall goals of the application.

For example, an app like Uber will have a different Microservice for finding a driver, handling payments, storing and managing reviews, user-related data etc.

I won't go into why to not use them or when to use them, it depends on the team size and various other things.

There are a few things I learnt that I want to share

• How do services communicate with others efficiently?

• Can we maintain Atomicity while working with Microservices ?

• Some things I didn't like

How do MicroServices Communicate with each other Efficiently?

While there are different ways of inter-service communication with different protocols like GRPC, Event Driven, Thrift etc., we followed the REST HTTP approach.(Behind the scenes uses HTTP2).

Simple Answer:

They communicate with API calls that doesn't have to go through the internet. That's it

You expose an endpoint and the consumer services call the API.

Importance of having a Mapping of ServiceNames, PortNumbers and BaseURL

While working with micro-services, there must be some code/package which handles the inter-service calls easily for you.

It should have a mapping between all the service names, with their port number.

For the local communication, the port and other things can be from the mapping but for other environments, we can take the necessary values from the env variables/ AWS secrets manager.

You need to call this function to get the appropriate URL. Something like the getUrl function below.

function getServiceUrl(serviceName: string, endPoint: string) {
    // if we want only one of the services to point to local
  const localUrlExists = process.env[`${serviceName}_URL`];
   if(localUrlExists?.length){
        return localUrlExists;
   }
  const isLocal = process.env.isLocal;
  if (isLocal) {
    const port = getPortNumber(serviceName); // stored mapping
    const baseUrl = constructBaseUrl(serviceName, isLocal);
    return `http://localhost:${port}/${baseUrl}/endPoint`;
  } else {
    const domain = process.env.domain;
    const baseUrl = constructBaseUrl(serviceName, isLocal);
    return `https://${domain}/${baseUrl}/endPoint`;
  }
}

function constructBaseUrl(serviceName: string) {
  // specific to your org.
  const version = process.env.version;
  const name = getServiceBaseName(serviceName); // a mapping
  return `api/${version}/name`;
}

function getPortNumber(serviceName:string) {
  const portMap = {
    userService: 4045,
    mapService: 4046,
  };
  if (!portMap[serviceName]) {
    // throw Error
  }
  return portMap[serviceName];
}

The idea is, Each services can be on the same domain while they have a different path.

At our company we have a separate internal package which holds this mapping information.

Imaginary Example:

https://uber.com/api/v2/userservice - points to userservice

https://uber.com/api/v2/reviewservice - points to review service

Isn't it Slow ?

Isn't it slower, if we are making API calls over the internet for interservice communication?

Yup. But we won't make the calls through the internet from a public DNS.

We create a private DNS entry for the domain to the Internal Load Balancer which routes the traffic internally as if two services are talking to each other while they are running locally.

so the service when making an API call checks

• if there is any private DNS record for the domain

• if yes, it routes the traffic to an internal load balancer

• internal load balancer decides what instance will handle the request (based on rules like round robin etc) and get the resource directly from that instance without calling it through the internet

If you have not configured a Private DNS and the Internal Load balancer, then the traffic will go through the normal Internet(slow).

Overall Overview for setting up a new service

• You have to Create a Mapping in the internal package.

• Update the package in every other service, or in the service where the new service will be used, to begin with

• Create a New Codebase with the mentioned BaseURL before the api starts ("/api/serviceName" ) and instal required packages.

• Do the necessary Infra changes like adding the baseURL mapping to the deployed service.(Don't have that deep understanding about this)

• If this is the first service, then you have to setup an external Load balancer, Internal Load balancer, Private, Public DNS and maybe some other steps which I might have missed.

Now imagine, every time you have a new environment like Staging or UAT, you have to do this all over again.

This is why I was saying if you are a small team, Just keep it monolith until you hire someone who takes care of these things. (probably a nice place to plug Qovery)

I have personally never touched the infra side of things but my architect was sweet enough to show me how these things were setup when I asked questions around this.

Few things which are annoying in Microservices

Let's say your PM want's a new feature like showing some extra data in a couple of pages.

For them it's just showing simple data in 2 pages, but for you,

you might have to modify 3-4 services,

get your code reviewed for all the services ,

make the necessary frontend changes

and get it merged to all the required branches. Oh dear this video is accurate 👇🏻

Also, Did I forget about test cases?

I am a big believer in unit testcases. It might not be helpful when you are developing the code but oh boi when you are refactoring the code and want to check if all the API's are working fine, this is a huge saver.

But another pain is, you have to write testcases independently and mock the API response. If there is a change in the API response, you have to go in and change the testcases in all the other servies which are consuming it.

Should we ensure Atomicity in Microservices ?

Sure yeah, why not.

If you are updating something to 3 different services and one of them failed to respond, you ideally should not make the changes in all the remaining services. But this is hard and slow.

I think the term is called Distributed Transactions. Well in my current workplace we haven't implemented something like this because it adds more overhead, plus might add a single point of failure.

So No, We don't maintain Atomoicity through services but we maintain Service Level Atomoicity using MongoDB and hope for the best.

await saveDataInCourse({ endDate }); // interservice call
await Quiz.updateOne({ quizId:"adsa"}, { $set: { endDate } }, { session: getSession() }); // updates the value in the service DB with sessions
await commitTransaction(); // commits the transaction when the function is called

Something like this where we are making an interservice call to save the quizData and then storing the quizData in the DB.

If the inter-service call fails, our Quiz Data doesn't save because the code won't reach commitTransaction where the actual commit happens to the DB - Good

if our inter-service call works but the QuizData.updateOne fails for some reason - Cry

I have not faced this issue though, But yeah before having any interservice logic, we can make sure our data which needs to be updated is validated in the code level and the DB mostly won't fail is what you can hope for. (especially if it's a managed service like atlas). Let me know, how you will handle this.

Working with MongoDB

MongoDB was the second NOSQL Database I tried after Firestore. It is similar to firestore but has so much more to offer like complex aggregations and compound indexes. (Both of which I have not worked on with firestore)

I have learnt a bunch of things while working on mongoDB from over an year.

Schema Design tips

Let's start with basics, The idea of designing Schema's is simple

Store data in a way you want to query data.

While SQL recommends you store Data in a Normalized Form (Keeping Data in seprate Tables without Redundancy ) . MongoDB let's you be free, keep all the data at one place if it's queried that way, no one will ask.

Let's say you want to keep all the grade information of a student with his attempt information.

{
    userId: "123",
    attempts: [ { attemptId: "456", attemptName: "sda" }]
    userName: "Hsb"
}

Indexes - The ESR Rule

A MongoDB index is like an index in a library, but for a database. It helps the computer find things in the database more quickly. Instead of going through the entire library to find your book, you know where your book is located through an index.

This is the only thing I knew when I started working with MongoDB. I had created a bunch of primary indexes for a couple of collections, I knew they were fast and consumed memory to store them.

But at some use cases at my work, just querying with one field was not enough, there were multiple fields in the query.

let's say you want to query all the PRO Users from Bangalore.

db.collection.find({ membershipType: "PRO", location: "Bangalore"})

Assuming we have added an index for membershipType, this will be fast but mongo has to still filter out the records for location:"Bangalore" without an index. That means if there are 10,000 users with 500 Pro users and 10 Pro users from bangalore, the Index will definitely help us limit the data to 500 records but it has to still filter out and get you just 10 users in memory.

That's why we create a compound index like

{ membershipType:1, location:1 }. Mongo will return just 10 users using the index.

Now if you want to get the latest Pro users from Bangalore and assuming you are having a field called createdAt, then you should create an compound index with

{ membershipType: 1, location: 1, createdAt: 1 }

db.users.find({ membershipType: "PRO", location: "Bangalore"}).sort({ createdAt: -1 })

Now if you want to get the latest Pro users from Bangalore who have spent more than 150K in total, then your query will be like

db.users.find({ 
membershipType: "PRO",
location: "Bangalore", 
amountSpent: { $gt: 150000 }
}).sort({ createdAt: -1 })

If you are thinking to add an index like { membershipType: 1, location: 1, amountSpent: 1, createdAt: 1 }

you might not get the optimal results. The right way to do this is like this

{ membershipType: 1, location: 1, createdAt: 1, amountSpent: 1 }

Yes, the order matters. It's based on a Rule called ESR.

Equality , Sort , Range.

membershipType: "PRO" is the equality match

.sort({ createdAt: -1 } is the sort

amountSpent: { $gt: 150000 } is the range operation

Our compound indexes should follow this rule to get data in the most efficient way.

Don't just add compound indexes everywhere, if you know that with a single index or a compound index with less fields is going to filter data with less records then you don't have to create index for all the terms for the query.

Adding Index to Array of Elements

MongoDB suggests you to have related data in one place.

Imagine that there is a quiz which can have a bunch of questions with this schema.

    {
  quizId: string,
  questions: [{ questionId: string, marks: number, title: string }],
  .......
};

Now if you want to see, in how many quizes the same question is used ? You can write a query like

{ "questions.questionId: "123" } // searching for the questionId 123

This will scan the entire document if it’s not indexed. Here is how you can create an index for this using mongoose

schema.index({ "questions.questionId": 1 });

This was a total new discovery for me.

(Also, the above was just an example to show array of elements index, instead of querying it with an index, I would just store and increment the count in the questions collection whenever it’s used in an quiz)

The concept of having correct indexes has helped me improve the performance of multiple data fetching API by effectively indexing the values.

Aggregation

Aggregation is a powerful framework in mongo which can help you get complex computed data right from the database.

You won't need this unless you have to show some complex data which comes different places like in a dashboard or need to format data in a particular way. You write a series of pipelines for operations and it finally works.

While skimming through the codebase in the first week of my joining,

I was weirded out about what's happening seeing some crazy aggregation pipelines. I was scared of it to be honest.

After few months, I had to use aggregation for a feature and discovered this playlist on Aggregation from Codedamn and this totally gave me the confidence/motivation to get started.

Then started exploring it more and started playing around with it using the MongoDB Compass Aggregation Tab, This is such a good thing in MongoDB Compass. You can visualise what each pipleine is doing

I will share a step by step guide into getting started with aggregation in my other post, coming soon :)

Built a Notion to Blog using the Official Notion API

I have a side hustle called tapthe.link which can redirect people directly to the youtube app instead of youtube.com while sharing links on Tiktok.

I wanted to explore writing some articles for boosting SEO and also wanted to learn something new.

Why Notion ?

I spend a lot of my time on notion every day, so something like notion to Blog is what I was looking into.

So 10 months back, I built app.tapthe.link/blog , where I just update something in my notion page and it's reflected the site within 5 minutes.

Features of the Blog

• Custom CTA at the end of each blog which comes from notion, if enabled

• Custom OG Images compatible for all social media platforms

• sitemap and robots.xml

• Table of Contents (straight out copied the UI from hashnode because of how good it is, I hope they don't mind 🙏🏻 )

The implementation used notion's Official API

-> The API returns data in JSON blocks

-> That get's converted into Markdown using Notion-to-md

-> Uses different Rehype plugins to format to HTML

-> Uses Custom Code to generate TOC

-> Get styled with tailwind typography

-> Blog Ready.

If you want to build your own notion to blog, check this repository out. You can add other custom features on top of this.

Why didn't I just use Rehype TOC ?

Well, I did that in the first, it just took me to 3 minutes to add a TOC. I was mind blown by how easy it is to add a TOC. But the UI sucked

In my mind, I wanted a TOC just like hashnode and I coudn't figure out how to customise Rehype TOC to make it look and work like that.

That's why I added my own custom logic which will parse the markdown, form the structure I need to build a TOC and then uses getBoundingClientRect to get the height after which the the TOC Dropdown has to be closed. It's not as perfect as hashnode, but I didn't want to spend a lot of time perfecting it since I had already spent a good 4-5 days on making this work.

Shoutout to my friend rahul who is there to listen to me share everything I build lol.

Improved Performances of API's

While working on a particular feature on top of an existing module I started noticing that the API was really slow and there was something really bad happening there.

I didn't really do some black magic and made things faster, just did some basic optimisations and brought down the API response times from 3-6 seconds to 200ms in one of the API. I have already written about it here in my previous blog post.

It was basically , Right Indexing + Reducing Time Complexity + Reducing Bandwidth by projecting only required fields + Concurrent GET Calls wherever possible.

Learnt about Pre Commit Hooks

We use swagger to document our API. It basically allows you to document your API's by writing some lengthy ass structure like

wait, let me show you.

Here is an API to fetch users from a particular location.

  /**
   *
   * @swagger
   * /api/userservice/users/?location={location}:
   *  get:
   *    summary: Fetch Users from a location
   *    tags: [Users]
   *    parameters:
   *      - in: query
   *        name: location
   *        schema:
   *          type: string
   *        example: false
   *        required: true
   *        description: location of the user
   *    responses:
   *      200:
   *        description: Successfully retrieved the user
   *        content:
   *          application/json
   */

This is how much code you have to add for a simple user fetching API from a location. This is used in most companies.

Now if you make any indentation mistakes or miss a semi column , the code build won't happen.

What used to happen is sometimes is ,

I used to make some silly mistakes like missing a semi column while writing the swagger doc and before pushing the code , I was not generating the docs everytime to verify if my documentation code was proper. The code used to get to develop and the deployment used to fail.

Why ? - There was a semi column missing haha

There is something called husky which will let you add some commands to be run before you commit your code to git. I added this command

npm run generate-docs
// will genereate docs if the syntax is proper, if not will throw error.

Has saved me so much of trouble.

Started Writing Unit Test Cases

Here is what Unit test cases are,

you as a developer will write some positive and negative test cases of your API.

Something like, if your API is expected to send a "pong" message when you send it "ping" and you receive the same message while running the API, the test was successful.

Some other testcase can be like this, If someone sends pingg instead of ping, your API will throw error like "Unkown Input" with 400 status code. If you get the same error message while sending the same input then your test was succeeded.

Initial Reaction

Oh boi, I felt like I am wasting a lot of time on this, what can go wrong ? In 10 days sprint , I need to develop features + also write testcases for the expectations which is already in my mind, what a waste of time.

Then came a refactor. Oh boi, unit test cases saved so much time. What do I mean ?

Unit testcases makes your code more maintainable for the future,

Let's say, If some other developer touches something and breaks it in the code to fix a bug which has impact on a lot of API's. When running the tests, all the API's which have dependency will fail. You realise you have made a mistake .

Downside: On every feature improvement, you have to not only develop new features but also change the testcases in all dependant APIs.

How to do it ?

You can write unit testcases with the help of Two Libraries, Mocha and Chai.

Here is an example

import { expect } from "chai";
import "mocha";

const httpClient = axios.create({
  baseURL: 'https://my-api.com/api/' // your baseurl
});

describe("Create a Bookmark for a Content", async () => {
  const courseId = Utils.generateUuid();
  const contentId= Utils.generateUuid();


 before(async () =>{
    await createContent(courseId, contentId);
  });

it('throws an error if course is not provided', async () => {
  await verifyAsyncError(
    `400 - "courseOfferingId may not be null, undefined or blank"`,
    async () => {
      const result = await httpClient.post(
        '/bookmarks/',
        {
          elementId: elementId,
        },
        {
          headers: {
            authorization: jwttoken,
          },
        }
      );
    }
  );
});

it('sucessfully creates a bookmark for content ', async () => {
  const result = await httpClient.post(
    '/bookmarks/',
    {
      courseOfferingId: courseOfferingId,
      elementId: elementId,
    },
    {
      headers: {
        authorization: jwttoken,
      },
    }
  );
  expect(_.isEqual(elementId, result.elementId));
  expect(_.isEqual(true, result.isBookmarked));
});

after(async () => {
  await httpClient.delete('/bookmarks/clear-depencencies', {
    headers: {
      authorization: jwttoken,
    },
  });
});

function verifyAsyncError(expectedErrMsg, callback) {
  let errMsg = '';
  try {
    await callback();
  } catch (err) {
    errMsg = err.message;
  }
  let substringMatch = errMsg.indexOf(expectedErrMsg) !== -1;
  if (!substringMatch) {
    console.error('Expected:', expectedErrMsg);
    console.error('Actual:', errMsg);
  }
  chai_1.expect(substringMatch).to.be.true;
}

There are Three Things you need to know.

• Describe - Describes a Feature you want to test

• before/after - What has to be done before and after running the testcases

• it - It's a function which accepts a description and a callback function to be executed.

But do note that,

If running testcases is not in your pre-deployment procedure and your developers are not updating their testcases on every feature improvement, then it's as good not keeping them in the codebase.

Data Patches in Production

So I don't know if I mentioned earlier, but I work at an EdTech company and we offer our LMS to some B2B Clients. Now sometimes few of these customers want to get their old data migrated to our LMS.

Something like

"We have these students who have taken the quiz on google form, and we want this to be imported to the LMS".

Or

"We have received the submission offline on google drive and want that to be imported to the LMS"

So a script should get this data from their excel sheet, convert it to the format your LMS expects, add all the dependencies like metadata etc and finally make it work.

This was a super fun and challenging task. Loved it

End

Now that I look into the past while writing this article, I have learnt a lot of new things, it's just the beginning, my stomach is not filled, I am hungry for more.

What's for 2023 ?

• I want to build more useful tools like tapthe.link

• I want to explore React Native and want to go deep into reading research papers and engineering blogs :)

If there is something here which you don't agree or is wrong, you can let me know, I am still a newbie and it's still day one for me.

I just did 3 basic things which improved performance in most of the API routes.

This blog is really helpful especially if you are new to Javascript so let's get right into the solution.

Do note that our REST APIs are built using Node.js+ typescript+mongoDB and this guide might use a few concepts that are applicable only to this.

Using Promise.all()

When reviewing old code, I could see something like this

const user = await getUser(userId);
const quizData = await getScore(activityId, userId);
const attemptData = await getAttemptData(activityId, userId);

This is just an example, not the actual code from work

Here is what's happening,

• We wait till the user data is fetched and then
• We wait for quizData to be fetched
• and then fetch the attemptData.

Now if you notice quizData is not dependant on user data and attemptData is not dependant on quizData, then why are we waiting until the previous promise is resolved?

We can concurrently fetch non-dependable values and literally 2X API performance.

Here is how you do it

const [user, quizData, attemptData] = await Promise.all([ 
getUser(userId), 
getScore(activityId, userId), 
getAttemptData(activityId, userId)
]);

Let's say your API was taking 700ms, this brings it down close to 300 - 350ms.

Reducing DB Calls

One of the other optimisations you can do is to reduce DB calls as much as possible. Let me explain

Let's say you have 5 userIds and you want to fetch user details for it.

Here is the bad way, You are getting user details one by one

const userIds = ['123', '345', '442', '567', '998'];

async function getUserDetails(userIds) {
const userDetails = [];
  for (const userId of userIds) {
    const userDetail = await fetchUserDetails(userId);
    userDetails.push(userDetail);
  }
  return userDetails;
}

const users = await getUserDetails(userIds);

This code becomes slower and slower when the user list grows.

Here we are making 5 different Inter-service API/DB calls to get 5 different resources.

Ideally, you shouldn't do this.

Here is how you reduce this to just one DB call

You can just do this

async function getUserDetails(userIds) {
  const userDetails = await fetchUserDetails(userIds);
  return userDetails;
}

As you can see there is just one function call where we are awaiting and the fetchUserDetails(userIds) should get all resources in one DB call.

For example in MongoDB here is what the implementation of this function will look like

async function fetchUserDetails(userIds){
const userDetails  = await User.find(userId: { $in: userIds  });
return userDetails;
}

In mongoDB you can use $in operator which can query against an array of values.

Avoiding cursor.skip in Pagination

You can skip this part if you don't use mongoDB.

Pagination is a concept used to divide the data into pages and send only required data one by one to save bandwidth and to have faster response times.

I still remember how my friend Shithij explained pagination in one line with an example of P**n Hub 3 years back. "The 1,2,3 button you use, till you get the right video"

We can implement Pagination in MongoDB using two methods called skip and limit

    // Page 1
    await User.find().limit(5)

    // Page 2
    await User.find().skip(5).limit(5)

    // Page 3
    await User.find().skip(10).limit(5)

cursor.limit is used to indicate the size of the document and cursor.skip is used to skip to the next set of documents.

To get Page 2, we say something like give me 5 documents after skipping 5 documents

But the problem with this in Mongo is that cursor.skip has to traverse through all the data until it finds the right one.

So Let's say there are 100K Records and you want the last page, mongo has to go through all documents from the first to skip through documents. There is no way around this if you are implementing Specific Page Pagination (Basic Pagination).

But for my project, we have a dashboard where a teacher can view all comments from students serially with Infinite Scrolling.

It's basically like viewing YouTube comment section

If you also want to implement something like this (serial pagination/ infinite scrolling) then you don't actually need cursor.skip

We can implement Key-Set Pagination and Optimise Speed

Source

How does this work?

In Key-Set Pagination, we use the _id property which Mongo Stores to retrieve the next set of records. Something like

     // Page 1
    await User.find().limit(5)

    // Page 2
    last_id = ...  # get last_id from the client
    await User.find({'_id': {'$gt': last_id}}).limit(5)

    // Page 3
    last_id = ... # get last_id from the client
    await User.find({'_id': {'$gt': last_id}}).limit(5)

The query to get Page 2 is something like "Get me 5 Documents whose _id is greater than last_id"

the reason why this works is that _id is stored in a sorted order based on timestamps. Here is a screenshot from the Mongo Doc

Basically, If you insert Two Documents into a Collection, The first Record's _id is less than the second _id

_id is indexed by default by MongoDB and even $gt and $lt comparisons can be done in O(1) [ Fast]

and your client can store the latest last_id and send it in the request body.

Extra Tip: If you want to get the latest documents while having serial pagination you can do something like this

   // Page 1
    await User.find().limit(5).sort({ _id: -1 })

    // Page 2
    last_id = ...  # get last_id from the client
    await User.find({'_id': {'$lt': last_id}}).sort({ _id: -1 }).limit(5)

    // Page 3
    last_id = ... # get last_id from the client
    await User.find({'_id': {'$lt': last_id}}).sort({ _id: -1 }).limit(5)

Miscellaneous Tips

Keep all Related Data Together

As simple as it looks, this is sometimes missed. You need to design the schema based on how and what you want to query so that you can query all data in one go.

So Plan properly yo

Try Avoiding For Loop Inside For Loop

If you are using a for-loop inside a for-loop(nested loop), there is usually a way to reduce it to one for-loop.

For Example, Let's say you want to send user information with game score which comes from a different source. You might do something like this

function getUserDetailsWithScore() {
  const userDetails = [
    { userId: '123', name: 'X' },
    { userId: '345', name: 'Y' },
  ];

  const gameResults = [
    { userId: '123', score: 50 },
    { userId: '345', score: 20 },
  ];

  for (const user of userDetails) {
    for (const result of gameResults) {
      if (result.userId === user.userId) {
        user.score = result.score;
      }
    }
  }
  return userDetails;
}

console.log(getUserDetailsWithScore());

If you write something like this which has 2 for-loops or you use Array.prototype.find() inside the for-loop, this is a bad practice and your time complexity becomes O(n^2).

Here is how I would do it.

function getUserDetailsWithScore() {
  const userDetails = [
    { userId: '123', name: 'X' },
    { userId: '345', name: 'Y' },
  ];

  const gameResults = [
    { userId: '123', score: 50 },
    { userId: '345', score: 20 },
  ];

  const userScoreMap = {};

  gameResults.forEach((result) => {
    userScoreMap[result.userId] = result.score;
  });

  for (const user of userDetails) {
    user.score = userScoreMap[user.userId] ? userScoreMap[user.userId] : 0;
  }
  return userDetails;
}

const userDetails = getUserDetailsWithScore();

We can find one common value in both collections and then can keep that as a key and create an ObjectMap, so our Key, in this case, becomes userId and we store score like this userScoreMap[userId] = score.

Do you know how much faster the second approach is?

If you run the same code but with userDetails and gameResults having 100K records, the code with a nested loop takes 27 seconds to run and the latter one takes 18 milli seconds. You get the idea, Try to avoid nested loops

Conclusion

These are a few basic optimisations you can do in your project and improve response times and bandwidth.

I hope you learnt something new and if you did go ahead and click on all the reaction emojis hashnode gives you.

Also If you have some other tips, do put them in the comment sections-

Twitter had like a different/modern Installation UI for its PWA. (Know more about PWAs here.)

I was like, Woah, this looks so dope and better than that boring installation screen, which most of the PWAs have, This is also similar to Playstore like UI with an image carousel

If you didn't know, here is how most of the PWA Installation Screen Look

Now I wanted this installation UI for my PWA too, so started digging and Initially, I thought this might be some custom implementation of the Installation UI but then, I visited https://twitter.com/manifest.json.

and I noticed a new property called screenshots in the manifest JSON file, and the src property had the same images which we saw in the installation screen.

Turns out, there is an option to turn this new Installation UI by just adding a bunch of screenshots of your app.

I just copied the same width and height of screenshots from twitter and created some images on Canva

you can follow any height and width which falls under this critera.

Now, time to add the screenshots property with images we created into the manifest.json

 "screenshots": [
    {
      "src": "https://abs.twimg.com/responsive-web/client-web/twitter-lite-data-saver-marketing.68059867.png",
      "sizes": "586x1041",
      "type": "image/png"
    },
    {
      "src": "https://abs.twimg.com/responsive-web/client-web/twitter-lite-explore-marketing.fd45b027.png",
      "sizes": "586x1041",
      "type": "image/png"
    },
    {
      "src": "https://abs.twimg.com/responsive-web/client-web/twitter-lite-timeline-marketing.befcdb47.png",
      "sizes": "586x1041",
      "type": "image/png"
    }
  ],

Replace the src and sizes according to your image location and size. The Image can be either a CDN link or can be in your public folder.

Final Result

Yay, Even my PWA has the same UI now.

That's it folks, I hope you update your PWA with this new UI and share this with your friends.

Also if you know any creators who make YouTube videos, tell them to checkout tapthe.link

For example, this article you are reading, is on blog.hrithwik.me which is essentially a Reverse proxy / Rewrite of hrithwik.hashnode.dev.

So you get the idea, this can be super useful for building SAAS with custom domains, Serving your blog from subdomain to subdirectory, or even serving content from a no-code website builder to your own domain for *FREE.

Well, there are a plethora of other things you can do with Cloudflare workers as a Reverse Proxy, but here are a few things this article covers.

(if you have worked with next.js, this is similar to next.js rewrites)

Things covered in this article

• Getting Started with Cloudflare Workers
• Simple Reverse Proxy/ Rewrites Example with Cloudflare workers
• How to Block Few words from your Articles in a few regions with Cloudflare Workers

What is Cloudflare Workers and Why use it?

For starters, Cloudflare Workers is a serverless architecture that can run JavaScript code on the edge.

That means you don't need to manage the infrastructure and scale it based on the traffic, also means your code is distributed at over 117+ countries and runs in the nearest location of the request with almost zero delay/cold start and is super quick (max 100ms).

It uses V8 Engine under the hood to run JavaScript but not a complete node.js environment so not all npm packages might work. But for most of our use case, this is more than enough.

Also, Cloudflare is super generous with its FREE Plan, which gives us 100K requests/day 😯

So in short, It's lightweight, fast and free to get started with.

Things you will need

• Basic knowledge of JavaScript and NPM modules.

• A Cloudflare Account ,

• Wrangler CLI, which makes our life easy while working with Cloudflare workers. Here is how you can install it globally.

npm i @cloudflare/wrangler -g

Getting Started with Cloudflare Workers

Ensure you have installed the Wrangler CLI

1. • Run wrangler generate <project-name> in your command prompt.

(This will create a bunch of files in the directory and the most important ones are wrangler.toml, index.js and package.json to be honest.)

1. Run wrangler login (Accept the consent after logging in and you are done.)
2. Run wrangler dev , which will start the local server and when you visit it you must get "Hello worker!" Response

Now Let's get to the meat of the topic

Cloudflare Workers as a Reverse Proxy / Doing Rewrites using Cloudflare Workers

Rewrites as the name suggests, allow us to proxy a website to our domain, for example,

Let's say I want content from this website something.site on my personal domain something.com,

I can write some small piece of code, which will fetch content from something.siteand then show that on something.com whenever someone visits something.com

here is how you can make this work.

paste the below code in your index.js and replace example.com with the domain you want to proxy / fetch content from.

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  const url = new URL(request.url);
  url.hostname = 'example.com';
  const data = await fetch(url.toString(),request);
  return data;
}

addEventListener listens to every request and passes it to our handleRequest Function. Here we are creating a new URL object and replacing its host with example.com. Now we fetch the data from the site(This is very similar to the fetch browser API ) with the original request headers etc and return the data.

I will be using my domain richurl.tech instead of example.com

and yay, it's working 🎉

Note: If you observe the network tab, on every reload you are making 35-50 requests, which means that you will exhaust your Free plan if you get 3000 -2000 requests per day.

I still feel the FREE plan is more than enough for most of your needs but if you want more you can look at the pricing here

How to Block Few words from your Articles in a few regions with Cloudflare Workers

If you want to create a cuss free mirror of a website, here is how you do it.

I will be using this website as an example https://whatthefuck.is/memoization . (Great Post by the way)

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  const url = new URL(request.url);
  url.hostname = 'whatthefuck.is';
  const data = await fetch(url.toString(),request);
  // Read response body.
  let text = await data.text();
  // Modify it.
  const cussWordRegex = /fuck/g;
  let modified = text.replace(cussWordRegex , "fork");
  // create a new response and return it
  return new Response(modified, {
    status: data.status,
    statusText: data.statusText,
    headers: data.headers
  })
}

the first three lines of code inside the handleRequest function remain the same, that is we are fetching content from another host website, but instead of straight out returning the response, we modify a few things and create a new response and send it.

(We get the HTML text data and replace cuss words with something else and send the response.)

Works like a charm. Here is a playground if you want to play around with this

for some reason if you decide not to to show cuss words only for certain regions like India or Korea (just example lmao) , here is how you do it

addEventListener('fetch', event => {
  event.respondWith(handleRequest(event.request))
})

async function handleRequest(request) {
  const url = new URL(request.url);
  url.hostname = 'whatthefuck.is';
  const data = await fetch(url.toString(),request);

  const country = request.cf.country;
  const cussFreeCountries = ['IN', 'KP'];

  if(cussFreeCountries.includes(country)){
    let text = await data.text();
    const cussWordRegex = /fuck/g;
    let modified = text.replace(cussWordRegex , "fork");
    return new Response(modified, {
      status: data.status,
      statusText: data.statusText,
      headers: data.headers
    })
  }

  return data;

}

Cloudflare workers also allow us to identify countries using request.cf.country, so we can probably keep an array of not allowed countries and replace cuss words if the request is coming from those countries otherwise just return the same original data.

So you get it, we are kind of intercepting data by fetching data from the original host and then modifying a few things and sending the response. You can even add styles and some other js code too if you want some additional functionality in your proxied site.

Let's Publish it

Run this command wrangler publish and this will publish your code to a subdomain of Cloudflare,

Here is how you add Cloudflare workers to your custom domain.

Step 1 - Add a route for your workers after selecting the domain in the dashboard

Here I am using my service paytm-tap (It's the project name), but you can choose the name of your project

Step 2 - Point your domain to a random IP address in Cloudflare

and it's done, your domain now runs Cloudflare workers 🎉

That's it folks, a small introduction to Cloudflare worker as a reverse proxy/rewrites.

There are several other use cases like using blog.domain.com as a reverse proxy at domain.com/blog etc which I will cover in the future if time permits.

I think if you play around with workers a little bit, you will figure out this by yourself too.

I do some random shitposting on Twitter, so let's stay in touch there.

If you have doubts or struck somewhere while using Cloudflare workers there are always people who help you here on the Cloudflare Discord

But the problem with Whatsapp Newsletter is, new subscribers can't access the old content.

So, I decided to create a simple project where you can access all the articles/resources which I have shared with the group, segregated into different sections.

Building a curation website like this is super easy.

By following this tutorial you will create your own curation website and will learn about creating different endpoints in the backend to access data on HarperDB.

For the backend I was planning on using an Express server and deploying it to a Digital Ocean instance.

Luckily, we don't need to spend extra cash on running another instance somewhere else.

Because HarperDB recently announced Custom Functions which lets you write endpoints to access and process data. It runs a Fastify server behind the scenes and can be deployed in the same instance as your DB. So I will be using this as the backend for our website.

Things to Do 📚

Here are few things we need to setup to create our own curation web app

• Create a HarperDB Cloud instance
• Create a schema and table for our data
• Create a custom function project
• Create different endpoints

Creating a HarperDB Cloud Instance ⚙️

First Sign up here studio.harperdb.io

then click on create new HarperDB Cloud Instance

Here we will be choosing cloud instance

There are different Instance plans here, I will choose the free option for now

Congratulations 🎉 You have successfully completed the first step of creating a cloud instance. You are now the owner of your very own HarperDB Cloud Instance!

Creating a Schema and Table 📋

Let's now create a table -> create a schema and -> add data.

Open your cloud Instance and create a schema for your table. I will name it "example".

Now create a table, I will name it "resources" and for the hash attribute, I will be using "id"

Time to add data...

Click on the plus symbol and here you can either send in one json object or multiple ones in an array. I will be pasting the below json:

[
  {
    "url": "https://matthewmcateer.me/",
    "title": "A Blog on AI and Blockchain",
    "description": "Biologist-turned-ML-Engineer in San Francisco. I blog about machine learning, biotech, blockchain, dogs, and more.",
    "cover": "https://matthewmcateer.me/photo-large.png",
    "category": "AI"
  },
  {
    "url": "https://future.a16z.com/a-taxonomy-of-tokens-distinctions-with-a-difference/",
    "title": "Designing Internet-Native Economies: A Guide to Crypto Tokens - Future",
    "description": "Crypto protocols, social apps, online communities, and marketplaces will need to deeply understand the interplay between fungible and non-fungible tokens to create their own internet economies.",
    "cover": "https://future.a16z.com/wp-content/uploads/2021/06/V7_GenericTopic_Crypto.jpg",
    "category": "Crypto"
  },
  {
    "url": "https://www.youtube.com/watch?v=R9ch1Jvaj7A",
    "title": "How to Create a Killer Marketing Message",
    "description": "how to go to any event n leave with potential clients",
    "cover": "https://img.youtube.com/vi/R9ch1Jvaj7A/maxresdefault.jpg",
    "category": "Marketing"
  }
]

Now we have successfully added data to our table. Time to create some API endpoints which we can use in the frontend.

Creating your first Custom Function 💾

Head over to the Functions Section and create a Project. I will name my project as api.

This gives you a file for adding routes, a file for adding helper functions and a static folder. I will discuss about helper functions later in the tutorial, Right now head over to the example.js file in the Routes folder and you will see some template code given. Let's remove it for now and just send "Hello world" as a response to check if our endpoint works.

'use strict';


module.exports = async (server) => {

  server.route({
    url: '/',
    method: 'GET',
    handler: () => {

      return "Hello World!";
    }
  });

}

This will send Hello World as the output when we hit our endpoint.

You can access this endpoint at -> {http://baseURL}/{projectName}

In our case it is https://functions-cloud-hsb.harperdbcloud.com/api

Nice, It's working!

Now let's create a route to fetch data by getting category from params.

{baseurl}/api/resources/:category

Here we are using a SQL query to select all the documents in the table resources which is from schema "example" where "category" comes from our params.

server.route({
    url: '/resources/:category',
    method: 'GET',

    handler: (request) => {
      request.body= {
        operation: 'sql',
        sql: `select * from example.resources where category="${request.params.category}"`
      };
 return hdbCore.requestWithoutAuthentication(request);

    }
  });

Also, notice that HarperDB allows access to the database with no authentication with their built-in method. If you are doing a POST request, do create some prevalidation which checks auth token etc.

To showcase this, I will create a a simple prevalidator like throwing an error if a particular category is not available in our list.

For this we can use helper functions. Let's create one called "checkForCategory"

Now import the function and send the request through the prevalidation

'use strict';
const checkForCategory = require('../helpers/checkForCategory');

  server.route({
    url: '/resources/:category',
    method: 'GET',
    preValidation: (request) => checkForCategory(request, logger),


    handler: (request) => {
      request.body= {
        operation: 'sql',
        sql: `select * from example.resources where category="${request.params.category}"`
      };



      return hdbCore.requestWithoutAuthentication(request);


    }
  });

Congratulations! You have successfully created a few endpoints to get data from your database . While also being hosted in the same instance. So, no need to deploy your backend somewhere else.

You can also do pagination etc here like

server.route({
    url: '/resources',
    method: 'GET',
    handler: (request) => {
      const size = request.query.size|| 10;
      const page = request.query.page || 0;
      const skip = parseInt(page)*parseInt(size);



      request.body= {
        operation: 'sql',
        sql: `SELECT * FROM example.resources ORDER BY category LIMIT ${size} OFFSET ${skip}`
      };
      return hdbCore.requestWithoutAuthentication(request);
    }
  })};

Frontend 📺

I sketched a simple UI using excalidraw Once that was done , I set up a React project and changed the app.js to this

import './App.css';
import { useState,useEffect } from 'react';

function App() {
  const [items, setItems] = useState([]);
  const [activeIndex, setActiveIndex] = useState("Marketing");
  const [selectedTag,setSelectedTag] = useState("Marketing")
  useEffect(()=>{
    function fetchData () {

      fetch(`https://functions-cloud-hsb.harperdbcloud.com/api/resources/${selectedTag}`, {
    method: 'GET', // or 'PUT'


  })
  .then(response => response.json())
  .then(data => {
    setItems(data);
  })
    }
    fetchData()


  },[selectedTag]);



  const tags=["Marketing","Startups","Podcasts","Product","Crypto","Learn"];



function ChangeResource(tag){

  setSelectedTag(tag);
  setActiveIndex(tag);
}

  return (
    <> 
    <div className="row">
     <div className="Boxtitle">
      <h1> Curated List of Best Articles and Resources 📚</h1>
     </div>
     </div>

     <div className="Boxcontent">

     {tags.map(tag=>(
       <div onClick={() => ChangeResource(tag)} key={tag} className={activeIndex === tag ? "sectionType active" : "sectionType"}> 
       <h3  > {tag} </h3>
       </div>
     ))}

     </div>
<div className="gridWrap"> 
 {items.map((item,key)=>{


    return(
        <div key={item.title} className="GridView">
     <a href={item.url} >
     <img   className="ImageStyle" src={item.cover} alt="" /> 
     <h3 style={{color:"white",width:"100%",maxWidth:"400px",textAlign:"center",display:"absolute"}}>  {item.title}  </h3>
     </a>
  </div> ) }
  )} 

    </div>





    </>
  );
}

export default App;

Here is the repo to the frontend code.

Here it is deployed on vercel-> https://smart-internet.vercel.app/

(Fun Fact : You can deploy your frontend on HarperDB too, if it is static . Read the FAQ section to know more)

You can find the backend code here

Misc 💭

Can we upload our Own Custom Functions?

Yes, you can. your local code should have routes and a proper project structure. You can start by cloning the official repo and change the code along the way.

How to Upload a Custom Functions to HarperDB?

At the time of writing this article, the best way to upload your local code to your cloud instance without having a local instance running is by creating a .tar file using 7zip or any tool and then converting it to a base64 string by uploading your tar to this website.

Then you need to make a POST request to your cloud instance. You will find your auth header and instance URL in the “config” section of your HarperDB instance.

While adding the auth header don't forget to add Basic before your token. (E.g. "Basic dsdfndfkdsad-")

Here is the request body with project property as api (since our project name was api) and the payload will be that long base64 encoded tar text. The file section is dummy but necessary so just copy what I have written.

{
    "operation": "deploy_custom_function_project",
    "project": "api",
    "file": "/tmp/144275e9-f428-4765-a905-208ba20702b9.tar",
    "payload": "aGVscGVycy8AAAAAAAAAAAAAAAAAAAAAAA........"
}

Congratulations, you just deployed a cloud function from your local project.

I hope you learned something new today. If you did, do give this a Like :)

Let me know how you will use HarperDB to create your next project.

I will go now and resume adding more resources from my whatsapp newsletter to this website

It's like a private space in the internet and getting new people to read it is hard.

Hashnode solves this. You can blog on your own domain and also get a community which reacts to your articles.

But when I was just scrolling down Hashnode feed for 20 minutes and I could see that there were many new faces and amazing new articles but had very little engagement . What if I tell you that with a few small tweeks , your blog could actually blow up 🚀.

I am not pro at blogging but as a YouTuber, I have learnt a thing or two about growing your online presence. Currently My YouTube Channel has 2M+ views and my blog has 50,000+ total views here on Hashnode.

Hashnode currently doesn't have a custom recommendation algorithm so chances that you get more exposure is still high , compared to medium or Dev.to

The Basics

If you recently started your blogging journey and confused what to write , here are some recommendations

• It's fine to share life experiences
• General Things like your tech interview experience
• Your first project
• Your hackathon experience ,etc . (Hashnode and Auth0 currently have a hackathon running where you could win $1000 ) .

When I recently called Tapas Adhikary for some guidance . I asked him how he is consistent and some suggestions for articles , here is what he had to tell

if you were searching something online and it didn't really match the exact problem you were trying to solve. After you figure out the solution to that problem, Write about it.

I have also observed one thing which even I was feeling initially, that is not writing about a particular topic because there are already some articles about it but

If you learnt something recently like react ,Vue or shifted from one framework to the other. Writing just makes your understanding strong, helps beginners and if you forget about something , you can get back to it.

Publishing Checklist

Here is a checklist after you publish your article.

• Have a Crisp Title focused on Viewers/ Problem
• Have a cover Photo ( checkout slickr by Savio Martin or CoverView by Rutik Wankhade)
• Make sure you add 5 relevant tags

Those were the basics which few still miss out.

Now that you have written an article. Getting initial traction is very important. Post your Blog on Twitter / Reddit and if you are ready for some hate then Hackernews aswell ( it's a rabbit hole, no one knows how something gets viral there, my last post got like 15K views for being number 4 for 3 hours ) .

But Hrithwik , I don't have Twitter or friends on Twitter , what can I do ??

Let's start with me. Just DM me on Twitter and I will retweet your article if I like it.

Also If you are from college, share your blog in your college dev discord / whatsapp groups. Joining Clubs in college has huge upside to your network (But do you know that Hashnode has it's own discord community ? )

This get's you those initial traction especially if your article is something very unique or your dev friends from college have not herd of. If something is interesting people automatically like and comment.

Now if you are serious about writing or anything in life , you need to have a group of friends in the same field. Support your friends , have small talks , collaborate and the support becomes mutual. Developers are one of the nicest and friendly people in the world (atleast everyone who write on hashnode ) .

So I am sure you would have guessed where I am going. Yes community.

• It's important to have atleast 20-30 people from the same group who are there for you always and you are too.
• Comment on others stories and react to their stories. This let's people know that you exist and supporting others is always good karma .
• Linkedin is unnoticed but if you have a bunch of tech connections on LinkedIn you have high chances to get the initial traction from there too .

Additional Tips

In the home page the first two articles are featured posts which stay on top for few hours and if you have more reactions than the third or fourth post you will be the third post and if the article is good enough for a larger audience , your article will be featured on Hashnode. Congratulations 🎉

The Second Boost

Hashnode promotes all the good article on their twitter , facebook and linkedin. You get the second boost from their newsletter too. Also some of my viewers also came from Daily. dev

That's it everyone. I hope you get more reach and grow your online presence.

Who am I ?

I am Hrithwik from Mysore , India . I am Curious about startups and tech in general.

I belong to different cults xD . I am a musician , a tech guy , cool college senior and I can also code. So I have multiple groups I hangout with.

I also run a whatsapp newsletter which people love , I share interesting things about product, code, startups and podcasts. Highly recommend you to join if you use whatsapp .

Also newsletter helps you convert first time viewers into recurring viewers . Hashnode has an inbuilt newsletter . Soon you will be able to send custom messages in the newsletter too .

I am learning react and next.js and it is very exciting. Will share my journey from vue to react soon. Follow here and stay tuned

Here are all my socials where we could chit chat .

Github Copilot is an VS Code extension which can autocomplete your code and also synthesize code by seeing your comments and function name. This is built using the same model and trained on billions of public code.

When I told my friends about Github Copilot the first question I got is

Can it do DSA Problems and help me crack into FAANG xD?

This was funny but I wanted to try it out and the results were mind blowing. AI could solve most of the DSA problems and I have shared some results here .

Later I tried using github copilot in some real development work like writing email validation , making api calls to github and even building a simple email sending client.

Now I have some clear idea on the limitations and things that Copilot is good at this movement .

Before moving on to the article if you prefer videos over an article, Here is my video on Solving DSA Problems using Github Copilot

The Good✨

Forgot the Syntax ? No worries. AI got your back

When building a project I had forgot status code for what response to send and writing a comment about it gave me a result. Also I had written python after long time and at one point I had forgot the syntax for slicing and this helped me there aswell.

Can help you with Email Validation and API Calls

As I shared before it was really easy to generate simple functions for calling some API's and write some regex code without learning it or depending on documentation. ( Not recommended if you are a beginner) .

Bioler Plate code

It can also help you get like a boiler plate code for express and helped me write some mongo schemes aswell .

Overall I just love how seamlessly Github Copilot can show you suggestions on vs code. Beautifully done

Limitations 🙃

Github Copilot for an first generation/ Beta product is way too good. But there are a lot of biased articles online so I just wanted to show the other side aswell.

Doesn't Follow the Best Practices

The code synthesized doesn't always follow the best practices or is not optimized to use it for backend development. Like in Javascript, it generates var and == than const and === which can lead to different bugs and shadowing.

Checkout my article on best practices for JS developers

Version Conflicts

Some Code Generated might not work for the version your codebase is on and can lead to conflicts. Also sometimes AI can give out a lot of unnecessary code. Developers with experience can definitely handle this out but what if a newbie directly starts with the help of AI, he will spend more time on stack overflow than writing actual code. ( oh wait that's how most of the developers are lmao)

Weird Copyright messages

When I tried to generate code for converting linkedlist into a tree which is like a famous DS problem, The AI started showing copyright messages in the comments and it was from a person called Stephen.

Hey Stephen, If you are reading this, please follow me on hashnode .

Frequently asked Questions

• What's Good ?

  It's great if you want to create functions like email validation or user schema or API request since it has enough data but again some code might not really be efficient.

• Can it help you crack DSA Problems (if it is legal) ?

  The Answer is not binary. it can help you give building blocks according to your needs and you are the one who should build the building.

• Will it Replace Humans ?

Absolutely NOT. As the name suggests , it's a copilot making your life easier. You are the pilot who can understand human narratives and stories to build product from code.

Let's Connect

Thanks for reading my article. I want to make new friends in the dev world. Let's connect on Twitter and LinkedIn .

Also a tiny Observation

After using this to write my project code for more than 2 hours I got so used to copilot that without using github copilot suggestion I literally didn't know what to write next in my code . I am a little scared. IT's a TRAP to make developers even lazy xD . Can't wait to see the pricing.

Few months back I built a tool called richurl.tech which is like a url shortner 🔗 but with super power ⚡ . When someone clicks on the richurl, it sends them directly to the YouTube app instead of opening videos on FB/Instagram/Linkedin Broswer.

The backend API was deployed on heroku. It's a cloud platform as a service where you can deploy apps built using python, node, Go, and many more.

If you are on a free tier on Heroku your app sleeps if it is not active for more than 30 minutes. But My back-end should never sleep since it causes delay to wake up the server and defeats the whole purpose of simplifying the user flow through the service.

Here is how I fixed it

Solution ✨

Adding a credit card on heroku gives you total 1000 hours/month , allows you to deploy more than 5 apps and allows custom domain mapping all for FREE.

First method

There are many ways on the internet by which you can stop your heroku from sleeping but by far this is the simplest way on the internet.

Follow this only if you have more than 750hours of dyno time since you won't run out of monthly quota.

Visit Kaffeine Shots and enter your heroku app. It basically pings your app every 30 minutes once and doesn't make it sleep except for the minimum 6 hours which is mandatory.

If you don't want this to continue or running out of quota then you can even stop this.

Second method

Hobby Dynos

Hobby Dynos are same as free dynos but they never sleep by default.

They cost $7/month but if you are a student like me you can sign up for GitHub Students beneifts and can get free hobby dynos for 2 years .

Currently I am using Hobby Dynos since I am a student.

That's it folks. If you have a friend who is a youtuber . Ask him to try out Rich URL while sharing his links on social media platform. I am very close to 100 followers on hashnode. Click on Follow :)

I am also available for free lance gigs or Full time offer. You can contact me on Twitter or on Linkedin

Do note that some browsers might not support all these API's

Clipboard API 📋

Implementing clipboard functionality is even simpler now with the clipboard API.

Using Clipboard API ,You can either Copy or Paste from your Clipboard

Here is how to copy

const text = "Some Information"
if (!navigator.clipboard) {
    // Clipboard API not available
    return
  }

 try {
    await navigator.clipboard.writeText(text)
    // copies to clipboard
  } catch (err) {
    console.error('Failed to copy!', err)
  }

Here is how to Paste from the user's clipboard

if (!navigator.clipboard) {
    // Clipboard API not available
    return
  }

 try {
    const text = await navigator.clipboard.readText()
    // copies info from clipboard to text variable
  } catch (err) {
    console.error('Failed to copy!', err)
  }

Share Target API 🔗

Let's say you have built a PWA where people upload photos or share links and allowing users to see your app while clicking on the share button makes your PWA look dope.

Add this in your mainfest.json file. If you don't know about manifest json file or PWA's in general .Check out my article on Converting your website into PWA

Manifest.json

"share_target": {
  "action": "/share-target/",
  "method": "GET",
  "params": {
      "text": "link",
 }
}

Now on the app side, you can get the data that the user has shared through a query parameter called link

The above code works only for links or text but what if you have to share images or files?

To implement for images/files checkout an article at web.dev here

File System API

Let's say you want a user to upload a file and the below code works for reading the file

<input type="file">

and you can save the file using download inside an anchor tag

<a href="" download="file.txt > Save File </a>

But you can't just edit the file which you have accessed from the above method. With File Share API you can edit the files you have read and save them which sends a new prompt to the user.

Do note that this is a new feature and only chrome 86 or greater supports this.

index.html

<button id="fs-open"> Upload File </button>

index.js

const getFile = document.getElementById('fs-open')

getFile.onClick = async() =>{
[fileHandle] = await window.showOpenFilePicker();
if (fileHandle.type === 'file') {
          const fileData = await fileHandle.getFile();
}
else if(fileHandle.type==='directory'){
          const dirName = 'directoryToGetName';

           // assuming we have a directory handle: 'currentDirHandle'
          const subDir = currentDirHandle.getDirectoryHandle(dirName, {create: true});
}

};

To save a file to disk, you can either use that filehandle that you got earlier or call showSaveFilePicker() to get a new filehandle.

async function saveFile(fileHandle) {
  if (!fileHandle) {
    fileHandle = await window.showSaveFilePicker();
  }
  const writable = await fileHandle.createWritable();
  await writable.write(contents);
  await writable.close();
}

You can even get directories and create files in the directories.

This is especially useful if you want to create Notepad Applications on the web. Read more about it here .

Web Storage API

Local Storage is great for storing information that shouldn't be sent to the server, it also doesn't have any expiration, and also it has more storage capacity on the other hand session storage can be used for a session and for a fixed amount of time.

Here is how to store let's say a username

localStorage.username= 'hrithwik'

now let's say you have to retrieve the username of the username without a further API call

let username = localStorage.username

Remember I told you that there is no expiration for this?

localStorage.removeItem('username')

User Media API

Let's say you are building the next zoom or an AR app. How do you access the camera ?

That can be done using navigator api

navigator.mediaDevices.getUserMedia({
  video: true,
  audio: true
})
// get's both audio and video from the user

This returns a promise and can be stored in a variable. Let's Display the stream on the browser now

<button id="video-on">Get access to camera</button> <video autoplay></video>

document
  .querySelector('#video-on')
  .addEventListener('click', async function init(e) {
    try {
      const stream = await navigator.mediaDevices.getUserMedia({
        video: true
        audio:true
      })
      document.querySelector('video').srcObject = stream
      document.querySelector('#video-on').setAttribute('hidden', true)
      setTimeout(() => {
        track.stop()
      }, 3 * 1000)
    } catch (error) {
      alert(`${error.name}`)
      console.error(error)
    }
  })

This hides the button after clicked and shows the video on the browser.

Bonus - Geo Location API

What if you are building an app that requires constant GPS Information from the users' device.

That's where GeoLocation API comes in. It gives you not only information regarding the longitude and latitude of users but also access to other various sensors which can be workaround with more use cases.

 if(!navigator.geolocation) {
    status.textContent = 'Geolocation is not supported by your browser';
  } else {
    status.textContent = 'Locating…';
    navigator.geolocation.getCurrentPosition(success, error);
  }

function sucess(position){
const latitude  = position.coords.latitude;
const longitude = position.coords.longitude;
mapLink = `https://www.openstreetmap.org/#map=18/${latitude}/${longitude}`;
}

function error(){
 status.textContent = 'Unable to retrieve your location';
}

You can even see if the device is in motion by adding a event listner.

window.addEventListner('devicemotion',(event) =>{
console.log(event)
}

If you want to discover more Web API's Here is a list of all Web API's

If you are on Hashnode and seeing this. Consider following my blog :)

Also, I love Filter Coffee

I use Instagram extensively and while surfing there I see a lot of influencers add their YouTube links in the bio and stories but when I click on their link it opens it on Instagram's white browser which sucks for viewers.

I have been a YouTuber myself and have seen my friends complaining about why they don't click my bio link. To Summarise below is the problem statement.

Update: this project got me selected as a winner for AWS Amplify Hackathon .

Problem

Content Creators / Influencers on YouTube and marketers who use Instagram, Facebook, and Linkedin for sharing their video links have observed that all these platforms use their own browsers while opening external links.

This might be good for Facebook as it makes people stay on their own platform for more time but not good for creators and end consumers.

End Consumers can't see videos in high quality and end up having a bad experience while creators can't get better ad revenues or better engagement since not all users would have logged in to YouTube on their browser.

My Solution ✨

I created richurl.tech which takes normal YouTube and Amazon links and converts them to magical rich URLs for absolutely FREE with unlimited redirects and unlimited links.

Generating Rich URL's for YouTube Increases the chances of viewers staying more on your video, better ad revenue, and better chances of commenting.

Generating Rich URL's for Amazon helps digital marketers increase the chances of users clicking the buy button resulting in better affiliate commission.

How does it work?

Let me try to put it in the simplest way. When someone generates a rich URL for their YouTube video, I create 3 links. one for the web(https), android(intent), and IOS. Later based on the user agent of the request I do a 301 redirect to the respective link.

Features 🔥

• Supports YouTube Videos/Channels
• Supports Amazon India
• Get click counts based on the device.
• Directly share the link from YouTube to richurl.tech or paste Links from Clipboard with one click.
• Resultant URL gets copied to your clipboard automatically so you can start sharing with ease.
• Progressive Web App

If you are a YouTuber this is a must try and let me know if you are liking it.

Developing Experience 😊

I had a basic idea of how to implement it but was not sure if it would work. That's when I started breaking the problem into small chunks and assigned myself small tasks every day till the 23rd of February to complete on Notion.

Notion is amazing. I created different pages for the Backend and frontend. Created boards and assigned work for every day.

Technology Stack

• Nuxt
• Buefy
• Mongo DB
• Express JS
• Amplify

In the first 6-8 days built the rest-api iteratively and added small features here and there when I got new ideas. The learnings from building Rest API were building auth, Relational Data with Moongoose, and the main logic to implement deep linking on the web. Learnings from building the front end side were Share Target API and Buefy .

The next 3 days setup the nuxt project handling and functionalities like Auth, Clipboard web API, web share api, and handling dynamic routes in Nuxt.

The final days(3 Days) were for building the UI side of things using Buefy Extention to Nuxt.

Challenges during the project ⚙️

• Deep Linking Functionality (it's different for Yt Video and channel)
• Implementing Auth in Nuxt (Once it's figured it's piece of cake but took me an entire day)
• BUEFY TABLES (Customization, handling clicks, and other small details are hard to find in the documentation and took me 2 days just to implement basic features in the frontend.)

Overall loved building this tool for marketers and influencers as I was new to most of the stack in this project and got to learn a lot by building this and pushed myself to write code every day. Shout out to Hashnode and Amplify for finally motivating me to build this.

Future Improvements

• Complete Statistics with Geo-Location and From location (on it right now)
• Add support for more social platforms
• Map to a custom domain for better branding ( not sure how this works. Need to dive deep in)

Here is the GitHub Repo - Click Here

This is a soft launch of RichURL.tech and my submission to Hashnode's Amplify Hackathon. Use it and please give feedback so that I can add more features and improve the existing ones.

Also, share this on Twitter with your other Youtube Friends so that they can be benefited.

If you have recently started learning JavaScript or a beginner I would highly recommend you to read the complete article so that you follow better practises while coding in JavaScript as it's a little different.

It's been a few months since I have started programming in JavaScript and these are some of the practises I and several other senior developers follow.

Don't use ' == '

In JavaScript there are both '== ' and '===' and you might be used to "==" in other programming languages and might continue using the same, But using it without knowing how it works might bring few bugs which might be difficult to track. Read my previous article to know more.

Use "let" over "var"

When declaring variables you might have come across 'var' and 'let' but "var" is not recommended as it brings some problems along with it. It will be easier to understand with an example.

for(var i=0;i<5;i++){
    console.log(i);
}
console.log(i)

// Output : 0 1 2 3 4 5

Notice that variable i is still accessible outside the block/for-loop which is bad. On the other hand let's write the same code using "let"

for(let i=0;i<5;i++){
            console.log(i);
        }
        console.log(i)
// Output : i is not defined

the output throws an error as the variable i is not accessible outside the for loop/block thanks to "let" as its block-scoped.

There is also something called as shadowing. For example

var a= 100;
function example(){
a=20;
} 
example();
// 20 just shadowed 100 of the global scope 
console.log(a);
// Output : 20

Here you can see that both the variables access the same memory space that's why changing the value inside a block changed the value of the variable outside.

'var' also leads to Hoisting which I will discuss later in the same article. Whenever you want to declare variables in the future where data might change use"let".

Go through some of the articles regarding the same from Caelin Sutch - Let vs var and Faith Gaiciumia - Difference between var and let .

Use Immutable variables or "const"

Use 'const' to declare variables as much as you can Unless the data changes in the variable like the 'i' inside the for-loop. Here are few examples

const x = 'SECRET'
console.log(x)
//Output: SECRET

// Here are few illegal things 
// number 1

const x = 'SECRET'
x='Hacking'
console.log(x)
// Error: Identifier 'x' has already been declared

// Number 2 
const x;
x="SECRET";
// Error: Missing initializer in const declaration

Read more about const here .

Use Function Expressions

With the introduction to ES6, there is a new way to write functions. Here are few examples

// Method 1  
let sum = function(a, b) {
    return a + b;
  };

//Method 2 
let sum = (a, b) => { // using arrow function
    return a + b;
  };

// Method 3 
let sum= (a,b) => a+ b ;  /* if there is only line inside function. 
There is no need for return */

//Method 4 (Annonymous Function)
(() =>{
    console.log("annonymous  functions")
  })();

// Notice our function doesn't have anything but still works.

This is called Function Expression. Having function expression instead of the normal function declaration also avoids Hoisting.

What is hoisting?

to explain hoisting in functions let me take an example.

exampleFunction(); // function call 

function exampleFunction(){ // function declaration

console.log("Hashnode is awesome")

}
// Output: Hashnode is awesome

Did you observe how calling exampleFunction() even before definition didn't throw any error.

Why did this happen ? In JavaScript unlike other languages an entire copy of function is copied in the execution context even before executing the code. But doing the same using function expressions would throw an error.

To understand that let's see what happens to variables declared using "var" keyword. Here the memory is allocated even before the execution of the program but it will have undefined value until it goes to the line where the variable is assigned with a value.

console.log("Before Assignment: ",a);

var a=10;

console.log("After assignment: ",a)

/* Output:

        Before Assignment: undefined

        After assignment: 10

The above is called variable hoisting. But coming back to function expressions.

exampleFunction();

const exampleFunction= () =>{

console.log("Hashnode is awesome");

}

// Output : ReferenceError: exampleFunction is not defined

In function expressions the values are not copied to memory even before execution which leads to an error.

Some developers leverage hoisting with function definition by declaring all the functions in the bottom and access it on top but it depends on you. I would suggest you use function expressions and call functions after declarations as it might lead to fewer errors and even reading becomes easier.

Use Pure Functions

Pure functions is a function where everytime you call that function you get the same result and it should not have side effects.

Why should you care ?

This helps in testing the function and debugging becomes really easy. Understanding this wih an example would be better . Below is an impure function which multiples all values in an array by 2.

array=[1,2,3,4,5];
const doubleArrayValuesImpure= (array)=>{

    for (const number in array) { 

   array[number] = array[number] *2 ; 

    }
    return array; 
    }
// changes the values in original array

Let's call the same function 2-3 times

console.log(doubleArrayValuesImpure(array)) ;
console.log(doubleArrayValuesImpure(array)) ;
console.log(doubleArrayValuesImpure(array)) ;

/*Output:
[ 2, 4, 6, 8, 10 ]
[ 4, 8, 12, 16, 20 ]
[ 8, 16, 24, 32, 40 ]
*/

Notice how the values are changing on the original array every time we call the function. Let's write a pure function with the same logic.

const doubleArrayValuesPure= () => array.map(number => number *2) ; 
// returns a new array with new values

calling the pure function

console.log("Pure Function results")
console.log(doubleArrayValuesPure(array));
console.log(doubleArrayValuesPure(array));
console.log(doubleArrayValuesPure(array));
/*Output:
[ 2, 4, 6, 8, 10 ]
[ 2, 4, 6, 8, 10 ]
[ 2, 4, 6, 8, 10 ]
*/

Notice how calling the same function 3 times didn't change the original array. Whenever we declare functions we should focus on writing pure functions unless we are changing a file or writing into database.

Name things properly

We developers are very lazy and usually use random names for variables and cry in a corner when we have to revisit the same code. You don't actually need to document your code if you have named things in the right way. Also use camel casing.

// BAD
let ad="124 Broadkill Rd #483";
let c="Milton";

// Good
let address="124 Broadkill Rd #483";
let city="Milton";

Use Destructing

Let's consider having an object called person.

const person= {
    name: "Hrithwik",
    age:20,
    number: 9999999999,
    jobStatus:"Give me a job please",
}

Now if you want to access multiple values which are inside an object you use object destruction instead of manually writing extra lines of code. Check this out.

//noob 
let name = person.name;
let age= person.age;
let phoneNumber= person.number;
let occupation= person.jobStatus


// Pro 
let { name,age,number, jobStatus} = person ;

But do note that the name of the variable outside the object should be the same as the ones inside. You can also destructuring arrays, read this article to know more.

Use Promises over Call Backs

I am not sure if you have discovered about handling asynchronous code. if you haven't skip this and read about it later.

Here is an example of asynchronous code which takes the user object and checks the commit name in the first repository of the GitHub user.

console.log("before");

getUser(1,(user)=>{
    console.log("username:",user.github);
    getRepo(user.github,(repo)=>{
        console.log("Repositories are " + repo);
        getCommits(repo[0],(commit)=>{
            console.log("Commits")
        })
    })
})


function getUser(id,callback   ){
    setTimeout(()=>{
        console.log("reading user from database")
    callback({ id:id*2,github:'hrithwikbharadwaj'});
},2000)
}

function getRepo(username,callback){
    setTimeout(()=>{
        console.log("reading repo from "+username);
        callback(['repo1','repo2','repo3']);
    },2000)

}
function getCommits(repo,callback){
    setTimeout(()=>{
        callback(repo);
    },300);
}

console.log("After")

Callbacks are cool unless there are a lot of callbacks inside callbacks.

Isn't it difficult to read? It's also called callback hell/Christmas tree problem. You can fix it by named functions but a better way to do it is by using promises. Here is the same code using promises.

console.log("before")

getUser(1)
.then(user=> getRepos(user.github)) // is user available ?
.then(repos=>getCommits(repos[0])) // oh great now find his first repo
.then(commits=>console.log("commit name: ",commits)); // found the repo ? cool now show the commits



function getUser(id){
    return new Promise((resolve,reject)=>{
        setTimeout(()=>{
        console.log("reading user from database")
    resolve({ id:id*2,github:'hrithwikbharadwaj'});
},2000);
    })

}

function getRepos(username){
     return new Promise((resolve,reject)=>{
          setTimeout(()=>{
        console.log("reading repo from "+username);
        resolve(['repo1','repo2','repo3']);
    },2000);
     });


}
function getCommits(repo){
   return new Promise((resolve,reject)=>{
        setTimeout(()=>{
        resolve(['commits']);
    },2000);
   });
}

console.log("After")

Read more about promises by Jome Favourite here on Hashnode.

Write Functions for everything

Most of the times we write functionality for many things but we don't name it. Keeping everything inside a function helps to again search for what you want easily. Also just keep one functionality inside one function.

// Bad code
let number1=2;
let number2=5;

let sum= number1+number2;

// Good
const sum = (number1,number2) =>{
    return number1+number2;
}

console.log(sum(2,3))

Eliminate Decimals without killing performance

You might have a situation where you need to generate random whole numbers between 0 to 1000 and we usually use math.floor or math.ceil

Math.floor(Math.random()*1000)

but instead of that try using

~~ (Math.random()*1000)

this can improve performance when micro-optimization of code. refer to this article to know more.

Deleting array like a pro

There are several ways to delete an array on javascript but this is my favourite way.

let numbers=[1,2,3,4];
numbers.length=0;

Most of the times we jump into a language without knowing its fundamentals but I hope with this article you would write better code in JavaScript .

You can never learn from just one article. Google concepts and read multiple articles, books and play around the code by making projects and that's how you learn.

Checkout a curration of learning resources for JavaScript by Rahul here .

Instead of using these services let me tell you about PWA's and how you can convert your website into a Progressive Web Application (PWA) by following 3 easy steps.

What's a PWA?

** Progressive Web Applications ** is a technology built by Google developers which allow developers to add native looks and some features like offline load & Push Notifications to your website.

Advantages of PWA

• Easy Installation for users: It just take few seconds to install PWAs and they are lightweight which is the reason why Instagram lite and Facebook lite is basically a PWA.

• Easy to build: Web technologies are easier to learn than native development and numbers of developers required to build a project is also less.

• Cost-Effective: With one code base your website can run on android phones, iPhones and Windows Computers.

• Better Performance: With image and code caching the performance is increased drastically.

Prerequisites

• Basics of HTML and JavaScript

• Current Website should load with HTTPS

• Current Website should be responsive

How can you do it?

These are the three files needed to convert your current website into an installable PWA.

• Manifest.json

• Service Worker

• Installable Navigator

Manifest JSON

This is the file which helps you add splash screens and icons to your app. In this file, you specify basic metadata like your app name, icon size, icon directory and start URL of your website.

Here is my Manifest.JSON which you can copy and replace it with your details.

{ 
  "name": "Hrithwik Bharadwaj",
  "short_name": "Hrithwik",
  "description": "Portfolio PWA of Hrithwik Bharadwaj",
  "icons": [
    {
      "src": "/assets/img/icons/512x512.png",
      "type": "image/png",
      "sizes": "512x512"
    },
    {
      "src": "/assets/img/icons/256x256.png",
      "type": "image/png",
      "sizes": "256x256"
    },
    {
      "src": "/assets/img/icons/192x192.png",
      "type": "image/png",
      "sizes": "192x192"
    }
  ],
  "start_url": "/",
  "scope": "/",
  "dir": "ltr",
  "background_color": "#111111",
  "display": "standalone",
  "theme_color": "#111111",
  "orientation": "portrait",
  "prefer_related_applications": false
}

This might be confusing to understand if you are a beginner so just check out this manifest generator to generate icons and later download the file.

Also don't forget to Include this in your HTML file

 <link href="/static/manifest.json" rel="manifest">

Service Worker

This file helps in making your app work offline by caching important files. If you have worked on React or Vue Projects this file is usually shipped along.

Here I am using WorkBox API for service worker as its very simple. Just create a file called sw.js and paste the below code there.

if ('undefined' === typeof window) {
    importScripts('https://storage.googleapis.com/workbox-cdn/releases/4.3.1/workbox-sw.js');
}

workbox.core.skipWaiting();
workbox.core.clientsClaim();

if (workbox) {

    console.log(`Yay! Workbox is loaded 🎉`);

    workbox.precaching.precacheAndRoute([
        {
            "url": "/",
            "revision": "1"
        }
    ]);

    workbox.routing.registerRoute(
        /\.(?:js|css)$/,
        new workbox.strategies.StaleWhileRevalidate({
            cacheName: 'static-resources',
        }),
    );

    workbox.routing.registerRoute(
        /\.(?:png|gif|jpg|jpeg|svg)$/,
        new workbox.strategies.CacheFirst({
            cacheName: 'images',
            plugins: [
                new workbox.expiration.Plugin({
                    maxEntries: 60,
                    maxAgeSeconds: 30 * 24 * 60 * 60, // 30 Days
                }),
            ],
        }),
    );

    workbox.routing.registerRoute(
        new RegExp('https://fonts.(?:googleapis|gstatic).com/(.*)'),
        new workbox.strategies.CacheFirst({
            cacheName: 'googleapis',
            plugins: [
                new workbox.expiration.Plugin({
                    maxEntries: 30,
                }),
            ],
        }),
    );
} else {
    console.log(`Boo! Workbox didn't load 😬`);
}

workbox.googleAnalytics.initialize();

Now that you have created an sw.js file just add the link to it on your HTML page.

<script src="sw.js"> </script>

Adding the installable component

This part is the one which will allow you to get automatic "install now" prompt.

   <script>
            if ('serviceWorker' in navigator) {
              navigator.serviceWorker
                .register("sw.js", {scope: '/'})
                .then(registration => {
                  console.log("ServiceWorker running");
                })
                .catch(err => {
                   console.log(err);
                })
            }

</script>

Copy this and add it in your HTML file.

Testing

After you add these things. To check if everything is working fine, Use Lighthouse in chrome and check if it is showing a PWA sign.

What next?

PWA's can have many other features and even have a different type of service worker. Above code was the easiest implementation for beginners.

Below are some small things which you can add.

Custom Install Button

You can do something like this where the button hides after installation using this code some simple JavaScript code and CSS.

Push Notifications

From BookMyShow to Instagram everyone sends Push Notifications from PWA to their users. This can be implemented using OneSignal or Firebase Cloud Functions

Conclusion

If you have a small team which has already built a mobile first website than it's easier to switch to PWA.

To be honest, it's a temporary fix until your native app is getting ready.

Yeah sure, PWA looks like a mobile app but it isn't as smooth as a native app. A lot of developers like me are waiting for PWA's to become smooth.

(Update: There is also a new kind of Installation Widget that twitter started using for it's PWA - Here is short guide on it.)

PDF Miner

pip install pdfminer

PDF Miner lets you extract text from pdf files. This can be useful if you are building a product which needs text from say resumes or reports.

There is also a similar library called PyreParser which is specially made just for extracting resume information(This tech is pretty much used in every company in the hiring process)

Pillow

pip install Pillow==2.2.1

Pillow is a library which allows for image manipulation & conversion. You can do a wide variety of things like adding text to an image or blurring an image. Below is a simple example of adding text to an image.

Below is the before and after running this code. We just added "hashnode is amazing" in the bottom right using the x y coordinates.

There is also a python package called moviepy which lets you do similar things but with videos.

OpenCV

pip install opencv-python

This is one of the booming packages and LinkedIn is filled with project demos built with OpenCV. You can do anything from resizing an image to Controlling Subway surfers from your hands through image recognition.

Above is a simple example to open a greyscale image and resize it.

Refer to these Resources to learn more about OpenCV( Edureka and Murtazas YouTube Channel )

Furl

pip install furl

Furl makes parsing and manipulating URLs easy for you. Here is an example of paths and query using furl

Output:

http://www.google.com/path?two=2&three=3

Scikit- learn

pip install scikit-learn

Scikit- learn can be used for a variety of applications which include classification, regression, clustering, model selection, naive Bayes’, grade boosting, K-means, and preprocessing.

Above is a snippet from one of my 2nd-year projects at engineering which predicts prices for houses in Bangalore. This uses Linear Regression and I didn't know the formula and other details but using scikit-learn I could easily use the concept to build a model.

Html2Text

pip install html2text

This allows you to convert HTML code to markdown with just one line of python code.
But you can't directly add websites instead of HTML code, using a beautiful soup can help you do that.

Output :

#  Hello world

**Html2 text is amazing**

[ My Blogs are cool ](blog.hrithwik.me/)

Beautiful Soup

pip install beautifulsoup4

Beautiful Soup allows you to parse HTML data from websites and get the data you want by finding the element you want using id or class name. This is one of the most used tools in the field of web scraping. (NOTE: Web Scrapping might be illegal on some websites )

Here We just parsed through Bhanu's Article and Generated a markdown from the URL using HTML2Text

Kivy

pip install kivy==1.11.1

Kivy is a really interesting GUI framework that you can use to create desktop user interfaces and mobile applications on both iOS and Android.

Kivy applications will not look like native apps on any platform. This can be an advantage if you want your application to look and feel different from the competition. There are even some alternatives like PyQt and Quirt.

Faker

pip install Faker

Faker is a Python package that generates fake data for you. This can be really useful for testing or bootstrapping your database.

You can even checkout radar which can generate random date and time.

Selenium

pip install selenium

Selenium is one of the testing frameworks which can be used for fun stuff like automation and building bots. You can automate anything like sending WhatsApp messages to increasing views for your blog.

This is a code which opens your website for n number of times in a browser. You can add different user-agents as well.

Conclusion

These were some packages which can do testing or build machine learning projects. Try using any one of them and let me know what you built using these Packages. Hit me up on Twitter or Linkedin .

What's the Difference?

The '==' Operator

This is called Loose equality comparison operator in JavaScript.

Let's take an example here when comparing a string and an integer with the same value.

let a=1;
let b='1'; 
console.log(a==b);

OUTPUT

true

or wait let's try this

let a=1;
let b=true; 
console.log(a==b);

OUTPUT

true

WHAT?? How is it true??

You may be wondering how this is possible? well because the '==' operator takes only value into consideration, not the type. Therefore matches boolean true to 1 by converting one of them.

The '===' Operator

This is called Strict equality comparison operator in JavaScript.

let a=1;
let b='1'; 
let c=true;
console.log(a===b);
console.log(a===c);

false
false

This returns false for the values which are not of a similar type. This operator performs type casting for equality. If we compare 1 with '1' using ===, then it will return a false value.

When to use the '==' Operator

I don't usually recommend to use this often but you can use this when

• Checking if a value is null/undefined
• When you know both the operators of the same type

To Summarize

• '==' Operator is Value based comparison

• '===' Operator is Type+Value based comparison

So most of the times you should be using triple equals insead of double equals as it's more precise, accurate and doesn't bring up bugs which are not easy to find.

I love speaking to new people on the internet. Hit me up on Twitter or Linkedin , Lets talk tech.

There are a few options you have like PythonAnyWhere or Nitrious.io but both of them aren't that reliable and easy. That's where Heroku comes in.

What's Heroku?

Heroku is a container-based cloud Platform as a Service (PaaS). You can use Heroku to deploy, manage, and scale modern apps. You don't need to add credit card details to deploy apps here.

Advantages of Heroku

• Deployable using Git
• Free of Cost with 450 Hours of runtime

🛠️Tools needed on your machine

Before we begin you need to install these on to your windows/mac machine with the Gunicorn Pip Package.

• Github
• Heroku CLI
• Gunicorn

💻Setting up Heroku

Now that you have the essential tools installed on your PC/laptop, Go to Heroku Signup and fill in your details by confirming your email address.

Open the project folder and open command line terminal there and do the following commands

heroku login

If everything is good you will see something like this

🦄 Three important files for Heroku

Before deploying the flask app to Heroku you need to create these 3 files in your project.

• The procfile
• Requieremnet.txt
• runtime.txt

The Proc File

In simple words, this file tells the Heroku machine what program to run by which dyno. In case you are interested in reading more about the Proc File check this article.

web: gunicorn app:app

Requirement.txt

This file contains all the packages which are imported in the app.py which needs to be installed. You can generate this using pipreqs utility or using the command "pip freeze". Also don't forget to add jinja2 and gunicorn inside requirements.txt file.

Flask_Compress==1.4.0
Flask==1.1.0
Flask_SSLify==0.1.5
requests==2.20.1
gunicorn
Jinja2
gevent

Runtime.txt

python-3.7.5

🚀 Deploying to Heroku

heroku create (your app name)

This would create a git repository and a Heroku app with the unique link generated.

D:\Deshik\HashNode\Developer Quotes>heroku create devquoteflask
Creating ⬢ devquoteflask... done
https://devquoteflask.herokuapp.com/ | https://git.heroku.com/devquoteflask.git

Now Heroku creates its own private git repo. You can either use it by typing the following command every time you add new changes to your project.

git init 
git commit -m "commit"
git push heroku master

or you can even connect your GitHub repository so that every time you push something to your main GitHub repository, Any changes made to the app will automatically be reflected in the website. Follow this article to connect your Github Repository to Heroku.

🌐 My Single Page Flask app

Here is a simple Flask app I built for this article, Which Gives you new Programmers Quote every single time you refresh - http://devquoteflask.herokuapp.com/.

Refer the full source code in case you have any doubt with the files

😮 What's the Catch?

The free tier of Heroku is actually pretty good. You are limited to 10k database rows and the dyno spools down when it hasn't been used in about 5 minutes.

It makes the initial page load quite slow and it won't be able to handle heavy traffic but it's a seriously generous package and great for development.

You are allowed to create only 5 projects with 450 hours of dynos per month without adding the credit card. By adding credit card you get extra 550 hours of dynos.

Now that I helped you save some cash for deployment. Follow me on Twitter and Linkedin to stay connected.

Also, Check out my previous article on How I Created a MEME Bot using Python

What is FireBase?

Firebase is Backend as a Service (BaaS) where it takes care of user authentication, Database and hosting all by itself.

My Experience with Firebase

I have personally built 3 production-level projects using most of the firebase features and I have faced 0 problems with it and the best part I haven't spent a single penny on it.

3 Reasons Why you Should Care about Firebase

• Experience - Firebase has the best documentation and has good support in case you are stuck somewhere.
• Price - It's free for your small/experimental projects and you can pay as you go if the project grows linearly
• Maximize Time - You can implement a lot of features in less time with small code

Go To console.firebase.com and explore by creating a project inside but before that have a look at this article.

What does Firebase offer?

• Authentication
• Firestore (NO SQL Database)
• Hosting
• Cloud Functions

Authentication

If you have to build an application which has “login” features you definitely know building an authentication system with encryption and other things involved is a pain.

With Firebase authentication you can let users log in with email or even let users log in to your product via Google Sign in with a small piece of Code. Here is the documentation for Google sign in method and email sign in method

FireStore

If you are a computer science student you might have learnt MYSQL which is a relation based SQL DBMS.

Firestore is No SQL database like Mongo DB which is made of Collections and Documents stored in JSON format. It has its own pros and cons. For beginners, this is very easy to learn, even querying/filtering is so much better.

Let's say that we have to keep information about a user and keep a record of his order history. For that, we will build 2 collections

-Users -Orders

which will have a document inside with information stored in JSON format.

Hosting

So you have now built a Web App using Firebase and need to host it somewhere. Well, Firebase has got your back. Make sure that you have a folder with the public as a name which would have your index.html file. Project Structure should look something like this Run the following commands to deploy your web app to firebase.

Firebase init

In Response, you get this

? Are you ready to proceed? Yes
? Which Firebase CLI features do you want to set up for this folder? Press Space to select features, then Enter to confirm your choices.
 ( ) Database: Deploy Firebase Realtime Database Rules
 ( ) Firestore: Deploy rules and create indexes for Firestore
 ( ) Functions: Configure and deploy Cloud Functions
>(*) Hosting: Configure and deploy Firebase Hosting sites
 ( ) Storage: Deploy Cloud Storage security rules
 ( ) Emulators: Set up local emulators for Firebase features

Choose the services as hosting and select a public folder where you should have your index.html inside. This is how it should look

=== Hosting Setup

Your public directory is the folder (relative to your project directory) that
will contain Hosting assets to be uploaded with firebase deploy. If you
have a build process for your assets, use your build's output directory.

? What do you want to use as your public directory? public
? Configure as a single-page app (rewrite all urls to /index.html)? Yes
? File public/index.html already exists. Overwrite? (y/N) N

After the initialization, you can deploy it with this code

firebase deploy

You might want to connect your own domain or might get stuck at something. Refer to the Firebase Hosting Documentation for better reference.

Cloud Functions

This is one of the favourite things about Firebase is cloud Functions. I might write a few articles down the lane so stay tuned.

Cloud Functions is a hosted, private, and scalable Node.js environment where you can run JavaScript code. Cloud Functions for Firebase integrates the Firebase platform by letting you write code that responds to events and invokes functionality exposed by other Firebase features.

And it has a lot of interesting use cases, like:

• You can send notification to users when data changes

• Image processing

You can host only 2 cloud Functions for free of cost in a Spark plan, just keep that in mind.

Is Firebase used in tech companies?

Yes. Usually by startups without a lot of time and resources. Firebase is a godsend.

But not in big corporates where securing user data is a bigger priority, they prefer building it inhouse instead of using BaaS like firebase.

(The other reason why few companies don't prefer firebase is that the product they are building might be for global users and we know that some countries have banned google and hence firebase won't work there.)

Don't' worry though . if you understand how NO-SQL works using firebase, It's pretty similar to mongo, so it won't be a challenge.

In conclusion:

Firebase is easily the best backend as a service for beginners or freelancers who are just getting off the ground.

I will soon write on how you can integrate Firebase to Flask apps to stay tuned to my newsletter and follow me here on Hashnode.

Most of my time is spent on Instagram and I use it mostly for the memes.

So, some of my friends told me that most of the memes are uploaded to Reddit and later spread to Instagram, and Reddit was the best place to go for funny content but I wasn't a big fan of the Reddit UI and didn't really get used to it.

One day I randomly thought, what if I could start a meme page that gets content from Reddit but without downloading and uploading the images manually and still be the first account to upload those memes on Instagram straight from Reddit?

And, I Built a BOT which uploads memes from some subreddits like r/dankmemes and r/memes and uploads it on my Instagram account - @thiru.bot

How did I do it?

At this stage, I was a beginner and knew few things about python, had some experience building flask apps, had some experience in web scrapping and had huge hunger to build this just to flex my skills with my friends.

Before I started building it I had to visualize what I needed?

• Reddit API & Basics of web scrapping to get the images and save them.
• Any Instagram API which lets me upload photos to my account.
• A server to host this later to make it work 24/7

Friends, When you are a beginner you just google things, mix code from different repositories, add your own code and make it work.

and that's what I did! There were no Instagram Reddit bot but I found a Reddit to twitter project which gave me a clarity of thought in building this.

The Working Instagram API

The Official Instagram API doesn't allow developers to upload photos or videos to post. Most of the other third party APIs also dint work but at last found this pip package.

pip install instabot

Later Tested if this worked by giving my Instagram credentials and uploaded a random photo from my computer.

from instabot import Bot
bot=Bot()
bot.login(username="thiru.jpg",password="followMeOnInstagram")

bot.upload_photo('hey.png',caption=("Test Upload "))

and it worked :)

Next step - Scrapping the latest images from subreddits

Getting Images from Reddit

To get the latest submissions from our subreddits we will be using PRAW API from subreddit. it's pretty simple not to worry.

Setting up Praw API

reddit = praw.Reddit(client_id='',
client_secret='',
username='',
password='',
user_agent='')

Creating an Reddit App

To Fill the above details you need to create an app on Reddit, Just Open https://ssl.reddit.com/prefs/apps/ and click on create app.

I am here creating an app with the name test app. Client_Id parameter (the one which is below personal use script in the image above), client_secret is the secret key available. For username and password give your Reddit username and password. User Agent can be blank or chrome.

Setting up subreddits to get memes

red='dankmemes'
subreddit = reddit.subreddit(red)
newMemes = subreddit.hot(limit=3)

I choose the subreddit 'dankmemes' and NewMemess Praw Class has all the hottest submissions from the subreddit. You can get any type of submissions (refer to official documentation here )

Validating and Picking only Photos

for submission in new_memes:
    if submission.is_self == True:
        print("Post was text, skipping to next post.")
        continue
    else:
        pass
url = submission.url
time.sleep(waitTime)

You want just photos to upload. So, You need to validate them and choose only the submissions which have photos.

Downloading the images

Before Storing the images from Reddit we check Two Things

• if the submission is an Video/text or an image using isImageLink(url) Function and it returns the extension name of the image(Eg: jpg,png)
• If the image was posted before using already_uploaded(id) function so that we don't post the same memes again.

You can check out those functions and full code on my GitHub Repo here also follow me on GitHub.

if isImageLink(url) and not already_uploaded(id):
    try:
        img=requests.get(submission.url)
        filename=str(counter)+'.'+IsImageLink(submission.url)
        filename=os.path.join('images', filename)
        imagefile=open(filename, 'wb')
        imagefile.write(img.content)
        imagefile.close()
        post_ids.append(submission.id) # A list of all submission ids
        log_insta(post_ids)
        photoAlbum.append({'File':filename, 'Title':submission.title})
        #dictionary of all photos
        counter+=1
    except Exception as e:
        print(e)

After checking, You can download the image and save it with the count+ the extension (Eg: 2.png or 3.jpg).

Later you save a list with all the post id.

Remember you had to check if the post was already uploaded. How will it know if it is posted already? Yes, you will be logging the post_id of the image using the function log_insta(post_ids)

def log_insta(post_ids):
    for post_id in post_ids:
        with open(POSTED_CACHE, 'a') as out_file:
            out_file.write(str(post_id) + '\n')

Downloading the images from Imgur

Some Redditors upload photos to Imgur, and to download those images, You will using beautifulSoup and scrape data from those submissions. After you download, the saving part is same as above so I won't be posting the whole code.

elif str(submission.url).lower().startswith('https://imgur.com') or str(submission.url).lower().startswith('http://imgur.com') and counter<max_images:

    try:
        html_page = urllib.request.urlopen(submission.url)
        soup = BeautifulSoup(html_page, 'lxml')
        images = []
        for img in soup.findAll('img'):
        images.append('https:'+img.get('src'))

        img=requests.get(images[0])
        filename=str(counter)+'.'+images[0][-3:]

Downloaded the images, What next?

Now that we have all the 3 images in the photo album Dictionary with the File and Title we can upload it to instagram using our unofficial Instagram API and give credits to the authors,subreddit and also add captionTags.

authors = ''.join(str(e + ', ') for e in authors)
captionTags='#binod #memes #dank'
for photo in photoAlbum:
    img=photo['File']
    captionText=photo['Title']
bot.upload_photo(img,caption=(captionText + '\n' +'Reddit Authors:'+authors+'\n' captionTags))

Later we delete the image we downloaded just not to waste disk space.

for filename in glob(filePath + '/*'):
os.remove(filename)

Now Put all this code together into a single Python File, Run it and Hola, You have now uploaded a meme from Reddit to Instagram from your Python Code In my case, I named the file bot.py, thought of telling you this as this might be useful later in the deployment part.

But the bot.py is run on the local machine and we can't just run this 24/7 on our local machine. Any Fix?

Deployment

To Run The Bot in a server for 24/7 you need to deploy this to cloud. I used Heroku to deploy this Instagram meme bot. Follow this article to set up a Heroku account and know the basics of Heroku. I will soon be writing an article on the same.How to Upload Flask apps to Heroku .

You need 3 files to be created for the deployment of your bot into Heroku .

• ProcFile
• Runtime.txt
• Requirements.txt

Runtime.txt

python-3.7.3

In Runtime.txt you specify what language your code is in.

The ProcFile

web: python3 bot.py
worker: python3 bot.py

Here you will be specifying the python file to be run by the dynos and note that I created two of these just to make sure one of them always stays on and doesn't sleep. You later need to scale them.

Requirements.txt

imageio==2.4.1
praw
requests
urllib3
keyboard
pillow
beautifulsoup4
instabot

Heroku in simple words creates a virtual machine so you need to mention what are the packages you want to install for your bot.py to work.

After Uploading these files into a single folder. It's time to scale your dynos

heroku ps:scale web=1 worker=1

A working meme BOT

But yeah , I need to work on the followers count lol.

Problems & Suggestions

• Instagram Detecting Bots
• Upload Frequency
• Heroku Running out of Credits
• Dynamic Tags

Instagram Detecting Bots

Instagram DOES NOT like Bots.

Initially, when I started this, my bot was detected and banned for 1 day and was not allowed to use any Instagram features like liking, etc for 1 week.

Solution which worked for me

It took me weeks to crack this but the reason why my bot was getting detected is that my posts were only from one subreddit and the authors were mostly same and hashtags as well so every post had somethings similar and with same upload frequency and Instagram started detecting.

With no options left, I had to stop giving credits to authors in the caption and created a list of subreddits and used random function in Python to select one subreddit and stoped using tags which fixed the issue.Upload frequency also matters and I will be speaking about it in the next paragraphs.

Upload Frequency

How many Posts should you upload at a time?

How many hours once should you upload?

The safe bet is 3 posts /hour

Heroku Running out of credits

Heroku gives 550 hours of free dynos when you create an account. I had used all the 550 dynos and all my projects on Heroku had shut down, luckily I gave my credit card information and was able to get an extra 450 hours of dynos, But that won't be sufficient if you want to run this bot forever as 30 days = 1800 hours. I usually was turning off the dynos when I was going to sleep and turning it on when required so worked. This is the solution that worked out for me and just to remind you, this is hosted for completely free of cost so nothing to complain.

Dynamic Tags

So, removing tags helped with not getting banned but I wanted better reach as my account had just 3 followers from weeks so created multiple lists of trending hashtags and wrote a code to choose the hashtags according to the subreddit and its working great.

What Did we learn from this?

• You Don't need to know everything to start building things. Just the Hunger is enough.
• The results don't come just in the first iteration. Experiment Fix experiment.
• You can't get a girlfriend by creating a Meme bot :(

Github Repository

I still haven't added a ReadMe file(working on it). But this article is sufficient to understand the working of the same.

Here is the complete Code - GitHub Repo

I am Hrithwik Bharadwaj a 20-year-old Full Stack Developer and a Tech Enthusiast🧑🏻‍💻. I always consumed knowledge in video📺 and never read books or blogs📚. Over time I learnt how important it is to read and continued reading and loved the experience. Now want to get into writing as writing makes your thought process clear.

Experience

I have experience working at a tech startup for about a year and learnt a lot of things there but still consider myself a beginner and love learning new things everyday.

The Motive

When I started out my journey as a developer with zero knowledge I didn't know where to start, what to do, which language to start with which made me really anxious. I don't want the same thing to happen with others so started this blog.

I want to help new developers make the right decisions with right tricks and teach the basics of developing in short stories.

In these short stories, I will be explaining and documenting about my past and upcoming projects in the simplest ways.

I choose this platform as its ad-free and has an amazing community.

Social Media Presence

• Funny Vlogs Instagram - @thiru.jpg
• Tech Videos YouTube - Subscribe Here!
• Random Thoughts Twitter - @HSBTechYt