Creating a URL Shortener Microservice

This weekend I’ve been working on URL Shortener Microservice – one of the API and Microservice FreeCodeCamp (FCC) projects. I’ve had a couple of frustrating days where any time I’ve had available to dedicate towards coding seemed to be very slow going. However, I am pleased to say I did manage to complete the required user stories. I think having a couple of weeks away from nodejs and express made this project trickier as I had to remind myself about many of the required concepts to complete the project.

The project comprised of the following challenges and below I will describe how I completed each part.

1. Hosting of NodeJS server via Express
2. REST APIs: mostly Post
3. Passing user parameters to API via html button
4. Connection to MongoDB to store user data
5. Query MongoDB to existing data
6. Checking whether a URL is real

I choose to develop the tool on Glitch as it provides a simple development environment with it’s auto save functionality and hosting dynamic pages. It is also very easy to export the code to your GitHub site. The FCC started project on glitch also comes with some boilerplate code to cover number 1 above.

Number 2 was immediately trickier as I jumped straight in to linking the APIs with the new MongoDB I had setup via mLab. Initially I had forgot to add myself as a User to the database, preventing any of the APIs I had created to work. I was then testing the API endpoints manually without making much progress, with additional confusion created as I was using some of the earlier FCC challenges as a starting point which included code that created and then deleted database entries, in particular the below code deleted all previously created entries related to a collection in the database:

const mongoose = require('mongoose');

mongoose.connect(process.env.MONGO_URI, { useNewUrlParser: true });
var schema = new mongoose.Schema({ name: {type: String, required: true}, age: Number, favoriteFoods: [String] });
var Person = mongoose.model('Person', schema);

var Person = require('./myApp.js').PersonModel;

Person.remove()

My breakthrough came when I downloaded Postman tool on my dev macbook. Postman is a fantastic tool for working with APIs. It provides the user with a simple UI that provides the ability to test your own API endpoints and visualise the responses. I instantly became more comfortable with the task at hand after installing Postman. I’ve used sparingly at work, so it didn’t come to mind as quickly as it should have!

Now armed with Postman I was able to quickly create a Postman collection and associate a number of new API query examples that would tell me if the MongoDB is accessible, test adding a new item and test querying the database.

Number 3 required creating a POST method that read the values of an input box. The associated html already contained the code that completed the attachment of the body data to be read in the API function.

<form action="api/shorturl/new" method="POST">
          <label for="url_input">URL to be shortened</label>
          <input id="url_input" type="text" name="url" value="https://www.freecodecamp.com">
          <input type="submit" value="POST URL">
        </form>

It was also required to extract a user input from the URL. This again was simple to solve as there had been a previous FCC challenge that provided this exact challenge of extracting parameters from the API URL. Although at first I was testing a POST API I quickly realised that this needed to be a GET api and googling the difference agrees “hat GET carries request parameter appended in URL string while POST carries request parameter in message body which makes it more secure…”. This particular URL parameter needed to be passed as a parameter into a database query, with the returned value then being used to redirect the browser window to another url. I ended up with the below:

//redirect the user to original url when valid shortener used
app.get("/api/shorturl/:short_url", function(req, res, next){
  const shortUrl = req.params.short_url;
  
  findURLByShortener(shortUrl, function(err, data) {
        if(err) { return next(err) }
        if(!data) {
          console.log('Missing `done()` argument');
          return next({message: 'This shortener does not exist, please add via form.'});
        }
        res.redirect(data.original_url);
      });
})

var findURLByShortener = function(urlShortener, done) {
  
  URL.findOne({short_url: urlShortener}, function(err,data){
      if(err) return done(err);
      done(null, data);
  });
    
};

The above snippet shows passing the shortURL parameter along with a standard callback method, which seems to be the suggested method for querying an external database – a standard also introduced in the earlier FCC challenges. The main benefit I could see is that it helps to handle synchronous actions as the accompanying callback is only processed after the err and data parameters are populated following the completion of the initial function call. In this case the line “if(err) { return next(err) }” will not be completed until findURLByShortener has returned a result – hopefully (null, data)

Number 4 required adding data to the MongoDB, querying existing data to see if short_url had already been created and finding the current maximum short_url number. FindOne mongoose function has already been shown in the above snippet. Finding the max value for a field in a MongoDB collection was new and required checking the Mongoose docs.

var findMaxShortenedURL = function(done) {
  // Find the max shortened url of all db URLs
  URL.aggregate()
    .group({ _id: null, maxShortener: { $max: '$short_url' }})
    .exec(function (err, res) {
      if (err) return done(err);
      done(null, res);
    });
};

The above function returned an array of javascript objects and I used the standard method to extract the required data. I then added 1 to the maximum value and made this the next short_url value for any new URL added via the front end.

The final part of the challenge which was hinted at in the FCC template ReadME was that default package dns.lookup(add, cb) should be used to check whether a website adde by a user was a working domain. Again this required some official docs reading, which thankfully contained a useful example and I ended up implementing the check in the following way:

const fullURL = req.body.url;
      var urlRegex = /https{0,1}:\/\//gi;
      hostAdd = fullURL.replace(urlRegex, '');

      dns.lookup(hostAdd, (err, address, family) => {
        console.log('address: %j family: IPv%s', address, family);
        if(!address){
          res.json({"error":"invalid URL"})
        } else { //....do some cool stuff 

Overall I found this mini project very challenging as it tested combining a number of skills that had been introduced fairly quickly in the earlier FCC challenges. I think my knowledge to link a front end to a database via APIs has definitely improved as this challenge required so many different concepts to work well. And main lesson learned is that I will definitely be using Postman from the get go next time!