Converting HTML entities to UTF-8 in VimScript

I wrote a small script to fetch a web page title, inserting it as amarkdown formatted link: [title](URL) One of of the issues I ran into was that some titles contain HTML entities and Vim has no built-in way to deal with this.

The solution I came up with wasn’t that grand, but the interesting part was that I learned about a UNIX utility, that also ships with MacOS, called textutil.

The solution was to do a system call passing a string with HTML entities, getting back a UTF-8 formatted string that VIM understands. Here is example that you can inVim:

let oldtitle ='&#128075; hello &lt; world &#x1f310; <   &#60 It&#8217;s a great day.'
let newtitle = system( 'echo ' . shellescape(oldtitle) . ' | textutil -convert txt -format html -stdin -stdout')
echo newtitle

Example output.

The special part I want to point your attention to is ' | textutil -convert txt -format html -stdin -stdout'. This pipes the title into the textutil command which does all the conversions for us.

You can find the full function below.

function! AddLink()
	let url = input('URL to add? ')
	if empty(url)
		return
	endif
	let html = system('curl -s ' . shellescape(url))
	let regex = '\c.*head.*<title[^>]*>\_s*\zs.\{-}\ze\_s*<\/title>'
	let title = substitute(matchstr(html, regex), "\n", ' ', 'g')
	let title = system( 'echo ' . shellescape(title) . ' | textutil -convert txt -format html -stdin -stdout')
	if empty(title)
		let title = 'Unknown'
	endif
	put ='[' . title . '](' . url . ')'
endfunction

On the fence about Neovim

I’ve been using vim for the last 5 years. I’ve written about trying this out 5 years ago and stuck to it. It’s a fantastic experience and I truly enjoy working with this editor. I’ve been rethinking it this week though, as I was updating my configuration to get auto-completion to work.

I needed to get my editor ready as I haven’t needed to do any serious coding in the last two years. We’re trying out a new requirement at work: all engineering leaders should contribute a medium-sized piece of code in a 3 month period. I’ll try an write more about this, but in simple terms, I have to make sure that I’m spending a little more time coding.

The Neovim editor was forked from Vim (the code has been copied and is currently being changed into something better or different). Its been around since 2015, so this blog post is late to the party. Many people have already embraced Neovim.

I see how amazing it is, but really, I’m not sure that I want to change it. It’s a big commitment. It will take time to learn about the “new” editor and its features. It will also take focus away from other things, so I’m two minds. It is also a new project that is slowly moving away from the Vim I’ve come to enjoy. Maybe this is just me overthinking.

The one pain I currently have is language server support. Vim can do this, but it requires configuration and running a few things on my device. I like a simpler editor setup, but I’m missing the simple built-in features modern editors come with.

I’ll try out my current setup for the next few months and see if I miss anything. If I do, I’ll try out Neovim.

One year at the helm of Vim

One year ago I started using an old text editor called Vim. I’m happy to say that I’m still sticking with it. I’m now very comfortable with VIM and more empowered translate Ideas into working solutions.

In this year I’ve become accustomed to working the terminal, embracing the VIM way while realizing that that coding is just one part of the challenge.

Continue reading “One year at the helm of Vim”

So I tried an editor called VIM

But Why?

I’ve tried many editors, but with each of them, I found a few things I didn’t like. I always knew about VIM and I knew how to exit vim ( link for the pun ), but never thought it serious enough to work in until I saw how magically other people were using it.  That’s when I started thinking about trying it out. I’ve been using it for little over a month now and this week marks the end fo the first week of using it at work.

First steps

I started googling around for the best guides on where to start. This led me to the book called Practical Vim. This is the only book a beginner needs. It covers all the most important things you need to know. It also gives you a great foundation from which to grow your very own .vimrc.

The IDE rabbit hole

On making the switch I realised that I’m seriously going to miss PhpStorm if I don’t figure out how to use VIM effectively.

Enter plugins, the magical stuff you drop in to give VIM super powers. Then enter plugin managers. The first one I tried was one called Pathogen. It worked like a charm, but then I realised I needed to add git submodules for each new plugin. The thing with vim is that you want to version control all the things so you can easily take your environment with you when switching machines. So I’ve got a plugin manager and I have that under version control, but adding so many submodules was not working out especially after the 5th plugin. I eventually switch to something that only keeps a few strings in your config called Plug.

Side note, if you want the best way to keep your dotfiles on GH see this: https://news.ycombinator.com/iterThem?id=11070797

Now that you have all the vim niceness you’ll soon realise you need a better way to manage the terminal. You need tabs, panes or some sort of split window system. There are many, many options. I looked at a few and decided on Tmux, the main reason for doing so is that it is terminal based and works on many platforms. I’m totally happy with this.

Mastery

I hear that people use this editor for years and still learn new things. I’ve seen from using it for a month that you’re alway tweaking your workflow. There’s always something new that you can add or change in your .vimrc.  The best thing about VIM is that no two people use it in the same way. It should become an extension of your thoughts so that the editor gets out of your way. I’m not there yet, but I can see how my muscle memory is forming day by day.

Portability

I can now freely move between machines and take my editor with me.  I has dot fileshttps://github.com/dwainm/dotfiles

My Struggles

In PhpStorm, Cmd + L gives you this magic wand that fixes code indentation a common coding standard issues. I miss it so much. I need to figure out how to do this in VIM.

Another struggle I have now is that I’m constantly pressing the wrong keys. This will probably get better with time. I found that I can use the numbers keys at the top fo the keyboard because I don’t know where keys 4-7 is in the dark. I’ll have to learn this.

Lastly I the struggle is just to adjust to a new workflow. One where I’m constantly tweaking things to better my productivity. I believe this will pay off over time and I’m excited about my new vim adventure.